2015+MA+Program+Evaluation

The 2015 evaluation of the MA program is based upon data collected during the calendar years of 2013-2014. The following are the two columns we are required to submit as the findings of our evaluation. See rubric following the table for the criteria for evaluating each learning outcome.

 (Added 1 Sept 2015 by SRW) || (Added 1 Sept 2015 by SRW) ||
 * LO || Conclusions Based on Evidence || Rating || Actions Taken or Planned ||
 * Scholarship || Of the 14 students who completed or left the program in 2013-2014, all 14 successfully met the criteria for this learning outcome by completing at least 2 of the following 3 criteria: passing Sections I and III from the master's exam with a score of 70% or higher, successfully defending their project or thesis proposal, and successfully defending their project or thesis. The percentage of students who met this learning outcome far exceeds 70%, the cutoff for satisfactory status for the program. || Excellent || Program students continue to excel at this learning outcome. No planned action at this time. (Added 1 Sept 2015 by SRW) ||
 * Research || Of the 14 students who completed or left the program in 2013-2014, 13 successfully met the criteria for this learning outcome by passing Section II and III with a score of 70% or higher and by successfully defending a project or thesis. Thus, 93% of the students met this learning outcome, which exceeds 70%, the cutoff for satisfactory status for the program. || Excellent || Program students are doing well on this program learning outcome. We continue the changes discussed in fall 2013 to the courses MthEd 590 and MthEd 591, requiring students to spend significantly more time on their own research much earlier in the program. In Fall 2014 we decided to alter the timing of the Master's Exam to allow students to get a quicker start on their own research. So far, it seems to be having a positive effect on students’ progress in the program and time to graduation, but it is still too early to know the long term effects. We will continue to monitor these changes.
 * Mathematics || Of the 14 students who completed or left the program in 2013-2014, 13 students completed Section IV of the master's exam with a score of 70% or more. Thus, 93% met this learning objective. . Thus, 93% of the students met this learning outcome, which exceeds 70%, the cutoff for satisfactory status for the program. || <span style="font-family: Arial,sans-serif; font-size: 10pt;">Excellent || <span style="font-family: Arial,sans-serif; font-size: 10pt;">The 93% pass rate for this round of program evaluation exceeds the 65% pass rate from last year’s evaluation. The graduate committee met during Fall 2013 to review past data and the list of students who were scheduled for graduation in 2014 and concluded that it was very likely that 70% or more of the students considered in the report this year would meet this learning outcome. There seems to have been clear improvement over the prior two years. The graduate committee will continue to monitor this learning outcome.
 * <span style="font-family: Arial,sans-serif; font-size: 10pt;">Teaching || <span style="font-family: Arial,sans-serif; font-size: 10pt;">Because some of our students are full-time teachers, we do not have full sets of teaching data for the 14 students who completed or left the program in 2013-2014. Of the eight students for which we have both faculty evaluation and student ratings data, seven received satisfactory ratings. Two additional students had only student ratings data, and based on these scores only, both received satisfactory ratings. For the four remaining students, we have no teaching data. All four were full-time teachers who did not work as TA's in our department.

<span style="font-family: Arial,sans-serif; font-size: 10pt;"> In summary, for the 10 students for which we have some teaching data, nine (90%) received satisfactory ratings. The remaining four students had no data available. || <span style="font-family: Arial,sans-serif; font-size: 10pt;">Good || <span style="font-family: Arial,sans-serif; font-size: 10pt;">These results raise two concerns. The first is that for four students, we have no teaching data available, and for an additional two students, we have only student ratings. This concern will be addressed in part by our having instituted faculty reviews of teaching for all our graduate students each semester. However, the problem remains that for our part-time students who do not work as TA's, we currently have no way of assessing this learning outcome. The Graduate Committee and the Learning Outcome Committee will consider ways that we can assess the development of teaching in our part-time students.

<span style="font-family: Arial,sans-serif; font-size: 10pt;"> The second concern is that even though nine of our students met this learning outcome, one did not. TAs currently receive both a semester-long mentoring experience in teaching during their second semester in the program, as well as continued mentoring while working as TAs. The graduate committee will review the teaching support that TAs are currently receiving to identify possible modifications that might lead to increasing the success of our TAs in engaging in their mathematics instruction. <span style="font-family: Arial,sans-serif; font-size: 13.3333330154419px;">(Added 1 Sept 2015 by SRW) || <span style="font-family: Arial,sans-serif; font-size: 13.3333330154419px;">which exceeds 70%, the cutoff for satisfactory status for the program. <span style="font-family: Arial,sans-serif; font-size: 10pt;">. || <span style="font-family: Arial,sans-serif; font-size: 10pt;">Excellent || <span style="font-family: Arial,sans-serif; font-size: 10pt;">The number of students meeting this objective improved slightly from 88% last year. We have observed that low scores are often related to inadequate progress in their individual research. As mentioned above, we have tried to encourage their progress by increasing expectations for individual research during their first year of courses, and more recently by moving the time of the Master's exam to allow students more time to focus on either own research, These changes seem to have resulted in a substantial improvement in the timely completion of proposals, resulting in an increase in the percent of first year graduate students who were able to defend their proposals before fall semester of their second year. We are always aware, however, that the nature of our review process makes judgments from single data points difficult. We will continue to monitor this situation. <span style="font-family: Arial,sans-serif; font-size: 13.3333330154419px;">(Added 1 Sept 2015 by SRW) || <span style="font-family: Arial,sans-serif; font-size: 13.3333330154419px;">(Added 1 Sept 2015 by SRW) ||
 * <span style="font-family: Arial,sans-serif; font-size: 10pt;">Professionalism || <span style="font-family: Arial,sans-serif; font-size: 10pt;">Of the 14 students who completed or left the program in 2013-2014, we had both advisor and class ratings of professionalism from the MA-PIBS instrument for 13. Of these 13, 12 met or exceeded expectations. For one student, we had only advisor ratings, all of which indicated that the students met or exceeded expectations. Overall, based on available evidence, 13 of 14 students, or 93%, met this learning outcome,
 * <span style="font-family: Arial,sans-serif; font-size: 10pt;">Spiritual Stewardship || <span style="font-family: Arial,sans-serif; font-size: 10pt;">Of the 14 students who completed or left the program in 2013-2014, we had both advisor and class ratings of professionalism from the MA-PIBS instrument for 13. For the remaining student, we had only advisor ratings. With the caveat, however, all 14 met or exceeded expectations on those items measuring this learning outcome. T <span style="font-family: Arial,sans-serif; font-size: 13.3333330154419px;">his exceeds the 70% cutoff for satisfactory status for the program. || <span style="font-family: Arial,sans-serif; font-size: 10pt;">Excellent || <span style="font-family: Arial,sans-serif; font-size: 10pt;">Program students continue to excel at this learning outcome. No planned action at this time.
 * 2015 Rubric for Assessing Program LOs**

The MA program will be assessed annually at the beginning of winter semester. Because the purpose of the assessment is to see if the learning outcomes have been met by the end of the program, it does not make sense to consider data from students who are still in the program. However, if data is limited to only students who either graduated or dropped out of the program the year before, the data points are insufficient to establish clear trends. Consequently, at each annual evaluation of the program, data will be considered for students who either graduated or dropped out over the past two years. While this means that the data from a particular student will be used in two consecutive evaluations of program learning outcomes, we nonetheless feel that data aggregated across two years will provide a fuller understanding of the success of our program than data from a single year.

In preparation to assess the program learning outcomes, each student who has graduated or dropped out of the program will be evaluated individually on each learning outcome to determine if that learning outcome was met by that student. The criteria for a learning outcome to be met by a student are given below. Once each student has been evaluated on each learning outcome, the number of students who have met each learning outcome will be tallied. If 70% of the students meet a particular learning outcome, then we will conclude that the program is satisfactory with respect to that learning outcome.


 * Criteria for Determining Whether a Student has Met a Particular Learning Outcome**

LO 1: Scholarship For each student, data from Sections I and III of the Master's Exam, from their Thesis/Project proposal defense and from their Thesis/Project final defense will be gathered. (If students were given a second chance on any of these measures, data will come from the last attempt.) Master's Exam Sections I and III: Students are considered satisfactory on this measure if the average of their scores on the four problems from Sections I and III is at least 70%. Thesis/Project proposal defense: Students are considered satisfactory on this measure if they passed (initially or eventually) their proposal defense. Thesis/Project final defense: Students are considered satisfactory on this measure if they passed (initially or eventually) their final defense. Students meet this learning outcome if their performance on any two of the three measures is satisfactory.

LO 2: Research For each student, data from Sections II and III of the Master's Exam, and from their Thesis/Project final defense will be gathered. (If students were given a second chance on either of these measures, data will come from the last attempt.) Master's Exam Sections II and III: Students are considered satisfactory on this measure if the average of their scores on the four problems from Sections II and III is at least 70%. Thesis/Project final defense: Students are considered satisfactory on this measure if they passed (initially or eventually) their final defense. Students completing a project meet this learning outcome if their performance on both measures is satisfactory. Students completing a thesis meet this learning outcome if their performance on the thesis defense is satisfactory.

LO 3: Mathematics For each student, data from Section IV of the Master's Exam will be gathered. (If students were given a second chance to take the Master's Exam, data will come from the last attempt.) Students meet this learning outcome if the average of their scores on the two problems from Section IV is at least 70%.

LO 4: Teaching For each student, teacher evaluation and student rating data from the last two courses taught will be gathered and the better of each of those measures will be used for analysis. For students to meet this learning outcome, they must receive a satisfactory rating on both measures.

STUDENT RATINGS: On the student ratings measure, there is an overall course rating, an overall instructor rating, and 23 sub-questions. Eighteen of the 23 sub-questions are rated on a 1-8 scale. If 9 or more of the sub-questions are rated below a 6 or if the overall instructor rating is below a 6, the student will receive an unsatisfactory rating on this measure. TEACHER EVALUATIONS: On the teacher evaluations measure, if the student receives a rating below a 3 in any of the 4 areas (classroom organization; instruction, discussion, and tasks; content; communication), they will receive an unsatisfactory rating on this measure.

LO 5: Professionalism For each student, two MA PIBS—the last MA PIBS from the project/thesis advisor and the MA PIBS from the last 3-credit-hour mathematics education course the student took—will be considered. For the student to be considered satisfactory on an item, the student must receive a rating of "meets" or "exceeds" expectations for that item on both MA PIBS. If a students is unsatisfactory on 3 or more items, then the student will be considered unsatisfactory in professionalism.

LO 6: Spiritual Stewardship For each student, items 1-3, 7, and 9 from two MA PIBS—the last MA PIBS from the project/thesis advisor and the MA PIBS from the last 3-credit-hour mathematics education course the student took—will be considered. For the student to be considered satisfactory on an item, the student must receive a rating of "meets" or "exceeds" expectations for that item on both MA PIBS. If a students is unsatisfactory on 2 or more items, then the student will be considered unsatisfactory in spiritual stewardship.