2014+MA+Program+Evaluation

The 2014 evaluation of the MA program is based upon data collected during the calendar years of 2012-2013. The following are the two columns we are required to submit as the findings of our evaluation. See rubric following the table for the criteria for evaluating each learning outcome.


 * LO || Conclusions Based on Evidence || Rating || Actions Taken or Planned ||
 * Scholarship || Of the 17 students who completed or left the program in 2012-2013, 16 successfully met the criteria for this learning outcome by completing at least 2 of the following 3 criteria: passing Sections I and III from the master's exam with a score of 70% or higher, successfully defending their project or thesis proposal, and successfully defending their project or thesis. The percentage of students who met this learning outcome far exceeds 70%, the cutoff for satisfactory status for the program. || Excellent || Program students continue to excel at this learning outcome. No planned action at this time. ||
 * Research || Of the 17 students who completed or left the program in 2012-2013, 14 successfully met the criteria for this learning outcome by passing Section II and III with a score of 70% or higher and by successfully defending a project or thesis. Thus, 82% of the students met this learning outcome, which exceeds 70%, the cutoff for satisfactory status for the program. || Good || Program students are doing well on this program learning outcome. The faculty met during Fall 2013 to discuss how to help students shorten the time it takes to complete their projects and theses. Changes were made to the courses MthEd 590 and MthEd 591, requiring students to spend significantly more time on their own research much earlier in the program. We will monitor this change to see how it affects the quality and timely completion of projects and theses. ||
 * Mathematics || Of the 17 students who completed or left the program in 2012-2013, 11 students completed Section IV of the master's exam with a score of 70% or more. Thus, only 65% of students met this learning outcome. However, all 17 students passed all of the courses they took for the program, many of which involved significant mathematical understanding. This evidence suggests that the program is barely meeting this learning outcome. || Fair || The graduate committee met during Fall 2013 to discuss the poor scores for this learning outcome during last year's evaluation of the program. The committee noted that the percent of students who met this outcome was erroneously reported. In fact, 61% of the students who were considered in last year's evaluation met this learning outcome, and not 39% as stated in last year's report. The graduate committee reviewed past data and the list of students who were scheduled for graduation in 2013 and concluded that it was very likely that 70% or more of the students considered in the report this year would meet this learning outcome. Thus, they wondered if last year's outcome was an anomaly. Discussion was tabled to see how this outcome fared in the current program assessment. The graduate committee will meet to discuss the undesirable outcome of this year's evaluation. ||
 * Teaching || Because we had just begun to collect data to evaluate this learning outcome, we do not have full sets of teaching data for the 17 students who completed or left the program in 2012-2013. Of the nine students for which we have both faculty evaluation and student ratings data, all nine received satisfactory ratings. Five students had only student ratings data, and based on these scores only, four received satisfactory ratings. For the three remaining students, we have no teaching data. Two of these three were full-time teachers who did not work as TA's in our department; the third had taken a leave and we had not begun to collect data for graduate students when she last taught.

In summary, for the 14 students for which we have some teaching data, 13 (93%) received satisfactory ratings. The remaining three students had no data available. || Good || These results raise two concerns. The first is that for three students, we have no teaching data available, and for an additional five students, we have only student ratings. This concern will be addressed in part by our having instituted faculty reviews of teaching for all our graduate students each semester. However, the problem remains that for our part-time students who do not work as TA's, we currently have no way of assessing this learning outcome. The Graduate Committee and the Learning Outcome Committee will consider ways that we can assess the development of teaching in our part-time students.

The second concern is that even though 13 of our students met this learning outcome. one did not. TAs currently receive both a semester-long mentoring experience in teaching during their second semester in the program, as well as continued mentoring while working as TAs. Poor performance on this learning outcome seems to be related to TAs' lack of experience in teaching inquiry-based mathematics. Research suggests that learning to teach inquiry-based mathematics is difficult, and may require multiple years to achieve proficiency. The graduate committee will review the teaching support that TAs are currently receiving to identify possible modifications that might lead to increasing the success of our TAs in engaging in inquiry-based mathematics instruction. ||
 * Professionalism || Of the 17 students who completed or left the program in 2012-2013, we had both advisor and class ratings of professionalism from the MA-PIBS instrument for 12. Of these 12, 10 met or exceeded expectations. For the remaining five, we had only advisor ratings, all of which indicated that the students met or exceeded expectations. Overall, 15 of 17 students, or 88%, met this learning outcome. || Good || Over the years we have noticed a trend for why students do not do well on the MA-PIBS. Low scores are often related to inadequate progress in their individual research. As mentioned above, we have tried to encourage their progress by increasing expectations for individual research during their first year of courses. In addition, during the spring and summer terms of 2014, one of the faculty met weekly with graduate students from the 2013-2014 cohort to help them maintain steady progress toward completing their thesis/project defense. These changes have resulted in a substantial improvement in the timely completion of proposals, resulting in a large increase in the percent of first year graduate students who were able to defend their proposals before fall semester of their second year. We expect that this improvement in student research progress will have a positive effect on this learning outcome in the future. ||
 * Spiritual Stewardship || Of the 17 students who completed or left the program in 2012-2013, we had both advisor and class ratings of professionalism from the MA-PIBS instrument for 12. For the remaining five, we had only advisor ratings. With the caveat, however, all 17 met or exceede expectations on those items measuring this learning outcome. || Excellent || Program students continue to excel at this learning outcome. No planned action at this time. ||


 * 2014 Rubric for Assessing Program LOs**

The MA program will be assessed annually at the beginning of winter semester. Because the purpose of the assessment is to see if the learning outcomes have been met by the end of the program, it does not make sense to consider data from students who are still in the program. However, if data is limited to only students who either graduated or dropped out of the program the year before, the data points are insufficient to establish clear trends. Consequently, at each annual evaluation of the program, data will be considered for students who either graduated or dropped out over the past two years. While this means that the data from a particular student will be used in two consecutive evaluations of program learning outcomes, we nonetheless feel that data aggregated across two years will provide a fuller understanding of the success of our program than data from a single year.

In preparation to assess the program learning outcomes, each student who has graduated or dropped out of the program will be evaluated individually on each learning outcome to determine if that learning outcome was met by that student. The criteria for a learning outcome to be met by a student are given below. Once each student has been evaluated on each learning outcome, the number of students who have met each learning outcome will be tallied. If 70% of the students meet a particular learning outcome, then we will conclude that the program is satisfactory with respect to that learning outcome.


 * Criteria for Determining Whether a Student has Met a Particular Learning Outcome**

LO 1: Scholarship For each student, data from Sections I and III of the Master's Exam, from their Thesis/Project proposal defense and from their Thesis/Project final defense will be gathered. (If students were given a second chance on any of these measures, data will come from the last attempt.) Master's Exam Sections I and III: Students are considered satisfactory on this measure if the average of their scores on the four problems from Sections I and III is at least 70%. Thesis/Project proposal defense: Students are considered satisfactory on this measure if they passed (initially or eventually) their proposal defense. Thesis/Project final defense: Students are considered satisfactory on this measure if they passed (initially or eventually) their final defense. Students meet this learning outcome if their performance on any two of the three measures is satisfactory.

LO 2: Research For each student, data from Sections II and III of the Master's Exam, and from their Thesis/Project final defense will be gathered. (If students were given a second chance on either of these measures, data will come from the last attempt.) Master's Exam Sections II and III: Students are considered satisfactory on this measure if the average of their scores on the four problems from Sections II and III is at least 70%. Thesis/Project final defense: Students are considered satisfactory on this measure if they passed (initially or eventually) their final defense. Students completing a project meet this learning outcome if their performance on both measures is satisfactory. Students completing a thesis meet this learning outcome if their performance on the thesis defense is satisfactory.

LO 3: Mathematics For each student, data from Section IV of the Master's Exam will be gathered. (If students were given a second chance to take the Master's Exam, data will come from the last attempt.) Students meet this learning outcome if the average of their scores on the two problems from Section IV is at least 70%.

LO 4: Teaching For each student, teacher evaluation and student rating data from the last two courses taught will be gathered and the better of each of those measures will be used for analysis. For students to meet this learning outcome, they must receive a satisfactory rating on both measures.

STUDENT RATINGS: On the student ratings measure, there is an overall course rating, an overall instructor rating, and 23 sub-questions. Eighteen of the 23 sub-questions are rated on a 1-8 scale. If 9 or more of the sub-questions are rated below a 6 or if the overall instructor rating is below a 6, the student will receive an unsatisfactory rating on this measure. TEACHER EVALUATIONS: On the teacher evaluations measure, if the student receives a rating below a 3 in any of the 4 areas (classroom organization; instruction, discussion, and tasks; content; communication), they will receive an unsatisfactory rating on this measure.

LO 5: Professionalism For each student, two MA PIBS—the last MA PIBS from the project/thesis advisor and the MA PIBS from the last 3-credit-hour mathematics education course the student took—will be considered. For the student to be considered satisfactory on an item, the student must receive a rating of "meets" or "exceeds" expectations for that item on both MA PIBS. If a students is unsatisfactory on 3 or more items, then the student will be considered unsatisfactory in professionalism.

LO 6: Spiritual Stewardship For each student, items 1-3, 7, and 9 from two MA PIBS—the last MA PIBS from the project/thesis advisor and the MA PIBS from the last 3-credit-hour mathematics education course the student took—will be considered. For the student to be considered satisfactory on an item, the student must receive a rating of "meets" or "exceeds" expectations for that item on both MA PIBS. If a students is unsatisfactory on 2 or more items, then the student will be considered unsatisfactory in spiritual stewardship.