2014+BS+Program+Evaluation

The 2014 evaluation of the BS program is based on data collected in the 2013-2014 academic year. The following are the three columns we are required to submit as the findings of our evaluation. For the rubric on which the conclusions are based, see the description following the table.
 * LO || Conclusions Based on Evidence || Rating || Actions Taken or Planned ||
 * Mathematics || Students in the program excelled on this learning outcome. All graduating seniors passed the Praxis exam. All six items from the CPAS and TWS that are related to this learning outcome met the cutoff averages of 3.1 and 1.4, respectively.

Program students passed all of the courses that address this learning outcome. || Excellent || Our program continues to do well with respect to this learning outcome. Despite this excellence, we are taking two steps to strengthen students' mathematical knowledge. First, we are meeting regularly (1-2 times a month) to identify what specific mathematical content will be addressed in each mathematics education course. We hope to be able to address a wider range of secondary school mathematics topics as a result of this realignment. Second, we are proposing a program change to require our students to take an introductory statistics course from the Statistics Department (currently not required). We are also proposing that our department take over the currently required statistics class and modify it so that it focuses on both the conceptual underpinnings of basic statistics and the knowledge needed to teach and assess learning in statistics. It is our hope that this program change will prepare our students adequately to teach AP statistics classes. ||
 * Understanding of Mathematics Learners || Of the items on the CPAS and TWS that are related to this learning outcome, 86% met the cutoff averages of 3.1 and 1.4, respectively.

Every graduate passed all of the courses that address this learning outcome. Direct and indirect measures suggest that this program learning outcome is being met. || Good || In last year's evaluation, we noted that we had achieved better inter-rater reliability on the CPAS and TWS. However, with the addition of new faculty during the last year, our CPAS and TWS scores are once again consistently lower across most items, making the benchmarks for averages of 3.5 and 1.6 for CPAS and TWS items unrealistic. Because most of these items assess learning across multiple courses, it is more likely that the drop in scores is due to inconsistency across raters instead of a sudden drop in the quality of instruction. Once again, it is necessary for those who will be assigning these scores to meet and come to some agreement about how to rate items. Additional complications in interpreting the scores resulted from (1) the CPAS being modified to fit with Utah Effective Teaching Standards, and (2) discontinuation of the PIBS and an unsuccessful implementation of its replacement, resulting in a loss of the information captured by this particular instrument. In order to address these challenges, we realigned the modified items on the CPAS so that they fit with our learning outcomes. After considering the average decreases in CPAS and TWS items between the two years, we decided that cutoff scores of 3.1 and 1.4 for CPAS and TWS items, respectively, are more appropriate standards for our program.

In response to last year's performance on this learning outcome, we have been meeting as a faculty to discuss how to improve our students' understanding of learner diversity. We are currently identifying particular learning goals for our students in relation to recognizing and addressing diversity and specifying the course(s) in which each learning goal should be addressed. Because this process is being conducted alongside our realignment of mathematics content (as noted above), the process is slow and time consuming. We hope to have these goals in place for Fall 2015. ||
 * Instructional Design for Mathematics Learning || Of the items on the CPAS and TWS that are related to this learning outcome, 93% met the cutoff averages of 3.1 and 1.4, respectively.

Every graduate passed all of the courses that address this learning outcome. Direct and indirect measures suggest that this program learning outcome is being met. || Excellent || Measures for this learning outcome suggest that are students are doing well with respect to instruction design. Despite this success, we are currently meeting as a whole faculty to identify specific genres related to teaching that we want our students to learn, create descriptions and examples of these genres, and identify in which course(s) each genre will be taught. Genres that we have already identified and described include concept description, task description, lesson plan, and unit description. We hope to have a coordinated plan for teaching these genres in place by Fall 2015. ||
 * Assessment of Mathematical Learning || Of the items on the CPAS and TWS that are related to this learning outcome, 86% met the cutoff averages of 3.1 and 1.4, respectively.

Every graduate passed all of the courses that address this learning outcome. Direct and indirect measures suggest that this program learning outcome is being met. || Good || In last year's report, we noted that graduates were marginal on items that measured preassessment and assessing learner's communication of mathematics. Because of the changes in the CPAS, the new items do not correspond directly with old items. Thus, it is not possible to tell if graduates still struggle in these areas. An analysis of this year's items on which the average fell below cutoffs suggest that our graduates may need more help in using and coordinating a variety of different assessments. The faculty will add this issue to their discussions of which genres our students should learn, and perhaps add additional genres which could address this issue. || 93% of CPAS and TWS items met the cutoff averages of 3.1 and 1.4, respectively. || Excellent || Our students' performance on this learning outcome continues to be strong. We will continue to monitor our students performance in this area. || 75% of items met the cutoff average of 3.1. || Fair || In our last evaluation, we noted that our students' understanding of learner diversity may be marginal. Because of the changes in the CPAS, the new items do not correspond directly with old items. Thus, it is it is not clear whether graduates still struggle in exactly the same ways. However, analysis of this year's items on which the average fell below cutoffs suggest that our graduates may need more help in getting to know students' individual strengths and weaknesses and in recognizing the effects of these differences on learning. We are currently addressing this issue in faculty discussions. ||
 * Professionalism || Of the items on the CPAS and TWS that are related to this learning outcome,
 * Spiritual Stewardship || Of the items on the CPAS that are related to this learning outcome,

2014 Rubric for Assessing Program LOs [UPDATE]
All LOs are measured by items from the CPAS, PIBS, and TWS. For the results of an item from one of these exams to be considered satisfactory, the mean of the scores on that item for all graduates of the academic year being assessed must be no lower than the following:


 * CPAS item: The mean must be no lower than 3.1 (on a scale from 1 to 5).


 * PIBS item: The PIBS is no longer used.


 * TWS item: The mean must be no lower than 1.4 (on a scale from 0 to 2).

To determine whether or not a particular learning outcome has been met, all of the CPAS, PIBS, and TWS items related to that LO will first be evaluated to see if they suggest satisfactory results. If 80% or more of the items are considered to be satisfactory, then we will conclude that the LO has been met. If less than 80% of the items are considered to be satisfactory, then we will conclude that the LO has not been met. For the //Mathematics// LO to be considered as having been met, there must also be a 95% pass rate of the //Praxis// exam by graduates of the academic year being assessed (a score of 135 is considered to be passing by the state of Utah).