2013+BS+Program+Evaluation

The 2013 evaluation of the BS program is based on data collected in the 2012-2013 academic year. The following are the two columns we are required to submit as the findings of our evaluation. For the rubric on which the conclusions are based, see the description following the table.


 * LO || Conclusions Based on Evidence || Rating || Actions Taken or Planned ||
 * Mathematics || All graduates passed the Praxis Exam. We computed the average score on the CPAS and TWS items used to measure this outcome. In total, 100% of the CPAS and TWS items met or exceeded the target average scores of 3.5 and 1.6, respectively.

On a national survey, students in our program indicated that their experience at BYU had contributed to their ability to analyze quantitative problems "very much." This was higher than the BYU average (p < .001, small effect size 0.37) and similar to comparable mathematics education programs nationwide.

All graduates passed all of the courses used as indirect measures. This suggests that our graduates have met this learning outcome. || Excellent || Our program remains consistently strong in this area. Our measures indicate that students leave our program with a solid mathematical background. We will continue to monitor graduates' achievements in this area to make sure that they stay at the same high level. ||
 * Understanding of Mathematics Learners || We computed the average score on the CPAS and TWS items used to measure this outcome. In total, 85% of the CPAS and TWS items exceeded the target average scores of 3.5 and 1.6, respectively. This number is above the 80% required for this outcome to be met.

All graduates passed all of the courses used as indirect measures. This supports our conclusion that our students performance on this learning outcome was satisfactory. || Good || There were three CPAS items related to this learning outcome for which the means fell below the 3.5 cutoff. All three means were within .05 of the cutoff. This suggests that graduates may be marginal in the area measured by these three items, namely an understanding of student diversity. We plan to continue to monitor this area to see if a trend develops.

At our last annual assessment, we witnessed a radical change in the CPAS scores. We felt that the drastic change was due to inter-rater reliability, since no major changes in our program had occured during the past few years. To address this issue, all faculty members whose responsibilities include filling out the CPAS for our majors met to discuss how the CPAS should be scored so that scores are consistent across raters. Because a score of 3 on a scale of 1 to 5 represents competence, our faculty felt that an expectation of an average of 4 across graduates was unrealistic. Scores of 5 are seldom given, so we feel that achieving a mean of 3.5 on a CPAS item would require that most students receive a 3 or higher on that particular item. Thus, we decided to use a mean of 3.5 as a cutoff in determining whether students in our program have met the characteristics measured by a CPAS item.

The CPAS instrument will be modified in the upcoming year to be aligned with the Utah Effective Teaching Standards instead of the INTASC Standards. We will need to reconsider the alignment of our measures when modifications of this instrument are completed. ||
 * Instructional Design for Mathematics Learning || We computed the average score on the CPAS and TWS items used to measure this outcome. In total, 100% of the CPAS and TWS items exceeded the target average scores of 3.5 and 1.6, respectively. This number is above the 80% required for this outcome to be met.

All graduates passed all of the courses used as indirect measures. This supports our conclusion that our students performance on this learning outcome was excellent. || Excellent || The performance of this year's graduates in this area is strong. We will continue to monitor graduates' achievements in this area to make sure that they stay at the same high level.

We noted in our last assessment that drastic changes in CPAS scores had led us to question inter-rater reliability for this instrument. Please see "Actions Planned or Taken" for learning outcome "Understanding of Mathematics Learners" for the actions taken to remedy this problem. ||
 * Assessment of Mathematical Learning || We computed the average score on the CPAS and TWS items used to measure this outcome. In total, 91% of the CPAS and TWS items exceeded the target average scores of 3.5 and 1.6, respectively. This number is above the 80% required for this outcome to be met.

All graduates passed all of the courses used as indirect measures. This supports our conclusion that our students performance on this learning outcome was satisfactory. || Good || There was one CPAS item and one TWS item related to this learning outcome for which the means fell below the 3.5 and 1.6 cutoffs, respectively. Both means were within within .07 of their respective cutoff. This suggests that graduates may be marginal in the area measured by these two items, namely preassessing students and assessing students' ability to communicate mathematics. We plan to continue to monitor these areas to see if a trend develops.

We noted in our last assessment that drastic changes in CPAS scores had led us to question inter-rater reliability for this instrument. Please see "Actions Planned or Taken" for learning outcome "Understanding of Mathematics Learners" for the actions taken to remedy this problem. ||
 * Professionalism || We computed the average score on the CPAS, PIBS, and TWS items used to measure this outcome. In total, 100% of the CPAS, PIBS, and TWS items exceeded the target average scores of 3.5, 2.6, and 1.6, respectively.This number is above the 80% required for this outcome to be met.

On a national survey, students in our program worked with other classmates during class "often." This was higher than the BYU average (p < .05, small effect size .32) and higher than comparable mathematics education programs nation-wide (p < .01, small effect size .46). Students in our program responded that their experience at BYU contributed to their development in working effectively with others "very much." This was higher than the BYU average (p < .01, small effect size .35) and higher than comparable mathematics education programs nationwide (p. 001, medium effect size .64).

All graduates passed all of the courses used as indirect measures. This supports our conclusion that our students performance on this learning outcome was excellent. || Excellent || The performance of this year's graduates in this area is strong. We will continue to monitor graduates' achievements in this area to make sure that they stay at the same high level.

We noted in our last assessment that drastic changes in CPAS scores had led us to question inter-rater reliability for this instrument. Please see "Actions Planned or Taken" for learning outcome "Understanding of Mathematics Learners" for the actions taken to remedy this problem.

Also, the scale for the PIBS changed from a four point scale to a three point scale. The cutoff for the means on PIBS items was modified to reflect this change. The previous cutoff had been 3.4. The previous cutoff was set by taking 80% of the range in the scale and adding it to the lowest value of the scale (i.e., .8(4-1) + 1 = 3.4). We used the same method to produce the new cutoff (i.e., .8(3-1) + 1 = 2.6). ||
 * Spiritual Stewardship || We computed the average score on the CPAS and PIBS items used to measure this outcome. In total, 88% of the CPAS and PIBS items exceeded the target average scores of 3.5 and 2.6, respectively. This number is above the 80% required for this outcome to be met.

On a national survey, students in our program reported that their experience at BYU had contributed to the development of a personal code of values "very much." This was similar to the BYU average but much higher than comparable mathematics education programs nationwide (p < .001, large effect size of .97). Students in our program also indicated that their experience at BYU had "very much" helped them develop a deepened sense of spirituality. This was higher than the BYU average (p < .05, small effect size of .28) and extremely higher than comparable mathematics education programs nationwide (p < .001, large effect size of 1.85).

All graduates passed all of the courses used as indirect measures. This supports our conclusion that our students performance on this learning outcome was satisfactory. || Good || There were two CPAS items related to this learning outcome for which the means fell below the 3.5 cutoff. Both means were within .05 of the cutoff. This suggests that graduates may be marginal in the area measured by these two items, namely an understanding of student diversity. We plan to continue to monitor this area to see if a trend develops.

We noted in our last assessment that drastic changes in CPAS scores had led us to question inter-rater reliability for this instrument. Please see "Actions Planned or Taken" for learning outcome "Understanding of Mathematics Learners" for the actions taken to remedy this problem.

Please see "Actions Planned or Taken" for the learning outcome "Professionalism for an explanation of how the PIBS items cuttoff was changed. ||

2013 Rubric for Assessing Program LOs
All LOs are measured by items from the CPAS, PIBS, and TWS. For the results of an item from one of these exams to be considered satisfactory, the mean of the scores on that item for all graduates of the academic year being assessed must be no lower than the following: To determine whether or not a particular learning outcome has been met, all of the CPAS, PIBS, and TWS items related to that LO will first be evaluated to see if they suggest satisfactory results. If 80% or more of the items are considered to be satisfactory, then we will conclude that the LO has been met. If less than 80% of the items are considered to be satisfactory, then we will conclude that the LO has not been met.
 * CPAS item: The mean must be no lower than 3.5 (on a scale from 1 to 5).
 * PIBS item: The mean must be no lower than 2.6 (on a scale from 1 to 3).
 * TWS item: The mean must be no lower than 1.6 (on a scale from 0 to 2).

For the //Mathematics// LO to be considered as having been met, there must also be a 95% pass rate of the //Praxis// exam by graduates of the academic year being assessed (a score of 135 is considered to be passing by the state of Utah).