INTERNATIONAL JOURNAL OF CHANGES IN EDUCATION
Research Article
Nov 15, 2024 1 0

Unintended Consequences: When Innovation in Pedagogy Impacts Student Evaluations

Rosemary Fisher 1 * ,
Chamila Perera 1,
Richard Laferriere 1
1 School of Business, Law and Entrepreneurship, Swinburne University of Technology, Australia* Corresponding Author
International Journal of Changes in Education, 1(4), 2024, 169-176, https://doi.org/10.47852/bonviewIJCE42023045
Publication date: Nov 15, 2024
Full Text (PDF)

ABSTRACT

Student evaluation of teaching surveys (SETS) is one of the most controversial tools used in higher education, globally. Recently the impact that SETS had on innovation in pedagogy has been raised as a potential problem. The teaching team experienced a gap between the SETS results generated for a final-year undergraduate class that utilized flipped learning as the innovation and an empirical research project using the same cohort of students that investigated the effectiveness of that pedagogy. Accordingly, from 2014, we established this crosssectional sequential study over seven semesters to understand how students used the SETS after experiencing this innovation. We conducted a thematic analysis on the 588 SETS results from a final-year undergraduate class studying at an Australian university and found resisting students used the SETS as a weapon with adoption of the innovation of the prime casualty. We recommend SETS be tailored for use where students experience innovation in pedagogy as the use of SETS by students may undermine the adoption of otherwise effective pedagogy and impact the willingness of faculty to pursue innovation in pedagogy.

KEYWORDS

flipped learning student evaluations pedagogy innovation

CITATION (APA)

Fisher, R., Perera, C., & Laferriere, R. (2024). Unintended Consequences: When Innovation in Pedagogy Impacts Student Evaluations. International Journal of Changes in Education, 1(4), 169-176. https://doi.org/10.47852/bonviewIJCE42023045
Harvard
Fisher, R., Perera, C., and Laferriere, R. (2024). Unintended Consequences: When Innovation in Pedagogy Impacts Student Evaluations. International Journal of Changes in Education, 1(4), pp. 169-176. https://doi.org/10.47852/bonviewIJCE42023045
Vancouver
Fisher R, Perera C, Laferriere R. Unintended Consequences: When Innovation in Pedagogy Impacts Student Evaluations. International Journal of Changes in Education. 2024;1(4):169-76. https://doi.org/10.47852/bonviewIJCE42023045
AMA
Fisher R, Perera C, Laferriere R. Unintended Consequences: When Innovation in Pedagogy Impacts Student Evaluations. International Journal of Changes in Education. 2024;1(4), 169-176. https://doi.org/10.47852/bonviewIJCE42023045
Chicago
Fisher, Rosemary, Chamila Perera, and Richard Laferriere. "Unintended Consequences: When Innovation in Pedagogy Impacts Student Evaluations". International Journal of Changes in Education 2024 1 no. 4 (2024): 169-176. https://doi.org/10.47852/bonviewIJCE42023045
MLA
Fisher, Rosemary et al. "Unintended Consequences: When Innovation in Pedagogy Impacts Student Evaluations". International Journal of Changes in Education, vol. 1, no. 4, 2024, pp. 169-176. https://doi.org/10.47852/bonviewIJCE42023045

REFERENCES

  1. McLaughlin, J. E., Roth, M. T., Glatt, D. M., Gharkholonarehe, N., Davidson, C. A., Griffin, L. M., ..., & Mumper, R. J. (2014). The flipped classroom: A course redesign to foster learning and engagement in a health professions school. Academic Medicine, 89(2), 236–243. https://doi.org/10.1097/ACM.0000000000000086
  2. O’Flaherty, J., & Phillips, C. (2015). The use of flipped classrooms in higher education: A scoping review. The Internet and Higher Education, 25, 85–95. https://doi.org/10.1016/j.iheduc.2015.02.002
  3. Karabulut-Ilgu, A., Jaramillo Cherrez, N., & Jahren, C. T. (2018). A systematic review of research on the flipped learning method in engineering education. British Journal of Educational Technology, 49(3), 398–411. https://doi.org/10.1111/bjet.12548
  4. Missildine, K., Fountain, R., Summers, L., & Gosselin, K. (2013). Flipping the classroom to improve student performance and satisfaction. Journal of Nursing Education, 52(10), 597–599. https://doi.org/10.3928/01484834-20130919-03
  5. Wasserman, N. H., Quint, C., Norris, S. A., & Carr, T. (2017). Exploring flipped classroom instruction in calculus III. International Journal of Science and Mathematics Education, 15, 545–568. https://doi.org/10.1007/s10763-015-9704-8
  6. Priem, R. L. (2018). Toward becoming a complete teacher of strategic management. Academy of Management Learning & Education, 17(3), 374–388. https://doi.org/10.5465/amle.2017.0237
  7. Canning, J. (2017). Conceptualising student voice in UK higher education: Four theoretical lenses. Teaching in Higher Education, 22(5), 519–531. https://doi.org/10.1080/13562517.2016.1273207
  8. Clayson, D. E., Frost, T. F., & Sheffet, M. J. (2006). Grades and the student evaluation of instruction: A test of the reciprocity effect. Academy of Management Learning & Education, 5(1), 52–65. https://doi.org/10.5465/amle.2006.20388384
  9. Darwin, S. (2017). What contemporary work are student ratings actually doing in higher education? Studies in Educational Evaluation, 54, 13–21. https://doi.org/10.1016/j.stueduc.2016.08.002
  10. Freeman, R. (2016). Is student voice necessarily empowering? Problematising student voice as a form of higher education governance. Higher Education Research & Development, 35(4), 859–862. https://doi.org/10.1080/07294360.2016.1172764
  11. Clayson, D. E. (2009). Student evaluations of teaching: Are they related to what students learn: A meta-analysis and review of the literature. Journal of Marketing Education, 31(1), 16–30. https://doi.org/10.1177/0273475308324086
  12. Uttl, B., White, C. A., & Gonzalez, D. W. (2017). Meta-analysis of faculty’s teaching effectiveness: Student evaluation of teaching ratings and student learning are not related. Studies in Educational Evaluation, 54, 22–42. https://doi.org/10.1016/j.stueduc.2016.08.007
  13. Linse, A. R. (2017). Interpreting and using student ratings data: Guidance for faculty serving as administrators and on evaluation committees. Studies in Educational Evaluation, 54, 94–106. https://doi.org/10.1016/j.stueduc.2016.12.004
  14. Chong, Y. S., & Ahmed, P. K. (2015). Student motivation and the ‘feel good’ factor: An empirical examination of motivational predictors of university service quality evaluation. Studies in Higher Education, 40(1), 158–177. https://doi.org/10.1080/03075079.2013.818643
  15. Lederman,D. (2020). Evaluating teaching during the pandemic. Retrieved from: https://www.insidehighered.com/digital-learning/article/2020/04/08/many-colleges-are-abandoning-or-downgrading-student-evaluations
  16. McAndrew, P. (2015). Innovating for learning: Designing for the future of education. In Proceedings of the 14th European Conference on E-Learning, 356–363.
  17. Samuel, M. L. (2021). Flipped pedagogy and student evaluations of teaching. Active Learning in Higher Education, 22(2), 159–168. https://doi.org/10.1177/1469787419855188
  18. Pratt, M. G. (2008). Fitting oval pegs into round holes: Tensions in evaluating and publishing qualitative research in top-tier North American journals. Organizational Research Methods, 11(3), 481–509. https://doi.org/10.1177/1094428107303349
  19. Rogers, E. M. (2003). Diffusion of innovations (5th ed.). USA: Free Press.
  20. Oreg, S. (2003). Resistance to change: Developing an individual differences measure. Journal of Applied Psychology, 88(4), 680–693. https://doi.org/10.1037/0021-9010.88.4.680
  21. Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50(2), 179–211. https://doi.org/10.1016/0749-5978(91)90020-T
  22. Benton, S. L., & Cashin, W. E. (2014). Student ratings of instruction in college and university courses. In M. B. Paulsen (Ed.), Higher education: Handbook of theory and research: Vol. 29 (pp. 279–326). Springer.
  23. Aleamoni, L. M. (1999). Student rating myths versus research facts from 1924 to 1998. Journal of Personnel Evaluation in Education, 13, 153–166. https://doi.org/10.1023/A:1008168421283
  24. Cohen, P. A. (1981). Student ratings of instruction and student achievement: A meta-analysis of multisection validity studies. Review of Educational Research, 51(3), 281–309. https://doi.org/10.3102/00346543051003281
  25. Davies, M., Hirschberg, J., Lye, J., Johnston, C., & Mcdonald, I. (2007). Systematic influences on teaching evaluations: The case for caution. Australian Economic Papers, 46(1), 18–38. https://doi.org/10.1111/j.1467-8454.2007.00303.x
  26. Falchikov, N., & Boud, D. (1989). Student self-assessment in higher education: A meta-analysis. Review of Educational Research, 59(4), 395–430. https://doi.org/10.3102/00346543059004395
  27. Schiekirka, S., & Raupach, T. (2015). A systematic review of factors influencing student ratings in undergraduate medical education course evaluations. BMC Medical Education, 15, 30. https://doi.org/10.1186/s12909-015-0311-8
  28. Spooren, P., Brockx, B., & Mortelmans, D. (2013). On the validity of student evaluation of teaching: The state of the art. Review of Educational Research, 83(4), 598–642. https://doi.org/10.3102/0034654313496870
  29. MacNell, L., Driscoll, A., & Hunt, A. N. (2015). What’s in a name: Exposing gender bias in student ratings of teaching. Innovative Higher Education, 40(4), 291–303. https://doi.org/10.1007/s10755-014-9313-4
  30. Richmond, A. S., Berglund, M. B., Epelbaum, V. B., & Klein, E. M. (2015). a + (b1) Professor–student rapport + (b2) humor + (b3) student engagement = (Ŷ) student ratings of instructors. Teaching of Psychology, 42(2), 119–125. https://doi.org/10.1177/0098628315569924
  31. Young, S., & Duncan, H. E. (2014). Online and face-to-face teaching: How do student ratings differ? MERLOT Journal of Online Learning and Teaching, 10(1), 70–79.
  32. Hessler, M., Pöpping, D. M., Hollstein, H., Ohlenburg, H., Arnemann, P. H., Massoth, C., ..., & Wenk, M. (2018). Availability of cookies during an academic course session affects evaluation of teaching. Medical Education, 52(10), 1064–1072. https://doi.org/10.1111/medu.13627
  33. Benton, S. L., Duchon, D., & Pallett, W. H. (2013). Validity of student self-reported ratings of learning. Assessment & Evaluation in Higher Education, 38(4), 377–388. https://doi.org/10.1080/02602938.2011.636799
  34. Kyriakides, L., Creemers, B. P. M., Panayiotou, A., Vanlaar, G., Pfeifer, M., Cankar, G., & McMahon, L. (2014). Using student ratings to measure quality of teaching in six European countries. European Journal of Teacher Education, 37(2), 125–143. https://doi.org/10.1080/02619768.2014.882311
  35. Boysen, G. A. (2015). Uses and misuses of student evaluations of teaching: The interpretation of differences in teaching evaluation means irrespective of statistical information. Teaching of Psychology, 42(2), 109–118. https://doi.org/10.1177/0098628315569922
  36. Hornstein, H. A., & Law, H. F. E. (2017). Student evaluations of teaching are an inadequate assessment tool for evaluating faculty performance. Cogent Education, 4(1), 1304016. https://doi.org/10.1080/2331186X.2017.1304016
  37. Vaillancourt, T. (2013). Students aggress against professors in reaction to receiving poor grades: An effect moderated by student narcissism and self-esteem. Aggressive Behavior, 39(1), 71–84. https://doi.org/10.1002/ab.21450
  38. Burke, L. A., Karl, K., Peluchette, J., & Evans, W. R. (2014). Student incivility: A domain review. Journal of Management Education, 38(2), 160–191. https://doi.org/10.1177/1052562913488112
  39. Davidovitch, N., & Eckhaus, E. (2019). Teaching students to think – Faculty recommendations for teaching evaluations employing automated content analysis. International Journal of Higher Education, 8(3), 83–93.
  40. Fisher, W., Orr, J., Page, J., Pelizzon, A., & Walsh, H. (2020). Student evaluations: Pedagogical tools or weapons of choice? Legal Education Review, 30(1), 1–28.
  41. Rodriguez, J. (2019). The weaponization of student evaluations of teaching: Bullying and the undermining of academic freedom. AAUP Journal of Academic Freedom, 10, 1–16.
  42. Feldmann, L. J. (2001). Classroom civility is another of our instructor responsibilities. College Teaching, 49(4), 137–140. https://doi.org/10.1080/87567555.2001.10844595
  43. Balasubramanian, A. (2003). From the student’s view: Student course evaluations. Retrieved from: https://cft.vanderbilt.edu/articles-and-essays/the-teaching-forum/from-the-students-viewstudent-course-evaluations/
  44. May, A., & Tenzek, K. E. (2018). Bullying in the academy: Understanding the student bully and the targeted ‘stupid, fat, mother fucker’ professor. Teaching in Higher Education, 23(3), 275–290. https://doi.org/10.1080/13562517.2017.1379482
  45. Nordstrom, C. R., Bartels, L. K., & Bucy, J. (2009). Predicting and curbing classroom incivility in higher education. College Student Journal, 43(1), 74.
  46. Guest, G., MacQueen, K. M., & Namey, E. E. (2011). Applied thematic analysis. USA: SAGE Publications.
  47. Patton, M. Q. (1990). Qualitative evaluation and research methods. USA: SAGE Publications.
  48. Hao, Y. (2016). Exploring undergraduates’ perspectives and flipped learning readiness in their flipped classrooms. Computers in Human Behavior, 59, 82–92. https://doi.org/10.1016/j.chb.2016.01.032
  49. Wilson, K. (2023). What does it mean to do teaching? Aqualitative study of resistance to flipped learning in a higher education context. Teaching in Higher Education, 28(3), 473–486. https://doi.org/10.1080/13562517.2020.1822312
  50. Burke, A. S., & Fedorek, B. (2017). Does “flipping” promote engagement?:Acomparison of a traditional, online, and flipped class. Active Learning in Higher Education, 18(1), 11–24. https://doi.org/10.1177/1469787417693487
  51. Roehling, P. V., Root Luna, L. M., Richie, F. J., & Shaughnessy, J. J. (2017). The benefits, drawbacks, and challenges of using the flipped classroom in an introduction to psychology course. Teaching of Psychology, 44(3), 183–192. https://doi.org/10.1177/0098628317711282
  52. Boud, D. (2001). Making the move to peer learning. In D. Boud, R. Cohen&J. Sampson (Eds.), Peer learning in higher education: Learning from and with each other (pp. 1–20). Routledge.
  53. Grebennikov, L., & Shah, M. (2013). Monitoring trends in student satisfaction. Tertiary Education and Management, 19(4), 301–322. https://doi.org/10.1080/13583883.2013.804114
  54. Loughry, M. L., & Tosi, H. L. (2008). Performance implications of peer monitoring. Organization Science, 19(6), 876–890. https://doi.org/10.1287/orsc.1080.0356
  55. Trevino, L. K., & Victor, B. (1992). Peer reporting of unethical behavior: A social context perspective. Academy of Management Journal, 35(1), 38–64.
  56. Etzioni, A. (1968). Social control: Organizational aspects. In D. L. Sills & R. K. Merton (Eds.), International encyclopedia of social sciences, Vol. 14 (pp. 369–402). Macmillan.
  57. Sitzmann, T., Ely, K., Brown, K. G., & Bauer, K. N. (2010). Self-assessment of knowledge: A cognitive learning or affective measure? Academy of Management Learning & Education, 9(2), 169–191. https://doi.org/10.5465/amle.9.2.zqr169
  58. Beatty, J. E. (2004). Grades as money and the role of the market metaphor in management education. Academy of Management Learning & Education, 3(2), 187–196. https://doi.org/10.5465/amle.2004.13500516
  59. Stroebe, W. (2020). Student evaluations of teaching encourages poor teaching and contributes to grade inflation: A theoretical and empirical analysis. Basic and Applied Social Psychology, 42(4), 276–294. https://doi.org/10.1080/01973533.2020.1756817
  60. Pérez, O. E. G., & Trevino, J. P. (2019). Learning-oriented assessment in action: Impact on students of physics for engineering. International Journal on Interactive Design and Manufacturing, 13(4), 1485–1501. https://doi.org/10.1007/s12008-019-00606-2
  61. Shibukawa, S., & Taguchi, M. (2019). Exploring the difficulty on students’ preparation and the effective instruction in the flipped classroom: A case study in a physiology class. Journal of Computing in Higher Education, 31(2), 311–339. https://doi.org/10.1007/s12528-019-09220-3
  62. McNatt, D. B. (2010). Negative reputation and biased student evaluations of teaching: Longitudinal results from a naturally occurring experiment. Academy of Management Learning & Education, 9(2), 225–242. https://doi.org/10.5465/amle.9.2.zqr225
  63. Karanicolas, S., & Snelling, C. (2010). Making the transition: Achieving content connectivity and student engagement through flexible learning tools. In Proceedings of the Distance Education Association of New Zealand (DEANZ) Conference, 1–13.
  64. Borch, I., Sandvoll, R., & Risør, T. (2022). Student course evaluation documents: Constituting evaluation practice. Assessment & Evaluation in Higher Education, 47(2), 169–182. https://doi.org/10.1080/02602938.2021.1899130

LICENSE

Creative Commons License
This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.