INTERNATIONAL JOURNAL OF CHANGES IN EDUCATION
Review
Feb 25, 2025 1 0

Recommendations for Integrating Automated Writing Evaluation with Evidence-Based Instructional Practices

Joshua Wilson 1 * ,
Tania Cruz Cordero 1,
Andrew Potter 2,
Matthew Myers 1,
Charles A. MacArthur 1,
Gaysha Beard 3,
Emily A. Fudge 3,
Alexandria Raiche 4,
Cristina Ahrendt 5
1 School of Education, University of Delaware, USA2 Department of Psychology, Arizona State University, US3 Red Clay Consolidated School District, USA4 Milford School District, USA5 University of Delaware, USA* Corresponding Author
International Journal of Changes in Education, 2(1), 2025, 46-54, https://doi.org/10.47852/bonviewIJCE42024011
Publication date: Feb 25, 2025
Full Text (PDF)

ABSTRACT

Automated writing evaluation systems are formative assessment systems that provide immediate, automated feedback on L1, L2, and EFL students’ writing in the form of writing quality scores and suggestions for revising. As such, these systems have the potential for alleviating some of the persistent barriers teachers face to implementing evidence-based writing instruction practices. However, simply adopting this technology without careful attention to how it is implemented will not guarantee instructional benefits. In this article, we draw on prior research to make recommendations to effectively integrate automated writing evaluation alongside evidence-based writing instruction practices to improve writing instruction and intervention, leveraging the affordances of this technology while addressing its limitations. Specifically, we discuss how researchers, interventionists, and educators using automated writing evaluation should develop students’ knowledge of underlying evaluation criteria; teach strategies for planning, drafting, and revising; supplement automated feedback with effective teacher-provided feedback; and enact goal setting and progress monitoring.

KEYWORDS

automated writing evaluation automated scoring writing writing instruction writing assessment automated feedback

CITATION (APA)

Wilson, J., Cordero, T. C., Potter, A., Myers, M., MacArthur, C. A., Beard, G., Fudge, E. A., Raiche, A., & Ahrendt, C. (2025). Recommendations for Integrating Automated Writing Evaluation with Evidence-Based Instructional Practices. International Journal of Changes in Education, 2(1), 46-54. https://doi.org/10.47852/bonviewIJCE42024011
Harvard
Wilson, J., Cordero, T. C., Potter, A., Myers, M., MacArthur, C. A., Beard, G., . . . Ahrendt, C. (2025). Recommendations for Integrating Automated Writing Evaluation with Evidence-Based Instructional Practices. International Journal of Changes in Education, 2(1), pp. 46-54. https://doi.org/10.47852/bonviewIJCE42024011
Vancouver
Wilson J, Cordero TC, Potter A, Myers M, MacArthur CA, Beard G, et al. Recommendations for Integrating Automated Writing Evaluation with Evidence-Based Instructional Practices. International Journal of Changes in Education. 2025;2(1):46-54. https://doi.org/10.47852/bonviewIJCE42024011
AMA
Wilson J, Cordero TC, Potter A, et al. Recommendations for Integrating Automated Writing Evaluation with Evidence-Based Instructional Practices. International Journal of Changes in Education. 2025;2(1), 46-54. https://doi.org/10.47852/bonviewIJCE42024011
Chicago
Wilson, Joshua, Tania Cruz Cordero, Andrew Potter, Matthew Myers, Charles A. MacArthur, Gaysha Beard, Emily A. Fudge, Alexandria Raiche, and Cristina Ahrendt. "Recommendations for Integrating Automated Writing Evaluation with Evidence-Based Instructional Practices". International Journal of Changes in Education 2025 2 no. 1 (2025): 46-54. https://doi.org/10.47852/bonviewIJCE42024011
MLA
Wilson, Joshua et al. "Recommendations for Integrating Automated Writing Evaluation with Evidence-Based Instructional Practices". International Journal of Changes in Education, vol. 2, no. 1, 2025, pp. 46-54. https://doi.org/10.47852/bonviewIJCE42024011

REFERENCES

  1. Graham, S. (2023). Writer(s)-within-community model of writing as a lens for studying the teaching of writing. In R. Horowitz (Ed.), The Routledge international handbook of research on writing (pp. 337–350). Routledge. https://doi.org/10.4324/9780429437991-27
  2. Graham, S., & Harris, K. R. (2017). Evidence-based writing practices: A meta-analysis of existing meta-analyses. In R. Fidalgo, K. R. Harris & M. Braaksma (Eds.), Design principles for teaching effective writing: Theoretical and empirical grounded principles (pp. 13–37). Brill. https://doi.org/10.1163/9789004270480_003
  3. National Center for Education Statistics. (2012). The Nation’s report card: Writing 2011 (NCES 2012–470). Retrieved from: https://nces.ed.gov/nationsreportcard/pdf/main2011/2012470.pdf
  4. Baker, N. L. (2014). “Get it off my stack”: Teachers’ tools for grading papers. Assessing Writing, 19, 36–50. https://doi.org/10.1016/j.asw.2013.11.005
  5. Shermis, M. D., & Wilson, J. (2024). The Routledge international handbook of automated essay evaluation. UK: Routledge.
  6. Cotos, E. (2023). Automated feedback on writing. In O. Kruse, C. Rapp, C. M. Anson, K. Benetos, E. Cotos, A. Devitt & A. Shibani (Eds.), Digital writing technologies in higher education: Theory, research, and practice (pp. 347–364). Springer. https://doi.org/10.1007/978-3-031-36033-6_22
  7. Deane, P. (2024). Automated writing trait analysis. In M. D. Shermis & J. Wilson (Eds.), The Routledge international handbook of automated essay evaluation (pp. 278–302). Routledge.
  8. Scherer, S., Graham, S., & Busse, V. (2024). How effective is feedback for L1, L2, and FL learners’ writing?A meta-analysis. Learning and Instruction, 93, 101961. https://doi.org/10.1016/j.learninstruc.2024.101961
  9. Wilson, J., & MacArthur, C. A. (2024). Exploring the role of automated writing evaluation as a formative assessment tool supporting self-regulated learning in writing. In M. D. Shermis & J. Wilson (Eds.), The Routledge international handbook of automated essay evaluation (pp. 197–220). Routledge. https://doi.org/10.4324/9781003397618-14
  10. Fu, Q. K., Zou, D., Xie, H., & Cheng, G. (2024). A review of AWE feedback: Types, learning outcomes, and implications. Computer Assisted Language Learning, 37(1–2), 179–221.https://doi.org/10.1080/09588221.2022.2033787
  11. Hockly, N. (2019). Automated writing evaluation. ELT Journal, 73(1), 82–88. https://doi.org/10.1093/elt/ccy044
  12. Shi, Z., Liu, F., Lai, C., & Jin, T. (2022). Enhancing the use of evidence in argumentative writing through collaborative processing of content-based automated writing evaluation feedback. Language Learning & Technology, 26(2), 106–128. https://doi.org/10125/73481
  13. Butterfuss, R., Roscoe, R. D., Allen, L. K., McCarthy, K. S., & McNamara, D. S. (2022). Strategy uptake in writing pal: Adaptive feedback and instruction. Journal of Educational Computing Research, 60(3), 696–721. https://doi.org/10.1177/07356331211045304
  14. Fang, Y., Roscoe, R. D., & McNamara, D. S. (2023). Artificial intelligence-based assessment in education. In B. du Boulay, A. Mitrovic & K. Yacef (Eds.), Handbook of artificial intelligence in education (pp. 485–504). Edward Elgar Publishing. https://doi.org/10.4337/9781800375413.00033
  15. Keller-Margulis, M. A., Mercer, S. H., & Matta, M. (2021). Validity of automated text evaluation tools for written expression curriculum-based measurement: A comparison study. Reading and Writing, 34(10), 2461–2480. https://doi.org/10.1007/s11145-021-10153-6
  16. Mercer, S. H., Cannon, J. E., Squires, B., Guo, Y., & Pinco, E. (2021). Accuracy of automated written expression curriculum based measurement scoring. Canadian Journal of School Psychology, 36(4), 304–317. https://doi.org/10.1177/0829573520987753
  17. Grimes, D., & Warschauer, M. (2010). Utility in a fallible tool: A multi-site case study of automated writing evaluation. The Journal of Technology, Learning, and Assessment, 8(6), 1–44.
  18. Perelman, L. (2020). The BABEL generator and e-rater: 21st century writing constructs and automated essay scoring (AES). Journal of Writing Assessment, 13(1), 1–9.
  19. Deane, P. (2013). On the relation between automated essay scoring and modern views of the writing construct. Assessing Writing, 18(1), 7–24. https://doi.org/10.1016/j.asw.2012.10.002
  20. Barrot, J. S. (2024). Trends in automated writing evaluation systems research for teaching, learning, and assessment: A bibliometric analysis. Education and Information Technologies, 29, 7155–7199. https://doi.org/10.1007/s10639-023-12083-y
  21. Shi, H., & Aryadoust, V. (2023). A systematic review of automated writing evaluation systems. Education and Information Technologies, 28(1), 771–795. https://doi.org/10.1007/s10639-022-11200-7
  22. Li, Z. (2021). Teachers in automated writing evaluation (AWE) system-supported ESL writing classes: Perception, implementation, and influence. System, 99, 102505. https://doi.org/10.1016/j.system.2021.102505
  23. Kellogg, R. T., Whiteford, A. P., & Quinlan, T. (2010). Does automated feedback help students learn to write? Journal of Educational Computing Research, 42(2), 173–196. https://doi.org/10.2190/EC.42.2.c
  24. Fleckenstein, J., Liebenow, L. W., & Meyer, J. (2023). Automated feedback and writing: A multi-level metaanalysis of effects on students’ performance. Frontiers in Artificial Intelligence, 6, 1162454. https://doi.org/10.3389/frai.2023.1162454
  25. Li, R. (2023). Still a fallible tool? Revisiting effects of automated writing evaluation from activity theory perspective. British Journal of Educational Technology, 54(3), 773–789. https://doi.org/10.1111/bjet.13294
  26. Ngo, T. T. N., Chen, H. H. J., & Lai, K. K. W. (2024). The effectiveness of automated writing evaluation in EFL/ESL writing: A three level meta-analysis. Interactive Learning Environments, 32(2), 727–744. https://doi.org/10.1080/10494820.2022.2096642
  27. Nunes, A., Cordeiro, C., Limpo, T., & Castro, S. L. (2022). Effectiveness of automated writing evaluation systems in school settings: A systematic review of studies from 2000 to 2020. Journal of Computer Assisted Learning, 38(2), 599–620. https://doi.org/10.1111/jcal.12635
  28. Palermo, C., & Thomson, M. M. (2018). Teacher implementation of self-regulated strategy development with an automated writing evaluation system: Effects on the argumentative writing performance of middle school students. Contemporary Educational Psychology, 54, 255–270. https://doi.org/10.1016/j.cedpsych.2018.07.002
  29. Stevenson, M., & Phakiti, A. (2014). The effects of computergenerated feedback on the quality of writing. Assessing Writing, 19, 51–65. https://doi.org/10.1016/j.asw.2013.11.007
  30. Link, S., Mehrzad, M., & Rahimi, M. (2022). Impact of automated writing evaluation on teacher feedback, student revision, and writing improvement. Computer Assisted Language Learning, 35(4), 605–634. https://doi.org/10.1080/09588221.2020.1743323
  31. Wang, E. L., Matsumura, L. C., Correnti, R., Litman, D., Zhang, H., Howe, E., ..., & Quintana, R. (2020). eRevis(ing): Students’ revision of text evidence use in an automated writing evaluation system. Assessing Writing, 44, 100449. https://doi.org/10.1016/j.asw.2020.100449
  32. Zhu, M., Liu, O. L., & Lee, H. S. (2020). The effect of automated feedback on revision behavior and learning gains in formative assessment of scientific argument writing. Computers & Education, 143, 103668. https://doi.org/10.1016/j.compedu.2019.103668
  33. Bai, L., & Hu, G. (2017). In the face of fallible AWE feedback: How do students respond? Educational Psychology, 37(1), 67–81. https://doi.org/10.1080/01443410.2016.1223275
  34. Conference onCollegeComposition andCommunication. (2022). Writing assessment: A position statement. Retrieved from: https://cccc.ncte.org/cccc/resources/positions/writingassessment
  35. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing (Standard No. xxx). American Educational Research Association.
  36. Wilson, J., Ahrendt, C., Fudge, E. A., Raiche, A., Beard, G., & MacArthur, C. A. (2021). Elementary teachers’ perceptions of automated feedback and automated scoring: Transforming the teaching and learning of writing using automated writing evaluation. Computers & Education, 168, 104208. https://doi.org/10.1016/j.compedu.2021.104208
  37. Mayfield, E., & Butler, S. (2018). District wide implementations outperform isolated use of automated feedback in high school writing. In Proceedings of the International Conference of the Learning Sciences, 23–27.
  38. MacArthur, C. A., Traga Philippakos, Z. A., May, H., & Compello, J. (2022). Strategy instruction with self-regulation in college developmental writing courses: Results from a randomized experiment. Journal of Educational Psychology, 114(4), 815–832. https://doi.org/10.1037/edu0000705
  39. Hayes, J. R. (1996). A new framework for understanding cognition and affect in writing. In C. M. Levy & S. Ransdell (Eds.), The science of writing: Theories, methods, individual differences, and applications (pp. 1–27). Routledge.
  40. Black, P., & Wiliam, D. (2009). Developing the theory of formative assessment. Educational Assessment, Evaluation and Accountability, 21(1), 5–31. https://doi.org/10.1007/s11092-008-9068-5
  41. Ranalli, J. (2021). L2 student engagement with automated feedback on writing: Potential for learning and issues of trust. Journal of Second Language Writing, 52, 100816. https://doi.org/10.1016/j.jslw.2021.100816
  42. Harris, K. R., & McKeown, D. (2022). Overcoming barriers and paradigm wars: Powerful evidence-based writing instruction. Theory into Practice, 61(4), 429–442. https://doi.org/10.1080/00405841.2022.2107334
  43. Barrot, J. S. (2023). Using automated written corrective feedback in the writing classrooms: Effects on L2 writing accuracy. Computer Assisted Language Learning, 36(4), 584–607. https://doi.org/10.1080/09588221.2021.1936071
  44. Nunes, A., Cordeiro, C., Rocha, R., Limpo, T., & Castro, S. L. (2024). “Breath, plan, write, and evaluate”: The effects of an SRSD intervention and instructional feedback on 4th graders’ writing and motivation. Frontiers in Education, 9, 1305771. https://doi.org/10.3389/feduc.2024.1305771
  45. Wijekumar, K., McKeown, D., Zhang, S., Lei, P. W., Hruska, N., & Pirnay-Dummer, P. (2024). We write automated scoring: Using ChatGPT for scoring in large-scale writing research projects. In M. D. Shermis & J. Wilson (Eds.), The Routledge international handbook of automated essay evaluation (pp. 178–194). Routledge. https://doi.org/10.4324/9781003397618-12
  46. Patthey-Chavez, G. G., & Ferris, D. R. (1997). Writing conferences and the weaving of multi-voiced texts in college composition. Research in the Teaching of English, 31(1), 51–90. https://doi.org/10.58680/rte19973872
  47. Dujinhower, H., Prins, F. J., & Stokking, K. M. (2012). Feedback providing improvement strategies and reflection on feedback use: Effects on students’ writing motivation, process, and performance. Learning and Instruction, 22(3), 171–184. https://doi.org/10.1016/j.learninstruc.2011.10.003
  48. Matsumura, L. C., Patthey-Chavez, G. G., Valdés, R., & Garnier, H. (2002). Teacher feedback, writing assignment quality, and third-grade students’ revision in lower and higher-achieving urban schools. The Elementary School Journal, 103(1), 3–25. https://doi.org/10.1086/499713
  49. Biber, D., Nekrasova, T., & Horn, B. (2011). TOEFL iBT™ research report: The effectiveness of feedback for L1-English and L2-writing development: A meta-analysis. Retrieved from: https://files.eric.ed.gov/fulltext/EJ1111175.pdf
  50. Zhang, Z., & Hyland, K. (2022). Fostering student engagement with feedback: An integrated approach. Assessing Writing, 51, 100586. https://doi.org/10.1016/j.asw.2021.100586
  51. Friedrich, L. (2019). Setting up the writing classroom. In S. Graham, C. A. MacArthur & M. Hebert (Eds.), Best practices in writing instruction (3rd ed., pp. 31–50). The Guilford Press.
  52. Philippakos, Z. A. (2017). Giving feedback: Preparing students for peer review and self-evaluation. The Reading Teacher, 71(1), 13–22. https://doi.org/10.1002/trtr.1568
  53. Patchan, M. M., & Schunn, C. D. (2016). Understanding the effects of receiving peer feedback for text revision: Relations between author and reviewer ability. Journal of Writing Research, 8(2), 227–265. https://doi.org/10.17239/jowr-2016.08.02.03
  54. Li, W. Y., Kau, K., & Shiung, Y. J. (2023). Pedagogic exploration into adapting automated writing evaluation and writing classes. Sage Open, 13(4), 21582440231209087. https://doi.org/10.1177/21582440231209087
  55. Liaqat, A., Munteanu, C., & Demmans Epp, C. (2021). Collaborating with mature English language learners to combine peer and automated feedback: A user-centered approach to designing writing support. International Journal of Artificial Intelligence in Education, 31(4), 638–679. https://doi.org/10.1007/s40593-020-00204-4
  56. Ferretti, R. P., Lewis, W. E., & Andrews-Weckerly, S. (2009). Do goals affect the structure of students’ argumentative writing strategies. Journal of Educational Psychology, 101(3), 577–589. https://doi.org/10.1037/a0014702
  57. Chung, H. Q., Chen, V., & Olson, C. B. (2021). The impact of self-assessment, planning and goal setting, and reflection before and after revision on student self-efficacy and writing performance. Reading and Writing, 34(7), 1885–1913. https://doi.org/10.1007/s11145-021-10186-x
  58. Camacho, A., Alves, R. A., & Boscolo, P. (2021). Writing motivation in school: A systematic review of empirical research in the early twenty-first century. Educational Psychology Review, 33(1), 213–247. https://doi.org/10.1007/s10648-020-09530-4
  59. Cordeiro, C., Limpo, T., Olive, T., & Castro, S. L. (2020). Do executive functions contribute to writing quality in beginning writers? A longitudinal study with second graders. Reading and Writing, 33(4), 813–833. https://doi.org/10.1007/s11145-019-09963-6
  60. Harris, K. R., Camping, A., & McKeown, D. (2023). A review of research on professional development for multicomponent strategy-focused writing instruction: Knowledge gained and challenges remaining. In F. de Smedt, R. Bouwer, T. Limpo & S. Graham (Eds.), Conceptualizing, designing, implementing, and evaluating writing interventions (pp. 101–136). Brill. https://doi.org/10.1163/9789004546240_007

LICENSE

Creative Commons License
This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.