Informing Automated Writing Evaluation Using the Lens of Genre: Two Studies
Journal: CALICO Journal
Genre serves as a useful lens to investigate the range of evidence derived from automated writing evaluation (AWE). To support construct-relevant systems used for writing instruction and assessment, two investigations were conducted that focused on post-secondary writing requirements and faculty perceptions of student writing proficiency. Survey research is described from a national study and a second site study at American University, a 4-year private university in Washington, DC, to illustrate writing requirements and perceptions of writing proficiency in school and workplace settings. A mixed-methods analysis of faculty focus groups in the site study afforded more detailed discussions that were used to understand student writing support needs. Through the lens of genre, study results illustrated how the role of AWE might be expanded and aligned with instruction in four-year post-secondary institutions.
Author: Jill Burstein, Norbert Elliot, Hillary Molloy
ACT. (2005). Crisis at the core: Preparing all students for college and work. Iowa City, IA: ACT. Retrieved from http://www.act.org/research/policymakers/pdf/crisis_report.pdf
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
Applebee, A. N., & Langer, J. A. (1984). Contexts for learning to write: Studies of secondary school instruction. Norwood, NJ: Ablex.
Applebee, A.N., & Langer, J. A. (2009). What is happening in the teaching of writing? English Journal, 95, 19–29.
Applebee, A.N., Lehr, F., & Austen, A. (1981). Learning to write in the secondary school: How and where. English Journal, 70, 78–82. http://dx.doi.org/10.2307/817387
Appleman, D., & Green, D. E. (1993). Mapping the elusive boundary between high school and college writing. College Composition and Communication, 44, 191–199. http://dx.doi.org/10.2307/358838
Banks, A. (2015). Ain’t No Walls behind the sky, baby! Funk, flight, freedom. College Composition and Communication, 67, 267–279.
Bazerman, C. (2010). A rhetoric of literate action: Literate action (vol. 1). Fort Collins, CO: WAC Clearinghouse and Anderson: Parlor Press.
Bazerman, C., & Paradis, A. (Eds.) (1991). Textual dynamics of the professions: Historical and contemporary studies of writing in professional communities. Madison, WI: University of Wisconsin Press.
Beck, S.W., & Jeffery, J.V. (2009). Genre and thinking in academic writing tasks. Journal of Literacy Research, 41, 228–272. http://dx.doi.org/10.1080/10862960902908483
Beigman-Klebanov, B. & Flor, M. (2013a). Argumentation-relevant metaphors in test-taker essays. In Proceedings of the First Workshop on Metaphor in NLP, Annual Meeting of the Association of Computational Linguistics, 11–20. Atlanta, GA: Association of Computational Linguistics.
Beigman Klebanov, B., & Flor, M. (2013b). Word association profiles and their use for automated scoring of essays. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, 1148–1158. Sofia, BG: Association for Computational Linguistics.
Beigman Klebanov, B., Madnani, N., Burstein, J., and Somasundaran, S. (2014). Content importance models for scoring writing from sources. In Proceedings of the Annual Meeting of the Association of Computational Linguistics, 247–252. Baltimore, MD: Association of Computational Linguistics. http://dx.doi.org/10.3115/v1/p14-2041
Biber, D., Nekrasova, T., & Horn, B. (2011). The effectiveness of feedback for L1-English and L2-writing development: A meta-analysis (RR-11-05, TOEFLiBT-14). Princeton, NJ: Educational Testing Service. Retrieved
Bridgeman, B., & Carlson, S. (1984). Survey of academic writing tasks. Written Communication, 1, 247–280. http://dx.doi.org/10.1177/0741088384001002004
Britton, J. (1992). Language and learning: The importance of speech in children’s development. London: Penguin.
Britton, J., Burgess, T., Martin, N., McLeod, A., & Rosen, H. (1975). The development of writing abilities, 11–18. Urbana, IL: National Council of Teachers of English.
Burstein, J., Elliot, N., Lentini, J, Molloy, H., Shore, J., Steinberg, J. & Vezzu, M. (2014). Investigating genre for writing measurement and automated writing evaluation, Coordinated Session at the Annual Meeting of the National Council on Educational Measurement, April 2014. Retrieved from .
Casner-Lotto, J., & Barrington, L. (2006). Are they really ready to work? Employers’ perspectives on the basic knowledge and applied skills of new entrants to the 21st century US workforce. New York: The Conference Board. Retrieved from
Coppola, N. (2010). The technical communication Body of Knowledge initiative: An academic-practitioner partnership. Technical Communication, 57, 11–25.
Cotos, E. (2011). Potential of automated writing evaluation feedback. CALICO Journal, 28, 420–459. Retrieved from http://dx.doi.org/10.11139/cj.28.2.420-459
Council of Writing Program Administrators. (2014). WPA outcomes statement for first-year composition (Revisions adopted 17 July 2014). WPA: Journal of Writing Program Administration, 38, 142–146.
Crusius, T. W. (1989). Discourse: A critique and synthesis of major theories. New York: Modern Language Association.
Eblen, C. (1983). Writing across-the-curriculum: A survey of a university faculty’s views and classroom practices. Research in the Teaching of English, 17, 343–348.
Elliot, N., & Kilduff, M. (1991). Technical writing in a technological university: Attitudes of department chairs. Journal of Technical Writing and Communication, 21, 411–424.
Faigley, L., & Romano, S. (1995). Going electronic: Creating multiple sites for innovation in a writing program. In J. Janangelo & K. Hansen (Eds), Resituating writing: Constructing and administering writing programs, 46–58. Portsmouth, NH: Boynton/Cook, 1995.
Ferris, D.R. (2014). Responding to student writing: Teachers’ philosophies and practices. Assessing Writing, 19, 6–23. http://dx.doi.org/10.1016/j.asw.2013.09.004
Gardner, S. & Nesi, H. (2013). A classification of genre families in university student writing. Applied Linguistics, 34, 25–52. http://dx.doi.org/10.1093/applin/ams024
Gere, A. R., Aull, L., Escudero, Lancaster, & Lei (2013). Local assessment: Using genre analysis to validate directed self-Placement. College Composition and Communication, 64, 605–633.
Graham, S., & Perin, D. (2007). A meta-analysis of writing instruction for adolescent students. Journal of Educational Psychology, 99, 445–476. http://dx.doi.org/10.1037/0022-06126.96.36.1995
Harris, J., & Hult, C. (1985). Using a survey of writing assignments to make informed curricular decisions. WPA: Journal of Writing Program Administration, 8, 7–14.
Heath, S. B. (1993). Rethinking the sense of the past: The essay as legacy of the epigram. In L. Odell (Ed.), Theory and practice in the teaching of writing: Rethinking the discipline, 105–131. Carbondale, IL: Southern Illinois University Press.
Herrington, A. (1985). Writing in academic settings: A study of the contexts for writing in two college chemical engineering courses. Research the Teaching of English, 19, 331–359.
Horowitz, D. (1986). What professors actually require: Academic tasks for the ESL classroom. TESOL Quarterly, 20, 445–462. http://dx.doi.org/10.2307/3586294
Jenkins, S., Jordan, M. K., & Weiland, P. O. (1993). The role of writing in graduate engineering education: A survey of faculty beliefs and practices. English for Specific Purposes, 12, 51–67. http://dx.doi.org/10.1016/0889-4906(93)90027-L
Kane, M. T. (2013). Validating the interpretation and uses of test scores. Journal of Educational Measurement, 50, 1–73. http://dx.doi.org/10.1111/jedm.12000
Kiuhara, S., Graham, S., & Hawken, L. (2009). Teaching writing to high school students: A national survey. Journal of Educational Psychology, 101, 136–160.
Krueger, R. A. & Casey, M. S. (2000). Focus groups: A practical guide for applied research (3rd ed.). Thousand Oaks, CA: Sage. http://dx.doi.org/10.1037/10518-189
Melzer, D. (2014). Assignments across the curriculum: A national study of college writing. Logan, UT: Utah State University Press.
Miller, C. (1984). Genre as social action. Quarterly Journal of Speech, 70, 151–167. http://dx.doi.org/10.1080/00335638409383686
Miller, C. (1994). In A. Freedman & P. Medway (Eds), Rhetorical community: The cultural basis of genre, 67–97. London: Taylor & Francis.
National Center for Education Statistics (2012). The Nation’s Report Card: Writing 2011 (NCES 2012–470). Washington, DC: Institute of Education Sciences, U.S. Department of Education, Washington, D.C. Retrieved from http://nces.ed.gov/nationsreportcard/pdf/main2011/2012470.pdf
National Research Council. (2012). Education for life and work: Developing transferable knowledge and skills in the 21st century. Committee on Defining Deeper Learning and 21st Century Skills, J.W. Pellegrino & M.L. Hilton (Eds). Board on Testing and Assessment and Board on Science Education, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.
Odell, L., Goswami, D., & Herrington, A. (1983). The discourse-based interview: A procedure for exploring tacit knowledge of writers in non-academic settings. In P. Mosenthal, L. Tamor, & S. Walmsley (Eds), Research on writing: Principles and methods, 220–236. New York: Longman.
Relles, S. R., & Tierney, W.G. (2014). Understanding the writing habits of tomorrow’s students: Technology and college readiness. Journal of Higher Education, 84, 477–505. http://dx.doi.org/10.1353/jhe.2013.0025
Rose, M. (1983). Remedial writing courses: A critique and a proposal. College English, 45, 109–128. http://dx.doi.org/10.2307/377219
Shermis, M., Burstein, J., Elliot, N., Miel, S., & Foltz, P. (2016). Automated writing evaluation: A growing body of knowledge. In C. MacArthur, S. Graham, & J. Fitzgerald (Eds), Handbook of writing research, 395–409. New York, NY: Guilford Press.
Somasundaran, S., Burstein, J., & Chodorow, M. (2014, August). Lexical chaining for measuring discourse coherence quality in test-taker essays. Paper presented at The 25th International Conference on Computational Linguistics (COLING), Dublin, IE.
Song, Y., Heilman, M., Beigman Klebanov, B., & Deane, P. (2014). Applying argumentation schemes for essay scoring. In Proceedings of the 1st workshop on Argumentation Mining, 69–78. Baltimore, MD: Association for Computational Linguistics. http://dx.doi.org/10.3115/v1/W14-2110
Sparks, J. R., Song, Y., Brantley, W., & Liu, O. L. (2014). Assessing written communication in higher education: Review and recommendations for next-generation assessment (ETS Research Report No. RR-14-37). Princeton, NJ: Educational Testing Service.
Thrift, N. (2005). Knowing capitalism. London: Sage.
Wardle, E. (2009). ‘Mutt genres’ and the goal of FYC: Can we help students write the genres of the university? College Composition and Communication, 60, 765–789.