Item Details

Informing Automated Writing Evaluation Using the Lens of Genre: Two Studies

Issue: Vol 33 No. 1 (2016) Automated Writing Evaluation

Journal: CALICO Journal

Subject Areas:

DOI: 10.1558/cj.v33i1.26374


Genre serves as a useful lens to investigate the range of evidence derived from automated writing evaluation (AWE). To support construct-relevant systems used for writing instruction and assessment, two investigations were conducted that focused on post-secondary writing requirements and faculty perceptions of student writing proficiency. Survey research is described from a national study and a second site study at American University, a 4-year private university in Washington, DC, to illustrate writing requirements and perceptions of writing proficiency in school and workplace settings. A mixed-methods analysis of faculty focus groups in the site study afforded more detailed discussions that were used to understand student writing support needs. Through the lens of genre, study results illustrated how the role of AWE might be expanded and aligned with instruction in four-year post-secondary institutions.

Author: Jill Burstein, Norbert Elliot, Hillary Molloy

View Full Text

References :

ACT. (2005). Crisis at the core: Preparing all students for college and work. Iowa City, IA: ACT. Retrieved from

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

Applebee, A. N., & Langer, J. A. (1984). Contexts for learning to write: Studies of secondary school instruction. Norwood, NJ: Ablex.

Applebee, A.N., & Langer, J. A. (2009). What is happening in the teaching of writing? English Journal, 95, 19–29.

Applebee, A.N., Lehr, F., & Austen, A. (1981). Learning to write in the secondary school: How and where. English Journal, 70, 78–82.

Appleman, D., & Green, D. E. (1993). Mapping the elusive boundary between high school and college writing. College Composition and Communication, 44, 191–199.

Association for Colleges and Research Libraries. (2014). Framework for information literacy for higher education. Chicago, IL: American Library Association. Retrieved from

Banks, A. (2015). Ain’t No Walls behind the sky, baby! Funk, flight, freedom. College Composition and Communication, 67, 267–279.

Bazerman, C. (2010). A rhetoric of literate action: Literate action (vol. 1). Fort Collins, CO: WAC Clearinghouse and Anderson: Parlor Press.

Bazerman, C., & Paradis, A. (Eds.) (1991). Textual dynamics of the professions: Historical and contemporary studies of writing in professional communities. Madison, WI: University of Wisconsin Press.

Beck, S.W., & Jeffery, J.V. (2009). Genre and thinking in academic writing tasks. Journal of Literacy Research, 41, 228–272.

Beigman-Klebanov, B. & Flor, M. (2013a). Argumentation-relevant metaphors in test-taker essays. In Proceedings of the First Workshop on Metaphor in NLP, Annual Meeting of the Association of Computational Linguistics, 11–20. Atlanta, GA: Association of Computational Linguistics.

Beigman Klebanov, B., & Flor, M. (2013b). Word association profiles and their use for automated scoring of essays. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, 1148–1158. Sofia, BG: Association for Computational Linguistics.

Beigman Klebanov, B., Madnani, N., Burstein, J., and Somasundaran, S. (2014). Content importance models for scoring writing from sources. In Proceedings of the Annual Meeting of the Association of Computational Linguistics, 247–252. Baltimore, MD: Association of Computational Linguistics.

Biber, D., Nekrasova, T., & Horn, B. (2011). The effectiveness of feedback for L1-English and L2-writing development: A meta-analysis (RR-11-05, TOEFLiBT-14). Princeton, NJ: Educational Testing Service. Retrieved

Bridgeman, B., & Carlson, S. (1984). Survey of academic writing tasks. Written Communication, 1, 247–280.

Britton, J. (1992). Language and learning: The importance of speech in children’s development. London: Penguin.

Britton, J., Burgess, T., Martin, N., McLeod, A., & Rosen, H. (1975). The development of writing abilities, 11–18. Urbana, IL: National Council of Teachers of English.

Burstein, J., Elliot, N., Lentini, J, Molloy, H., Shore, J., Steinberg, J. & Vezzu, M. (2014). Investigating genre for writing measurement and automated writing evaluation, Coordinated Session at the Annual Meeting of the National Council on Educational Measurement, April 2014. Retrieved from

Burstein, J., Tetreault, J., & Chodorow, M. (2013). Holistic annotation of discourse coherence quality in noisy essay writing. [Special issue]. Dialogue and Discourse, 4. Retrieved from

Casner-Lotto, J., & Barrington, L. (2006). Are they really ready to work? Employers’ perspectives on the basic knowledge and applied skills of new entrants to the 21st century US workforce. New York: The Conference Board. Retrieved from

Coppola, N. (2010). The technical communication Body of Knowledge initiative: An academic-practitioner partnership. Technical Communication, 57, 11–25.

Cotos, E. (2011). Potential of automated writing evaluation feedback. CALICO Journal, 28, 420–459. Retrieved from

Council of Writing Program Administrators. (2014). WPA outcomes statement for first-year composition (Revisions adopted 17 July 2014). WPA: Journal of Writing Program Administration, 38, 142–146.

Crusius, T. W. (1989). Discourse: A critique and synthesis of major theories. New York: Modern Language Association.

Eblen, C. (1983). Writing across-the-curriculum: A survey of a university faculty’s views and classroom practices. Research in the Teaching of English, 17, 343–348.

Elliot, N., & Kilduff, M. (1991). Technical writing in a technological university: Attitudes of department chairs. Journal of Technical Writing and Communication, 21, 411–424.

Faigley, L., & Romano, S. (1995). Going electronic: Creating multiple sites for innovation in a writing program. In J. Janangelo & K. Hansen (Eds), Resituating writing: Constructing and administering writing programs, 46–58. Portsmouth, NH: Boynton/Cook, 1995.

Ferris, D.R. (2014). Responding to student writing: Teachers’ philosophies and practices. Assessing Writing, 19, 6–23.

Gardner, S. & Nesi, H. (2013). A classification of genre families in university student writing. Applied Linguistics, 34, 25–52.

Gere, A. R., Aull, L., Escudero, Lancaster, & Lei (2013). Local assessment: Using genre analysis to validate directed self-Placement. College Composition and Communication, 64, 605–633.

Graham, S., & Perin, D. (2007). A meta-analysis of writing instruction for adolescent students. Journal of Educational Psychology, 99, 445–476.

Harris, J., & Hult, C. (1985). Using a survey of writing assignments to make informed curricular decisions. WPA: Journal of Writing Program Administration, 8, 7–14.

Heath, S. B. (1993). Rethinking the sense of the past: The essay as legacy of the epigram. In L. Odell (Ed.), Theory and practice in the teaching of writing: Rethinking the discipline, 105–131. Carbondale, IL: Southern Illinois University Press.

Herrington, A. (1985). Writing in academic settings: A study of the contexts for writing in two college chemical engineering courses. Research the Teaching of English, 19, 331–359.

Horowitz, D. (1986). What professors actually require: Academic tasks for the ESL classroom. TESOL Quarterly, 20, 445–462.

Jenkins, S., Jordan, M. K., & Weiland, P. O. (1993). The role of writing in graduate engineering education: A survey of faculty beliefs and practices. English for Specific Purposes, 12, 51–67.

Kane, M. T. (2013). Validating the interpretation and uses of test scores. Journal of Educational Measurement, 50, 1–73.

Kiuhara, S., Graham, S., & Hawken, L. (2009). Teaching writing to high school students: A national survey. Journal of Educational Psychology, 101, 136–160.

Krueger, R. A. & Casey, M. S. (2000). Focus groups: A practical guide for applied research (3rd ed.). Thousand Oaks, CA: Sage.

Melzer, D. (2009). Writing assignments across the curriculum: A national study of college writing. College Composition and Communication, 61, W240–W261. Retrieved from

Melzer, D. (2014). Assignments across the curriculum: A national study of college writing. Logan, UT: Utah State University Press.

Miller, C. (1984). Genre as social action. Quarterly Journal of Speech, 70, 151–167.

Miller, C. (1994). In A. Freedman & P. Medway (Eds), Rhetorical community: The cultural basis of genre, 67–97. London: Taylor & Francis.

National Center for Education Statistics (2012). The Nation’s Report Card: Writing 2011 (NCES 2012–470). Washington, DC: Institute of Education Sciences, U.S. Department of Education, Washington, D.C. Retrieved from

National Governors Association. (2014). Common core state standards initiative. Washington, DC: National Governors Association. Retrieved from

National Research Council. (2012). Education for life and work: Developing transferable knowledge and skills in the 21st century. Committee on Defining Deeper Learning and 21st Century Skills, J.W. Pellegrino & M.L. Hilton (Eds). Board on Testing and Assessment and Board on Science Education, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.

Odell, L., Goswami, D., & Herrington, A. (1983). The discourse-based interview: A procedure for exploring tacit knowledge of writers in non-academic settings. In P. Mosenthal, L. Tamor, & S. Walmsley (Eds), Research on writing: Principles and methods, 220–236. New York: Longman.

Pimentel, S. (2013). College and career readiness standards for adult education. Washington, DC: US Department of Education, Office of Vocational and Adult Education. Retrieved from

Relles, S. R., & Tierney, W.G. (2014). Understanding the writing habits of tomorrow’s students: Technology and college readiness. Journal of Higher Education, 84, 477–505.

Rose, M. (1983). Remedial writing courses: A critique and a proposal. College English, 45, 109–128.

Shermis, M., Burstein, J., Elliot, N., Miel, S., & Foltz, P. (2016). Automated writing evaluation: A growing body of knowledge. In C. MacArthur, S. Graham, & J. Fitzgerald (Eds), Handbook of writing research, 395–409. New York, NY: Guilford Press.

Somasundaran, S., Burstein, J., & Chodorow, M. (2014, August). Lexical chaining for measuring discourse coherence quality in test-taker essays. Paper presented at The 25th International Conference on Computational Linguistics (COLING), Dublin, IE.

Song, Y., Heilman, M., Beigman Klebanov, B., & Deane, P. (2014). Applying argumentation schemes for essay scoring. In Proceedings of the 1st workshop on Argumentation Mining, 69–78. Baltimore, MD: Association for Computational Linguistics.

Sparks, J. R., Song, Y., Brantley, W., & Liu, O. L. (2014). Assessing written communication in higher education: Review and recommendations for next-generation assessment (ETS Research Report No. RR-14-37). Princeton, NJ: Educational Testing Service.

Thrift, N. (2005). Knowing capitalism. London: Sage.

Wardle, E. (2009). ‘Mutt genres’ and the goal of FYC: Can we help students write the genres of the university? College Composition and Communication, 60, 765–789.