How can Writing Tasks be Characterized in a way serving Pedagogical Goals and Automatic Analysis Needs?
Issue: Vol 33 No. 1 (2016) Automated Writing Evaluation
Journal: CALICO Journal
Subject Areas:
Abstract:
The paper works out and addresses a central question in the field of Intelligent Computer-Assisted Language Learning (ICALL): How can language learning tasks be conceptualized and made explicit in a way that supports the pedagogical goals in modern Foreign Language Teaching and Learning (FLTL) and at the same time provides an explicit characterization of the Natural Language Processing (NLP) requirements to provide feedback to learners completing those tasks? We argue that the successful implementation of language learning tasks to be automatically assessed by means of NLP-based feedback generation strategies demands a design process considering both pedagogical and computational requirements as equally important.
Extending well-established work in Task-Based Language Teaching (TBLT) and TBLT testing we propose a framework that helps us (i) elucidate the formal features of foreign language learning activities, (ii) characterize the gap between expected and actually elicited learner language, and (iii) assess how variability in learner responses impacts computational techniques for the automatic analysis of learner language.
To validate our approach we apply our framework to two specific writing tasks for learners of English as a Foreign Language (EFL). Our analysis highlights the relevance of spelling out the pedagogical and linguistic goals of language learning tasks in order to successfully characterize the language variation in learner responses needed to design effective pedagogical and NLP strategies.Given the combination of design- and data-driven perspectives, the framework supports an iterative approach to the creation of language learning tasks and ICALL materials.
Author: Martí Quixal, Detmar Meurers
References :
Amaral, L., & Meurers, D. (2011). On using Intelligent Computer-Assisted Language Learning in Real-Life Foreign Language Teaching and Learning. ReCALL, 23 (1), 4–24. Retrieved from http://purl.org/dm/papers/amaral-meurers-10.html. http://dx.doi.org/10.1017/S0958344010000261
Bachman, L. F., & Palmer, A. S. (1996). Language testing in practice: Designing and developing useful language tests. Oxford: Oxford University Press.
Badia, T., Díaz, L., Quixal, M., Ruggia, A., Garnier, S., & Schmidt, P. (2004). Individualised NLP-enhanced feedback for distance language learning. In Proceedings of ICALT. IEEE Computer Society, 2004. ISBN 0-7695-2181-9. URL http://www.computer.org/csdl/proceedings/icalt/2004/2181/00/21810729.pdf. http://dx.doi.org/10.1109/icalt.2004.1357638
Bailey, S., & Meurers, D. (2008). Diagnosing meaning errors in short answers to reading comprehension questions. In J. Tetreault, J. Burstein, & R. De Felice (Eds), Proceedings of the 3rd Workshop on Innovative Use of NLP for Building Educational Applications (BEA-3) at ACL’08, 107–115, Columbus, Ohio. URL http://aclweb.org/anthology/W08-0913. http://dx.doi.org/10.3115/1631836.1631849
Borin, L. (2002) What have you done for me lately? The fickle alignment of NLP and CALL. Paper presented at the EuroCALL 2002 pre-conference workshop on NLP in CALL, 14 August 2002, Jyvaskyla, Finland. URL http://k2xx.spraakdata.gu.se/personal/lars/pblctns/EuroCALL2002-NLP-WS.pdf
Brown, H. D. (2007). Principles of language learning and teaching, 5th edition. London: Pearson Education.
Chapelle, C. A. (2001). Computer applications in second language acquisition: Foundations for teaching, testing, and research. Cambridge: Cambridge University Press. http://dx.doi.org/10.1017/CBO9781139524681
Douglas, D. (2000). Assessing languages for specific purposes. Cambridge: Cambridge University Press.
Ellis, R. (2003). Task-based language learning and teaching. Oxford: Oxford University Press.
Estaire, S., & Zanón, J. (1994). Planning classwork: A task-based approach. Educational Language Teaching. Oxford: Macmillan-Heinemann.
González-Lloret, M., & Ortega, L. (2014). Towards technology-mediated TBLT. In M. González-Lloret, & L. Ortega (Eds), Technology-mediated TBLT: Researching Technology and Tasks, Volume 6, 1–22. Amsterdam/Philadelphia: John Benjamins.
Granger, S. (2003). Error-tagged learner corpora and CALL: A promising synergy. CALICO Journal, 20 (3), 465–480. URL http://purl.org/calico/Granger03.pdf
Granger, S., Gilquin, G., & Meunier, F. (Eds) (2015). The Cambridge handbook of learner corpus research. Cambridge: Cambridge University Press. http://dx.doi.org/10.1017/CBO9781139649414
Heift, T., & Schulze, M. (2007). Errors and intelligence in computer-assisted language learning: Parsers and pedagogues. New York: Routledge.
Housen, A., & Kuiken, F. (2009). Complexity, accuracy, and fluency in second language acquisition. Applied Linguistics, 30 (4), 461–473. http://dx.doi.org/10.1093/applin/amp048
King, L., & Dickinson, M. (2013). Shallow semantic analysis of interactive learner sentences. In Proceedings of the eighth workshop on innovative use of NLP for building educational applications, Atlanta, GA USA. URL http://aclweb.org/anthology/W13-1702.pdf
Leacock, C., & Chodorow, M. (2003) . C-rater: Automated scoring of short-answer questions. Computers and the Humanities, 37, 389–405. http://dx.doi.org/10.1023/A:1025779619903
Leacock, C., Chodorow, M., Gamon, M., & Tetreault, J. (2014). Automated grammatical error detection for language learners (2nd ed.) vol. 25. San Rafael: Morgan & Claypool Publishers.
Littlewood, W. (2004). The task-based approach: Some questions and suggestions. ELT Journal, 58 (4), 319–326. http://dx.doi.org/10.1093/elt/58.4.319
Lüdeling, A. (2008). Mehrdeutigkeiten und Kategorisierung: Probleme bei der Annotation von Lernerkorpora. In M. Walter & P. Grommes (Eds.), Fortgeschrittene Lernervarietäten: Korpuslinguistik und Zweispracherwerbsforschung, 119–140. Tübingen: Max Niemeyer Verlag.
Meurers, D. (2012). Natural language processing and language learning. In C. A. Chapelle (Ed.), Encyclopedia of Applied Linguistics, 4193–4205. Oxford: Wiley, Oxford. URL http://purl.org/dm/papers/meurers-12.html. http://dx.doi.org/10.1002/9781405198431.wbeal0858
Meurers, D. (2015). Learner corpora and natural language processing. In S. Granger, G. Gilquin, & F. Meunier (Eds) The Cambridge handbook of learner corpus research. Cambridge: Cambridge University Press. http://purl.org/dm/papers/meurers-15.html. http://dx.doi.org/10.1017/CBO9781139649414.024
Nagata, N. (2009) Robo-Sensei’s NLP-Based Error Detection and Feedback Generation. CALICO Journal, 26 (3), 562–579.
Nunan, D. (2004) Task-based language teaching. Cambridge: Cambridge University Press. http://dx.doi.org/10.1017/CBO9780511667336
Quixal. M. (2012). Language Learning Tasks and Automatic Analysis of Learner Language. Connecting FLTL and NLP in the design of ICALL materials supporting effective use in real-life instruction. PhD thesis, Universitat Pompeu Fabra, Barcelona and Eberhard-Karls-Universität Tübingen. URL http://www.sfs.uni-tuebingen.de/~quixal/pubs/Quixal-12.pdf
Quixal, M., Badia, T., Boullosa, B., Díaz, L. & Ruggia, A. (2006). Strategies for the generation of individualised feedback in distance language learning. In Proceedings of the Workshop on Language- Enabled Technology and Development and Evaluation of Robust Spoken Dialogue Systems of ECAI 2006, Riva del Garda, Italy, September.
Quixal,M., Preuß, S., Boullosa, B. & García-Narbona (2010). AutoLearn’s authoring tool: A piece of cake for teachers. In Proceedings of the NAACL HLT 2010 Fifth Workshop on Innovative Use of NLP for Building Educational Applications, 19–27, Los Angeles, June. URL http://www.aclweb.org/anthology/W10-1003.pdf
Reznicek, M., Lüdeling, A., & Hirschmann, H. (2013). Competing target hypotheses in the Falko corpus: A flexible multi-layer corpus architecture. In A. Díaz-Negrillo, N. Ballier, & P. Thompson (Eds), Automatic treatment and analysis of learner corpus data, Volume 59, 101–123. Amsterdam: John Benjamins. http://dx.doi.org/10.1075/scl.59.07rez
Richards, J. & Rodgers, T. (2001). Approaches and methods in language teaching (2nd ed.). Cambridge: Cambridge University Press. http://dx.doi.org/10.1017/CBO9780511667305
Salaberry, M. R. (1996). A theoretical foundation for the development of pedagogical tasks in computer mediated communication. CALICO Journal, 14 (1), 5–34. Retrieved from https://webspace.utexas.edu/mrs2429/www/Salaberry1996CALICO.pdf
Schmidt, P., Garnier, S., Sharwood, M., Badia, T., Díaz, L., Quixal, M., Ruggia, A., Valderrábanos, A. S., Cruz, A. J., Torrejon, E., Rico, C. & Jimenez, J. (2004). ALLES: Integrating NLP in ICALL applications. In LREC-2004. Conference on Language Re- sources and Evaluation. http://www.lrec-conf.org/proceedings/lrec2004/pdf/3.pdf
Schulze, M. (2010). Taking ICALL to Task. In M. Thomas & H. Reinders (Eds), Task-based language teaching and technology, 63–82. London/New York: Continuum.
Thomas, M., & Reinders, H. (Eds) (2010). Task-based language learning and teaching with technology. London/New York: Continuum.
Weischedel, R. M., Voge, W. M., & M. James. An artificial intelligence approach to language instruction. Artificial Intelligence, 10 (3), 225–240. http://dx.doi.org/10.1016/S0004-3702(78)80015-0
Willis, J. (1996). A framework for task-based learning. Boston, MA: Longman Addison-Wesley.
Ziai, R., Ott, N., & Meurers, D. (2012). Short answer assessment: Establishing links between research strands. In Proceedings of the 7th Workshop on Innovative Use of NLP for Building Educational Applications (BEA-7) at NAACL-HLT, 190–200, Montreal, June. Association for Computational Linguistics. http://aclweb.org/anthology/W12-2022.pdf