Item Details

A methodology for longitudinal research on EFL written production: Capturing writing multidimensionality by combining qualitative and quantitative procedures

Issue: Vol 3 No. 1 (2016) Mixed Methods

Journal: Journal of Research Design and Statistics in Linguistics and Communication Science

Subject Areas: Linguistics

DOI: 10.1558/jrds.30221


This article critically examines the most challenging aspects of the methodology used in a broader research endeavour aimed at measuring the longitudinal impact of Content Language Integrated Learning (CLIL) on the compositions in English as a Foreign Language (EFL) produced by secondary education learners. A pretest/posttest design with repeated measures was used, which appears to be the only way to control for differences in task complexity (Larsen-Freeman 2006) despite claims of carry-over effects (Abbuhl et al. 2014). The major methodological concern was to describe writing in all its complexity and multidimensionality (Alanen et al. 2010). The decisions regarding the communicative written tasks administered, the building of the corpus, and especially the mixed methods for evaluating the written production obtained were made accordingly. All things considered, a combination of quantitative and qualitative methods to measure and characterize EFL writing is advocated in order to capture all its richness.

Author: Maria Gené-Gil, Maria Juan-Garau, Joana Salazar-Noguera

View Original Web Page

References :

Abbuhl, R., Gass, S., and Mackey, A. (2014). Experimental research design. In R. J. Podesva and S. Devyani (eds) Research Methods in Linguistics, 116–134. Cambridge: Cambridge University Press.

Ai, H. and Lu, X. (2013). A corpus-based comparison of syntactic complexity in NNS and NS university students’ writing. In A. Díaz-Negrillo, N. Ballier, and P. Thompson (eds) Automatic Treatment and Analysis of Learner Corpus Data, 249–264. Amsterdam: John Benjamins.

Alanen, R., Huhta, A., and Tarnanen, M. (2010). Designing and assessing L2 writing tasks across CEFR proficiency levels. In I. Bartning, M. Martin, and I. Vedder (eds) Communicative Proficiency and Linguistic Development: Intersections between SLA and Language Testing Research, 21–56. European Second Language Association.

Amengual Pizarro, M. (2003). Análisis de la fiabiliad en las puntuaciones holísticas en items abiertos. Unpublished PhD Thesis. Universidad Complutense de Madrid.

Argüelles Álvarez, I. (2004). Evaluación y calificación de resúmenes de textos expositivos en el aula de ILE / IFE: la guía ‘BABAR’. Ibérica 8: 81–99.

Barquin, E. L. (2012). Writing development in a study abroad context. PhD thesis. Universitat Pompeu Fabra.

Bruton, A. (2011). Is CLIL so beneficial, or just selective? Re-evaluating some of the research. System 39(4), 523–532.

Bulté, B., and Housen, A. (2012). Defining and operationalising L2 complexity. In A. Housen, F. Kuiken, and I. Vedder (eds) Dimensions of L2 Performance and Proficiency. Complexity, accuracy and fluency in SLA, 21–46. Amsterdam: John Benjamins.

Celaya, M. L., and Navés, T. (2009). Age-related differences and associated factors in foreign language writing. Implications for L2 writing theory and school curricula. In R. M. Manchón (ed.) Writing in Foreign Language Contexts. Learning, Teaching, and Research, 130–155. Bristol: Multilingual Matters.

Chaudron, C., Martín Úriz, A., and Whittaker, R. (2005). Innovación pedagógica: los centros, los alumnos y los profesores. In A. Martín Úriz and R. Whittaker (eds) La composición como comunicación: una experiencia en las aulas de lengua inglesa en bachillerato 41–60. Madrid: Ediciones de la Universidad Autónoma de Madrid.

Collins, L., Halter, R. H., Lightbown, P. M., and Spada, N. (1999). Time and the distribution of time in L2 instruction. TESOL Quarterly 33 (4): 655–680.

Cummins, J. (1999). Alternative paradigms in bilingual education research: Does theory have a place?. Educational Researcher 28 (7): 26–32.

Dalton-Puffer, C. (2011). Content-and-language integrated learning: From practice to principles? Annual Review of Applied Linguistics 31: 182–204.

Eguiluz Pacheco, J., and Vega Santos, C. M. (1996). Criterios para la evaluación de la producción escrita. Didáctica del Español como LE. Expolingua 1996: 75–94.

Escobar Urmeneta, C. (2007). Mesurament de competències lingüístiques en aules AICLE, en contextos multilingües: Allò que els números amaguen. In O. Guasch and L. Nussbaum (eds) Aproximacions a la noció de competència multilingüe 133–146. Bellaterra: Servei de Publicacions de la UAB,

Gené-Gil, M. (2016). Efecto del AICLE sobre la producción escrita en estudiantes de secundaria de inglés como lengua extranjera: Un estudio longitudinal. PhD dissertation. Universitat de les Illes Balears.

Graesser, A. C., and McNamara, D., S. (2011). Computational analyses of multilevel discourse comprehension. Topics in Cognitive Science 3: 371–398.

Housen, A., and Kuiken, F. (2009). Complexity, accuracy, and fluency in second language acquisition. Applied Linguistics 30 (4): 461–473.

Hunt, K. W. (1964). Grammatical Structures Written at Three Grade Levels. Urbana, IL: National Council of Teachers of English.

Hunt, K. W. (1970). Recent measures in syntactic development. In M. Lester (ed.) Readings in Applied Transformation Grammar, 4–21. New York: Holt / Rinehert / Winston.

Jacobs, H. L., Zinkgraf, S. A., Wormuth, D. R., Hartfiel, V. F., and Hughey, J. B. (1981). Testing ESL Composition: A Practical Approach. Rowley: Newbury House Publishers.

Jiménez Catalán, R. M, Ruiz de Zarobe, Y., and Cenoz Iragui, J. (2006). Vocabulary profiles of English foreign language learners in English as a subject and as a vehicular language. Views, Vienna English Working Papers 15 (3): 23–27.

Juan-Garau, M., and Salazar-Noguera, J. (2015). Content-based Language Learning in Multilingual Educational Environments. London: Springer.

Kuiken, F., and Vedder, I. (2007). Task complexity and measures of linguistic performance in L2 writing. International Review of Applied Linguistics in Language Teaching IRAL 45 (3): 261–284.

Kuiken, F., and Vedder, I. (2008). Cognitive task complexity and written output in Italian and French as a foreign language. Journal of Second Language Writing 17 (1): 48–60.

Kuiken, F., Vedder, I., and Gilabert, R. (2010). Communicative adequacy and linguistic complexity in L2 writing. In I. Bartning, M. Martin, and I. Vedder (eds) Communicative Proficiency and Linguistic Development: Intersections between SLA and Language Testing Research, 81–100. European Second Language Association.

Larsen-Freeman, D. (2006). The emergence of complexity, fluency, and accuracy in the oral and written production of five Chinese learners of English. Applied Linguistics 27 (4): 590–619.

Lasagabaster, D. (2008). Foreign language competence in content and language integrated courses. The Open Applied Linguistics Journal 1: 3142.

Lasagabaster, D. (2015). Different approaches to bi- or multilingualism and their effect on language attitudes. In M. Juan-Garau and J. Salazar-Noguera (eds) Content-based Language Learning in Multilingual Educational Environments, 13–30. London: Springer.

Laufer, B., and Nation, P. (1995). Vocabulary size and use: Lexical richness in L2 written production. Applied Linguistics 16 (3): 307–322.

Llinares, A., and Whittaker, R. (2007). Talking and writing in a foreign language in CLIL context: A linguistic analysis of secondary school learners of geography and history Resla. Revista Española de Lingüística Aplicada 2007: 83–91.

Lorenzo, F., Moore, P., and Casal, S. (2011). On complexity in bilingual research: The causes, effects, and breadth of content and language integrated learning – a reply to Bruton (2011). Applied Linguistics 32 (4): 450–455.

Lorenzo, F., and Rodríguez, L. (2014). Onset and expansion of L2 cognitive academic language proficiency in bilingual settings: CALP in CLIL. System 4: 64–72.

Lu, X. (2010). Automatic analysis of syntactic complexity in second language writing. International Journal of Corpus Linguistics 15 (4): 474–496.

Lu, X. (2012). The relationship of lexical richness to the quality of ESL learners’ oral narratives. The Modern Language Journal 96 (ii): 190–208.

Lynch, B. K. (1996). Language Programme Evaluation: Theory and Practice. Cambridge: Cambridge University Press.

Manchón, R., Roca de Larios, J., and Murphy, L. (2009). The temporal dimension and problem-solving nature of foreign language composing processes: Implications for theory. In R. M. Manchón (ed.) Writing in Foreign Language Contexts. Learning, Teaching, and Research, 77–101. Bristol: Multilingual Matters.

Maxwell-Reid, C. (2010). Content and Language Integrated Learning (CLIL): The influence of studying through English on Spanish students’ first-language written discourse. Text & Talk 30 (6): 679–699.

McCarthy, P. (2005). An assessment of the range and usefulness of lexical diversity measures and the potential of the measure of textual, lexical diversity (MTLD). PhD thesis. University of Memphis.

McNamara, D. S., Graesser, A. C., McCarthy, P. M., and Cai, Z. (2014). Automated Evaluation of Text and Discourse with Coh-Metrix. Cambridge: Cambridge University Press.

McNamara, T. F. (1996). Measuring Second Language Performance. New York: Longman.

Mollet, E., Wray, A., Fitzpatrick, T., Wray, N. R., and Wright, M. J. (2010). Choosing the best tools for comparative analyses of texts. International Journal of Corpus Linguistics 15 (4): 429–473.

Moreno, V. (2014). El aprendizaje de la competencia escrita. In C. Lomas (ed.) La educación lingüística, entre el deseo y la realidad. Competencias comunicativas y enseñanza del lenguaje. Barcelona: Octaedro.

Norris, J. M., and Ortega, L. (2009). Towards an organic approach to investigating CAF in instructed SLA: The case of complexity. Applied Linguistics 30 (4): 555–578.

Ojeda Alba, J., and Jiménez Catalán, R. M. (2010). The vocabulary of food in EFL learners’ compositions. Cauce. Revista internacional de Filología, Comunicación y sus Didácticas 3: 307–319.

Pallotti, G. (2009). CAF: Defining, refining and differentiating constructs. Applied Linguistics 30 (4): 590–601.

Pérez-Cañado, M. L. (2012). CLIL research in Europe: Past, present, and future. International Journal of Bilingual Education and Bilingualism 15 (3): 315–341.

Pérez-Vidal, C., and Juan-Garau, M. (2009). The effect of Study Abroad (SA) on written performance. EUROSLA Yearbook 9: 269–295.

Pérez-Vidal, C., and Roquet, H. (2015). CLIL in context: Profiling language abilities. In M. Juan-Garau and J. Salazar-Noguera (eds) Content-based Language Learning in Multilingual Educational Environments, 237–255. London: Springer.

Pérez-Vidal, C. (2014). Language Acquisition in Study Abroad and Formal Instruction Contexts. Amsterdam: John Benjamins.

Roca de Larios, J., Manchón Ruiz, R., and Murphy, L. (2007). Componentes básicos y evolutivos del proceso de formulación en la escritura de textos en lengua materna y lengua extranjera. Resla. Revista Española de Lingüística Aplicada 20: 159–183.

Roquet, H., and Pérez-Vidal, C. (in press). Do productive skills improve in content and language integrated learning contexts? The case of writing. Applied Linguistics.

Ruiz de Zarobe, Y. (2010). Written production and CLIL: an empirical study. In C. Dalton-Puffer, T. Nikula and U. Smit Language Use and language learning in CLIL Classrooms, 191–209. Amsterdam: John Benjamins.

Ruiz de Zarobe, Y. (2015). The effects of implementing CLIL in education. In M. Juan-Garau and J. Salazar-Noguera (eds). Content-based Language Learning in Multilingual Educational Environments, 51–68. Londres: Springer.

Ruiz de Zarobe, Y., and Celaya, M. L. (2011). AICLE a través de las lenguas: resultados de la investigación en dos comunidades bilingües/CLIL across languages: research outcomes in two bilingual communities. TESI. Teoría de la Educación: Educación y Cultura en la Sociedad de la Información 12 (3): 200–214.

Sanchez, L. (2014) The role of age on the development of written competence in L4 English: Evidence from a Spanish / German CLIL context. In A. Otwinowska and G. De Angelis (eds) Teaching and Learning in Multilingual Contexts: Sociolinguistic and Educational Perspectives, 125–144. Clevedon: Multilingual Matters.

Schoonen, R. (2012). The validity and generalizability of writing scores. In E. Van Steendam, M. Tillema, G. Rijlaarsdam, and H. Van den Bergh (eds) Measuring Writing: Recent Insights into Theory, Methodology and Practices, 1–22. Leiden / Boston, MA: Brill.

Silva, T. (1993). Toward an understanding of the distinct nature of L2 writing: The ESL research and its implications. TESOL Quarterly 27 (4): 657–677.

Thelen, E., and Corbetta, D. (2002). Microdevelopment and dynamic systems: Applications to infant motor development. In N. Granott and J. Parziale (eds) Microdevelopment: Transition Processes in Development and Learning, 59–79. Cambridge: Cambridge University Press.

Torruella, J., and Capsada, R. (2013). Lexical statistics and typological structures: A measure of lexical richness. Social and Behavioral Sciences 95: 447–454.

Vermeer, A. (2000). Coming to grips with lexical richness in spontaneous speech data. Language Testing 17: 65–83.

Whittaker, R., Llinares, A., and McCabe, A. (2011). Written discourse development in CLIL at secondary school. Language Teaching Research 15 (3): 343–362.

Wolcott, W. and Legg, S. M. (1998). An Overview of Writing Assessment: Theory, Research and Practice. Urbana, IL: National Council of Teachers of English.

Wolfe-Quintero, K., Inagaki, S., and Kim, H. (1998). Second Language Development in Writing: Measures of Fluency, Accuracy and Complexity. Honolulu, HI: University of Hawai’i.

Yoon, H., and Polio, C. (2014). A longitudinal study of linguistic complexity in two genres. Communication presented at Colloquium on Crosslinguistic Aspects of Linguistic Complexity in SLA Research. Brussels.