Item Details

Five statistical models for Likert-type experimental data on acceptability judgments

Issue: Vol 3 No. 2 (2016)

Journal: Journal of Research Design and Statistics in Linguistics and Communication Science

Subject Areas: Linguistics

DOI: 10.1558/jrds.30822


This paper contributes to the ongoing debate over Likert scale experiments, in particular the issues of how to treat acceptability judgment data (as ordinal or interval) and what statistical model is appropriate to apply. We analyze empirical data on native speakers’ intuitions regarding marginal change-of state verbs in Russian (e.g. ukonkretit’ ‘concretize’, ovnešnit’ ‘externalize’) and compare the outcomes of five statistical models (parametric and non-parametric tests): (1) ANOVA; (2) Ordinal Logistic Regression Model; (3) Mixed-Effects Regression Model for Ordinal data; (4) Regression Tree and Random Forests Model; and (5) Classification Tree and Random Forests Model. We make four claims: (1) all five models are appropriate for this data to a greater or lesser degree; (2) overall, the outcomes of parametric and non-parametric tests applied to this data provide comparable results; (3) Classification Tree and Random Forests Model is the most appropriate, informative, and user-friendly regarding this data; and (4) the use of a culturally entrenched grading scale is an advantage.

Author: Anna Endresen, Laura A. Janda

View Original Web Page

References :

Baayen, R. H. (2008). Analysing Linguistic Data. A Practical Introduction to Statistics using R. Cambridge: Cambridge University Press.

Baayen, R. H., Janda, L. A., Nesset, T., Endresen, A., and Makarova, A. (2013). Making choices in Slavic: Pros and cons of statistical methods for rival forms. Russian Linguistics 37: 253–291.

Bermel, N. and Knittl, L. (2012). Corpus frequency and acceptability judgments: A study of morphosyntactic variants in Czech. Corpus Linguistics and Linguistic Theory 8 (2): 241–275.

Blaikie, N. (2003). Analyzing Qualitative Data. London: SAGE Publications Ltd.

Cantos Gómez, P. (2013). Statistical Methods in Language and Linguistic Research. Sheffield: Equinox Publishing.

Christensen, R. H. B. (2015). Ordinal – Regression Models for Ordinal Data. R package version 2015.6-28. Software and manual retrieved on 19 May 2017 from

Christensen, R. H. B. and Brockhoff, P. B. (2013). Analysis of sensory ratings data with cumulative link models. Journal de la Société Française de Statistique 154 (3): 58–79.

Cohen, L., Manion, L., and Morrison, K. (2000). Research Methods in Education, 5th ed. London: Routledge Falmer.

Collins, C., Guitard, S. N., and Wood, J. (2009). Imposters: An online survey of grammaticality judgments. NYU Working Papers in Linguistics 2: Papers in Syntax. Retrieved on 19 May 2017 from

Dąbrowska, E. (2010). Naive vs. expert intuitions: An empirical study of acceptability judgments. The Linguistic Review 27: 1–23.

Dubois, D. (2013) Statistical reasoning with set-valued information: Ontic vs. epistemic views. In C. Borgelt, Gil, M. A., Sousa, J. M. C., and Verleysen, M. (Eds) Towards Advanced Data Analysis by Combining Soft Computing and Statistics. Studies in Fuzziness and Soft Computing 285: 119–137. Berlin/Heidelberg: Springer-Verlag.

Endresen, A. (2013). Samostojatel’nye morfemy ili pozicionnye varianty? Morfologičeskij status russkix pristavok o- i ob- v svete novyx dannyx: korpus i èksperiment [Distinct morphemes or positional variants? Morphological status of the Russian prefixes o- and ob- in the light of new evidence: corpus and experiment]. Voprosy jazykoznanija 6: 33–69.

Endresen, A. (2014). Non-Standard Allomorphy in Russian Prefixes: Corpus, Experimental, and Statistical Exploration. Doctoral dissertation. University of Tromsø: The Arctic University of Norway. Retrieved on 19 May 2017 from

Faraway, J. J. (2006). Extending the Linear Model with R: Generalized Linear, Mixed Effects and Nonparametric Regression Models. Boca Raton, FL: Chapman and Hall/CRC.

Gardner, P. L. (1975). Scales and statistics. Review of Educational Research 45: 43–57.

Grilli, L. and Rampichini, C. (2012). Multilevel models for ordinal data. In R. S. Kenett and S. Salini (Eds) Modern Analysis of Customer Surveys: with Applications using R, 391–408. Chichester: John Wiley and Sons.

Harrell, F. E. (2001). Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis. New York: Springer series in Statistics.

Haspelmath, M. (2002). Understanding Morphology. London: Oxford University Press.

Jaccard, J. and and Wan, C. K. (1996). LISREL Approaches to Interaction Effects in Multiple Regression. Thousand Oaks, CA: SAGE Publications.

Jamieson, S. (2004). Likert scales: How to (ab)use them. Medical Education 38: 1212–1218.

Janda, L. A. (Ed.) (2013). Cognitive Linguistics: The Quantitative Turn. The Essential Reader. Berlin and Boston, MA: De Gruyter Mouton.

Kapatsinski, V. (2013). Conspiring to mean: Experimental and computational evidence for a usage-based harmonic approach to morphophonology. Language 89: 110–148.

Keller, F. and Asudeh, A. (2001) Constraints on linguistic coreference: Structural vs. pragmatic factors. In J. D. Moore and Stenning, K. (Eds) Proceedings of the 23rd Annual Conference of the Cognitive Science Society, 483–488. Mahawah, NJ: Lawrence Erlbaum Associates.

Kim, J.-O. (1975). Multivariate analysis of ordinal variables. American Journal of Sociology 81: 261–298.

King, B. M. and Minium, E. W. (2008). Statistical Reasoning in the Behavioral Sciences. Hoboken, NJ: Wiley.

Knapp, T. R. (1990). Treating ordinal scales as interval scales: An attempt to resolve the controversy. Nursing Research 39 (2): 121–122.

Labovitz, S. (1967). Some observations on measurement and statistics. Social Forces 46: 151–160.

Labovitz, S. (1970). The assignment of numbers to rank order categories. American Sociological Review 35: 515–524.

Lavrakas, P. J. (2008). Encyclopedia of Survey Research Methods. Thousand Oaks, CA: SAGE Publications.

Likert, R. (1932). A Technique for the Measurement of Attitudes. Doctoral dissertation. Columbia University. Series Archives of Psychology 22: 5–55. NY: The Science Press. Retrieved on 19 May 2017 from

Pell, G. (2005). Use and misuse of Likert scales. Medical Education 39 (9): 970.

R Development Core Team. (2010). R: A Language and Environment for Statistical computing. R Foundation for Statistical Computing. Vienna, Austria. ISBN 3-900051-07-0.

Rietveld, T. and van Hout, R. (2005). Statistics in Language Research: Analysis of Variance. Berlin and New York: Mouton de Gruyter.

Schütze, C. T. (1996). The Empirical Base of Linguistics: Grammaticality Judgments and Linguistic Methodology. Chicago, IL and London: The University of Chicago Press.

Strobl, C., Malley, J., and Tutz, G. (2009). An introduction to recursive partitioning: Rationale, application, and characteristics of classification and regression trees, bagging, and random forests. Psychological Methods 14 (4): 323–348.

Tagliamonte, S. A. and Baayen, R. H. (2012). Models, forests and trees of York English: Was/were variation as a case study for statistical practice. Language Variation and Change 24 (2): 135–178.

Townsend, Ch. E. (1968). Russian Word-Formation. Bloomington, IN: Slavica Publishers. Reprint edition from 2008.