Abstract:

This study provides specific evidence regarding the extent to which quantitative measures, common sense notional measures, and comprehensive measures adequately characterize spontaneous, although engaged, speech. As such, the study contributes to the growing body of literature describing the current limits of automatic systems for evaluating spoken proficiency, provides examples of the essential nature of various notional and comprehensive variables, supports continued development of hybrid systems, and includes suggestions for the possible utilization of additional variables for automatic analyses. Data for this study were gathered and analyzed as follows. After 4 weeks of activities related to career development, 20 native English speaking college freshmen made recordings in English explaining their career preferences. Three experiments were then conducted. Experiment 1 analyzed the recordings according to current quantitative analyses used in fully automatic evaluations of fluency. Experiment 2 examined the recordings through a perception study according to common everyday notions of fluency. Experiment 3 analyzed the recordings according to an adaptation of the comprehensive rubrics used by the Educational Testing Service (ETS) for evaluating oral proficiency. The comprehensive evaluation (Experiment 3) provided the most insight, and temporal quantitative measures (Experiment 1) provided the least insight concerning the proficiency of the 20 speakers.

Author: Garry Molholt, María Jose Cabrera, V. K. Kumar, Philip Thompsen