Speaker
Description
This presentation was funded by a JALT research grant. Spoken fluency is a crucial concept, but also notoriously hard to operationalize. Since automated solutions fail reliably extract even the simplest fluency metrics, this presentation introduces a brief training program to facilitate the collection of large amounts of reliable human annotations. The program has been tested over a period of 80 hours. Output from all participants shows a high degree of agreement, indicating robust learning effects.
Abstract section 4: Outcomes/results
The GAP Workshop has been tested over a period of 80 hours by 8 collaborators who worked in small groups and without supervision. Collaborators had varying degrees of familiarity with linguistics, speech analysis and annotation in Praat, including complete novelty. The analysis of the output from all participants shows a high degree of agreement, both between annotators and with the reference annotation, indicating robust learning effects. Post-session feedback was collected, analyzed and used for the improvement of the script, of the training materials and of the workshop as a whole.
Abstract section 5: References
[1] Fillmore (1979). On fluency. In Fillmore, Kempler & Wang (eds.), Individual differences in language ability and language behavior. New York: Academic Press.
[2] Riggenbach (2000). Perspectives on fluency. Ann Arbor: University of Michigan Press.
[3] Coretta et al. (2023). Multidimensional Signals and Analytic Flexibility: Estimating Degrees of Freedom in Human-Speech Analyses. Advances in Methods and Practices in Psychological Science 6(3).
[4] Tavakoli (2016). Fluency in monologic and dialogic task performance. International Review of Applied Linguistics in Language Teaching 54(2).
[5] Suzuki, Kormos & Uchihara (2021). The Relationship Between Utterance and Perceived Fluency: A Meta-Analysis of Correlational Studies. The Modern Language Journal 105(2).
[6] Lennon (1990). Investigating Fluency in EFL: A Quantitative Approach. Language and Learning 40(3).
[7] Sato (2014). Exploring the construct of interactional oral fluency. System 45.
[8] De Jong, Pacilly & Heeren (2021). PRAAT scripts to measure speed fluency and breakdown fluency in speech automatically. Assessment in Education: Principles, Policy & Practice 28(4).
[9] McDougall & Duckworth (2017). Profiling fluency: An analysis of individual variation in disfluencies in adult males. Speech Communication 95.
[10] Cole, Mahrt & Roy (2017). Crowd-sourcing prosodic annotation. Computer Speech & Language 45.
[11] Boersma & Weenink (2025). Praat: doing phonetics by computer. https://www.praat.org.
[12] Ishikawa (2023). The ICNALE guide. London: Routledge.
Abstract section 2: Contribution/research questions
Therefore, to ensure progress in the field, two developments are needed: (a) exploring advanced metrics for the quantification of spoken fluency [9] and (b) facilitating the collection of large amounts of reliable human annotations [10]. This presentation addresses the latter goal, by introducing researchers, language teachers, and unexperienced collaborators to a training program for the nuanced quantification of spoken fluency. The program takes approximately 2 hours and requires no previous knowledge in speech analysis.
Abstract section 1: Relevance
Spoken fluency is a crucial concept both for language research [1] and for education practice [2]. Due to the inherently multi-dimensional nature of speech [3], however, fluency remains hard to operationalize in dialogic contexts [4], and its quantification still relies on few simple fluency metrics [5], such as speech rate (i.e. words per minute), either pruned (i.e. excluding hesitations) or unpruned [6,7]. Moreover, even the extraction of such simple metrics relies on time-consuming manual annotations. Alternatives based on artificial intelligence remain unreliable, even for low-stakes contexts [8], and still require a comparison with the gold standard of human annotation.
Abstract section 3: Content/method
The main component of the program is a custom Praat [11] script containing a Guided Annotation Procedure (GAP) for the analysis of fluency, complete with examples [12], a 40-minutes videotutorial and a 10-pages manual. Users are instructed on how to use Praat, how to segment speech and silence, which intervals to flag as disfluencies, and how to provide syllable counts. After providing their annotation of a test file, users receive automated feedback on their performance.
| Title | GAP: A guided procedure for the annotation of speech fluency |
|---|---|
| Teaching Context | College and university education |