Speaker
Description
Lacking a consensus on the value of AI usage in language learning,teachers of second language writing may wish to limit its use and identify when it has been used. However, can teachers identify machine-generated text and distinguish it from human-written sentences?
This poster describes the pilot study comparing the ability of language teachers, AI checkers and large language models (LLMs) to identify machine-generated text, defined as that produced by an LLM (e.g. ChatGPT) or machine-translation (MT) software (e.g. Google Translate).
The survey comprises 24 extracts of 6 types: 1) real student writing from the pre-AI era; 2) text machine-generated using an LLM; 3) text translated from an entirely Japanese text using MT; 4 and 5) real student writing with AI/MT elements; 6) real student writing “polished” using Grammarly (considered mixed human/machine).
In the survey, respondents are asked to identify which sentences in each text appear machine-generated. Then, they are asked to give an overall evaluation of the text.
The steps undertaken to produce the extracts and design the mechanics of the survey will be described, itself coded in HTML and JavaScript using Claude AI. Participants can participate in the pilot and sign up to join the full study later.
| Presentation location | In person (Kumamoto) |
|---|