Speaker
Description
This study examines whether authorship can be distinguished through process-based evidence of engagement with written text. The Content Restoration Authorship Familiarity Test (CRAFT) assesses writers’ ability to restore lexical and structural elements of texts. Sixty undergraduate English education majors completed CRAFT tasks based on self-authored and LLM-generated texts. CRAFT scores strongly differentiated conditions (Cohen’s d = 5.34), suggesting that access to cognitively encoded compositional decisions provides an interpretable, process-based approach to authorship assessment in educational contexts.
Abstract section 2: Contribution/research questions
RQ1. To what extent does performance on the Content Restoration Authorship Familiarity Test (CRAFT), administered immediately after text production, distinguish between human-authored and LLM-generated texts?
RQ2. To what extent do different components of the CRAFT (e.g., word-choice restoration, structural reconstruction) differentially contribute to distinguishing between human-authored and LLM-generated texts?
Abstract section 4: Outcomes/results
Results showed a clear and robust separation between human-authored and LLM-generated writing conditions. Participants consistently demonstrated high levels of authorship familiarity for texts they had written themselves, while performance was substantially lower for LLM-generated texts, yielding non-overlapping score distributions across conditions. This pattern indicates that the CRAFT battery reliably distinguished genuine authorship from surrogate text production in the study context. The findings suggest that authorship-familiarity–based assessment provides interpretable, task-relevant evidence of writing engagement and authorship, offering a viable alternative to probabilistic AI-detection tools for educational assessment.
Abstract section 1: Relevance
Writing remains a central form of evidence for learning and assessment across language education contexts. However, the rapid normalization of LLM-assisted writing complicates the relationship between student texts, cognitive engagement, and authorship. While LLMs offer pedagogical benefits, they also challenge the integrity of writing-based assessment. Current institutional responses, such as prohibitions, disclosure policies, and automated detection tools, are difficult to enforce and offer limited instructional value. This work constitutes an alternative assessment approach that moves beyond surveillance, supporting principled, pedagogically meaningful distinctions between genuine student authorship and surrogate text production in the LLM era.
Abstract section 3: Content/method
The study piloted the Content Restoration Authorship Familiarity Test (CRAFT) with future Korean teachers of English at a university in Seoul. Participants wrote two essays on the same topic: one composed independently by hand and one generated using a large language model. They then completed the CRAFT battery for both texts, which assessed their ability to recall and reconstruct lexical choices and textual structure. Performance patterns were compared across human-authored and LLM-generated conditions.
| Title | Authorship Familiarity as a way to Distinguis Human and LLM- |
|---|---|
| Teaching Context | College and university education |