News from the Chinese Room: an explorative study on using ChatGPT as linguistic informant

This exploratory study explores whether large language models can serve as variationist linguistic informants by comparing GPT‑4 (ChatGPT) outputs with human productions of diminutive suffixes in Austrian dialectal varieties of German. Human data stems from Wenker sentence translation tasks spanning eight diminutive contexts, while synthetic data were elicited via prompts that manipulated shots as well as explicit dialect labels. Both datasets were annotated for diminutive suffixes. A chi‑square test showed a significant overall divergence between human and synthetic suffix distributions (χ² = 79.227, p = 1.98e‑14; Cohen’s ω ≈ 0.799). For individual dialect regions, Fisher’s exact tests (with hapax excluded) were significant in Alemannic, Bavarian‑Alemannic, and South Bavarian, but not in East‑, South‑, or West‑Middle Bavarian. Qualitatively, ChatGPT overproduced diminutives in general and standard forms (e.g., -chen, -lein) and underproduced non‑standard -lan; it also introduced rare or dubious forms, inflating apparent diversity. As a consequence, current LLMs do not reliably model regional morphological variation and cannot replace human informants.

Keywords: Language Variation, Large Language Models, Diminutives, AI-generated language