Le Colloque > Résumés > Bonnard Warren

As applied linguistics continues to evolve, generative artificial intelligence (GenAI) tools based on large language models (LLMs), like ChatGPT, offer novel opportunities for genre analysis. This study addresses the under-researched area of US judicial opinions, exploring how GenAI can be applied to rhetorical move-step analysis (Swales, 1990), a task that traditionally relies on human expertise. Judicial opinions present unique challenges due to their complex, context-dependent nature, contrasting with more frequently studied genres like research article introductions (Lavissière & Bonnard, 2024). In discourse analysis, AI has predominantly been applied to tasks at the sub-sentence level, such as pragmatic annotation (Yu et al., 2024) or function-to-form analysis (Curry et al., 2024), yielding mixed results. Its potential for analyzing rhetorical structures in specialized genres remains largely unexplored, except in research article introductions, where ChatGPT has achieved high levels of accuracy (Kim & Lu, 2024). Therefore, key research questions include: How effectively can AI capture rhetorical information in specialized legal texts? How can AI tools be integrated into corpus-assisted discourse studies? What potential do these tools have for learners needing Language for Specific Purposes (LSP) when engaging with understanding the structure of these complex texts?

Preliminary data were collected from a test using ChatGPT to analyze SCOTUS majority opinions. The tool was prompted with a coding scheme and an annotation guide for rhetorical steps. The initial results demonstrate poor to moderate success, with 36% accuracy in identifying rhetorical steps and 76% accuracy in identifying classes, i.e., broader discourse categories reflecting the organization of SCOTUS opinions. These accuracy levels are significantly lower than those achieved by trained human annotators on the same texts or by ChatGPT on research article introductions. While this represents a preliminary stage of data collection and analysis, efforts to refine prompts and explore alternative methodologies to improve the tool's analytical performance are underway. Detailed findings will be reported at the conference.

These early results suggest that ChatGPT currently has limited potential for advancing rhetorical discourse analysis in less-studied specialized genres. In the absence of a robust conceptual framework, such as the one formulated by Swales and widely applied to article introductions, the tool lags behind human annotators in terms of interpretative depth, even when provided with the same annotation guidelines. This underscores the ongoing necessity of manual annotation to train AI models for such tasks. Additionally, the study situates its results within broader discussions of GenAI's role in applied linguistics, emphasizing how GenAI-assisted methods can complement corpus-assisted research and be integrated into teaching specialized genres, particularly in LSP contexts.

References

Curry, N., Baker, P., & Brookes, G. (2024). Generative AI for corpus approaches to discourse studies: A critical evaluation of ChatGPT. Applied Corpus Linguistics, 4(1), 100082.

Kim, M., & Lu, X. (2024). Exploring the potential of using ChatGPT for rhetorical move-step analysis: The impact of prompt refinement, few-shot learning, and fine-tuning. Journal of English for Academic Purposes, 71, 101422.

Lavissière, M. C., & Bonnard, W. (2024). Who's really got the right moves? Analyzing recommendations for writing American judicial opinions. Languages, 9(4), 119.

Swales, J. M. (1990). Genre analysis. Cambridge University Press.

Yu, D., Li, L., Su, H., & Fuoli, M. (2024). Assessing the potential of LLM-assisted annotation for corpus-based pragmatics and discourse analysis: The case of apology. International Journal of Corpus Linguistics.