In order to meet the rapidly increasing demand for item pools for large-scale assessments, automatic item generation (AIG) emerged about thirty years ago, using pre-programmed algorithms to automatically construct large numbers of test items with predictable item parameters. The rapid progress in natural language processing due to transformer networks has enabled large language models to handle a variety of natural language processing tasks, i.e., translation, text summarization, question answering, and writing text, at a level similar to humans. This study has carried out an empirical study on the human and transformer-based collaborative AIG framework for course-based item generation performances of several GenAI models for translation tasks of the English examination in China. The results show that: 1) Most GenAI models can successfully generate English-Chinese and Chinese-English sentence translation items. 2) Most GenAI models can generate both English-Chinese and Chinese-English text translation passages. 3) Readability of the generated passages is analyzed, and content validity of the generated sentence translation items and text translation passages is verified by subject matter experts. This study highlights that GenAI models help reduce teachers’ burdens of repetitive and time-consuming human item writing tasks if handled properly.
References
[1]
Attali, Y., Runge, A., LaFlair, G. T., Yancey, K., Goodwin, S., Park, Y. et al. (2022). The Interactive Reading Task: Transformer-Based Automatic Item Generation. FrontiersinArtificialIntelligence,5, Article 903077. https://doi.org/10.3389/frai.2022.903077
[2]
Baker, T., & Smith, L. (2019). Educ-AI-Tion Rebooted? Exploring the Future of Artificial Intelligence in Schools and Colleges. Nesta Foundation Website. https://media.nesta.org.uk/documents/Future_of_AI_and_education_v5_WEB.pdf
[3]
Chiu, T. K. F. (2023). The Impact of Generative AI (GenAI) on Practices, Policies and Research Direction in Education: A Case of ChatGPT and Midjourney. InteractiveLearningEnvironments,32, 6187-6203. https://doi.org/10.1080/10494820.2023.2253861
[4]
Crompton, H., & Burke, D. (2023). Artificial Intelligence in Higher Education: The State of the Field. InternationalJournalofEducationalTechnologyinHigherEducation,20, Article No. 22. https://doi.org/10.1186/s41239-023-00392-8
[5]
Dhawan, S., & Batra, G. (2021). Artificial Intelligence in Higher Education: Promises, Perils, and Perspectives. ResearchGate. https://www.researchgate.net/publication/348910302
[6]
Feng, Q. H., & Chen, K. F. (2008). An Elementary Coursebook on Chinese-English Translation. Higher Education Press.
[7]
Garner, B. A. (2009). Black’s Law Dictionary (9th ed., p. 1499). WEST.
[8]
Hughes, A. (2003). Testing for Language Teachers (2nd ed.). Cambridge University Press.
[9]
Humes, A. (1983). Discourse Type and Composition Research. Southwest Regional Laboratory Working Paper, WP 2-83 /01.
[10]
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F. et al. (2023). ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education. LearningandIndividualDifferences,103, Article ID: 102274. https://doi.org/10.1016/j.lindif.2023.102274
[11]
Law, L. (2024). Application of Generative Artificial Intelligence (GenAI) in Language Teaching and Learning: A Scoping Literature Review. ComputersandEducationOpen,6, Article ID: 100174. https://doi.org/10.1016/j.caeo.2024.100174
[12]
Lee, P., Fyffe, S., Son, M., Jia, Z., & Yao, Z. (2023). A Paradigm Shift from “Human Writing” to “Machine Generation” in Personality Test Development: An Application of State-Of-The-Art Natural Language Processing. JournalofBusinessandPsychology,38, 163-190. https://doi.org/10.1007/s10869-022-09864-6
[13]
Lin, Z., & Chen, H. (2024). Investigating the Capability of ChatGPT for Generating Multiple-Choice Reading Comprehension Items. System,123, Article ID: 103344. https://doi.org/10.1016/j.system.2024.103344
[14]
Lu, D. K. (2025). Subversion and Reconstruction: The “Butterfly Effect” in Education Triggered by DeepSeek and Measures of Response. Journal of Xinjiang Normal University, 46, 144-152.
[15]
Lu, O. H. T., Huang, A. Y. Q., Tsai, D. C. L., & Yang, S. J. H. (2021). Expert-Authored and Machine-Generated Short-Answer Questions for Assessing Students’ Learning Performance. Educational Technology & Society, 24, 159-173.
[16]
Ouyang, F., Zheng, L., & Jiao, P. (2022). Artificial Intelligence in Online Higher Education: A Systematic Review of Empirical Research from 2011 to 2020. EducationandInformationTechnologies,27, 7893-7925. https://doi.org/10.1007/s10639-022-10925-9
[17]
Qin, H. W., & Wang, K. F. (2010). Comparison and Translation Between English and Chinese. Foreign Language Teaching and Research Press.
[18]
Rottenberg, A. T., & Winchell, D. H. (2021). The Structure of Argument (10th ed., pp. 667-668). Bedford/St. Martin’s.
[19]
Russel, S., & Norvig, P. (2010). Artificial Intelligence—A Modern Approach. Pearson Education.
[20]
Sayin, A., & Gierl, M. (2024). Using OpenAI GPT to Generate Reading Comprehension Items. EducationalMeasurement:IssuesandPractice,43, 5-18. https://doi.org/10.1111/emip.12590
[21]
Shu, D. F., & Zhuang, Z. X. (2008). ModernForeignLanguageTeaching:Theory,PracticeandMethod(RevisedEdition). Shanghai Foreign Language Education Press.
[22]
Siefring, J. (2004). TheOxfordDictionaryofIdioms (2nd ed.). Oxford University Press.
[23]
Sommer, M., & Arendasy, M. (2025). Automatic-and Transformer-Based Automatic Item Generation: A Critical Review. Journal of Intelligence, 13, Article 102. https://doi.org/10.3390/jintelligence13080102
[24]
Song, Y. S., Du, J. L., & Zheng, Q. H. (2025). Automatic Item Generation for Educational Assessments: A Systematic Literature Review. InteractiveLearningEnvironments,33, 1-20.
[25]
Wang, D. F. et al. (1996). AChinese-EnglishDictionaryofIdioms. Sichuan People’s Publishing House.
[26]
Wang, Y., & Zhang, Z. (2024). The Worries and Solutions to ChatGPT AI Translation. ChineseTranslatorsJournal,No. 2, 95-102.
[27]
Xu, J. J. (2024). BFSUReadabilityAnalyzer3. https://corpus.bfsu.edu.cn
[28]
Yuan, C. X. (2023). Baidu’sNetProfitIncreasedby10%LastYear,BettingAIEra,PlanningtoIntegrateMultipleMainstreamBusinesseswithErnieBot. Securities Daily. http://www.zqrb.cn/gscy/qiyexinxi/2023-02-23/A1677080208200.html
[29]
Zawacki-Richter, O., Bai, J. Y. H., Lee, K., Slagter van Tryon, P. J., & Prinsloo, P. (2024). New Advances in Artificial Intelligence Applications in Higher Education? InternationalJournalofEducationalTechnologyinHigherEducation,21, Article No. 32. https://doi.org/10.1186/s41239-024-00464-3
[30]
Zhang, P. J. (2018). ACourseinEnglish-ChineseTranslation(RevisedEdition). Shanghai Foreign Language Education Press.
[31]
Zhong, W. H., & Zhao, J. F. (2015). Interpretation of Key Points in the National Standards for Teaching Quality of Bachelors’ Translation and Interpreting Programs. ForeignLanguageTeachingandResearch(Bimonthly),47, 289-296.
