AI-Generated Assessment Items: Quality, Validity, and the Role of Human Oversight

Abstract

The rapid development of generative artificial intelligence has created new opportunities for educational assessment, particularly in the development of test items. Large language models (LLMs) can generate multiple-choice questions, constructed-response prompts, distractors, scenarios, and alternative item versions within seconds, potentially reducing the time and resources traditionally required for item development. However, the ability to generate a plausible question does not necessarily demonstrate that the resulting item is suitable for educational measurement. This paper examines current evidence on the quality of AI-generated assessment items and considers the implications of their use for validity, cognitive alignment, fairness, technical quality, and psychometric functioning. A structured review and conceptual analysis of recent literature on LLM-based automatic item generation was conducted, with particular attention to research published between 2023 and 2026. The review indicates that AI-generated items can achieve promising levels of linguistic, content, and, in some contexts, psychometric quality. Nevertheless, important limitations remain, including factual inaccuracies, construct misalignment, insufficient cognitive demand, weak distractors, response-pattern artifacts, cultural sensitivity concerns, and substantial variation according to model, prompt design, and assessment domain. Building on emerging human-in-the-loop approaches, this paper proposes an evidence-based quality assurance framework that differentiates among tasks suitable for AI assistance, tasks requiring professional judgment, and claims requiring empirical evidence. The paper argues that as item generation becomes increasingly scalable, the principal challenge shifts from producing assessment content to validating it. Generative AI may therefore change rather than eliminate the role of assessment professionals, placing greater emphasis on specification, review, validation, and accountable decision-making.