Instructors’ perceptions of a large language model framework for generating automated multiple-choice questions from video transcripts

Video-Based Learning has become a central modality in digital education, with embedded Multiple-Choice Questions (MCQs) recognized as an effective strategy for promoting engagement and assessing comprehension. However, manually creating high-quality MCQs is challenging for instructors. This study presents a framework that leverages Large Language Models (LLM) to automate MCQ generation from video transcripts while ensuring alignment with established item-writing criteria. Unlike most prior approaches, the framework explicitly integrates the Item-Writing Flaws (IWF) checklist into the generation prompt, rather than using it solely for post-hoc evaluation. It segments video transcripts into topics, extracts and classifies key concepts, and applies structured prompting strategies to generate MCQs adhering to the IWF checklist and additional criteria from the literature, including relevance, difficulty, contextual specificity, and answerability. The framework was evaluated through a mixed-methods study involving seven instructors from diverse disciplines who assessed 257 generated MCQs. The results indicate that the majority of items met established quality guidelines. Specifically, 84.05% of the questions were rated as clear, 86.38% contained plausible distractors, and 93.39% were directly answerable based on the video content. The main issues identified concerned relevance, reflecting occasional difficulty in distinguishing illustrative examples from core instructional content, and linguistic precision, with a tendency to overgeneralize instructors’ statements. In semi-structured interviews, instructors reported that the framework could substantially reduce MCQ preparation time while requiring only minor refinements to improve question specificity. Overall, the findings suggest that embedding pedagogical quality principles directly into LLM prompting represents a viable and promising direction for assessment automation.

Giacomo Cassano, Daniel Ebbert, Negin Mirriahi, Nicoletta Di Blas

09/2026

Educational Technology Research and Development