- 54 views
- 84 downloads
Can AI Offer Effective Feedback for Open-Ended Questions? Results from Fine-Tuned LLMs for Automatic Feedback Generation
-
- Author / Creator
- Mazzullo, Elisabetta
-
The value of timely, personalized, detailed, and actionable feedback for learning is widely recognized; yet its provision is time-consuming and unfeasible in large-scale contexts. Current automatic feedback generation (AFG) methods lack customization and educator involvement. Large language models (LLMs), with their abilities to analyze and generate text, could address these limitations: existing studies using out-of-the-box LLMs and prompting strategies achieved positive results, but space for improvement remains. Parameter efficient fine-tuning can customize pre-trained LLMs using limited data and memory. Also, open-source LLMs can offer cost and privacy advantages with comparable performance to proprietary models. This study explores fine-tuning both closed and open-source LLMs for AFG. Using Meta’s Llama-2-7B and OpenAI’s GPT-3.5-turbo, we generated feedback for open-ended responses to situational judgment questions. The models were fine-tuned on a small set of hand-crafted feedback examples, using prompting strategies in the training instruction. The final model was evaluated by two independent judges and test experts. In addition, a user satisfaction survey was conducted for participants to interact with the model as test takers and evaluate their feedback. Our findings suggest that fine-tuning produces better results when using GPT, and it underscores the importance of effective training instructions. The fine-tuned model achieved a high user satisfaction (84.8%) and largely met desired structural qualities (72.9%). Also, it successfully generalized across different items, and provided feedback aligned with the given instructions, functioning similarly well, regardless of performance level, English learner status, or whether the respondent is currently a student. However, there remain instances where outputs contain linguistic mistakes, fail to provide focused suggestions, or feel quite generic. Suggestions on how these issues might be addressed, implications, and ethical considerations are discussed.
-
- Subjects / Keywords
-
- Graduation date
- Fall 2024
-
- Type of Item
- Thesis
-
- Degree
- Master of Education
-
- License
- This thesis is made available by the University of Alberta Library with permission of the copyright owner solely for non-commercial purposes. This thesis, or any portion thereof, may not otherwise be copied or reproduced without the written consent of the copyright owner, except to the extent permitted by Canadian copyright law.