- 156 views
- 146 downloads
Conversation-Based Assessments: Measuring Student Learning with Human-Like Communication
-
- Author / Creator
- Yildirim-Erbasli, Seyma Nur
-
In recent years, conversational agents have been widely used in education to support student learning. Conversational agents have the capability to enhance learning by improving interaction, motivation, feedback, and personalization. To date, researchers have designed and used different types of conversational agents––such as virtual teaching assistants, tutors, and peers (or learning companions)––to support the student learning process. In addition to the instructional use of conversational agents, there have been new attempts in recent years to design and use conversational agents for educational assessments (i.e., conversation-based assessments: CBA), with the goal of improving students’ assessment experiences. To address the limited research on CBA, this research aimed to design a CBA as a formative assessment to assess higher education students’ knowledge and provide support and feedback to scaffold their learning. This study introduced a CBA with selected-response (multiple-choice and true-false) and constructed-response (short-answer) tests and evaluated its performance based on intent classification and confidence score. CBA was designed using Rasa—an artificial intelligence-based tool—and deployed to Google Chat to share with students in two sections of an undergraduate-level course, Educational Assessment. One section of the course (n1 = 290) was provided with only the selected-response format while the other section (n2 = 119) was provided with both CBA formats following course instructors’ availability and preference to use CBA in their sections. A survey was administered after students experienced CBA to investigate their attitudes toward CBA. The unique total number of students who took selected-response and constructed-response tests and completed the survey are 98, 21, and 61 respectively. In addition, CBA was evaluated by students in another undergraduate-level course, Introduction to Human Computer Interaction, (n3 = 106) to find usability issues through a cognitive walkthrough. The conversation data showed that CBA with both selected-response and constructed-response items produced high standard accuracy measures and confidence scores for each intent (i.e., student response). CBA with selected-response items interpreted all student responses accurately and chose the appropriate conversation paths (F1-measure of 100% and the confidence score of 1 for each intent). In comparison, CBA with constructed-response items consistently matched student responses to the appropriate conversation paths for the most part (F1-measures ranged from 89% to 100%, and confidence scores ranged from 0.30 to 0.99). The findings suggest that ensuring the accuracy of CBA with constructed-response items is more challenging than CBA with selected-response items. According to the survey data, most of the students reported positive attitudes toward CBA. Student reactions to CBA and regular assessments (e.g., online quizzes) were very similar. The findings from the cognitive walkthrough of CBA showed its usability, however, several important usability problems were also reported to improve the user interaction with CBA. Highly accurate dialogue moves within CBA, positive student attitudes toward CBA, and usability indicators suggest the utility of CBA in measuring student knowledge and skill as well as enhancing their assessment experiences. Overall, this study indicates the promise of conversational agents in developing more interactive assessments to measure higher education students’ knowledge and skill as well as enhance their assessment experiences through a more interactive assessment environment.
-
- Graduation date
- Fall 2022
-
- Type of Item
- Thesis
-
- Degree
- Doctor of Philosophy
-
- License
- This thesis is made available by the University of Alberta Library with permission of the copyright owner solely for non-commercial purposes. This thesis, or any portion thereof, may not otherwise be copied or reproduced without the written consent of the copyright owner, except to the extent permitted by Canadian copyright law.