Teaching Effectiveness Advisor chatbot brings AI to SETs

GenAI in Teaching and Learning Faculty Fellow Jared DeForest uses AI to solve problems -- to support students with a custom chatbot that personalizes learning or to decrease administrative loads he experiences as department chair of Environmental and Plant Biology, for example. 

His most recent excursion into custom chatbot use focused on student evaluation of teaching (SET) to improve instruction. Like his colleagues, DeForest often receives much positive feedback and also feedback that seems important and worth taking seriously but that was difficult to interpret or act upon. Comments from students were often emotionally resonant and/or well-intentioned but also unclear or vague. 

So DeForest took up another chatbot challenge, training a Teaching Effectiveness Advisor using Ohio University and Center for Teaching, Learning and Assessment policies and norms, expectations outline by the Advancing Ohio Higher Education Act and research on student feedback to improve instruction. The chatbot accepts SET reports, peer observation reports, syllabi, course information teaching reflections and philosophies -- whatever materials faculty deem relevant.

In response to the data provided, the chatbot recognizes patterns (themes, strengths, recurring issues), interprets SET data and provides recommendations for actionable, low-burden teaching strategies and approaches to try. DeForest established response-rate rules for SETs. The chatbot offers no interpretation and refers users to CTLA for assistance and consultation if student response rates on the provided SETs are below 20%. Limited and cautious, contextualized interpretation takes place at below 50% response rate.

The chatbot is coded to protect materials uploaded; however, faculty should be careful about types of data submitted and understand the limitations of AI and custom chatbots. The chatbot is designed for FORMATIVE use only. DeForest does not have access to any of the materials uploaded. Ohio faculty who would like to try out the Teaching Effectiveness Advisor chatbot may access it through this QR code. 

DeForest is interested in feedback from Ohio faculty who choose to experiment with the custom chatbot. He may be reached by email. 

Best Practices for Using this Chatbot

Start with context to improve clarity

• Share multiple sections/semesters of SETs
• Include your syllabus, modality and enrollment context 
• Add your teaching philosophy and/or course goals

Summarize first, look for patterns

• Ask for high-level summary and assessment
• Focus on recurring themes
• Follow patterns into the specifics

Review detailed SETs after you have a framework

• Use the chatbot to explore possible interpretations
• Pair with your own reflection and judgment
• Treat suggestions as options, not prescriptions

Staff from the CTLA and the Office of Instructional Design are available to provide additional and individual consultation on instructional strategies, course design and pedagogical approaches.