Intelligent tutoring systems can support adult learners, but they currently face challenges in adoption due to sociotechnical barriers, while large language models show potential but make too many errors to be suitable as standalone tutoring solutions.

Objective: This dissertation examines how to design and deploy scalable intelligent tutoring systems (ITS) specifically for adult learners, investigating user needs, adoption challenges, and learning outcomes, while also evaluating the potential of large language models (LLMs) to support mathematics tutoring.

Methods: The research adopts a multi-method approach across three key studies:

1. The development and deployment of "Apprentice Tutors," a web-based intelligent tutoring platform for college algebra topics, followed by focus groups with students and teachers to identify sociotechnical factors affecting adoption.
2. Analysis of a two-year deployment of Apprentice Tutors at a state technical college, examining usage patterns, demographics, and learning outcomes through quantitative analysis of user data.
3. Evaluation of Large Language Models (including GPT-3.5 Turbo, GPT-4, GPT-4o, o1-mini, and o1-preview) for math tutoring using two novel assessment methods: using intelligent tutors as testbeds for problem-solving accuracy and interactive prompting to evaluate tutoring dialogue quality.

Key Findings:

1. On Tutor Adoption and Usage:
   • Apprentice Tutors had an overall adoption rate of approximately 14.8% among adult learners.
   • As deployment scaled from 4 to 58 classes, the rate of initial tutor use decreased from 62.84% to 13.78%, suggesting challenges in scaling educational technology.
   • Adult learners primarily used tutors during evening hours (5 p.m. to 12 a.m.) and Thursday was the peak usage day, reflecting the need for flexibility.
   • Older learners and female students showed higher engagement with the tutors than younger and male counterparts.
2. On Learning Outcomes:
   • Evidence showed decreasing error rates with practice within the tutors, indicating skill improvement.
   • Preliminary evidence suggested a positive correlation between tutor usage and course performance, with tutor users achieving higher mean grades on Unit 1 assessments.
3. On LLM Evaluation:
   • LLMs achieved reasonable final answer accuracy (average 85.5% across models) on college algebra problems.
   • However, only 56.6% of interactive LLM tutoring dialogues were entirely correct, suggesting they make errors in nearly half of all tutoring interactions.
   • While 90% of LLM dialogues demonstrated high pedagogical quality (good scaffolding, clear explanations), they prioritized getting the final answer right over ensuring students understand the process.

Implications: The research contributes to our understanding of how to improve intelligent tutoring systems for adult learners:

1. It identifies key sociotechnical factors affecting adoption, including the need for intuitive interfaces, alignment with course content, and teacher support.
2. It demonstrates that adult learners can benefit from intelligent tutors when they are designed to accommodate flexible learning schedules and diverse needs.
3. It suggests that while LLMs show promise for certain aspects of tutoring (generating hints, providing motivation), they are not yet reliable enough to serve as standalone tutors without oversight to ensure accuracy.
4. The work provides actionable design guidelines for improving educational technology for adult learners, including incorporating human elements, streamlining onboarding, and demonstrating clear value.

Limitations:

1. The non-experimental design makes it challenging to establish causal relationships between tutor usage and improved academic performance.
2. The studies were conducted at a single technical college, which may limit generalizability to other educational contexts.
3. The LLM evaluation was focused specifically on college algebra problems, not covering other subject areas.
4. The LLM evaluation did not assess impacts on real students due to concerns about potential harm from incorrect tutoring.

Future Directions:

1. Implementing randomized controlled trials to more rigorously establish the causal effects of tutor use on adult learning outcomes.
2. Expanding tutor coverage beyond mathematics to other disciplines and assessing differences in adoption and effectiveness.
3. Exploring how LLMs might be integrated with traditional intelligent tutoring systems to leverage their respective strengths while mitigating limitations.
4. Investigating how to better support teachers in adopting and implementing intelligent tutoring systems at scale.

Title and Author: "Designing and deploying scalable intelligent tutoring systems to enhance adult education" by Adit Gupta

Published On: March 2025

Published By: Drexel University (doctoral dissertation)