Using an automated writing evaluation (AWE) system for K-12 students leads to positive cumulative effects on ELA performance that peak after three years of implementation.
Objective: To investigate the long-term effects of statewide implementation of Utah Compose (UC), an AWE system, on students' state test English Language Arts (ELA) performance over four years (2015-2018).
Methods:
- Quasi-experimental design using inverse probability of treatment weighting (IPTW)
- Sample: 134,425 students in Grades 4-10 in Utah
- Data collected from:
- Utah State Board of Education (student demographics and ELA scores)
- Measurement Incorporated (UC usage data)
- National Center on Educational Statistics (school data)
- Analysis using stabilized IPTW and regression models
Key Findings:
- First year of UC use showed positive effects (effect size = 0.12)
- Second year increased effect size by 0.06
- Third year further increased by 0.02
- Total cumulative effect size after three years was 0.20
- Effects plateaued after three years
- Benefits diminished each subsequent year
- Lower usage observed in high school grades compared to elementary/middle school
Implications:
- AWE shows immediate positive impact in first year
- Long-term implementation beneficial up to three years
- Findings support AWE as effective educational technology
- Results encourage sustained, long-term AWE usage
- Highlights need to address curriculum constraints in higher grades
Limitations:
- Focus only on overall ELA performance rather than writing specifically
- Dichotomous treatment of UC exposure rather than usage intensity
- Limited teacher-level measures available
- Study confined to naturalistic implementation without usage requirements
- Results may differ from controlled implementation settings
Future Directions:
- Explore AWE effects on different state-level outcomes
- Identify which AWE features drive positive effects
- Investigate impact of usage intensity
- Examine effects of different usage sequences
- Study AWE implementation in more controlled settings
- Include additional teacher-level variables
Title and Authors: "Exploring the Long-Term Effects of the Statewide Implementation of an Automated Writing Evaluation System on Students' State Test ELA Performance" by Yue Huang, Joshua Wilson, and Henry May
Published On: December 16, 2024
Published By: International Journal of Artificial Intelligence in Education