A productive-failure-based workshop using AI tools and Scratch programming did not significantly improve algorithmic thinking skills in secondary students during a short intervention period, with most learning failures occurring during the algorithm development phase.
Objective: The main goal of this study was to investigate whether the productive failure approach, integrated with AI tools (specifically Teachable Machine) and Scratch programming, can effectively foster algorithmic thinking in secondary school students. The research also aimed to identify the types of productive failures students experience during algorithmic thinking development and understand how the problem-solving process is structured within this educational framework.
Methods: The researchers employed a mixed-methods single case study design with a within-subject approach, analyzing the same 23 secondary school students (aged 13-14) before and after the workshop intervention. The study utilized both quantitative and qualitative data collection methods. Quantitatively, students completed a reliable algorithmic thinking pre- and post-test (Cronbach's α = 0.8) consisting of 24 items. Qualitatively, 7 groups (14 students) were video-recorded during their problem-solving processes, with data analyzed through qualitative content analysis using MAXQDA software. The intervention consisted of a four-module workshop: Module 0 (knowledge activation in school), Module 1 (online machine learning basics with Teachable Machine), and Modules 2-3 (productive failure approach implementation in a university teaching-learning lab). The productive failure design followed a Problem-Solving followed by Instruction (PS-I) structure, where students first attempted to solve complex programming tasks (creating Tetris and rock-paper-scissors games) before receiving targeted instruction. Treatment fidelity was ensured through consistent worksheets, program recordings, and video documentation across all participants.
Key Findings: The quantitative analysis revealed no significant improvement in algorithmic thinking skills after the workshop intervention. The Wilcoxon test showed no significant differences between pre- and post-test scores (W = 133, p = 0.859), with a low effect size (biserial rank correlation coefficient = 0.299). The standard deviation increased from 2.95 to 4.55, indicating greater diversity in student performance post-intervention. Qualitatively, the study identified 294 productive failures across all participants, with the majority (76.5%) occurring during solution development phases. Students demonstrated stronger performance in understanding problems compared to solving them, with most worksheet responses showing weak implementation in the problem-solving category. The analysis revealed that productive failures occurred primarily in the first half of each module, after which students alternated more successfully between solving and analyzing problems. Programming quality showed mixed results: Module 2 (Tetris) demonstrated better flow control implementation, while Module 3 (rock-paper-scissors) showed superior logical structure implementation, with no weak programs categorized in the final module.
Implications: The findings contribute important insights to the field of AI in education by demonstrating that short-term interventions may be insufficient for developing complex cognitive skills like algorithmic thinking. The study provides valuable evidence about the productive failure approach in computer science education, an area where this methodology has been rarely tested. The research reveals that most learning difficulties occur during the algorithm development phase, suggesting educators should focus additional support and scaffolding during this critical stage. The integration of AI tools like Teachable Machine with traditional programming environments shows promise for engaging students with machine learning concepts while developing computational thinking skills. The study also highlights the importance of considering individual learning differences and the need for adaptive learning pathways that can accommodate varying skill levels and prior knowledge.
Limitations: Several significant limitations constrain the study's generalizability and impact. The sample size was relatively small (N = 23), with an even smaller qualitative analysis group (N = 14), limiting statistical power and broader applicability. The pilot study was conducted exclusively with one class of secondary school students aged 13-14 in a specific educational context, restricting generalizability across different age groups, educational systems, and cultural contexts. The short duration of the intervention (four 90-minute modules) may have been insufficient to observe meaningful changes in algorithmic thinking development. The study lacked a control group and was not fully randomized, preventing causal conclusions about the effectiveness of the productive failure approach. Additionally, the researchers could not control for external factors such as peer collaboration and assistance outside the formal workshop structure. The study did not assess students' baseline algorithmic thinking before Module 0, making it difficult to determine the specific impact of different workshop components.
Future Directions: Future research should implement longer-term interventions with multiple measurement points to better capture the development of algorithmic thinking skills over time. Studies should include larger, more diverse samples across different educational contexts, grade levels, and cultural backgrounds to improve generalizability. Researchers should explore the integration of AI tutoring systems and adaptive learning technologies to provide personalized scaffolding during the problem-solving process. Investigation into the relationship between productive failures and specific misconceptions about algorithmic concepts would provide valuable insights for curriculum design. Future work should also examine the effectiveness of combining computational thinking 1.0 (traditional algorithmic problem-solving) with computational thinking 2.0 (machine learning and data-driven approaches) within the productive failure framework. The development of more sophisticated category systems for analyzing productive failures in computer science education, as well as experimental designs with proper control groups, would strengthen future research in this area.
Title and Authors: "Fostering Algorithmic Thinking within a Productive-Failure-based Workshop utilizing AI" by Frauke Ritter and Nadine Schlomske-Bodenstein.
Published on: The publication date appears to be 2025 based on the conference proceedings reference.
Published by: 2025 IEEE Global Engineering Education Conference (EDUCON).