Higher confidence in GenAI tools is associated with less critical thinking among knowledge workers, while higher self-confidence correlates with more critical thinking despite requiring greater effort.

Title and Authors: "The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers" by Hao-Ping (Hank) Lee, Advait Sarkar, Lev Tankelevitch, Ian Drosos, Sean Rintel, Richard Banks, and Nicholas Wilson.

Published On: April 26-May 01, 2025 Published By: CHI Conference on Human Factors in Computing Systems (CHI '25)

Objective: The study aimed to investigate how knowledge workers perceive the enaction of critical thinking when using GenAI tools and examine when and why they experience increased or decreased effort in critical thinking due to GenAI use.

Methods:

• Conducted an online survey with 319 knowledge workers who use GenAI tools at least once per week
• Collected 936 real-world examples of GenAI tool use in work settings
• Used both quantitative analysis (regression models) and qualitative analysis of free-text responses
• Measured participants' perceived enaction of critical thinking and effort across six cognitive activities based on Bloom's taxonomy
• Analyzed task-specific factors (confidence in self, AI, and evaluation) and user factors (tendency to reflect, trust in GenAI)

Key Findings:

• Workers with higher confidence in GenAI tools showed less critical thinking engagement
• Those with higher self-confidence demonstrated more critical thinking despite perceiving it as requiring more effort
• GenAI tools shift critical thinking effort in three ways:
  1. From information gathering to information verification
  2. From problem-solving to AI response integration
  3. From task execution to task stewardship
• Users perceive decreased effort for cognitive activities when using GenAI tools
• Higher trust in GenAI correlates with reduced perceived effort in critical thinking activities

Implications:

• Highlights the need for balanced design approaches that maintain critical thinking while leveraging AI benefits
• Suggests the importance of developing user domain expertise alongside AI tool usage
• Demonstrates the shift in knowledge work from direct task execution to oversight and verification
• Provides insights for designing GenAI tools that support rather than diminish critical thinking

Limitations:

• Study conducted only in English with English-speaking participants
• Sample skewed toward younger, more technically skilled participants
• Reliance on self-reported data which may not accurately reflect actual critical thinking practices
• Possible conflation between reduced effort in using GenAI and reduced effort in critical thinking
• Subjective nature of self-confidence measures

Future Directions:

• Conduct longitudinal studies tracking changes in AI usage patterns and critical thinking processes
• Explore cross-linguistic and cross-cultural perspectives on GenAI usage
• Investigate the relationship between subjective and objective expertise in critical thinking with GenAI
• Develop alternative measures of critical thinking beyond self-reporting
• Study the long-term effects of GenAI use on cognitive abilities and work practices
• Research ways to enhance awareness, motivation, and ability to think critically when using GenAI tools
• Examine how specific tools can evolve to better support critical thinking in different tasks