Undergraduate students show a discrepancy between their AI application competence and understanding of AI principles, with confidence in AI tools being influenced more by perceived tool limitations than personal ability.

Objective: To understand how differences in AI confidence and knowledge influence students' relationships with AI by examining undergraduate perspectives to redefine AI literacy and inform curriculum development.

Methods:

• Explanatory sequential mixed methods design
• Initial AI literacy survey adapted from Ng et al. with 84 respondents
• Focus group interviews with 8 participants
• Survey analyzed using R and ggplot2
• Data collection through Spring-Summer 2024
• Analysis of both quantitative and qualitative responses

Key Findings:

• Students showed higher competence in using AI applications than understanding underlying AI principles
• Significant gender differences in AI confidence levels between males and females
• Student confidence in AI's capability as a learning tool was limited by beliefs about AI tool efficacy rather than personal ability
• Students believed AI literacy would benefit students across all fields
• Participants viewed AI's integration into the workforce positively, though saw it as inevitable
• Lower correlation between AI confidence and knowledge than expected

Implications:

• Need to differentiate between AI application knowledge and understanding of AI principles
• Importance of combining surveys with objective assessments
• Value of considering both confidence in using AI and confidence in AI's capabilities
• Need for targeted support to address gender-related disparities
• Potential for AI literacy to bridge workplace equity gaps

Limitations:

• Small focus group sample size (n=8)
• Survey may not distinguish between tool usage and understanding
• Potential overestimation of AI knowledge due to tool familiarity
• Limited geographic diversity
• Gender imbalance in survey respondents

Future Directions:

• Development of more nuanced assessment tools
• Research on gender-related disparities in AI confidence
• Investigation of actual versus perceived AI knowledge
• Studies on effective AI literacy curriculum development
• Exploration of AI's role in workplace equity

Title and Authors: "Leveraging Undergraduate Perspectives to Redefine AI Literacy" by Jack Ebert and Kristina Kramarczuk

Published On: To be presented February 26-March 1, 2025

Published By: Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1 (SIGCSE TS 2025)