A comprehensive scoping review of AI literacy assessments reveals that most studies focus on K-16 students using questionnaires and computer-based tests to primarily assess AI knowledge, with significant gaps in reliability/validity reporting and limited assessment of practical AI usage skills.
Objective: This scoping review aimed to systematically evaluate and synthesize existing empirical research on AI literacy assessment by examining assessment tools, forms of assessment, and reliability and validity evidence across different application domains, while developing an AI literacy framework suitable for people of all ages and countries.
Methods: The researchers conducted a scoping review following established protocols, searching Web of Science and Scopus databases for peer-reviewed journal articles published between 2019-2024 that included "AI literacy" or "Artificial Intelligence literacy" in their titles. After applying inclusion/exclusion criteria and removing duplicates, 36 studies were selected for analysis. The researchers employed content analysis procedures with coding schemes based on four research questions, achieving 90% inter-rater reliability. Data extraction focused on study characteristics, assessment constructs, tools used, and psychometric properties. The analysis involved both quantitative summaries of study features and qualitative thematic analysis of assessment approaches.
Key Findings: The review revealed several important patterns in AI literacy assessment research. Most studies (75%) focused on K-16 students, with 77% specifically targeting primary and secondary school students, while only 25% addressed early childhood, teacher education, or adult populations. Regarding assessment constructs, four main areas emerged: "knowing and understanding AI" (31 studies), "AI ethics" (12 studies), "affect towards AI" (8 studies), and "use of AI" (4 studies). The most commonly used assessment tools were questionnaires (12 studies), followed by surveys (8 studies) and tests (5 studies). Computer-based assessments were most prevalent (47%), followed by paper-based (25%) and various hybrid approaches. Geographically, most research originated from Hong Kong, mainland China, and the United States. A critical finding was that only 63.8% of studies reported reliability evidence, and fewer than 20% provided validity evidence, indicating significant gaps in psychometric rigor.
Implications: The findings highlight several important implications for the field of AI literacy assessment. The predominant focus on K-16 education suggests a need for more research on early childhood and adult AI literacy assessment. The heavy emphasis on assessing AI knowledge rather than practical application skills indicates that current assessments may not adequately prepare learners for real-world AI interaction. The lack of robust psychometric evidence in many studies raises concerns about the quality and reliability of existing assessment tools, potentially limiting their effectiveness in educational settings. The geographic concentration of research suggests a need for more diverse, culturally responsive assessment approaches. The review also reveals an urgent need for more comprehensive teacher preparation in AI literacy, as educators cannot effectively teach what they don't understand themselves.
Limitations: Several limitations affect the scope and generalizability of this review. The study only included journal articles, excluding conference papers that might contain relevant assessment tools or approaches. The geographic concentration of studies primarily in Hong Kong, mainland China, and the United States limits the cultural diversity and international applicability of findings. The focus exclusively on journal publications may have excluded innovative assessment approaches presented at conferences or in other venues. Additionally, the rapid evolution of AI technology means that some assessments may quickly become outdated, and the review's timeframe may not capture the most recent developments in generative AI and its implications for literacy assessment.
Future Directions: The researchers recommend several key areas for future investigation. First, there's a critical need to develop and validate AI literacy assessments for kindergarten children, requiring age-appropriate methods that account for limited reading and comprehension abilities. Second, more research should focus on teacher education and professional development, including the development of comprehensive assessment tools to measure educators' AI competency. Third, future assessments should move beyond knowledge evaluation to include practical application skills, potentially through real-world AI projects and hands-on activities. Fourth, researchers should prioritize reporting robust reliability and validity evidence to ensure assessment quality and enable broader adoption. Fifth, there's a need for more diverse assessment approaches, including interviews, observations, and performance-based evaluations to provide comprehensive understanding of AI literacy. Finally, the field would benefit from more culturally diverse research that considers different educational contexts and cultural perspectives on AI literacy.
Title and Authors: "A scoping review of empirical research on AI literacy assessments" by Jiahong Su (University of Hong Kong), Xinyu Chen (University of Hong Kong), Samuel Kai Wah Chu (Hong Kong Metropolitan University), and Xiao Hu (University of Hong Kong).
Published On: Accepted May 14, 2025 (Published online August 25, 2025)
Published By: Education Tech Research Dev, published by Springer and the Association for Educational Communications and Technology