Students with high AI literacy demonstrate more sophisticated and collaborative interaction patterns with AI in academic writing tasks, leading to better writing performance compared to students with low AI literacy.
Title and Authors: "Students' prompt patterns and its effects in AI-assisted academic writing: Focusing on students' level of AI literacy" by Jinhee Kim, Seongryeong Yu, Sang-Soog Lee & Rita Detrick
Published On: January 24, 2025 Published By: Journal of Research on Technology in Education
Objective: To analyze how students with different levels of AI literacy design prompts when interacting with generative AI for academic writing tasks, and to examine how these different prompt patterns affect writing performance.
Methods:
- Studied 19 international university students enrolled in preparatory courses
- Used pre-survey to categorize students into high and low AI literacy groups
- Analyzed student-AI chat histories using content analysis and categorical data analysis
- Visualized interaction patterns using Gephi 0.10.1
- Evaluated writing performance through expert assessment
- Conducted follow-up interviews for qualitative insights
Key Findings:
- High AI literacy students:
- Used more descriptive, context-based prompts
- Engaged in collaborative interactions with AI
- Demonstrated better writing performance across content, structure, and expression
- Showed more sophisticated cognitive engagement
- Low AI literacy students:
- Used more general, basic prompts
- Showed student-directed, transactional approach
- Relied more heavily on AI-generated content
- Demonstrated limited depth in AI interactions
Implications:
- Educational GenAI systems should be designed to support diverse cognitive engagements
- Teachers should guide students toward interactive rather than passive AI engagement
- AI literacy development should integrate both technical and cognitive understanding
- Need for structured learning objectives that foster higher-order thinking in AI-assisted writing
Limitations:
- Small, non-representative sample size
- Binary classification of AI literacy levels
- Focus on specific writing task type
- Analysis limited to one side of student-AI interaction
- Study conducted in English only
Future Directions:
- Investigate additional student characteristics beyond AI literacy
- Develop more nuanced AI literacy classification
- Study diverse writing tasks and topics
- Analyze both sides of student-AI dialogue
- Conduct larger-scale studies with more diverse populations
- Examine long-term impacts of AI use on writing skills
- Consider cross-cultural and multilingual perspectives