The integration of ChatGPT in a computer science course shows predominantly positive effects on student learning and creativity despite reliability concerns, with most students using it for multiple educational purposes.

Objective: The main goal of this study was to explore students' perceptions of ChatGPT during a computer science course, particularly examining their awareness levels, usage patterns, anticipated effects on learning and creativity, and perceptions regarding reliability and security.

Methods: The researchers integrated ChatGPT into Microsoft Teams private discussion groups for a basic Web Development course at Tampere University during Spring 2023. ChatGPT was implemented through Power Automate Cloud Flow and was activated when students used the keyword "Kiran" in their messages. Students were divided into experimental groups (with access to ChatGPT through Teams) and control groups (without direct access through Teams, though they could still use ChatGPT independently). The researchers collected data through two voluntary electronic surveys: a pre-survey at the beginning of the course (N=190) and a post-survey at the end (N=111), with 109 students completing both surveys. The surveys included both Likert-scale questions and open-ended questions. The data was analyzed using both quantitative methods (for Likert-scale responses) and qualitative content analysis (for open-ended responses).

Key Findings:

• Nearly all students (except two) had heard about ChatGPT before the course began, learning about it through news, social media, and friends.
• Approximately 58% of students in the experimental group and 38% in the control group reported using ChatGPT on a weekly basis by the end of the course.
• Students primarily used ChatGPT for multiple purposes: debugging code, tutoring, enhancing comprehension, writing explanations, finding examples, and summarizing texts.
• The amount of positive implications of using ChatGPT outweighed the negative implications when considered from understanding, learning, and creativity perspectives.
• Students' confidence in ChatGPT's ability to enhance comprehension decreased from pre-survey (63.2%) to post-survey (39.6%), but appreciation for its productivity features increased (from 35.8% to 47.7%).
• Approximately 70% of students encountered challenges while using ChatGPT, primarily related to the chatbot not understanding questions or providing incorrect information, which required reformulating queries.
• Students were skeptical about ChatGPT's ability to protect personal information, with low trust levels reflected in both pre and post-surveys.
• Positive comments related to creativity (idea generation, productivity, understanding, and problem-solving) outweighed negative comments, with over 80% of mentions conveying positivity.

Implications: The findings suggest that integrating ChatGPT into educational settings can be beneficial when implemented thoughtfully. The study indicates that:

• ChatGPT can serve as a valuable supplementary tool for enhancing learning experiences, particularly in computer science education.
• Students appreciate ChatGPT's ability to provide alternative explanations, quick responses, helpful summaries, and examples.
• The technology can assist with productivity-related tasks like code generation, debugging, and summarizing content.
• Educators should focus on training students in effective prompting techniques, critical evaluation of outputs, and ethical usage to maximize benefits.
• The integration of AI tools should complement learning processes rather than replace them, ensuring students still engage in critical thinking and problem-solving.

Limitations: The study had several limitations:

• The absence of quantitative usage data made it difficult to understand the frequency and depth of student interactions with ChatGPT.
• The control group still had access to ChatGPT and other language models outside the platform, making the experimental setting "not pure."
• Some survey questions may have introduced bias by presupposing positive contributions of ChatGPT to the learning experience.
• The study was conducted in a specific educational context (a Web Development course), which might limit the generalizability of findings.
• The self-reported nature of surveys may have introduced biases in the data.

Future Directions: The researchers suggest several areas for future research:

• Deeper exploration of the long-term effects of integrating language models into educational settings over extended periods.
• Investigation of student interactions with ChatGPT and similar tools in diverse learning contexts beyond computer science courses.
• Development of training methods for prompting, efficient usage, and critical and ethical use of ChatGPT.
• Exploration of solutions to address reliability issues, such as implementing large language models locally to address privacy concerns.
• Further research on how to best integrate these tools to complement students' learning processes without replacing essential aspects of learning.

Title and Authors: "Call me Kiran" – ChatGPT as a Tutoring Chatbot in a Computer Science Course" by Jaakko Rajala, Jenni Hukkanen, Maria Hartikainen, and Pia Niemelä from Tampere University, Finland.

Published On: October 03–06, 2023

Published By: 26th International Academic Mindtrek Conference (Mindtrek '23) proceedings, published by ACM (Association for Computing Machinery), New York, NY, USA.