Objective: This study aimed to provide the first comprehensive meta-review exploring the scope and nature of Artificial Intelligence in Education (AIEd) research specifically in higher education settings, by synthesizing secondary research publications to map the field, understand its current state, and suggest directions for future development.

Methods: The researchers conducted a tertiary review (review of reviews) of 66 AIHEd evidence syntheses published between 2018 and July 2023. They systematically searched multiple databases including Web of Science, Scopus, ERIC, EBSCOHost, IEEE Xplore, ScienceDirect, and ACM Digital Library, and supplemented these with citation searches through OpenAlex, ResearchGate, and Google Scholar. Reviews were included if they synthesized AI applications solely in formal higher or continuing education, were published in English, appeared in journal articles or full conference papers, and contained a method section. The researchers extracted data on publication characteristics, authorship patterns, review types, AI applications, quality assessment, key findings, and research gaps, then analyzed this data using narrative synthesis and produced interactive evidence gap maps.

Key Findings:

• The review identified that systematic reviews (66.7%) were the most common evidence synthesis type, followed by scoping reviews (12.1%).
• Adaptive systems and personalization (54.5%) were the most discussed AI applications in higher education, followed by profiling and prediction systems (48.5%).
• North America (27.3%), Europe (24.2%), and Asia (22.7%) had relatively balanced representation in authorship, though South and Central America (4.5%) were underrepresented.
• Research collaboration was common (89.4% of publications), but primarily occurred within domestic settings (71.2%) rather than internationally.
• Quality assessment revealed that 65% of reviews were of critically low to medium quality, with concerning findings including 31.8% of studies searching only one or two databases, 51.5% not reporting inter-rater reliability, and 45.5% not undertaking quality assessment.
• The primary benefits of AI in higher education included personalized learning (38.7%), improved student understanding, positive learning outcomes, and reduced administrative time for educators (32.3% each).
• The main challenges were lack of ethical consideration (29%), curriculum development issues, infrastructure problems, insufficient teacher technical knowledge, and shifting authority in educational settings (22.6% each).

Implications: This meta-review provides a significant foundation for the future development of AIHEd research by mapping the field comprehensively for the first time. The findings highlight that for AI to achieve its potential in higher education, researchers must strengthen three key areas: ethics, collaboration, and methodological rigor. The review also serves to prevent research waste by identifying existing knowledge and gaps, enabling more targeted and valuable future research efforts.

Limitations: The review acknowledged several limitations, including that the protocol was not pre-registered in an official systematic review repository, only the first 500 records in Google Scholar were considered (versus the recommended 1000), and the review was limited to English-language publications. Additionally, the authors noted that the quality assessment tool developed was imperfect, as the distance between quality categories (yes, no, partly) may not be equal.

Future Directions: The authors identified three critical areas for future development:

1. Increased ethics: Future AIHEd research must pay greater attention to ethical considerations, including participant consent, data collection and storage procedures, biases perpetuated through data, and embedding ethical AI as a topic throughout higher education curricula.
2. Increased collaboration: The field would benefit from greater collaboration in developing AI applications, designing AI curricula, conducting interdisciplinary research, and forming international research partnerships.
3. Increased rigor: The authors call for enhanced methodological robustness in AIHEd research, including more comprehensive database searching, better reporting of data extraction methods, increased use of quality assessment, and greater transparency through modern evidence synthesis tools.

Title and Authors: "A meta systematic review of artificial intelligence in higher education: a call for increased ethics, collaboration, and rigour" by Melissa Bond, Hassan Khosravi, Maarten De Laat, Nina Bergdahl, Violeta Negrea, Emily Oxley, Phuong Pham, Sin Wang Chong, and George Siemens.

Published On: 2024 (Volume 21, Issue 4)

Published By: International Journal of Educational Technology in Higher Education