Meta-analysis in Education

Ever since ChatGPT arrived, there has been a wave of excitement, skepticism, and curiosity about whether - and how - it actually helps students. Now, a systematic review and meta-analysis by Deng et al. in Computers & Education has pulled together findings from 69 experimental studies, shedding new light on what ChatGPT means for teaching and learning. What Did the Research Reveal? 1️⃣ Stronger Academic Performance Studies show that ChatGPT-assisted interventions often lead to higher grades and better written work - especially in language-rich subjects. One caveat? Many experiments did not make it clear whether students were allowed to use ChatGPT during exams, raising questions about genuine mastery versus AI-assisted output. 2️⃣ Positive Motivation - But Mostly for College Learners University students typically felt more engaged and motivated. In K-12 settings, however, the motivational boost was not as pronounced - suggesting a need for age-appropriate strategies and scaffolds. 3️⃣ Perceived Gains in Higher-Order Thinking Learners reported enhanced creativity and critical thinking. The big “but”: most studies relied on self-reports, so future work needs objective assessments (e.g., problem-solving tasks or performance-based measures) to confirm actual skill growth. 4️⃣ Reduced Mental Effort, Uncertain Self-Efficacy ChatGPT may lighten cognitive load - learners felt tasks were less “taxing.” At the same time, studies showed a mixed or non-significant effect on self-efficacy, implying we need a deeper look at whether students gain real confidence or just convenience. What This Means for Educators & Academics? 1️⃣ Design Rich Assessments: To spot genuine skill gains, use project-based tasks that demand application and originality. 2️⃣ Spell Out Tech Policies: Clearly specify whether and how learners can use ChatGPT - especially for graded work. 3️⃣ Look for Long-Haul Impact: Do not just check excitement levels right after introducing ChatGPT; measure whether those positive vibes (and scores) persist weeks or months down the road. 4️⃣ Mind the Methods: If you are studying ChatGPT’s educational impact, conduct power analyses (to ensure you have enough participants) and randomize group assignments to get the most reliable data. This meta-analysis provides early - but promising - evidence that ChatGPT can enrich students’ learning experiences. The next step? Refining the methods, tracking long-term outcomes, and ensuring actual learning gains are assessed - not just AI’s ability to produce polished outputs. Reference: Deng, R., Jiang, M., Yu, X., Lu, Y., & Liu, S. (2024). Does ChatGPT enhance student learning? A systematic review and meta-analysis of experimental studies. Computers & Education, 105224. https://lnkd.in/eXe8agAT

UPDATE May 2026: This paper was retracted on April 22, 2026 due to discrepancies in the meta-analysis. See my new post for context. Important new evidence on ChatGPT in education: Wang & Fan's (2025) meta-analysis of 51 studies shows we're at an inflection point. The technology demonstrably improves learning outcomes, but success depends entirely on implementation. The research reveals optimal conditions: sustained use (4-8 weeks), problem-based contexts, and structured support for critical thinking development. Effect sizes tell the story; large gains for learning performance (g=0.867), moderate for critical thinking (g=0.457). Quick fixes don't work. Thoughtful integration does. Particularly compelling: ChatGPT excels in skills development courses and STEM subjects when used as an intelligent tutor over time. The key? Providing scaffolds like Bloom's taxonomy for higher-order thinking tasks. As educators, we have emerging empirical guidance for AI adoption. Not whether to use these tools, but how to use them effectively - maintaining rigor while enhancing accessibility and engagement. The future of education isn't human or AI. It's human with AI, thoughtfully applied.

𝗗𝗼 𝘆𝗼𝘂 𝗵𝗮𝘃𝗲 𝗲𝗻𝗼𝘂𝗴𝗵 𝗼𝗳 𝗮 𝗴𝗿𝗼𝘄𝘁𝗵 𝗺𝗶𝗻𝗱𝘀𝗲𝘁 𝘁𝗼 𝗮𝗰𝗰𝗲𝗽𝘁 𝘁𝗵𝗮𝘁 𝗺𝗮𝘆𝗯𝗲 𝗶𝘁 𝗶𝘀𝗻'𝘁 𝗮 𝗿𝗲𝗮𝗹 𝘁𝗵𝗶𝗻𝗴? The text below is from a Twitter thread by Brooke Macnamara reporting the results of her #systematicreview and #metaanalysis of #growthmindset interventions on students’ academic achievement. It's long but the nuance is important. First, some findings from the systematic review: • studies authored by researchers with financial incentives to report positive effects were > 2.5x as likely to report positive effects • > 90% of studies had confounds in their study design • some studies found null results but interpreted them as significant anyway (inc. highly-cited studies) • some studies didn’t adjust for clustering, leading them to erroneously report significant effects (inc. a highly-cited study) • 97% of samples were not preregistered. In fact, there were more studies that described themselves as preregistered that were not preregistered than there were actual preregistered studies. • many studies never tested whether students’ mindsets were affected by the intervention • of the studies that tested whether students’ mindsets were affected by the intervention, many found no evidence of a mindset change Meta-analysis 1: Across all studies, we found a small effect. No theoretically-meaningful moderators were significant. We tested for publication bias using multiple approaches (Egger’s, Duval & Tweedie’s, PET-PEESE). All suggested publication bias. When correcting for publication bias, the overall effect was non-significant. Meta-analysis 2: We next tested if any effects from growth mindset interventions were from the assumed cause—change in students’ mindsets from the intervention. Here, we included all studies that demonstrated the intervention changed treatment students’ mindsets. < 25% of studies demonstrated the intervention changed treatment students’ mindsets. For these studies, the overall effect was non-significant. Meta-analysis 3: We focused on the highest quality evidence. We aimed to only include interventions that changed students’ mindsets & met 100% best practices—e.g., no confounds, full blinding, active control group, no authors with financial COIs. No study met these criteria. We had to considerably lower the threshold for what was considered the highest-quality studies in the growth mindset intervention literature. Among the highest-quality studies available, the effect on academic achievement was not significant. We then conducted over 200 meta-analytic models examining adherence to every combination of best practice criteria. As the number of best practices adhered to increased, the number of significant models decreased. Preprint here: https://lnkd.in/egQctJPt

🚨 New publication in Motivation Science! 🚨 Why do some teachers, parents, coaches, and leaders adopt highly controlling approaches, while others are far more supportive of people’s motivation? I’m excited to share our new systematic review and meta-analysis—the first of its kind—examining the antecedents of interpersonal communication styles within a Self-Determination Theory (SDT) framework. 🔍 While previous SDT reviews have focused on the consequences of interpersonal styles (need-supportive vs. need-thwarting), none have systematically reviewed what drives these styles in the first place. 📊 Our synthesis draws on 122 studies across education, parenting, sport, healthcare, and more. We: Identified 59 candidate antecedents of interpersonal styles. Grouped them into 13 general factors and 3 higher-order themes: 1️⃣ Socio-contextual factors 2️⃣ Motivators’ personal factors 3️⃣ Motivators’ perceptions of motivatees’ motivation & behaviour Integrated these findings into a model extending the “classic SDT sequence.” 💡 This classification system can help: Pinpoint moderators of intervention effectiveness Identify new targets for interventions to foster more need-supportive styles 📄 Read the full paper here: https://lnkd.in/dhAeCjBf #SelfDeterminationTheory #Motivation #MetaAnalysis #SystematicReview #Psychology #Education #Coaching #Parenting #Leadership Thank you to the action editor Marylène Gagné and two anonymous reviews for a constructive round of reviews. Andreas Ivarsson Dennis Bengtsson Chris Lonsdale Jennie Hancox Eleanor Quested

"Meta-analyses" are supposed to be a "gold standard" evidence synthesis method, and many have been produced about the effects of AI on learning and education. Most of these reviews tend to report "statistically significant" effects on outcomes. A new pre-print "meta-meta-review", however, appears to throw a bucket of cold methodological water on the fast-growing effort to measure the impact of AI on learning: "In summary, our results highlight severe issues in the literature on the effect of AI/LLMs on learning. The results are plagued by publication bias and display extreme between-study heterogeneity. While we believe it is a priori plausible that AI/LLMs may have a positive impact on learning, the current empirical evidence base is insufficiently diagnostic and does not warrant concrete recommendations for educational practice or policy." The paper is a hard read unless you love stats, but that's a blunt summary of the authors' conclusion. The evidence just isn't there as the basis for making policies or designing practices to boost learning outcomes. But it raises, I think, some other big Qs about the production of evidence about AI in education too. Like, can you really measure if a technology affects learning, while stripping it out of its contexts of use and isolating it as a causal determinant on your metric of choice? What should professional associations involved in AIED research do/say about the seemingly substandard science in their field? Are "learning outcomes" actually sufficient proxies for learning itself if what's being measured may be what students have outsourced to a language model? What are the incentives for researchers to publish "positive" results, and to incorporate these in meta-analyses, while downplaying other negative or inconclusive findings? At the very least, the pre-print reminds us that the evidence on AI effects in education is very much not a settled matter of objective scientific fact, but increasingly a site of scientific controversy. It's an intriguing contest - a politics of method and evidence production you could say - for social scientific attention. https://lnkd.in/e6HF6kQ6

How to Do a Meta-Analysis (For Dummies — Like I Once Was) Meta-analysis sounds like rocket science, but it’s really just a fancy way to say: "I’m going to collect a bunch of studies on one topic, and combine their results to get a clearer answer." Here’s how ANYONE (yes, even you!) can start doing one — step-by-step: Step 1: Pick ONE clear research question Example: “Does intermittent fasting reduce breast cancer recurrence?” Keep it specific. Not “fasting and cancer” — that’s a black hole. Step 2: Set your inclusion criteria (aka rules) Decide ahead of time: What kind of studies will you include? Which years? Which populations? Which outcomes? Step 3: Search the literature Use PubMed, Google Scholar, Embase, Cochrane Library. Learn a few keywords + Boolean tricks (like AND, OR, NOT). Start saving the articles in a folder or reference manager (Zotero, Mendeley). Step 4: Screen the studies (PRISMA flow) Yes, you have to actually read them! Remove duplicates Skim titles and abstracts Read full texts Record which ones you included and excluded (and why) Step 5: Extract the data Make a spreadsheet. For each study, pull: Author, Year Sample size Population Intervention Control group Outcome results This is your data table — it’s gold. Step 6: Assess quality and bias You want solid, reliable studies. Use tools like: Cochrane Risk of Bias Tool Newcastle-Ottawa Scale This helps weed out weak or sketchy studies. Step 7: Do the actual analysis If you're new, use RevMan (free and beginner-friendly) or R with packages like meta or metafor. You’ll calculate things like: Risk Ratios Odds Ratios Confidence Intervals And make a forest plot — the iconic visual summary. Step 8: Interpret results Look for trends: Is the effect consistent? Are there outliers? Is it statistically significant? Also consider heterogeneity (I²) — are the studies saying wildly different things? Step 9: Write the paper (Use PRISMA Guidelines!) Follow the format: Abstract Introduction Methods (your search strategy, criteria, tools) Results (with figures and tables) Discussion (your interpretation) Conclusion Add references (AMA or APA style). Step 10: Submit it or present it Aim for a journal, or make a research poster or conference presentation. You just did a meta-analysis. Yes, YOU! #MetaAnalysis #ResearchMadeEasy #MedicalResearch #ForBeginners #SystematicReview #PRISMA #LinkedInAcademia #PhysicianScientist

There is a moment in every researcher’s journey when the evidence feels too scattered to hold. Too many papers. Too many methods. Too many conclusions pointing in different directions. I have lived that moment myself, standing in front of a question that refused to stay simple. And in those moments, meta-analysis becomes more than a technique. It becomes a way to bring coherence back into the chaos, a way to let many voices converge into one truth that can guide real decisions. That is why this guide matters. Because when you understand how to synthesize evidence with clarity and rigor, you step into a deeper form of authorship, one where your work does not just inform, it illuminates. Every piece of research holds a fragment of the truth. But no single study can tell the whole story. 1. Define your question clearly. A precise research question is the anchor of every meta-analysis. Use the PICO framework (Population, Intervention, Comparison, Outcomes) to ensure focus and reproducibility. 2. Establish inclusion and exclusion criteria. Be transparent about which studies qualify and why. Criteria should align with your research goals and be documented before data collection begins. 3. Develop a comprehensive search strategy. Search multiple databases systematically, use Boolean operators, and document every step. A strong meta-analysis depends on an exhaustive and reproducible search. 4. Select studies rigorously. Screen titles and abstracts carefully, then review full texts. Use multiple reviewers to minimize bias and maintain consistency. 5. Extract data with precision. Develop a structured extraction sheet. Record key variables such as study design, participants, effect sizes, and outcomes. Accuracy here determines validity later. 6. Assess study quality. Use standardized tools to evaluate bias, methodology, and overall reliability. Poor-quality studies can distort the pooled results. 7. Analyze data appropriately. Use statistical models (fixed or random effects) according to study heterogeneity. Always present confidence intervals and sensitivity analyses. 8. Interpret findings responsibly. Discuss strengths and limitations honestly. Place results within the broader context and highlight implications for future research or practice. Download the full guide and explore the eight steps and let it support you as you build research Which of these eight steps feels the most challenging for you? Leave a comment and let me know! ______________________________ 📌 This is Prof. Samira Hosseini. I’ve helped 12,000+ ambitious academics go from struggling with publishing papers in Q1 journals, limited visibility, and poor citation records to building a solid research trajectory and high 𝘩-index. Book a free Strategy Call, and we can dive into your challenges in top-tier journal publication and citation and see how I can best assist you: https://lnkd.in/ezqV64dX

Finally today, I would like to share a relatively recent meta-analysis entitled, "The effect of ChatGPT on students’ learning performance, learning perception, and higher-order thinking: Insights from a meta-analysis" by Wang and Fan (2025) (https://lnkd.in/eeRcFMva ). Synthesizing 51 experimental and quasi-experimental studies (201 studies screened from 2022–2025), the review provides statistically robust answers to a question that has fueled global debate: Does ChatGPT actually improve learning? This study found that: (1) appropriate learning scaffolds or educational frameworks (e.g., Bloom’s taxonomy) should be provided when using ChatGPT to develop students’ higher-order thinking; (2) the broad use of ChatGPT at various grade levels and in different types of courses should be encouraged to support diverse learning needs; (3) ChatGPT should be actively integrated into different learning modes to enhance student learning, especially in problem-based learning; (4) continuous use of ChatGPT should be ensured to support student learning, with a recommended duration of 4–8 weeks for more stable effects; (5) ChatGPT should be flexibly integrated into teaching as an intelligent tutor, learning partner, and educational tool. Across 72 independent effect sizes, ChatGPT demonstrates a large positive impact on learning performance (g = 0.867) and moderate positive effects on both learning perception (g = 0.456) and higher-order thinking (g = 0.457). These effects were exceptionally stable, supported by strong sensitivity analyses and low risk of publication bias—reinforced by symmetrical funnel plots shown on pp. 12–13 and high fail-safe N values for all outcomes. The study’s moderator analyses reveal when ChatGPT is most effective. It produced the strongest gains in skills and competencies courses, problem-based learning, and interventions lasting 4–8 weeks. For higher-order thinking, impacts were highest when ChatGPT served as an intelligent tutor, especially in STEM contexts. Notably, ChatGPT’s influence did not vary significantly by grade level, suggesting that when appropriately scaffolded it can support learners from secondary to postsecondary education. The authors conclude with clear guidance for educators and institutions: integrate ChatGPT within coherent learning frameworks (Bloom’s taxonomy), expand its use across disciplines, design learning activities that position the tool as a cognitive partner, and sustain use over several weeks to stabilize learning gains. Reference Wang, J., & Fan, W. (2025). The effect of ChatGPT on students’ learning performance, learning perception, and higher-order thinking: Insights from a meta-analysis. Humanities and Social Sciences Communications, 12, 621.

**Research Alert** A new meta-analysis from researchers in Beijing further demonstrates that the direct instruction of social emotional skills through curriculum has the greatest effect on students' development of social emotional competence. "Intervention approach was another determining feature that moderated the effectiveness of SEL programs. Results showed that comprehensive and curriculum-based SEL programs had relatively large effect sizes compared to supplemental ones. In particular, the effect size of supplemental SEL programs was not significantly different from zero, indicating a negligible effect. On the contrary, the effect sizes of comprehensive interventions (ES = 0.21) and curriculum-based interventions (ES = 0.20) were significant."

Meta-analysis shows practically significant effects for many of Richard Mayer's multimedia learning principles. Effect of these principles were most consistent across text and diagram materials, less so for animation, games, and simulations. Effects highest for transfer and inference outcomes.