Evaluating Educational Innovations through Data

Your programme works. You have data to prove it. Then the hard questions came: 'How do you KNOW it was YOUR intervention?' 'Which parts must stay the same when we replicate this in 12 countries?' 'Why did it work in the first place?' Silence. You're not alone in not having the answers. Most programme (innovative or traditional) can't answer these questions because they collected activity data, not evidence for scale. Here's what you should be measuring at each stage instead: 📍 Early stage (Pilot): Don't just count participants. Measure: Did it work? Was it feasible? Do users actually want this? 📍 Mid-stage (Acceleration): Don't just report more numbers. Measure: What are the core elements that CAN'T change? What CAN flex for different contexts? 📍 Scale stage: Don't just show reach. Measure: Can you prove YOUR intervention caused the change? Can others sustain it without you? UNICEF's Innovation MEL Toolbox breaks down exactly what evidence you need at each stage (from ideation to scale) including practical tools like: →Theory of Change for different stages →Contribution Analysis (when RCTs aren't possible) →Fidelity & Adaptation Monitoring →Scaling Approach frameworks Whether you're testing something new, expanding what works, or adapting proven approaches to new contexts, this document is for you. 🔥 If this resonated, follow me. I break down Monitoring and Evaluation (M&E) concepts daily with practical, implementable tips that are grounded in facilitation experience across sectors. #MonitoringAndEvaluation

Unpacking the impact of digital technologies in Education This report presents a literature review that analyses the impact of digital technologies in compulsory education. While EU policy recognizes the importance of digital technologies in enabling quality and inclusive education, robust evidence on the impact of these technologies is limited especially due to its dependency from the context of use. To address this challenge, the literature review presented here, analyses the focus, methodologies, and results of 92 papers. The report concludes by proposing an assessment framework that emphasizes self-reflection tools, as they are essential for promoting the digital transformation of schools. The literature review on the impact of digital technologies in education revealed several key findings: - Digital technologies influence various aspects of education, including teaching, learning, school operations, and communication. - Factors like digital competencies, teacher characteristics, infrastructure, and socioeconomic background influence the effectiveness of digital technologies. - The impact of digital tools on learning outcomes is context-dependent and influenced by multiple factors. - Existing evidence on the impact of digital tools in education lacks robustness and consistency. The assessment framework proposed in the report offers a structured approach to evaluating the effectiveness of digital technologies in education: 1. Identify contextual factors influencing technology impact. 2. Map stakeholders and their characteristics. 3. Assess integration into learning processes and practices. 4. Utilize self-reflection tools like the Theory of Change. 5. Provide evaluation criteria aligned with the framework. 6. Adapt existing tools for technology assessment. 7. Consider digital competence frameworks for organizational maturity. Implications and recommendations for policymakers and educators based on the report findings include: - Recognizing the contextual nature of technology use. - Focusing on creating rich learning environments. - Adopting a systems approach to studying technology impact. - Ensuring quality implementation and professional development. - Developing policies for monitoring and evaluation. - Encouraging further research on technology impact. By following these recommendations, stakeholders can leverage digital technologies effectively to improve teaching and learning outcomes in educational settings. https://lnkd.in/eBEN5XQg

Program evaluation serves as a cornerstone for improving implementation, measuring outcomes, and enhancing accountability in programs across diverse sectors. This comprehensive Program Evaluation Toolkit, crafted with contributions from the Regional Educational Laboratory at Marzano Research, offers a step-by-step framework designed to support evaluators at every stage of the evaluation process. From planning and logic models to data collection, analysis, and dissemination of findings, this guide equips practitioners with the tools and resources needed to drive evidence-based decisions. Emphasizing both the practical and theoretical aspects of evaluation, the toolkit aligns its methodologies with internationally recognized standards, ensuring rigor and applicability across local, state, and federal programs. Each module is designed to build the capacity of users, guiding them through crafting measurable evaluation questions, identifying quality data sources, selecting robust designs, and interpreting findings in meaningful ways that address key stakeholder needs. Designed for program managers, policymakers, and evaluators, this toolkit transforms evaluation from a compliance exercise into a strategic tool for learning and improvement. By leveraging its structured approach, users can not only assess program effectiveness but also identify pathways for innovation and sustainability, ultimately fostering greater impact in the communities they serve.

Edtech is often criticised for poor quality, misuse of student data and limited learning impact (I’ve voiced those concerns myself several times). But we can’t hold systems accountable without first showing what good or exceptional performance looks like. Once that’s clear, we can create competitive pressure and drive improvement.  ⬇️ Excited to finally share our paper in HSCC Springer Nature that outlines key benchmark criteria for high-quality EdTech. The paper summarises the work our research group has been doing over the past three years. It focuses on educational impact and edtech’s added value for students’ learning. 📚 After an extensive literature review and cross-sector consultations, we’ve developed a multidimensional framework grounded in the “5Es” — efficacy, effectiveness, ethics, equity, and environment.  Efficacy and Effectiveness combine experimental evidence with process-focused metrics and pedagogical implementation studies. Broader metrics focus on ethical data processing, inclusive and equitable approaches and edtech’s environmental impact. 👇 The fifteen tiered impact indicators already guide a comprehensive and flexible evaluation process of international policymakers, educators, EdTech developers and certification bodies (see EduEvidence - The International Certification of Evidence of Impact in Education and our case studies). 🙏 Huge thanks to all who contributed, especially through our participatory Delphi process. Your insights were invaluable! Nicola Pitchford Anna Lindroos Cermakova Olav Schewe Janine Campbell /Rhys Spence Jakub Labun Samuel Kembou, PhD Tal Havivi/ Ayça Atabey Dr. Yenda Prado Sofia Shengjergji, PhD Parker Van Nostrand David Dockterman Stephen Cory Robinson Andra Siibak Petra Vackova Stef Mills Michael H. Levine  #EdTech #ImpactMeasurement #5Es #EdTechQuality #EdTechStandards 👇 Read here or download from:

Context matters. When evaluating program impact, it’s easy to jump to conclusions based on early data or anecdotes. But education—especially when working with students from varied backgrounds—is complex. As I shared with a client recently, student outcomes at a community college are shaped by a multitude of factors: • Personal characteristics and family responsibilities • Socio-psychological pressures • Housing or food insecurity • Work schedules and caregiving roles No single program can solve all of these challenges. That’s why evaluation must be context-sensitive. It’s not just about if a program works—but how, for whom, and under what conditions or circumstances. (Realist evaluation 💡) We’re now exploring a pilot study model to help isolate variables and better understand both the short and long-term impact of student success interventions like embedded tutoring and proactive coaching. Caseload sizes, coach training, and holistic student needs all matter in delivering the right support at the right time. Let’s resist the urge to over-simplify. The students—and the data—deserve better.