VMEvalKit 🎥🧠

A framework to score reasoning capabilities in video generation models at scale, through cognitive tasks. We make it very convenient to add models, add tasks, run inferences, run scoring, manage datasets and display results. It's permissively open-source, and we welcome everyone to join us and build in public together! 🚀

🏆 Leaderboard (Under Construction) • 🔍 Preliminary results

VMEvalKit provides unified access to 40 video generation models across 11 provider families:

VMEvalKit provides access to 3 external benchmark datasets and 6 local task generation engines:

VideoThinkBench Subsets:

• arc_agi_2 - Abstract reasoning (1,000 tasks)
• eyeballing_puzzles - Visual estimation (1,050 tasks)
• visual_puzzles - Pattern recognition (496 tasks)
• text_centric_tasks - Math & multimodal reasoning (1,453 tasks)

All tasks follow the unified First Frame → Final Frame format with text prompts, enabling consistent evaluation across diverse reasoning domains.

VMEvalKit aims to provide an infrastructure for reasoning research in video models at scale:

• 🎯 Task Creation at Scale: Create question dataset of many different cognitive tasks programmatically at scale and our framework makes sure the dataset to be well-organized.
• 🚀 Model Inference at Scale: Easy one-click inference of the entire question dataset across many video models (commercial APIs + open-source) with automatic resume, error handling, and structured output management, and automatically sync the inference results into the dataset.
• ⚖️ Scoring Pipeline: Human scoring via web interface and AI scoring via automated MLLM scoring, also automatically sync the scoring results into the dataset.
• ☁️ Dataset Management: Manage question datasets from task creation, inference results from video models, and scoring results from humans or MLLM pipelines. Provides AWS S3 integration with version tracking and built-in logging for reproducibility.

We have completed running a question dataset of chess, maze, Sudoku, mental rotation, and Raven's Matrices on latest video models. Checkout our raw results videos on this website. Here are a few examples.

1. Clone the repository

git clone https://github.com/hokindeng/VMEvalKit.git
cd VMEvalKit

2. Initialize submodules - good for optional open-source models and datasets

git submodule update --init --recursive

3. Configure environment - Copy the example environment file and add your API keys

cp env.template .env

4. Set up Python environment – Recommended: use a fresh virtual environment

python -m venv venv
source venv/bin/activate

Alternatively, you can use other tools like uv for faster install (uv venv), or conda if your usecase has cross-language dependencies.

5. Install dependencies:

pip install -r requirements.txt
pip install -e .

For open-source video generation and evaluator models, please refer to Open Source Models for detailed installation instructions.

Here's a complete workflow from creating questions to scoring results:

# Generate 5 chess and maze questions each
python examples/create_questions.py --task chess maze --pairs-per-domain 5

# Output: Creates data/questions/ with chess_task/ and maze_task/ folders

# List available models
python examples/generate_videos.py --list-models

# set Luma key in .env file 
# Run on specific model (e.g., Luma Ray 2)
python examples/generate_videos.py --model luma-ray-2 --task chess maze

# Output: Creates data/outputs/pilot_experiment/ with generated videos

# Option A: Human scoring via web interface
python examples/score_videos.py human

# Option B: Automated GPT-4O scoring
python examples/score_videos.py gpt4o

# Launch web dashboard to explore results
cd web && ./start.sh
# Open http://localhost:5000 in your browser

# AWS S3 (enterprise backup)
python data/s3_sync.py --log

That's it! You now have:

• ✅ Custom reasoning questions in data/questions/
• ✅ Generated videos in data/outputs/
• ✅ Scoring results in data/scorings/
• ✅ Interactive dashboard to explore everything

Tips:

• Start small: --pairs-per-domain 2 for quick testing
• Use --task-id chess_0001 to run specific questions
• Try different models: --model openai-sora-2 veo-3.0-generate

Solving Chess

Solving Maze

Mental Rotation

Raven's Matrices

Sudoku Solving

Every VMEvalKit dataset consists of Task Pairs - the basic unit for video reasoning scoring:

Each Task Pair consists of three core components:

• 📸 Initial state image (first_frame.png): shows the starting point or problem to be solved
• 🎯 Final state image (final_frame.png): illustrates the goal state or solution
• 📝 Text prompt (prompt.txt): provides natural language instructions for the video model

There is also an accompanying question_metadata.json file with rich metadata. Each task pair is organized in its own folder (data/questions/{domain}_task/{question_id}/) containing all four files.

from vmevalkit.runner.inference import InferenceRunner

# Initialize runner - creates structured output directories
runner = InferenceRunner(output_dir="data/outputs")

# Generate video showing reasoning process
result = runner.run(
    model_name="luma-ray-2",
    image_path="data/questions/maze_task/maze_0000/first_frame.png",
    text_prompt="Navigate the green dot through the maze corridors to reach the red flag",
    question_data={"id": "maze_0000", "domain": "maze"}  # Optional metadata
)

# Each inference creates a self-contained output folder:
print(f"📁 Output folder: {result['inference_dir']}")
# Contains:
# ├── video/generated_video.mp4    # The generated video
# ├── question/                    # Input data archive
# │   ├── first_frame.png         # Input image
# │   ├── prompt.txt              # Text prompt
# │   └── question_metadata.json  # Task metadata
# └── metadata.json               # Complete inference record

┌─────────────────────────────────────────────────────────────────────────────┐
│                               InferenceRunner                               │
│        Top-level orchestrator: manages workflow, batching, and output       │
└───────────────────────┬─────────────────────────────────────────────────────┘
                        │      Dynamic Model Loading (importlib)              
                        ▼                                                    
┌─────────────────────────────────────────────────────────────────────────────┐
│                              MODEL_CATALOG                                  │
│  Unified model registry:                                                    │
│    - Lists all available models (both API and open-source)                  │
│    - Records provider family, wrapper paths, model meta-info                │
│    - No imports of implementations (pure config)                            │
└───────────────────────┬─────────────────────────────────────────────────────┘
                        │      importlib.import_module() dynamically loads   
                        ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                       Model Implementations (Two Flavors)                   │
│ ┌────────────────────────────┬────────────────────────────────────────────┐ │
│ │          Commercial Models             │      Open-Source Models        │ │
│ │       (Closed Source Services)         │    (Local Implementations)     │ │
│ ├────────────────────────────┼────────────────────────────────────────────┤ │
│ │ LumaWrapper  +  LumaService           │ LTXVideoWrapper  +  LTXService  │ │
│ │ VeoWrapper   +  VeoService            │ HunyuanWrapper   +  HunyuanSvc  │ │
│ │ RunwayWrapper+  RunwayService         │ VideoCrafterWrapper+VCService   │ │
│ │ ...                                   │ DynamiCrafterWrapper+DynService │ │
│ └────────────────────────────┴────────────────────────────────────────────┘ │
│   - Each Wrapper implements unified VMEvalKit interface                     │
│   - API Services handle endpoints, retries, S3-upload (when needed)         │
│   - Open-source backends directly invoke local model code                   │
└─────────────────────────────────────────────────────────────────────────────┘

See Inference Guide for details.

# Create questions for specific domains
python examples/create_questions.py --task chess maze --pairs-per-domain 10

# Generate videos with selected models  
python examples/generate_videos.py --model luma-ray-2 veo-3.0-generate --task chess maze

# Score results
python examples/score_videos.py human
python examples/score_videos.py gpt4o

# Using runner modules directly
python -m vmevalkit.runner.score human \
  --experiment pilot_experiment \
  --annotator "John Doe" \
  --port 7860 --share

python -m vmevalkit.runner.score gpt4o \
  --experiment pilot_experiment \
  --output-dir data/scorings \
  --temperature 0.1 

Use as Python Modules

from vmevalkit.eval import HumanScorer

scorer = HumanScorer(experiment_name="pilot_experiment")
scorer.launch_interface(share=True, port=7860)

See Scoring Guide for details.

VMEvalKit uses a hierarchical structure for organizing all data:

data/
├── questions/                       # Task datasets
│   ├── vmeval_dataset.json         # Master dataset manifest
│   ├── chess_task/                 # Chess puzzles (mate-in-1 scenarios)
│   │   └── chess_0000/
│   │       ├── first_frame.png     # Initial chess position
│   │       ├── final_frame.png     # Solution position
│   │       ├── prompt.txt          # Move instructions
│   │       └── question_metadata.json  # Task metadata
│   ├── maze_task/                  # Maze solving challenges
│   ├── raven_task/                 # Raven's progressive matrices
│   ├── rotation_task/              # 3D mental rotation
│   └── sudoku_task/                # Sudoku puzzles
│
├── outputs/                         # Model inference results
│   └── pilot_experiment/           # Experiment name
│       └── <model_name>/           # e.g., openai-sora-2, luma-ray-2
│           └── <domain>_task/      # e.g., chess_task
│               └── <task_id>/      # e.g., chess_0000
│                   └── <run_id>/   # Timestamped run folder
│                       ├── video/
│                       │   └── model_output.mp4
│                       ├── question/
│                       │   ├── prompt.txt
│                       │   └── first_frame.png
│                       └── metadata.json
│
├── scorings/                        # Scoring results
│   └── pilot_experiment/
│       └── <model_name>/
│           └── <domain>_task/
│               └── <task_id>/
│                   ├── human-score.json     # Human scoring scores
│                   └── GPT4OScorer.json     # GPT-4O scoring scores
│
└── data_logging/                    # Version tracking
    ├── version_log.json            # Version history
    └── versions/                   # Version snapshots

Upload your dataset to S3:

# AWS S3
python data/s3_sync.py --log  # Upload with version logging
python data/s3_sync.py --date 20250115  # Upload with specific timestamp

See Data Management for details.

You could quickly host your results on an interactive dashboard at: http://localhost:5000

# Navigate to web directory
cd web

# Option 1: Use startup script (recommended)
./start.sh

# Option 2: Manual startup
source ../venv/bin/activate
python app.py

See Web Dashboard for details.

You can add new video generation models and reasoning tasks with minimal effort:

Adding New Models

Add any video generation model (API-based or open-source) with just a few steps:

# Example: Adding a new model wrapper
from vmevalkit.models.base import BaseVideoModel

class MyModelWrapper(BaseVideoModel):
    def generate_video(self, image_path, text_prompt, **kwargs):
        # Your model's video generation logic
        return video_path

Then register it in MODEL_CATALOG.py:

"my-model": {
    "provider": "mycompany",
    "wrapper_path": "vmevalkit.models.my_model.MyModelWrapper",
    ...
}

See Adding Models Guide for details.

Adding New Tasks

Create new reasoning tasks programmatically at scale:

from vmevalkit.tasks.base_task import BaseTask

class MyTask(BaseTask):
    def generate_task_pair(self, ...):
        # Generate initial and final states
        initial_state = self.create_initial_state()
        final_state = self.create_final_state()
        prompt = self.create_prompt()
        
        return {
            "first_frame": initial_state,
            "final_frame": final_state, 
            "prompt": prompt,
            "metadata": {...}
        }

VMEvalKit handles all the dataset organization, inference pipelines, and evaluation infrastructure automatically.

See Adding Tasks Guide for details.

VMEvalKit is meant to be a permissively open-source shared playground for everyone. If you’re interested in machine cognition, video models, evaluation, or anything anything 🦄✨, we’d love to build with you:

• 🧪 Add new reasoning tasks (planning, causality, social, physical, etc.)
• 🎥 Plug in new video models (APIs or open-source)
• 📊 Experiment with better evaluation metrics and protocols
• 🧱 Improve infrastructure, logging, and the web dashboard
• 📚 Use VMEvalKit in your own research and share back configs/scripts
• 🌟🎉 Or Anything anything 🦄✨

💬 Join us on Slack to ask questions, propose ideas, or start a collab: Slack Invite 🚀

📚 Core Documentation:

• Inference Guide - Complete guide to running inference, supported models, and architecture
• Scoring Guide - Human and automated scoring methods
• Data Management - Dataset organization, S3 sync, and version tracking
• Adding Models - How to add new video generation models
• Adding Tasks - How to create new reasoning tasks
• Web Dashboard - Interactive results visualization

Here we keep track of papers spinned off from this code infrastructure and some works in progress.

• "Video Models Start to Solve Chess, Maze, Sudoku, Mental Rotation, and Raven's Matrices"

This paper implements our experimental framework and demonstrates that leading video generation models (Sora-2 etc) can perform visual reasoning tasks with >60% success rates. See results.

Apache 2.0