Vibe Bench - LLM App Benchmark Tool

Vibe Bench is a benchmark for testing and evaluating fully LLM-generated applications across different technology stacks. It helps to identify the best technology stack for "vibe coding" in a given use case.

1. Generate the complete application using a given use-case specific prompt.
2. Use a second prompt to test and debug the app before running it.
3. Run the app in a Docker container. If it doesn't work, use up to two additional prompts (depending on the stack) to fix the issues.
4. Manually test the app using a given test suite and log the results in an evaluation checklist.

The table below tracks the evaluation scores of LLM-generated applications across different technology stacks. For full testing checklist, refer to each run's folder under /apps. First prompt is the app generation prompt. Second prompt asks to add more test and self-test the application. We allow a maximum of 100 tool use calls for each prompt, if the app still doesn't work after both prompts the run is considered failed.

Test runs below are grouped by use case and technology stack. They are ordered by increasing number of prompts required to get a working application.

Other stack ideas to test: Astro + Tailwind, TanStack Router + React, SvelteKit, Gatsby + React, Nuxt.js + Vue, Hugo + Go templates, WordPress + PHP, Django + Python, Ruby on Rails.

Other stack ideas to test: Python + Pygame, JavaScript + Phaser, C# + Unity, C#/C++ + Godot, C++ + Unreal Engine, C++ + SFML, Java + LibGDX, Rust + Bevy.

Other stack ideas to test: Go + Bubble Tea, Python + Textual, C++ + FTXUI, JavaScript + Blessed, C + ncurses, Nim + Illwill, Haskell + Brick, Elixir + Ratatouille, Zig + libvaxis

A tool for automatically running and evaluating web applications generated by LLMs across different technology stacks.

• Automatically detects app type (Node.js, Python, Ruby, Go, etc.)
• Parses setup and running instructions from README files
• Creates Docker containers to run apps in a sandboxed environment
• Runs basic functionality tests against running apps
• Generates benchmark reports with success rates by tech stack
• Interactive mode for manual testing of apps
• Generates prompts for different technology stacks from a template

• Docker
• Docker Compose
• Python 3.9+

1. Create a directory structure for your generated apps:

apps/
├── app1/  # First generated app
├── app2/  # Second generated app
└── app3/  # Third generated app

2. Run the benchmark tool:

# Using Docker Compose (recommended)
docker-compose up

# Or using Python directly
python benchmark.py ./apps

3. View results in the benchmark_results directory.

The tool includes a prompt generator that can create tech stack-specific prompts from a template:

# List available tech stacks
python benchmark.py --list-stacks

# List available templates
python benchmark.py --list-templates

# Generate a prompt for a specific stack and framework
python benchmark.py --generate-prompt --stack typescript --framework react

# Use a specific template
python benchmark.py --generate-prompt --stack python --framework django --template generic_web_app_template.md

Generated prompts will be saved to the output directory (default: benchmark_results).

To run a single app in interactive mode for manual testing:

# Using Docker Compose
APP_NAME=your_app_name docker-compose run interactive

# Or using Python directly
python benchmark.py ./apps --interactive --app-name your_app_name --port-start 3000

This will start the app and keep it running until you press Ctrl+C, allowing you to interact with it manually through your browser at http://localhost:3000.

To run a single app in a Docker container:

python app_launcher.py path/to/app --port 3000

To run benchmark tests on all apps in a directory:

python benchmark.py path/to/apps/dir --output-dir benchmark_results --port-start 3000

1. Prompt Generation: Generate technology stack-specific prompts from a template.

2. App Detection: The tool scans each app directory to identify the technology stack based on files like package.json, requirements.txt, etc.

3. Instruction Parsing: It attempts to parse setup and start instructions from the README file. If not found, it uses heuristics based on the detected stack.

4. Docker Container Creation: For each app, a Dockerfile is created based on the detected stack and instructions.

5. Container Launch: The app is launched in a Docker container, exposing it on a specified port.

6. Testing: Basic functionality tests are run against the running app.

7. Results Collection: Results are collected and saved to the output directory.

Prompt templates use the ${language} and ${framework} placeholders to customize for different tech stacks. You can create your own templates in the prompts directory.

All apps are run in isolated Docker containers to ensure they cannot access the host system. The benchmark runner itself can also be run in a Docker container for an extra layer of isolation.

MIT