TokenMizer

Keep your AI context alive across sessions.

Graph-backed memory · session checkpointing · intelligent compression
Drop-in proxy for Claude, GPT, Gemini, Grok, DeepSeek, Ollama — any LLM.

Every AI session has a context limit. When you hit it:

• The model forgets every decision, rationale, and context built over hours
• You waste 10–30 minutes re-explaining the project every new session
• Large files (CSV, PDF, Excel) eat your entire token budget instantly

TokenMizer is a local proxy between your app and any LLM. Every request goes through a pipeline that builds a live knowledge graph, compresses inputs, caches responses, and auto-checkpoints before context runs out.

Your App  →  TokenMizer (:8000)  →  Claude / GPT / Gemini / any LLM
                    │
          ┌─────────┴──────────────┐
          │   6-Layer Pipeline     │
          │   L0  File Intel       │  CSV/PDF/Excel → schema + sample
          │   L1  Compression      │  15–40% input reduction
          │   L2  Output Trim      │  5–15% output reduction
          │   L3  Semantic Cache   │  100% on repeated queries
          │   L4  Graph Memory     │  session continuity
          │   L5  Prompt Cache     │  90% on repeated system prompts
          └────────────────────────┘

History is never deleted. "Why did we switch from React to Next.js?" — always answerable.

# Recommended
pip install "tokenmizer[anthropic,cache]"

# All providers
pip install "tokenmizer[anthropic,openai,gemini,cohere,cache]"

# No key? Use Ollama (free, local)
brew install ollama && ollama pull llama3
pip install tokenmizer

export TOKENMIZER_ANTHROPIC_API_KEY=sk-ant-...
# or: TOKENMIZER_OPENAI_API_KEY, TOKENMIZER_GEMINI_API_KEY, etc.

tokenmizer serve
# → Proxy:     http://localhost:8000/v1/chat/completions
# → Dashboard: http://localhost:8000
# → API docs:  http://localhost:8000/docs

from openai import OpenAI

client = OpenAI(
    api_key="your-key",
    base_url="http://localhost:8000/v1",  # ← only this changes
)

response = client.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[{"role": "user", "content": "Let's build an auth service"}],
    extra_body={"session_id": "my-project"},  # enables graph memory
)

# Add TokenMizer as a plugin marketplace
/plugin marketplace add Shweta-Mishra-ai/tokenmizer

# Install
/plugin install tokenmizer@Shweta-Mishra-ai/tokenmizer

Then use skills directly:

/tokenmizer:checkpoint my-project      → save session to graph memory
/tokenmizer:resume my-project          → load previous session (300 tokens)
/tokenmizer:resume my-project full     → full 600-token context
/tokenmizer:analyze /data/sales.csv    → analyze file (99% token savings)
/tokenmizer:stats                      → token savings report

Add to ~/.claude/settings.json:

{
  "mcpServers": {
    "tokenmizer": {
      "command": "python",
      "args": ["-m", "tokenmizer.mcp.server"],
      "env": { "TOKENMIZER_URL": "http://localhost:8000" }
    }
  }
}

Cursor / Continue.dev / any OpenAI-compatible tool:

API Base URL:  http://localhost:8000/v1

tokenmizer checkpoint my-project
tokenmizer resume my-project

Goal: Build FastAPI auth service with JWT + PostgreSQL
Done: Project setup | User model | Login endpoint | Fix 422 | 18 tests passing
In progress: Refresh token rotation
Decided: PostgreSQL (concurrent writes) | bcrypt | Redis for refresh tokens
Changed: ~~React~~ → Next.js (better SEO)
Files: api/auth.py, api/models.py, config.py
Continue: Implement token refresh endpoint

247 tokens replaces 25,000+ tokens of conversation history.

from tokenmizer.filters.file_intelligence import FileIntelligence

fi = FileIntelligence()
result = fi.process(open("sales.csv","rb").read(), "sales.csv",
                    token_budget=500, query="which regions underperforming")
# 412,000 tokens → 447 tokens  (99.9% saved)

TokenMizer complements — does not replace — these tools:

Tip: If using Caveman, set terse_output: enabled: false in tokenmizer.yaml to avoid conflicting system prompts.

# tokenmizer.yaml
provider: anthropic
default_model: claude-sonnet-4-6

graph_checkpoint:
  enabled: true
  trigger_at_percent: 0.85
  use_llm_extraction: false     # true = 80%+ recall, needs key (~$0.001/turn)

compression:
  enabled: true

cache:
  enabled: true
  max_size: 10000

state_backend: memory           # memory | redis (production)

All settings via env vars: TOKENMIZER_PROVIDER, TOKENMIZER_API_KEY, etc.

# Quick start
docker-compose up tokenmizer

# With Redis (production)
ANTHROPIC_API_KEY=sk-ant-... docker-compose up

# With proxy auth
TOKENMIZER_API_KEY=strong-key docker-compose up

• API key auth — TOKENMIZER_API_KEY (constant-time comparison)
• Secret/PII redaction applied once at ingestion, before graph storage, checkpoint storage, AND every LLM call (main chat and the background extraction model — these are separate, the redaction gap between them was a real bug, now fixed)
• Session-isolated cache (sensitive data never shared across sessions)
• Basic prompt-injection keyword filter — catches copy-pasted jailbreak templates only; not a security boundary against a motivated adversary. See SECURITY.md for exactly what it does and doesn't catch.
• CORS restricted to configured origins by default

python benchmarks/checkpoint_accuracy/runner_v2.py
pytest tests/ -v

Benchmark v2 — Graph vs plain Summary (3 sessions, heuristic-only):

Avg resume size: 246 tokens vs ~1,500+ tokens of raw history.

Enable use_llm_extraction: true for hybrid extraction (LLM + heuristic merge).

On LLM/hybrid recall numbers — read this before trusting any percentage here: earlier versions of this README quoted "90-100% hybrid recall" sourced from runner_v3.py's MockLLMProvider. That mock sampled its fake output directly from the same ground-truth dict used to score recall — circular by construction, guaranteed to look good regardless of what the real extraction logic did. It measured nothing about actual LLM extraction quality. That number has been removed rather than replaced with a better-sounding one we can't back up.

What runner_v3.py now actually does:

• Default mode verifies HybridExtractor.merge()'s logic contract against fixtures with deliberately known overlap (corroborated / LLM-only / heuristic-only items) — confirms merge never drops an item either source found, and applies confidence tiers (0.95 corroborated, 0.80 LLM-only, 0.65 heuristic-only) correctly. This is a real, non-circular check, but it's a logic-contract test, not a recall measurement.
• --live mode calls a real configured provider (ANTHROPIC_API_KEY or OPENAI_API_KEY) and scores its actual output against ground truth. This is the only path that produces a number meaningful enough to put in a table. Run it yourself — we're not publishing a live-mode number here because n=3 sessions is too small a sample to generalize, and publishing one without a large, ongoing benchmark would just be swapping one unsubstantiated number for another.

Heuristic-only numbers above (76-100%) ARE real, deterministic, reproducible measurements — runner_v2.py runs actual heuristic extraction against actual ground truth with no LLM and no mocking involved, which is why those numbers are presented with confidence and the LLM ones currently are not.

Engineers ask this every time. Honest answers:

Why not just use Git history? Git stores what changed, not why you decided to change it. You can't ask Git "what did we decide about auth?" or "why did we switch from MySQL to PostgreSQL?" TokenMizer stores decisions with trigger, reason, and evidence — not diffs.

Why not RAG (retrieval-augmented generation)? RAG retrieves relevant chunks — it doesn't model decision state. If you switched from bcrypt to Argon2 mid-session, RAG might retrieve both and confuse the model about which is current. TokenMizer tracks decision supersession explicitly: old decision is marked SUPERSEDED, new decision is ACTIVE. Resume context only includes current state.

Why not a plain summary at the start of each session? Summaries lose structure. You can't query "all superseded decisions" or "what triggered the auth change" from a blob of text. Our benchmark shows graph memory preserves +5% more information than a summary baseline — and unlike summaries, the graph is queryable, editable, and grows incrementally without re-summarizing everything each turn.

Why not Mem0 or Zep? Mem0 and Zep store facts ("user prefers Python"). TokenMizer stores decisions with rationale — the full causal chain: what was decided, what replaced it, why, what evidence triggered the change, and how confidence shifted. If you need "remember my name across sessions," use Mem0. If you need "remember that we switched from PostgreSQL to SQLite because of cost, and here's the evidence," use TokenMizer.

Why not just a longer context window? Longer context = higher cost + slower inference + model attention dilution on long histories. TokenMizer compresses a 50-turn session into ~246 tokens of structured context — not by summarizing, but by extracting what actually matters: goals, active decisions, current tasks, recent errors.

tokenmizer serve [--port 8000]
tokenmizer checkpoint <session-id>
tokenmizer resume <session-id> [--level standard|full|critical]
tokenmizer stats

Note on file analysis: /tokenmizer:analyze (used from inside Claude Code, see Claude Code Integration above) is real and works — it's a plugin skill (.claude-plugin/skills/analyze/) that calls FileIntelligence directly via an inline Python snippet, independent of the CLI/API layer. What does not exist is a bare tokenmizer analyze <file> terminal command or a /api/analyze HTTP endpoint — useful if you want file analysis from a plain shell or a non-Claude-Code tool (Cursor, a script, curl, etc.) rather than inside Claude Code specifically. Found during a documentation accuracy pass: an earlier version of this README listed tokenmizer analyze <file> in this CLI section as if it were a cli.py command — it never was. Removed from here rather than left in place pointing at something that would fail. Tracked as a real, wanted gap — contributions adding a /api/analyze endpoint + thin CLI wrapper (following the existing pattern in cli.py) are welcome.

See CONTRIBUTING.md. Graph extraction contributions are the highest priority.

git clone https://github.com/Shweta-Mishra-ai/tokenmizer
pip install -e ".[dev]"
pytest tests/ -v && ruff check tokenmizer/

MIT © Shweta Mishra