the SDK
The evaluation engine for RAG, agents, conversations, and document AI. Validate graders, detect flaky behavior, compare runs with uncertainty, and keep the evidence local from one Python API.
pip install multivon-eval
Explore the SDK →Apache 2.0 · Python · local-first · v0.16.1
Know whether your AI product actually got better.
multivon-eval is an open-source Python framework that validates your graders, separates product changes from statistical noise, and turns every run into evidence your team can inspect. Use it for RAG, agents, conversations, and document AI.
We set ourselves a 50% determinacy gate. We scored 20.9%. We published it. see the track record →44 evaluators · Python 3.10–3.14 · no signup · no telemetry · runs locally or in CI
multivon-evallive output · real local run · no API key
$ pip install multivon-eval
$ python -m multivon_eval
multivon-eval demo · customer support bot
6 cases · deterministic tier · no API key
Pass Rate: 83.3% [44%–97% Wilson 95% CI]
Avg Score: 0.83 [0.50–1.00 bootstrap 95% CI]
Score dist: p10 0.50 · p50 1.00 · p90 1.00
⚡ Power warning: 6 cases — min detectable change
at 80% power is ~74%. Add cases to catch less.Evaluate
Any Python callable · LangChain · LangGraph · OpenAI Agents SDK
Judge with
Anthropic · OpenAI · Gemini · Ollama · OpenAI-compatible models
Ship through
pytest · GitHub Actions · local HTML · JSON · CSV · JUnit
Most eval tools grade your model. Multivon also grades the measurement.
A score is useful only when the cases, graders, sample size, and infrastructure support it. multivon-eval checks each layer and keeps the reasoning attached to the result.
Before the run
Validate the eval before you trust its score.
Run your graders against reference answers and known-bad contrasts. Broken tasks and miscalibrated graders surface before they become fake model failures.
See grade-your-graders →Against human labelsCalibrate automated judges instead of accepting arbitrary defaults.
Per-model thresholds and their provenance travel with the package. A judge change warns instead of silently changing the meaning of pass and fail.
Inspect the calibration evidence →On every runKnow when an apparent improvement is only noise.
Confidence intervals, power warnings, repeated-trial reliability, error budgets, and paired comparisons appear in the report—not in a separate statistics project.
Explore reliability features →After the runKeep the evidence local, portable, and reviewable.
Export HTML, JSON, CSV, and JUnit, or build an audit package with calibration data, a manifest, and an offline verifier. No hosted account required.
See the audit format →Why validate the measurement?
In our published pilot, three frameworks using the same 100 items, judge, and seed reached barely-above-chance agreement on which answers hallucinated (κ ≈ 0.04). Framework choice changed 33 verdicts in one pair. Inspect the method and raw results →
What are you trying to evaluate?
Start with the problem you recognize. Each path leads to a worked example first and the implementation details second.
Document AI
I ship AI that reads PDFs, contracts, claims, or medical records.
Test whether a model reads the visible document instead of trusting hidden text, missing footnotes, or losing table structure. PDF Hell supplies 510 generated cases with exact, code-based answers.
RAG / faithfulness
My RAG model hallucinates from retrieved context.
Check whether an answer is supported by the retrieved context, relevant to the question, and consistently graded. Start from your own traces and generate a runnable evaluation suite in one command.
Agents
I need to grade my agent’s tool calls and trajectories.
Check whether an agent chose the right tools, passed the right arguments, and took a sensible path to the result. LangGraph and OpenAI Agents SDK traces work out of the box.
Compliance
Procurement needs an audit pack our enterprise customers will accept.
Turn any run into a tamper-evident evidence pack with the cases, answers, results, and compliance mappings included. Use an on-prem model so production data stays inside your environment.
Agent safety
My LLM coding agent is sending PII or secrets upstream.
multivon-guard is a local proxy that intercepts your agent’s outbound traffic and redacts secrets before they leave your machine. Auto-detects Claude Code, Codex, Cursor, Aider, Continue. Now in private beta.
Request early access →The four public packages are Apache 2.0 and free forever. They share one engine (multivon-eval) and one methodology (QAG + code-based ground truth). multivon-guard is in private beta; email for access.
An eval vendor should be auditable.
When we measure ourselves and fail, we publish the number. Four from the record below; each links to an artifact we can’t edit after the fact.
- 2026-05-24
pdfhell Opus 4-7 headline finding retracted
A provider temperature bug had been silently scored as wrong answers; the “0% on all seven trap families” claim was an eval artifact, not a model failure. Retracted, leaderboard re-run and corrected, notice left permanently in the repo README.
- 2026-06-11
Determinacy gate run on real repos — failed at 20.9%
Scanner v3 (0.10.1) measured 278 prompt call sites across aider, gpt-researcher, open-interpreter, letta, and pr-agent: 20.9% statically resolvable, below the 50% gate we set ourselves. Published with the per-repo table on the epic; the runtime recorder was promoted to the priority path past the static ceiling.
- 2026-07-13
Three frontier models voted 0.16.0 not ready — so it wasn't released
The pre-release verification round — three independent model reviews — voted unanimously NOT ready, on four major defects including a crash that fired after the paid model run but before results were saved. All four were fixed with ~40 tests and the release re-verified to a unanimous green in round 2 before tagging. The vote and its outcome are recorded in the release notes.
- 2026-08-16
The 47% error rate we published against ourselves is fixed in 0.16.1
A reasoning model spends part of its output budget thinking before it answers. Our judge calls capped that budget at 100 tokens for a yes/no verdict and 512 for claim extraction, so the judge was cut off mid-thought and returned an empty verdict — the 47.3% error rate we published in the study's ablation instead of quietly repairing. 0.16.1 floors the per-call ceiling at 2048 tokens for reasoning-tier judges. Every other judge is byte-identical, and an explicit larger max_tokens is never lowered.
Four Apache 2.0 packages + one in private beta.
Each package solves a piece of the AI-evaluation problem and composes with the others. All four are Apache 2.0: three on PyPI, eval-action on GitHub; multivon-guard is in private beta. No telemetry.
the benchmark
510 adversarial PDFs across 17 trap families that fool AI document readers, like an invoice whose visible total differs from its hidden text layer. The correct answer comes from code, so no AI is grading another AI. Drops straight into CI.
pip install pdfhell
PDF Hell docs →the agent surface
MCP server exposing 22 evaluation tools. Drop into Claude Desktop, Claude Code, Cursor, Cline, or OpenCode and your AI coding agent calls evals directly mid-edit.
pip install multivon-mcp
Wire it up →the CI gate
GitHub Action that runs multivon-eval on PRs, posts a diff comment with regressions, and gates merges on safety-class failures.
uses: multivon-ai/eval-action@v1
View action on GitHub →in private beta
multivon-guard▲ Early Access
the runtime gatekeeper
Evaluation alone isn't enough. multivon-guard catches what your AI is about to send before it hits the wire: secrets, PII, API keys. Local proxy, auto-detects Claude Code / Codex / Cursor / Aider / Continue. Now in private beta.
mailto:hello@multivon.ai
Request early access →Bring your own API key. Local-first by default. Hash-chained audit logs ship with every run.
Apache 2.0. Free forever. No telemetry.
Three packages are on PyPI; eval-action is on GitHub. Stars and issues are welcome. For custom adversarial trap families, on-prem deployment, or paid integration support, see /commercial (inbound only, no fake tiers).