Most eval tools hand you one number. multivon-eval puts a confidence interval on every score, so you can tell whether a change actually moved the metric or just moved the noise. The LLM-judge thresholds are calibrated against human labels, not guessed.
We set ourselves a 50% determinacy gate. We scored 20.9%. We published it. see the track record →44 evaluators · one API · runs on Ollama, LM Studio, vLLM, or any OpenAI-compatible server
No signup · No telemetry · First eval in ~3 min
$ pip install multivon-eval
$ python -m multivon_eval demo
multivon-eval demo · customer support bot
6 cases · deterministic tier · no API key
Pass Rate: 83.3% [44%–97% Wilson 95% CI]
Avg Score: 0.83 [0.50–1.00 bootstrap 95% CI]
Score dist: p10 0.50 · p50 1.00 · p90 1.00
⚡ Power warning: 6 cases — min detectable change
at 80% power is ~74%. Add cases to catch less.κ ≈ −0.01 · 58% disagreement between multivon-eval and DeepEval (RAGAS, earlier run: 40% / 20% vs each).
Same judge, same dataset, same seed (n=50). The worst pair agrees barely better than chance; all three disagree more than you’d hope.
Read the benchmark →F1 0.830 [0.70–0.92] on held-out HaluEval-Sum at the calibrated threshold.
Wilson + bootstrap CIs on every number, judge thresholds calibrated against human labels.
See multivon-eval →Pixels-only nearly inverts the PDF Hell leaderboard: GPT-5 94.7→67.6, Opus 4-7 79.4→85.9.
Same 170 cases rasterised locally at 150 dpi — no embedded text layer — and the PDF leader drops 27.1 points while Opus takes the lead.
Per-model deltas →A single pass-rate number can’t tell you whether a change was real or noise. So the default output ships confidence intervals, a warning when your sample is too small to trust, and a tamper-evident audit trail — enough to stand behind a shipping decision. Tracing tells you what your agent did; this tells you whether it got better, with the statistics to prove it.
0.73vs 0.04
multivon-eval · DeepEval · F1 on ragtruth-sum (n=100, gpt-4o-mini judge)
Same judge, same dataset, same seed, side-by-side. DeepEval’s default threshold flags almost nothing — F1 0.04(recall 0.02). Multivon’s defaults hit F1 0.729; at best-tuned thresholds it’s 0.84 vs 0.61. The gap is the default threshold: DeepEval ships an uncalibrated one, multivon’s is calibrated against human labels. Code in the repo — re-run it yourself.
Read the benchmark →Default outputA pass rate of 84% with CI [62%, 95%] tells you it’s indistinguishable from 75% — and the framework tells you so. When N is too small to detect a 10-point shift, stderr warns before the run, not after the wrong release.
Statistical rigor docs →Compliance--audit-pack out.zip bundles every test case, answer key, run JSON, JUnit XML, and a SHA-256 manifest. Hash-chained NDJSON inside; suite-hash + library-version locked into every report.
See the audit format →A/B that refuses to lieexp.compareruns paired McNemar tests with Benjamini–Hochberg correction across evaluators, reports Cohen’s h, and tells you how many more cases you need at 80% power — instead of letting two noisy runs crown a winner.
Experiments docs →PDF Hell is an adversarial benchmark — 510 procedurally-generated cases across 17 trap families that break the assumptions document-AI vendors silently rely on. Six hand-authored, 11 discovered by an autoresearch loop. Code-based ground truth, not LLM-as-judge. The current leaderboard exposes per-trap blind spots in GPT-4o (hidden-OCR text-layer trust), Anthropic's premium tier (a 3.5pt-footnote trap that inverts on locally-rasterised pixels — a PDF-ingestion failure, not vision), and three different models (text-run fragmentation).
multivon-eval ships Faithfulness, ContextPrecision, and ContextRecall as QAG-graded evaluators with calibrated thresholds per judge. Pass-rate gates your CI without you babysitting an LLM judge. Bootstrap CLI generates a runnable suite + adversarial seed cases (30 by default, up to 500 behind duplicate + hardness gates, every reject reported) + calibrated thresholds from your own traces in one command — and when your prompts change later, the staleness report tells you which cases were authored against the old ones.
8 agent-native evaluators — ToolCallAccuracy, ToolCallNecessity, TrajectoryEfficiency, StepFaithfulness, AgentMemoryEval. LangGraph + OpenAI Agents SDK tracers wired natively. Plus 22 MCP-callable tools so Claude Code / Cursor / Cline run the eval mid-edit. And multivon-eval simulate drives adaptive multi-turn personas against your agent — transcripts scored by the conversation evaluators, always labeled simulated.
Hash-chained NDJSON, SHA-256 manifest, EU AI Act / NIST AI RMF / HIPAA / DPDP India mappings. Generate one from any pdfhell or multivon-eval run — free, no signup. JudgeConfig(base_url=…) routes judging through any OpenAI-compatible endpoint (Ollama, vLLM, on-prem) so production data never leaves the VPC.
multivon-guard is a local proxy that intercepts your agent's outbound traffic and redacts secrets before they leave your machine. Auto-detects Claude Code, Codex, Cursor, Aider, Continue. Now in private beta.
Request early access →The five public packages above are Apache 2.0 and free forever. They share the same engine (multivon-eval) and the same methodology (QAG + code-based ground truth). multivon-guard is in private beta — email for access.
An eval vendor should be auditable. Every entry below links to an artifact we can’t edit after the fact — PyPI release history, GitHub issues, dated posts.
multivon-eval, DeepEval, and RAGAS — same judge, same dataset, same seed (n=50) — disagree on 56% of cases; the worst pair agrees barely better than chance. A result that indicts the category we sell in, published anyway.
A provider temperature bug had been silently scored as wrong answers; the “0% on all seven trap families” claim was an eval artifact, not a model failure. Retracted, leaderboard re-run and corrected, notice left permanently in the repo README.
A peer review caught the 0.9.4 “held-out” claim being in-distribution — the evaluator's threshold was calibrated on the same dataset it was tested on. 0.9.5 corrected the claim, 0.9.6 fixed runtime bugs the same review caught in the generated bootstrap template, 0.9.7 fixed a threshold-vs-default mismatch in the held-out reproducer. All four releases left on PyPI — yanking them would erase the record.
Scanner v3 (0.10.1) measured 278 prompt call sites across aider, gpt-researcher, open-interpreter, letta, and pr-agent: 20.9% statically resolvable, below the 50% gate we set ourselves. Published with the per-repo table on the epic; the runtime recorder was promoted to the priority path past the static ceiling.
Opt-in pytest --record-prompts captures rendered prompt fingerprints at the same three SDK surfaces the static scanner reads. Each source keeps its own trust tier: the static scan proves prompt text, recordings prove only the k-of-N renderings observed, and template prompts stay out of scope. Case-to-site bindings are propose-only.
The cross-modality run exposed that two autoresearch trap families (zero_width_space_split, unicode_confusable_total) rendered visible tofu boxes where they claimed visual normality — the “visually identical” premise was false. Both were redesigned in pdfhell 0.6.1 (adjacent-text-run fragmentation; a digit-zero T0TAL confusable) and a sixth glyph_clean validation gate now pins the invariant. Alongside it, multivon-eval 0.11.1 shipped scanner v4 with an explicit UNSCANNABLE tier — honest UNKNOWN over confident wrong.
Pick the ones your use case needs, ignore the rest. All Apache 2.0, all available via Python SDK and MCP.
No LLM judge needed. Cheap, fast, reproducible.
QAG decomposition. Wilson 95% CIs on every aggregate. Thresholds calibrated against human labels; inter-judge agreement κ 0.60-0.80.
Trajectory-level scoring for multi-step agents.
Procurement-grade checks. PII redaction + schema enforcement.
Image-grounded faithfulness + document QA scoring.
Multi-turn dialog quality, not just single-turn answers.
Run-to-run stability across N samples.
Full list: docs.multivon.ai/evaluators · agent-callable via MCP
The left tab is a real run: GPT-4o against the current pdfhell mini-v4-sample suite (170 cases, 17 trap families). The other four tabs are illustrative shapes — faithfulness, experiment comparison, plain-English checks, and persona simulation. Real numbers and reproducer commands for every model we've scored live on the leaderboard.
Eval suites rot quietly: prompts drift away from the cases written against them, coverage gaps stay invisible, and hand-written scripts test the conversations you imagined instead of the ones real users have. Keeping cases current is part of the tool here, not something you bolt on later.
multivon-eval staleness diffs every prompt call site against the committed baseline and names the cases authored against the old prompts — CHANGED, REMOVED, ADDED, honestly UNKNOWN for what static analysis can't read. --fail-on turns it into a CI gate.
bootstrap --n-seed-cases generates in batches behind duplicate and hardness gates, then reports exactly what survived: “generated 500, accepted 431 — dropped 38 duplicates, 12 malformed.” No silent caps; every accepted case carries its gates and hardness score.
The generation toolkit covers deterministic mutations and template grids (no LLM, no cost), judge-verified contrast pairs, span-grounded doc-QA with refusal-bait unanswerables, and simulate-transcript export. Every generator stamps provenance, routes through the dedupe gates, and reports its rejects.
multivon-eval assess runs a free, deterministic preflight over your traces — count, field completeness, near-duplicates, PII density — before generation runs. No vanity score, and it warns rather than blocks, so a WARN can't break a CI. It runs automatically inside bootstrap and generate.
multivon-eval simulate drives adaptive multi-turn conversations: a persona LLM with a goal and a temper converses with your model, and the transcripts are scored by the conversation evaluators. Every output is labeled simulated, so synthetic users stay out of your production metrics.
JUnit XML output renders natively in every CI runner. A 10-line YAML wires PDF Hell into a pull-request gate. multivon-eval’s pytest plugin slots into your existing test suite without a special harness.
# .github/workflows/pdfhell.yml
name: PDF Hell
on: [pull_request]
jobs:
pdfhell:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v5
- run: uvx pdfhell run \
--model anthropic:claude-sonnet-4-6 \
--suite mini \
--junit results.xml \
--audit-pack audit.zip \
--fail-threshold 0.85
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
- uses: actions/upload-artifact@v4
if: always()
with:
name: pdfhell
path: |
results.xml
audit.zipEach package solves a piece of the AI-evaluation problem and composes with the others. All five are Apache 2.0 — three on PyPI, eval-action and eval-framework-benchmark on GitHub; multivon-guard is in private beta. No telemetry.
the SDK
44 evaluators across deterministic, LLM-judge (QAG), agent-trace, conversation, compliance, and multimodal categories, plus a new bootstrap CLI that proposes a tuned eval suite from your product description + sample traces. Calibrated thresholds, hash-chained audit logs, pytest plugin.
the benchmark
510 adversarial PDFs across 17 trap families that fool AI document readers — like an invoice whose visible total differs from its hidden text layer. The correct answer comes from code, so no AI is grading another AI. Drops straight into CI.
the agent surface
MCP server exposing 22 evaluation tools. Drop into Claude Desktop, Claude Code, Cursor, Cline, or OpenCode and your AI coding agent calls evals directly mid-edit.
the CI gate
GitHub Action that runs multivon-eval on PRs, posts a diff comment with regressions, and gates merges on safety-class failures.
the receipts
Head-to-head benchmark vs DeepEval + RAGAS on hallucination detection. Same judge, same dataset, same seed, fully reproducible. multivon-eval F1 0.79 vs DeepEval 0.0 at defaults.
the runtime gatekeeper
Evaluation alone isn't enough. multivon-guard catches what your AI is about to send before it hits the wire, secrets, PII, API keys. Local proxy, auto-detects Claude Code / Codex / Cursor / Aider / Continue. Now in private beta.
Bring your own API key. Local-first by default. Hash-chained audit logs ship with every run.
What's new across the stack
The view --dir report browser (0.15.0), the input-quality gate (0.14.0), the generation toolkit (0.13.0), and the staleness and recorder layers underneath. See the full roadmap →
multivon-eval view --dir opens a local report browser: a sortable index of every run in a directory, each report rendered in full, and a diff between any two that stacks both runs' judge reasons on the cases that flipped. Read-only, local-first, no new dependencies — it extends the existing view command rather than adding a service.
Read more →assess_input and multivon-eval assess run a free, deterministic preflight over your traces — trace count, field completeness, near-duplicate ratio, PII density — before any generation spend. There's no 0-100 score (the vanity metric the gate exists to prevent); it warns rather than blocks, so a WARN can't break a CI. Runs automatically inside bootstrap and generate.
Read more →mutate_cases applies deterministic robustness mutations (free); cases_from_template expands parametric grids (free); generate_contrast_pairs writes judge-verified unfaithful twins; span-grounded doc-QA records source offsets and can mix in refusal-bait unanswerables; simulate --export-cases turns transcripts into cases. Every generator stamps provenance, routes through the dedupe gates, and reports its rejects.
Read more →multivon-eval simulate drives your bot through persona-driven, adaptive multi-turn conversations: a simulator LLM plays the user, reacts to each reply, and stops at a hard budget ceiling. Every transcript carries provenance and is labeled simulated, not real traffic, so the results can't sneak into production metrics.
Read more →bootstrap --n-seed-cases scales seed-case generation to 500 behind duplicate and hardness gates. The generation report states exactly what survived ("generated 500, accepted 431") and counts every reject, so a thin suite can't masquerade as a big one.
Read more →Opt-in pytest --record-prompts captures rendered prompt fingerprints at call time, the way past the 20.9% static-resolvability ceiling. Static scans prove prompt text; recordings prove only the k-of-N renderings observed; templates stay out of scope. Case-to-site bindings are propose-only.
Read more →multivon-eval staleness diffs a live scan of your repo's prompt call sites against a committed prompt_baseline.json: CHANGED (with before/after fingerprints and the cases bound to that prompt), REMOVED, ADDED, and UNKNOWN for dynamic prompts static analysis can't read. staleness stamp binds cases to the prompts they exercise; bootstrap writes the baseline automatically. Every report opens with a determinacy headline and closes with a blind-spots footer. --fail-on gates CI.
Read more →multivon-eval bootstrap --product PRODUCT.md --traces TRACES.jsonl emits a tuned EvalSuite + adversarial seed cases + calibrated thresholds + a forwardable DISCOVERY_REPORT.md. Single LLM call, ~$0.12 per bootstrap. The fastest path from "I don't know what to eval" to a runnable suite — and 0.12.0's --n-seed-cases scales it to 500 gated cases.
Read more →The 0.10-era substrate under bootstrap, in one card: auto_evaluators(case) ranks evaluators heuristically in microseconds (zero LLM cost); generate_adversarial_cases targets 10 failure modes with stress_test routing metadata; an N-shot judge-noise filter keeps only validated-hard cases (+0.80 mean failure-rate separation); a local PII/secret scan redacts before any trace leaves your machine; thresholds calibrate from your own traces (p25 of baseline scores).
One-command installer for the three bundled Claude Code skills (eval-bootstrap, eval-audit, eval-explain). Symlinks SKILL.md files from the wheel into ~/.claude/skills/ so pip install -U multivon-eval propagates skill edits without re-running the installer.
Read more →An autoresearch loop (Karpathy pattern) where a rotation of Opus 4-7, GPT-5, and Gemini 2.5 Pro propose adversarial PDF traps against an 8-model eval panel. Six validation gates filter candidates before any vision-eval spend (glyph_clean added in 0.6.1 after the tofu-box bug). $88 total across two overnight runs produced 11 surviving trap families — now in mini-v4 (510 cases). The agent does not merge its own work; a human curator promotes from keep/.
Read more →The full mini-v4 suite is 510 cases across 17 trap families; the public leaderboard runs the 170-case mini-v4-sample (first 10 seeds/family). The sample surfaces three real per-trap blind spots: GPT-4o on hidden-OCR invoices (0/10), Anthropic's premium tier on a 3.5pt-footnote trap (0/10), and three models on text-run fragmentation (0/10, measured pre-0.6.1-redesign). The 2026-06-12 pixels runs traced all three to PDF ingestion, not vision.
Read more →Every public package is on PyPI today. Stars and issues welcome on GitHub. For custom adversarial trap families, on-prem deployment, or paid integration support, see /commercial — inbound only, no fake tiers.
Run the current pdfhell mini-v4-sample suite (170 procedurally generated PDFs, 17 trap families) against gpt-4o. It passes most of the suite at 81% overall, but never catches the hidden-OCR mismatch — the visible page text says one number, the buried OCR layer says another, and the model returns the OCR layer every time. The numbers below are a real run — same suite, same seeds, reproducible.