multivon-eval v0.16.1 on PyPIFull changelog →

Roadmap

What we’re building next, what comes later, and what we are deliberately not building. No dates more specific than quarters, no features that aren’t on someone’s branch. Finished work moves to the changelog.

Framing principles

→No promised launch dates more specific than quarters, and no aspirational features that aren't actually being built.
→Every shipped item is verifiable: links to the CHANGELOG entry, commit SHA, or published PyPI artifact.
→Every planned item names a specific user-facing outcome you would see, not a vague capability area.
→Open source, Apache-2.0, runs on your machine. No managed-service tier. No hosted eval-database. No fine-tuning-as-a-service. Those are explicitly not in the cards.
→The roadmap is readable as steps toward one thesis: an autonomous swarm of eval agents that lives inside your system. Every item below is groundwork for that future, not random feature accretion.
→If a feature is hard to commit to honestly, it goes under "Exploring", not "Q3". We'd rather under-promise.

In flight Q3 2026 target Long-term vision

In flight

July 2026 — in-progress

multivon-eval watch <dir> daemon

Planned

Long-running process that re-runs the configured suite on file or git changes in a watched directory. Debounced. Reuses the existing EvalSuite and CostTracker. First step toward continuous eval that doesn't require a CI trigger. Re-scoped after 0.10.0: the watcher will run the staleness report on file change, so prompt drift surfaces in your terminal before any suite re-run spends judge tokens.

What you’d see

Edit a prompt file, save it, see the suite re-run in your terminal within a second. No CI round-trip.

sync — propose-and-review case refresh

Planned

Consumes the staleness JSON report and proposes case-file diffs for human review when bound prompts change. Never auto-commits — confidently-wrong refreshes would poison the suite, so every proposed diff requires explicit approval. The deliberate other half of the 0.10.0 read-only report.

What you’d see

multivon-eval sync proposes updated cases for every CHANGED prompt; you review and apply, or reject. Nothing is rewritten without your sign-off.

multivon-eval#8 →

Staleness gate in eval-action

Planned

Surfaces the staleness report in the eval-action PR workflow: warn-only $GITHUB_STEP_SUMMARY line first (works today as a documented one-liner with --format markdown), per-category fail-on as an action input later. Gate-by-default is deliberately avoided: failing PRs on ADDED prompts punishes adoption.

What you’d see

Every PR shows a staleness summary in the Actions run page without any gating; teams opt into --fail-on per category when ready.

eval-action#1 →

MetricsSink interface (file / stdout / Datadog)

In progress

Pluggable sink so every EvalReport can be streamed to a destination of your choice. Three built-in adapters: file (JSONL append), stdout (rich-rendered), Datadog (gauge per evaluator + count per status). No hosted sink — bring your own observability.

What you’d see

suite.run(sink=DatadogSink(api_key=...)) lights up your existing Datadog dashboards with calibrated eval metrics. No multivon-hosted middleware.

Cross-source ingest adapters

In progress

First-class adapters for LangSmith, Sentry, and S3 so traces stored across systems can be loaded into a single eval run without you writing custom ETL. Extends the existing load_traces aliasing work (LangFuse, Phoenix already covered).

What you’d see

load_traces('langsmith://project/foo') and load_traces('sentry://issue/1234') return EvalCase lists ready to feed a suite.

eval-watch Claude Code skill

Planned

Fourth bundled skill wrapping the watch daemon. Slash command that starts the daemon scoped to the current repo, surfaces failures in chat, and proposes fixes when a regression appears. Sibling to /eval-bootstrap, /eval-audit, /eval-explain.

What you’d see

/eval-watch in Claude Code keeps your eval running while you code; agent narrates regressions as they happen.

Swarm coordination skeleton — multivon-eval swarm <repo>

Planned

Proof-of-concept that coordinates three named subagents (auditor, attributor, explainer) against a repo and surfaces a unified report. Skeleton only. The real swarm is the long-term thesis below; this is the first walking version.

What you’d see

multivon-eval swarm <repo> produces a markdown report combining a suite audit, a prompt-change attribution, and per-failure explanations from three independent agents.

Q3 2026 target

Q3 2026 — planned

Truly cross-corpus held-out evaluation (TruthfulQA + FaithBench)

Planned

HaluEval-QA and HaluEval-Sum share CNN/DailyMail structure. Adding TruthfulQA and FaithBench as held-out targets gives a genuinely cross-corpus generalization figure. Will also re-run at n=500–1000 to tighten CIs (current held-out n=60 is honestly wide).

What you’d see

/eval gains a second held-out tile: F1 with a 95% CI on a corpus the calibration set has never touched. Plus narrower CIs on the existing held-out claim.

Phase 2 prompt attribution — sidecar + majority voting

Planned

Adds a non-prompt-change signal (git diff classifier over pyproject, model configs, dependencies) ANDed against Haiku majority-voted attribution. When the sidecar fires `present`, attribution refuses rather than confidently misattributing a mixed-cause regression. Design doc: feature_prompt_attribution_phase2_sidecar_design.md.

What you’d see

On any PR, the eval-action comment shows per-case attribution with HIGH/MED/LOW confidence, or skips and tells you both prompts AND non-prompt code changed and the data doesn't support causal attribution.

Async / batch QAG

Planned

QAG evaluators (Faithfulness, Hallucination, ContextPrecision) currently don't override aevaluate, so async paths fall back to sync. Closes a known limitation flagged in iter-2 peer review. Pairs with batch judging via the existing OpenAI/Anthropic batch APIs for 50% cost reduction on long runs.

What you’d see

suite.run_async(workers=32) actually parallelizes QAG judge calls. Long benchmark runs (n=1000+) finish 5–10x faster at half the API cost.

Supply-chain hardening for eval-action

Planned

Cosign signature on the published container image, SBOM (CycloneDX) attached to each tagged release, immutable @v1.x.y tags alongside the mutable @v1. Closes r/MLOps procurement-tier concerns from launch review.

What you’d see

cosign verify succeeds against the eval-action image. Procurement teams get an SBOM file in the release assets. Pinning to a specific SHA stays stable.

Pluggable baseline source

Planned

Today multivon-eval compare pairs cases by sequential index within an input. Adding a pluggable baseline source (git ref, JSON file, S3 bucket, MetricsSink read-back) lets compare answer "did my prompt change help vs the version on main 30 days ago" without manual artifact wrangling.

What you’d see

multivon-eval compare --baseline git:origin/main proposal.json --markdown produces a PR-ready diff against whatever main was when the run happened.

Trust & safety evaluator family

Planned

Extend the shipped Toxicity and Bias judges into full coverage of the trustworthiness dimensions benchmarks like TrustLLM (ICML 2024) popularized: robustness (input-perturbation + prompt-injection resistance), harmful-content / jailbreak refusal, and machine ethics. Truthfulness (Faithfulness/Hallucination) and privacy (PIIEvaluator) already ship; this closes the remaining dimensions.

What you’d see

suite.add(Robustness()) / Jailbreak() / MachineEthics(): gate a model on trustworthiness the same way you gate on faithfulness today, each carrying the usual Wilson CI and calibrated threshold.

Long-term vision

12–18 months

Autonomous eval swarm

Exploring

The underlying thesis. Every roadmap item above is a step toward one thing: a swarm of eval agents that goes into your system, reads your code and your data and your logs wherever they're stored, and continuously updates eval metrics without a human writing or maintaining the eval suite. The watch daemon is the trigger. MetricsSink is the egress. Ingest adapters are how the swarm sees your traces. The Phase 2 attribution sidecar is how it stays honest about causality. Swarm coordination is the orchestration layer. Once those pieces are mature, the eval suite stops being a thing you write — it becomes a thing your repo grows.

What you’d see

You install one process. It discovers your prompts, your traces, your code-change history, and your existing observability stack. It proposes evals, calibrates them, runs them continuously, attributes regressions, and ships a weekly health report. You never write a suite by hand.

What we are deliberately NOT building

Exploring

A managed-SaaS eval dashboard is not on the roadmap. Neither is a hosted eval-database, and neither is fine-tuning-as-a-service. multivon-eval is and will remain Apache-2.0, runs on your machine, sinks metrics to observability tools you already own. If the right answer for you is a hosted dashboard, the answer is Datadog or Phoenix or your own — not us. The commercial surface (if any) will be support and integration consulting, not a tier above the OSS.

What you’d see

Forever-free local install. No required cloud account. No data-egress to multivon servers. Your eval results stay in your infrastructure.

Want something on this list sooner?

File an issue on multivon-eval or email us. Real demand is how we re-prioritise. Every “planned” item ships in dependency order, not alphabetical order.

File a GitHub issue hello@multivon.ai Recent releases →