Benchmark · method disclosed · 2026-06-26
multivon-eval vs DeepEval vs RAGAS
on hallucination detection.
Every framework claims accuracy on faithfulness and hallucination detection, but none publish a side-by-side comparison. This one holds everything constant: the same judge model, the same 100 cases, the same random seed for every framework. The measured configuration and a machine-readable result summary are published below.
What this means
The framework you choose can change which answers are called hallucinations. Defaults therefore need to be checked against human labels, not accepted as interchangeable scores.
Results at default thresholds
Dataset: ragtruth-sum (n=100)(RAG-Truth summarization split, 100-case stratified sample with human labels). Each row is one framework's faithfulness/hallucination metric scored against the human labels at the framework's default threshold.
Judge: claude-haiku-4-5
| Framework | Threshold | F1 | Precision | Recall | Latency (ms) | Errors |
|---|---|---|---|---|---|---|
| multivon-eval | 0.90 | 0.690 | 0.615 | 0.784 | 10066 | 0 |
| DeepEval | 0.50 (default) | 0.000 | 0.000 | 0.000 | — | 100 |
| RAGAS | — | — | — | — | — | 0 |
Judge: gpt-4o-mini
| Framework | Threshold | F1 | Precision | Recall | Latency (ms) | Errors |
|---|---|---|---|---|---|---|
| multivon-eval | 0.90 | 0.729 | 0.912 | 0.608 | 10341 | 0 |
| DeepEval | 0.50 (default) | 0.038 | 1.000 | 0.020 | 12720 | 0 |
| RAGAS | 0.50 (default) | 0.038 | 1.000 | 0.020 | 136122 | 4 |
Reading the table.At default thresholds, DeepEval scores F1 0.000 with the claude-haiku judge (every case errors) and F1 0.038 with gpt-4o-mini (recall 0.02; it flags almost none of the labeled hallucinations). multivon-eval's F1 is 0.690 (claude-haiku) and 0.729 (gpt-4o-mini). Default-vs-default is the comparison most users get when they install each framework and run with the documented configuration.
What happens if you tune the threshold?
Some of DeepEval's poor performance at default settings is a threshold issue. Below, F1 across a threshold sweep on the gpt-4o-mini judge. multivon-eval's best F1 is 0.837 at threshold 0.95. DeepEval's best F1 is 0.609 at threshold 0.95. Even at best-tuned thresholds, multivon-eval has a ~37% F1 advantage. Threshold sweeps are computed on the test set, so read them as upper bounds, not held-out estimates.
| Threshold | multivon-eval F1 | DeepEval F1 |
|---|---|---|
| 0.30 | 0.000 | 0.000 |
| 0.50 | 0.000 | 0.038 |
| 0.60 | 0.075 | 0.107 |
| 0.70 | 0.210 | 0.194 |
| 0.80 | 0.418 | 0.384 |
| 0.90 | 0.729 | 0.569 |
| 0.95 | 0.837 | 0.609 |
Inspect the recorded result
Download the exact configuration and table above as JSON. The standalone cross-framework harness is not public yet, so we do not label this run independently reproducible. The public SDK benchmark directory contains related evaluator benchmarks and raw artifacts.
curl -O https://multivon.ai/data/framework-benchmark-summary.json
# Related public SDK benchmarks and raw artifacts:
git clone https://github.com/multivon-ai/multivon-eval
cd multivon-eval/benchmarksDatasets: HaluEval QA and Summarization (100-case stratified samples each), plus the ragtruth-sum split. Judges tested: Claude Haiku 4.5, GPT-4o-mini. Same random seed across all runs. Inspect the machine-readable summary or the public SDK benchmarks.
Calls we made
- Same judge for all frameworks.We don't let each framework use a different judge model. If we did, the comparison would measure judge quality, not framework quality.
- Same threshold semantics.Each framework's documented "default threshold" was used as-is, then thresholds were swept in the second section so you can see how each behaves at its best.
- RAGAS included as of ragas 0.4.3. It errored on every case in the prior harness; it now completes (4/100 cases still error, disclosed above) and runs ~15× slower than the other two.
- Where multivon-eval loses, it's documented. See COMMENTARY.mdin the repo for cases where multivon-eval flagged a hallucination the human label said was correct, or vice versa. We don't hide them.