Partners in Evaluation
Key Takeaways
- Muse Spark 1.2 is the current top model by Harvey final score at 25.42%, followed by Muse Spark 1.3 Max at 23.75% and Muse Spark 1.3 at 22.92%.
- Qwen 3.8 27B is the strongest open-weight model in this run at 11.25%, just ahead of Kimi K3 at 10.83%, while DeepSeek V4 matches GPT 5.5 on final score at 3.75%.
- Top models satisfy most individual criteria, with criteria pass rates around 90%.
Benchmark
The Legal Agent Benchmark is a benchmark recently released by Harvey to test the ability of models to support legal work in an agentic setting. There are two datasets as of today: the public set and the held-out test set. The results here are from the held-out set, and initial results have already been released by Harvey.
Each task asks an agent to produce legal work against a set of task-specific criteria. The agent is provided with six tools:
Read File, Edit File, Write File, Glob, Bash, and Grep. It also has three skills: docx, pptx, and xlsx.
Reported results use the same methodology as Harvey’s initial leaderboard. Criteria pass rate is included to show how often models satisfy individual requirements, even when they do not fully resolve the task.
Results
Among models with a nonzero final score, MiMo V2.5, DeepSeek V4 Flash 0731, Muse Spark 1.1, and Muse Spark 1.2 form the cost/performance frontier.
Muse Spark 1.2 leads on the overall Harvey final score at 25.42%, with Muse Spark 1.3 Max second at 23.75%, Muse Spark 1.3 third at 22.92%, Muse Spark 1.1 fourth at 20.00%, and Grok 4.6 fifth at 15.83%. Claude Fable 5, tied for seventh at 11.25%, fell back to Claude Opus 4.8 on 4 tasks; counting those as failures gives a no-fallback score of 10.42%. The criteria pass rates are much higher: Muse Spark 1.3 Max reaches 94.74%, Muse Spark 1.2 94.52%, Muse Spark 1.1 92.86%, and Grok 4.6 92.52%.
The leaderboard can be filtered by task type and switched between task pass rate and criteria pass rate. Models perform best on task resolution in Energy/Natural Resources and Healthcare/Life Sciences, while criteria pass rates averaged across all models are highest in Intellectual Property, Corporate M&A, and Data Privacy/Cybersecurity.
Harvey grades a task as resolved only if every criterion passes. A model can satisfy most individual criteria and still miss task resolution credit.
There is a clear trend: strong models and agents satisfy most criteria, around 90% for top models. The remaining gaps are large enough that task resolution stays low even when criterion-level performance looks strong.
Average tool count per item, across the six tools in the benchmark harness.
Models heavily prefer Bash and Read File. Write File appears regularly for some models, while Edit File, Glob, and Grep are lower-volume.
The available skills rely on shell commands in their instructions and to run their scripts. The tools also often encourage models to read files through the harness.
The skills do not directly emphasize Edit File, Write File, Glob, or Grep. Those tools still appear in traces, but less consistently than Bash and Read File. Grep is not well-suited to the binary format of .docx, .pptx, and .xlsx files. Likewise, Edit is useful for text-based files such as .md files, not for these filetypes.
Average skill invocations per item, across the three skills available to the agent.
Skill usage is dominated by docx, followed by xlsx. pptx is used less often, but it is not absent.
Methodology
We use Harvey’s generation and grading protocol in the same environment, with internet access disabled.
Harvey grades each submission with two LLM judges. Each judge computes a task pass rate. A task passes only if 100% of its criteria pass, and Harvey’s final score is the average of the two judge task pass rates.
The two judges were GPT 5.5 and Claude Sonnet 4.6. GPT 5.5 used medium reasoning and Claude Sonnet 4.6 was not modified.
While running the benchmark, we found that redline criteria needed DOCX tracked changes preserved when reading submitted files so judges could see inserted and deleted text. We fixed that bug and merged it upstream in harveyai/harvey-labs#76. The scoring rubric is unchanged.
The benchmark was modified to use our model library, an abstraction over various LLM provider APIs, and to run on Valkyrie, our framework for running agentic benchmarks. These are infrastructure changes and do not impact model performance.
To improve judge performance and reduce cost, we split the instruction prompt provided to each judge so common elements could be cached. This did not modify prompt content outside of caching.
