🔍 Read the full analysis: Z.ai’s Legal AI Pass Rate And Ranking: What The Benchmark Shows on ThorstenMeyerAI.com
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal recorded a 19.17% all-criteria pass rate on Vals AI’s version of Harvey’s Legal Agent Benchmark, placing seventh among 81 models in the archived leaderboard. It passed 92.5% of individual criteria, but the benchmark counts a task as complete only when every criterion is met. The score is independently listed by Vals AI; Z.ai’s characterization of the result as state of the art among open-source models has an unspecified comparison set.
Z.ai’s GLM-5.3-FlashX-Legal achieved a 19.17% all-criteria pass rate on Vals AI’s leaderboard for Harvey’s Legal Agent Benchmark, ranking seventh among 81 models in the archived results. The score means the model met every required criterion on about one in five benchmark tasks, even though it passed 92.5% of individual criteria; the distinction matters because one missed requirement makes an entire task a failure under Harvey’s grading rule.
Vals AI’s archived Version 1 leaderboard, last updated October 18, 2026, lists GLM Legal at 19.17% and a cost of $0.15 per test. Z.ai’s October 13 blog post describes the result as 19.2% and identifies Vals as the independent evaluator. The model is built on GLM-5.3-Flash, according to the company’s announcement.
The leaderboard also lists the GLM-5.3-Flash base model at 6.67% all-pass and $0.57 per test. The difference between the two displayed scores is 12.50 percentage points, matching Z.ai’s stated improvement. The legal model’s 92.5% criterion pass rate is a separate measure: it counts individual rubric requirements, not fully completed tasks.
Z.ai’s developer documentation describes the model as tuned for legal document understanding, reasoning and task execution, including research, contract review and drafting. The company says it underwent legal-domain mid-training and post-training, with reinforcement learning in tens of thousands of task environments modeled on legal workflows. Those are company descriptions of its development process, not independent findings about performance beyond the published benchmark results.
Why the All-Pass Score Matters
The gap between 92.5% of criteria passed and 19.17% of tasks completed illustrates how scoring rules shape the meaning of a legal AI benchmark. A model may handle many details correctly across a matter yet fail the task because it misses one required deadline, citation, recommendation or other rubric item. For legal work, that distinction is consequential: a nearly complete document can still contain an omission that requires a lawyer to catch and correct.
The result is relevant to firms considering AI for research, review or drafting, but it is not evidence that the model can safely handle unsupervised legal work. Harvey’s benchmark tests work products against detailed criteria on a private set of tasks. It does not, by itself, establish how the model performs across every jurisdiction, matter type, firm workflow or real client file. Procurement decisions also involve questions that a pass-rate figure cannot answer, including how client data is handled and who verifies citations and conclusions.
Cost is another point of comparison, not a complete measure of value. Vals lists GLM Legal at a lower cost per test than several higher-ranked entries, but the source material does not establish that costs are directly comparable across all usage conditions or that benchmark performance translates into a particular firm’s savings.
Popular right now
Shark HydroVac MessMaster Corded Wet Dry Vacuum and Mop (WD161)
Dreame G10 Pro Wet and Dry Vacuum Mop
Breville BES870XL Barista Express Espresso Machine, Brushed Stainless Steel
As an affiliate, we earn on qualifying purchases.
How Harvey Grades Legal Tasks
Harvey released its Legal Agent Benchmark as open source on May 6, 2026. The benchmark contains 1,250 tasks across 24 practice areas. Each task gives an agent an instruction and a client matter with relevant materials, then asks it to produce work for review. More than 75,000 expert-written rubric criteria cover items such as facts, conclusions, citations, deadlines, dollar amounts and formatting.
Under Harvey’s all-pass grading, every criterion must pass for the task to count as complete. The public leaderboard uses a private holdout set. Vals says its holdout contains 120 tasks, with internet access disabled; two language-model judges grade each submission, and the score averages their task pass rates. This procedure offers a consistent comparison under the benchmark’s rules, while still making the score dependent on the selected tasks, rubric and judging system.
Harvey’s own May 26 baseline reported a 7.1% all-pass rate as the best result among the frontier models it tested in that run. Harvey said the models completed fewer than 10% of tasks end to end in aggregate. That earlier result provides context for the benchmark’s difficulty, but it is a different run and should not be treated as a direct comparison with the later Vals leaderboard.
“A task is marked complete only if every criterion passes.”
— Harvey, describing the benchmark’s scoring rule
Limits of the Published Results
Z.ai calls the result state of the art among open-source models, but the source material does not identify the comparison set behind that claim. It also does not establish that the legal model’s weights are publicly available; the article’s source author says they found no published weights. The benchmark ranking should not be taken as confirmation that the model qualifies as open source under a particular definition.
The displayed results are tied to a specific leaderboard version and evaluation procedure. The Vals archive lists 19.17%, while Z.ai rounds the score to 19.2%. The documentation page is undated, and the source material identifies the October 13 blog post and October 12 Vals release listing as the dates associated with the announcement. It is not clear from the supplied information whether later leaderboard versions or model updates have changed the ranking.
A source author also reports a four-task spot check in which the legal model cited rulings the author could not find. That small check is not a benchmark and does not establish the model’s general error rate. The supplied material does not include enough detail to independently evaluate those examples. More broadly, the reported benchmark does not settle whether citations, legal reasoning or data protections will meet the requirements of a particular firm or jurisdiction.
What Firms Should Verify
The immediate next step for readers comparing models is to consult the specific Vals leaderboard version and its evaluation details, rather than relying only on Z.ai’s rounded announcement figure or its state-of-the-art description. A current ranking, if available, should be checked separately because the figures cited here come from an archive updated October 18, 2026.
Firms evaluating the system will need to test it on work suited to their own practice and require human review of outputs, especially citations, deadlines and legal conclusions. They will also need to establish how client materials are processed, what contractual and security protections apply, and which staff members are responsible for checking drafts before they are used or shared. Z.ai’s documentation recommends connecting the model to services for laws, cases and company data, but the supplied information does not detail the controls or terms for those integrations.
No further benchmark update or independent audit is specified in the source material. Until one is published, the confirmed picture remains the archived Vals score, its all-or-nothing grading rule and the limits of what that evaluation can establish.
Key Questions
What does GLM Legal’s 19.17% score mean?
It is the share of benchmark tasks on which the model met every required criterion, averaged across the evaluation judges. Missing even one criterion makes a task fail under Harvey’s all-pass rule.
How can the model pass 92.5% of criteria but only 19.17% of tasks?
The measures use different units. The 92.5% figure counts individual rubric criteria; the 19.17% figure counts whole tasks that had no failed criteria. A task can contain many passed criteria and still fail overall.
Where did the seventh-place ranking come from?
Vals AI’s archived Version 1 leaderboard lists GLM Legal seventh among 81 models, with a 19.17% pass rate. The archive was last updated October 18, 2026.
Does the benchmark prove the model is ready to provide legal advice?
No. The result measures performance on a defined set of benchmark tasks. It does not establish that the model can work without lawyer review or meet the requirements of every firm, matter or jurisdiction.
Is Z.ai’s open-source performance claim independently established?
Vals independently lists the benchmark score, but Z.ai’s claim that it is state of the art among open-source models uses a comparison set that is not specified in the supplied material. The source material also does not confirm that the legal model’s weights are publicly available.
Source: ThorstenMeyerAI.com
