🔍 Read the full analysis: We Tested 42 AI Setups With 15 Questions About Law on ThorstenMeyerAI.com
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
TL;DR
Thorsten Meyer AI reports that its Lexassay test gave 42 AI setups the same 15 questions on German, U.S. and EU law. The top score was 79 out of 100, while legal-focused GLM-5.3 FlashX Legal scored 8; the test author says its one-run-per-question design and Claude-based grading limit what the results can show.
Thorsten Meyer AI says its Lexassay test put 42 AI setups through 15 questions about German, U.S. and EU law, with results ranging from 8 to 79 out of 100. The report found that GLM-5.3 FlashX Legal, which its maker Z.ai describes as optimized for legal workflows, scored 8; the findings matter because the questions include consequential tasks such as calculating court deadlines, while the test also carries limitations and a disclosed grading conflict.
Lexassay’s leading setup, Claude Opus 5.5, scored 79, followed by GPT-6 Astra at 72 and GPT-6.1 Sol at 71. The report says even the top scorer fully met every criterion on only eight of the 10 public tasks. It also says the legal-focused GLM model scored zero on the hidden set and gave no answer on two questions. Its base model scored 3, according to the report.
The author says the test covered legal deadlines, recent changes in law, leading court decisions and questions containing false premises. Models answered without internet access or tools. Each answer was assessed against a prepared answer with three to six weighted criteria, and citations were checked against official databases. In the published runs, the report says it confirmed 2,644 citations as real and identified 804 that were invented or did not support the model’s stated claim.
The test’s example of a deadline question involves two German civil cases with judgments served on April 27, 2027. The report says the one-month appeal deadline falls on May 28 in Munich because May 27 is a public holiday in Bavaria, but remains May 27 in Berlin. It says most strong models answered correctly, while one put Berlin’s deadline a day late and another placed the deadline for appeal grounds a month too late. These are findings reported by Lexassay, not an independent audit of the models.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Why Legal Answers Need Checking
The results highlight a gap between fluent legal responses and answers that are accurate enough to rely on. A missed court deadline can affect whether an appeal is accepted, while an unsupported citation can make a confident-sounding answer misleading. The test author advises readers to treat AI as a starting point, verify cited laws and judgments in the original sources, and consult a lawyer about deadlines, money or dealings with an authority.
The report also cautions against treating its rankings as a definitive guide to legal AI. The test measures models answering from their own internal knowledge, rather than systems that can retrieve and work from supplied legal documents. Lexassay’s results therefore speak to a particular use case: a chatbot responding without research tools. They do not establish how the same models would perform in a lawyer’s document-based workflow.
Popular right now
Ninja Foodi 10-Qt DualZone Air Fryer, 6-in-1, Black (DZ302)
iRobot Roomba 105 Robot Vacuum with LiDAR Navigation
De'Longhi Dinamica Plus Fully Automatic Espresso Machine
As an affiliate, we earn on qualifying purchases.
How Lexassay Scored the Models
The 15 questions comprise 10 published tasks and five hidden tasks. Meyer says the hidden questions are intended to make it harder for future models to have encountered the answers online; only their score is published. Different effort settings, which control how long a model may reason before answering, were tested separately, so the 42 setups do not necessarily represent 42 distinct underlying models.
Lexassay deducts points when citations are invented or misattributed, and applies a five-percent penalty to an answer that lacks a “not legal advice” note or a pointer to a lawyer. The author calls the exercise a spot check rather than a strict benchmark: each setup answered each task once, meaning small score differences may reflect noise. The test is run by Thorsten Meyer AI, which also operates Lexassay.
That relationship matters when reading the results: Claude agents graded the answers, and a Claude model ranked first. Meyer disclosed this conflict and says the method page and public answers, gold answers and citation checks are available for scrutiny. The source also cites a different legal-agent benchmark from Vals AI, where models work with supplied documents; it says that benchmark ranks some of the same systems differently. The two tests measure different setups and should not be read as interchangeable rankings.
“Optimized for professional legal workflows, with a focus on document understanding, legal reasoning, and task execution.”
— Z.ai, in developer documentation as quoted by Thorsten Meyer AI
Limits Behind the Scores
The results do not show how the models perform across repeated trials, since the author says there was one answer per setup and task. The test also excludes internet access and tools, and its grading relies on Claude agents despite the disclosure that a Claude model ranked first. The source says the public answers and grading materials can be checked, but it does not provide an independent replication in the material supplied here.
It is also unclear from the source how scores would change with retrieval systems, updated model versions, different prompts or professional review. The report’s citation totals cover its published runs; they are not presented as error rates for every legal AI use. The Vals AI comparison uses supplied documents and a different benchmark, so the differing rankings do not by themselves establish which test better predicts real-world legal work.
What Readers Should Verify
The source does not announce a date for a follow-up test or a new evaluation. Readers can review the Lexassay method page and its published answers, gold answers and citation checks, which Meyer says are available. Any later comparison would be more informative if it reported repeated runs and made clear whether models used search, legal databases or supplied documents.
For an immediate legal question, the report’s practical recommendation is to check each cited statute or ruling in its original source and seek advice from a lawyer when a deadline, financial claim or government authority is involved. Lexassay says it is testing AI systems, not providing legal advice.
Key Questions
What did the Lexassay test find?
Thorsten Meyer AI reports scores from 8 to 79 out of 100 across 42 AI setups answering 15 legal questions. Claude Opus 5.5 led at 79, while GLM-5.3 FlashX Legal scored 8. These are results from the test’s specific no-tools setup, not a general measure of legal competence.
Does the test show which AI is best for legal work?
No. The author describes Lexassay as a spot check, with one answer per setup and task and no internet or tools. A separate legal-agent benchmark uses supplied documents and, according to the report, produces different rankings.
Why is the grading conflict relevant?
Thorsten Meyer AI runs Lexassay, and Claude agents graded the answers; a Claude model also ranked first. The author disclosed that conflict and says the method and public grading materials are available, but the relationship is relevant when readers assess the results.
Can I rely on a chatbot for a legal deadline?
The report advises against taking a deadline from a chatbot. It recommends checking legal sources and contacting a lawyer when a deadline, money or an authority is involved. The test is not legal advice.
Source: ThorstenMeyerAI.com
