How To Evaluate AI Models For A News Pipeline: Lessons From 71 Tests

🔍 Read the full analysis: How To Evaluate AI Models For A News Pipeline: Lessons From 71 Tests on ThorstenMeyerAI.com

Age 18–24?Offer from Amazon

Prime made for students and young adults

• Fast, free delivery for dorm and study essentials
• Prime Video and Amazon Music included
• Member-only deals

Try Prime for Young Adults →Free trial for eligible 18–24 year olds

As an affiliate, we earn on qualifying purchases.Hide

Thorsten Meyer says a test of 71 AI model configurations for producing news articles found that GPT-6 Astra scored highest, but GPT-6 Luna was chosen for production because it cost far less while meeting his quality needs. The October 13 report also says two models in the site’s backup chain produced fabricated quotes or unusable output and were removed on October 7.

Meyer tested models using the production prompt and real news sources, generating English and German editions. The standard test used one long source of 6,340 characters and one short source of 619 characters; he says short sources are common in his workflow. Two AI systems, GPT-6.1 Sol and Claude Opus 5.5, judged the articles on source fidelity, fabrication, clarity about uncertainty, news value, structure and polish, with language quality also scored for German.

GPT-6 Astra scored 4.00 out of 5 in both languages in the initial test, with a source-fidelity score of 5.00. In a follow-up robustness run covering 12 articles, it scored 3.93, with a reported 95% bootstrap interval of 3.83–4.02. Meyer put its cost at about $0.12 per article, compared with roughly $0.0012 for GPT-6 Luna, and said Astra took about 50 seconds per article versus 17 seconds for Luna.

Luna scored 3.67 in English and 3.68 in German in the initial results. Its robustness-run score was 3.67, and its source-fidelity score was 4.71 out of 5; Meyer said all its articles in that run produced valid output. The report says Luna cost about $0.0013 per article in production during the previous week. Meyer selected it for routine use, saying the goal was to choose the least expensive model that reliably clears a quality threshold.

Why the Top Score Was Not Enough

The results highlight a practical trade-off for publishers automating routine coverage: the highest-scoring model may not be the production choice when per-article costs and speed matter. Meyer reports Astra performed better, but Luna’s much lower reported price makes it more practical for a high-volume pipeline, provided its output meets the publisher’s standards.

The test also points to a less visible risk in automated publishing: fallback models can affect accuracy when a primary service is unavailable. Meyer says his two former backups produced invented quotations or failed the pipeline’s output checks. Removing them may reduce the chance of those specific failures, though the report does not establish how often either model would have been used in live production.

For readers, fabricated quotations are not merely a formatting problem. They can make an article attribute statements to people who did not say them. The report’s findings support evaluating each model on the actual editorial task and checking failover routes, rather than relying on general-purpose rankings alone.

How the News-Writing Test Worked

Meyer describes the work as a task-specific evaluation, not a general ranking of AI capability. Each configuration received the same real-source news-writing task and production prompt, with articles prepared for English and German editions. The test included API, subscription, free-tier and locally run open-weight models.

The initial evaluation used only two source documents, so Meyer cautioned against treating small score gaps as meaningful: he regarded differences below about 0.3 points as possible noise. He said the follow-up run, which tested leading configurations on 10 additional real sources—mostly short ones—provided a more reliable ranking. Articles counted only if they produced valid output after the same repair step used in production.

One additional comparison tested GPT-6 Luna at six reasoning-effort settings. Meyer reported no reliable improvement from higher effort: the maximum setting cost about 12 times more and took about 12 times as long as the no-effort setting, while score differences stayed within the test’s uncertainty. He kept the no-effort setting in production.

“The right model is the cheapest one that clears your quality bar reliably, not the one at the top of the chart.”

— Thorsten Meyer, in the October 13 report

Limits of the Reported Results

The results come from Meyer’s own evaluation, rather than an independently published or peer-reviewed study. The report describes two sources in the standard test and a 12-article robustness run, but it does not provide enough information here to assess how representative those documents are of all topics, source types or newsroom workflows. The scores also rely on two AI judges, whose evaluations may not match those of human editors or readers.

It is not clear from the report how the models would perform across a larger, independently selected source set, or how often the removed backups would have been called in live operation. The report identifies fabricated-quote examples, but does not quantify the frequency of that problem across all 71 configurations. Its cost figures are tied to Meyer’s stated test and production conditions and may vary with usage, pricing or prompt changes.

Production Use and Further Testing

Meyer says GPT-6 Luna remains the production model for the news route, with reasoning effort set to none. GPT-6 Astra was not selected for routine use because of its reported cost, despite its higher test score. The two named backup models were removed on October 7 after the evaluation.

The report does not specify a schedule for another test or name a replacement failover model. A useful next step would be to test candidates across a larger and more varied set of real news sources, then check both article quality and output validity under the same production conditions. Until those results are available, the reported ranking should be read as a finding about one publisher’s workflow, not a universal model league table.

Key Questions

Which model scored highest in the test?

GPT-6 Astra scored 4.00 out of 5 in both English and German in the initial test. It scored 3.93 across 12 articles in the follow-up robustness run.

Why did Meyer choose GPT-6 Luna for production?

Meyer reports that Luna scored 3.67 in the robustness run and produced valid articles throughout it, while costing about $0.0013 per article in recent production. Astra scored higher but cost about $0.12 per article in the test.

Did higher reasoning effort improve the articles?

Not reliably in this evaluation. Meyer says score differences across six GPT-6 Luna effort settings were within the test’s noise, while the maximum setting cost and took about 12 times more than the no-effort setting.

What happened to the former backup models?

Meyer says GPT-4.1 nano was flagged for invented quotations and had a source-fidelity score of 1.00. He says locally run gpt-oss-120b produced no valid articles in four attempts and also generated invented quotes. Both were removed from the news route on October 7.

Do these results establish the best model for every newsroom?

No. The findings reflect one publisher’s prompt, sources, languages and evaluation process. The report itself notes the limited initial source sample, so other newsrooms would need to test models against their own editorial requirements and workflows.

Source: ThorstenMeyerAI.com