📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.

Italy’s Minerva-3B, a large-scale sovereign language model trained entirely from scratch on 2.5 trillion tokens, scored just 4.9% on the INVALSI Italian school-exam benchmark, despite high technical performance. This development highlights the complex challenge of producing truly country-specific language understanding through scale alone, and raises questions about the investment needed for meaningful linguistic and knowledge depth.

Minerva was developed by Sapienza University of Rome’s NLP group, led by Roberto Navigli, with support from Italy’s national supercomputing consortium CINECA and funding from Italy’s PNRR. The project trained models ranging from 350 million to 7 billion parameters, with approximately half of the training data in Italian, totaling 2.5 trillion tokens. The models outperform comparable multilingual models on Italian benchmarks, demonstrating technical success in scale and language specificity.

However, despite this technical achievement, Minerva-3B scored only 4.9% on the INVALSI Italian school-exam benchmark, a result considered near chance. Researchers concluded that dataset size and parameter count, while important, are insufficient alone to handle complex language tasks, especially in academic content. This finding suggests that the current scale may still be inadequate for deep country-specific language understanding, challenging assumptions about the sufficiency of scale in sovereign-LLM projects.

Implications of Minerva’s Low Exam Score for Sovereign LLM Strategies

The low performance of Minerva-3B on the Italian academic benchmark underscores a key challenge for European sovereign LLM initiatives: scaling up native-language data and parameters alone may not suffice to develop models with deep country-specific knowledge. This finding questions the assumption that larger models trained on more data automatically translate into better language understanding, especially for complex, domain-specific tasks. It suggests that future efforts must consider more nuanced approaches to data quality, diversity, and targeted training to achieve meaningful language and knowledge depth, impacting how European nations plan their AI sovereignty strategies.

The European Sovereign LLM Debate and Italy’s Approach

Italy’s Minerva project emerged as a counterpoint to the broader European debate over sovereign LLM development, which often centers on whether to train models from scratch or adapt existing multilingual models through continuation pre-training. Unlike Portugal’s AMÁLIA project, which layered European Portuguese onto a multilingual foundation, Italy chose to build Minerva entirely from scratch, using a large, dedicated Italian dataset and significant computational resources. The project was publicly transparent, releasing weights, data, and code, and was supported by Italy’s national AI strategy and infrastructure investments. Prior to the recent exam results, Minerva was considered a technical success, outperforming comparable models on Italian benchmarks, but the exam score revealed deeper limitations.

Historically, European sovereign-LLM projects have debated the trade-offs between scale, data specificity, and resource investment. Italy’s approach aimed to demonstrate the feasibility of a fully native-language model, but the recent performance data complicates the narrative, suggesting that even large-scale, native-language training may not automatically produce the desired country-specific knowledge depth.

“Minerva’s low exam score reveals the structural challenge facing European sovereign-LLMs: how much native-language investment is truly needed?”

— Thorsten Meyer

Unanswered Questions About Scaling and Knowledge Depth

It remains unclear how different training methodologies, data quality, and model architectures might influence the ability of sovereign LLMs to develop deep country-specific knowledge. The recent Minerva results suggest that simply increasing scale may not be enough, but the optimal strategies for achieving meaningful language and domain understanding are still under investigation. Additionally, the long-term implications of these findings for European AI sovereignty efforts are not yet fully understood, and ongoing research is needed to clarify these issues.

Future Research and Model Iterations in European Sovereign AI

Researchers involved in the Minerva project plan to continue iterating on their models, including ongoing experiments with continual training and data refinement. The team has indicated that subsequent versions may address some of the current limitations, aiming to improve performance on complex tasks like academic assessments. Meanwhile, policymakers and AI strategists are likely to reassess investment levels and methodological approaches, emphasizing the need for more targeted, quality-focused data collection and training strategies to achieve deeper country-specific language understanding.

Key Questions

Why did Minerva score so low on the Italian school exams?

The low score suggests that scale and data quantity alone are insufficient for developing deep, country-specific knowledge, especially in complex academic domains. It indicates a need for more targeted training approaches.

Does this mean European sovereign LLMs are not worth pursuing?

Not necessarily. It highlights that current approaches may need refinement. Larger scale alone may not produce the desired knowledge depth, but strategic improvements could still make sovereign models valuable.

What are the next steps for Italy’s Minerva project?

The team plans to continue model development, including experiments with continual training and data quality improvements, to enhance performance on complex tasks.

How does this impact the European AI sovereignty debate?

It emphasizes the importance of considering scale, data quality, and training methodology, potentially prompting a reassessment of current strategies and investments.

Source: ThorstenMeyerAI.com