🔍 Read the full analysis: What GPT-6 Astra’s AI Repair Says About Editing Model Weights on ThorstenMeyerAI.com
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
TL;DR
A November 2, 2026 blog post reports that GPT-6 Astra found and removed a direction in two layers of Qwen3.5-0.8B, improving its MATH-500 score from 7.9% to 48.1% without gradient descent. The result is an early, non-peer-reviewed experiment: 33.1% of edited-model answers still reached the 32,768-token limit, and the search used substantial testing time and compute.
A team of four researchers listed with MIT CSAIL says it used OpenAI’s GPT-6 Astra to edit two layers of the open-weight model Qwen3.5-0.8B, raising its reported MATH-500 score from 7.9% to 48.1% without gradient descent. The result, published November 2 in a blog post with logs and an edited model, addresses a known repetition problem—but it is an early, non-peer-reviewed experiment, and 33.1% of the revised model’s answers still hit the token limit.
The authors—Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab—say GPT-6 Astra traced the looping behavior to an internal direction associated with the MLPs in layers 15 and 19. It subtracted that direction from 64 output columns in each layer, changing 131,072 weights, or about 0.016% of the model’s 0.8 billion weights. The researchers’ Hugging Face upload shows differences in the two layers’ down-projection tensors, according to the source account.
The reported MATH-500 results rose with the search budget: scores were 12.3% after 10 forward passes, 14.1% after 25, and 48.1% after 50. The share of answers reaching the 32,768-token limit fell from 93.0% for the base model to 87.2%, 86.4% and 33.1%, respectively. On GSM8K, the reported score reached 67.1% after 50 passes, compared with 13.8% for the base model; on IFEval, it reached 62.3%, compared with 21.0%.
“Without training” in the post means the researchers prohibited gradient descent, backward passes and fitting weights to a loss. It does not mean the change was made without testing or iteration. The winning run lasted 20.3 hours and involved 158 turns, 80 Python runs and 50 forward passes, each of which could evaluate up to 512 problems. The researchers supplied practice problems but withheld the final benchmark sets from Astra. They also report that the 10- and 25-pass runs produced much smaller gains.
A Targeted Fix, Not New Math Ability
The result is relevant to organizations that run open-weight models because it suggests a model’s behavior might sometimes be changed by inspecting and editing a small part of its weights, rather than retraining the full system. If such methods prove reliable, they could offer a way to address specific failure patterns in deployed models. This report does not establish that the method works broadly, however: it covers a particular small model and a particular looping problem.
The distinction between repair and capability gain matters. The Qwen model card already warns about thinking loops, and the authors say that “fixing loops, however, is not the same as making a model better at math.” A substantial part of the reported score increase may reflect the model finishing answers that previously ran into the token cap. The remaining 33.1% rate shows the loop was reduced, not eliminated.
The method also places weight editing in a safety-sensitive area. The post points to prior research describing single-direction edits that can remove refusal behavior from open chat models. That does not mean the Qwen experiment removed safeguards; it does show that localized weight changes can potentially alter behavior in ways that operators would need to evaluate before adopting a patch.
Popular right now
Breville BES870XL Barista Express Espresso Machine, Brushed Stainless Steel
Apple AirPods Pro 3 Wireless Earbuds with Active Noise Cancellation
Ninja Crispi Glass Air Fryer (Stone) – 4‑in‑1
As an affiliate, we earn on qualifying purchases.
How the Weight Search Worked
The work appears in a blog post titled “Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights,” published November 2, 2026. The authors list MIT CSAIL affiliations and describe the results as “very early”; the post is not a peer-reviewed paper and is not on arXiv, according to the source account. It describes eight experiments, including attempts that did not succeed. In one, Astra could not identify which of two fine-tuned Gemma models had been trained to flatter a user.
For the Qwen experiment, Astra was run as gpt-6-astra through the Codex command-line tool at its highest reasoning setting, according to the published logs. The team disabled built-in Codex tools and provided a custom Python environment to inspect and modify the 24-layer model. The setup was CPU-only and had no internet access. Astra could not run the model directly; each forward pass let the researchers observe responses to a batch of problems or texts.
The researchers supplied 7,469 GSM8K and 7,275 MATH practice problems, while withholding the evaluation sets. The source account says OpenAI was not listed as an author, was not thanked in the post, and had made no statement about the experiment. The post’s results should therefore be read as the researchers’ report, not as an independently endorsed OpenAI finding.
““Fixing loops, however, is not the same as making a model better at math.””
— The researchers, as quoted in the November 2 blog post
Limits of the Reported Evidence
The benchmark figures come from the researchers’ blog post and published logs, not a peer-reviewed paper. The source account reports no independent replication, and the experiment does not show whether the edit generalizes to other model sizes, architectures, or tasks. The gains on three evaluations are reported for this model and setup only.
It is also unclear how robust the edited model is under different prompts, sampling settings, or longer evaluation runs. The source describes identical sampling settings for the base and edited MATH-500 comparison, but does not establish how often the edited weights might affect unrelated behavior. Nor does the reported reduction in looping explain whether every failure case shares the same internal cause.
The phrase “without training” can obscure the resources involved: the edit did not use gradient descent, but the winning search used 20.3 hours and 50 forward passes, while smaller search budgets delivered far less improvement. The source does not establish the cost or reproducibility of applying this procedure to larger models. OpenAI’s role beyond providing the model used for the experiment is also not described as a collaboration.
Replication and Broader Model Tests
The next meaningful test is whether independent researchers can reproduce the edit and its benchmark results using the published logs, weight files and procedure. Further evaluations would need to check whether the lower loop rate persists across prompts and tasks, and whether the change has unwanted effects on other model behaviors.
The authors’ post presents the work as early research rather than a finished tool or a general replacement for training. The source material does not give a schedule for a peer-reviewed paper, follow-up experiments or an OpenAI response. For now, readers and model operators should treat the numerical gains as reported results awaiting wider validation, not as a demonstrated general-purpose way to improve model weights.
Key Questions
What did GPT-6 Astra change in Qwen3.5-0.8B?
According to the researchers’ post, Astra identified an internal direction linked to repetitive answers and subtracted it from 64 output columns in each of layers 15 and 19. The reported edit changed 131,072 weights.
Did the model stop looping completely?
No. The reported share of MATH-500 answers reaching the 32,768-token limit fell from 93.0% to 33.1% after the 50-pass edit, so the problem was reduced but remained.
Was this done without training?
The researchers prohibited gradient descent, backward passes and fitting weights to a loss. But the process still used iterative testing and took 20.3 hours in the winning run. “Without training” does not mean without computation or experimentation.
Has the result been peer-reviewed?
No peer-reviewed publication is identified in the source material. The findings were published as a blog post with logs, and the authors called them very early results.
Does this show that GPT-6 Astra can improve any model?
No. The report documents one successful edit to a small Qwen model, alongside experiments with more limited or unsuccessful outcomes. It does not establish that the method generalizes to other models or problems.
Source: ThorstenMeyerAI.com
