TL;DR
- Checking whether a student’s math answer is right or wrong is largely solved — most frontier and several open-weight models exceed 90% accuracy on that task. Identifying why a wrong answer is wrong is not: the best of eight models benchmarked here reached only 73% accuracy on misconception detection, with 10–24% false-positive rates.
- Fine-tuning on roughly 3,400 annotated student workings improved every open-weight model tested — gains of 12–40% relative to their baseline — meaningfully narrowing the gap with closed models without large-scale data or compute.
- The annotated dataset keeps growing (50,000–100,000 new student workings a month), and further human annotation and testing are planned to see how far fine-tuned accuracy can go, and what threshold is needed for real classroom use.
- We thank Thinking Machines for supporting our experiment through the Tinker Research Grant.
Automated grading tools typically check whether a final answer is correct. That leaves out information that matters for instruction: where in a student’s reasoning something went wrong. Two students can arrive at the same incorrect answer for different reasons — one through a simple arithmetic slip, another through a genuine misunderstanding of the underlying concept — and each calls for a different response.
Detecting that distinction reliably is a harder problem than answer-checking — but it matters more for building critical thinking at scale. Most frontier models, and a number of open-weight models, already exceed 90% accuracy at identifying whether a student’s answer is right or wrong. Identifying why it’s wrong is a different task, and one that no model tested here can yet do reliably.
A gap remains before classroom-ready accuracy
A set of 400 annotated student workings on math problems was used to evaluate eight models — a mix of closed, proprietary systems and open-weight models — on their ability to identify the specific misconception in an incorrect answer. All models were given the same frozen, pre-extracted text, so differences reflect reasoning ability rather than variation in text extraction. The highest-scoring model, Gemini 3.5 Flash, correctly identified the misconception in 73% of cases, with a 10% false-positive rate. The remaining models scored lower, including the three open-weight models tested (Nemotron Ultra 550B, Gemma 4 31B, and Nemotron Super 120B), which ranged from 45% to 59% before any additional training.
None of the eight models reached a level of accuracy that would be considered reliable for classroom use. A model that misidentifies the source of error in a quarter or more of cases risks directing teacher s and students toward the wrong kind of help, and false-positive rates of 10–24% mean a non-trivial share of correct reasoning would be flagged as incorrect.
Proof of concept: fine-tuning open-weight models
Can you ‘teach’ a model by providing it good amount of genuine student workings annotated with misconception and errors? To begin to have some answer to this question, we fine-tuned three open-weight models, Nemotron Ultra 550B, Gemma 4 31B and Nemotron Super 120B on our own annotated data about 3,400 labeled student workings obtained from our Practice feature available to our live cohort-based classes.
The result is promising and provides a plausible path towards scaling open-ended assessments beyond a handful of well-resourced schools. Every open-weight model improved, and by a lot: Nemotron Super 120B went from 45% to 63% (a 40% relative gain), Gemma 4 31B from 52% to 64% (up 23%), and Nemotron Ultra 550B from 59% to 66% (up 12%). False-positive rates fell in every case too, most sharply for Gemma 4 31B (24% down to 11%) and Nemotron Super 120B (19% down to 9%).
Support from Thinking Machine
All fine-tuning experiments in this work were run on Tinker, Thinking Machines‘ fine-tuning API. It allowed us to fine-tune open models ranging from 27B to 550B parameters without managing any GPU infrastructure. This work was supported by Tinker Research Grant. Our thanks to the Thinking Machines team for making these experiments possible.
What's next?
These early fine-tuning results offer an important signal: reliable misconception-detection technology may not require frontier-model-scale investment. Fine-tuning existing open-weight models — even on a comparatively modest, domain-specific dataset — already produced meaningful gains, pointing to a lower-cost path toward closing the gap with frontier models. Here are the next steps:
- Full expert annotation. Move from the ~3,400 expert-checked examples used in this benchmark to fully expert-annotated evaluation and fine-tuning datasets, built on genuine student workings rather than partial or synthetic coverage, so the training signal reflects how students actually reason.
- Independently credible benchmarks. Partner with established educational institutions to construct and validate evaluation datasets whose results can be trusted as an independent measure of model capability, rather than relying on benchmarks built and scored in-house.
- Real-world deployment barriers. Identify the physical and technological constraints — device access, connectivity, integration into existing teacher workflows — that would need to be resolved before misconception detection could actually be used in a formal classroom setting.
