Same AI, Opposite Outcomes: The Design Choice Nobody's Talking About
September 10, 2026 · Talon Tutoring TeamA study published in PNAS in 2025 gave roughly 1,000 high school students across 50 classrooms in Turkey access to AI while practicing math — then took it away for the final exam. What happened in between is the clearest evidence yet for a design principle Talon has been built around from the start: the AI matters less than what it's allowed to give a student, and when.
Students were split into three groups: no AI, unrestricted ChatGPT, and the same underlying model reconfigured to tutor — meaning it withheld the answer until the student reasoned through the step, instead of providing it on request.
The same model produced opposite results
With AI available during practice, both groups looked good compared to the no-AI group: unrestricted ChatGPT scored 48% higher, the tutor-built version 127% higher. Then the AI was removed for the final exam. Unrestricted ChatGPT students scored 17% lower than the no-AI group. The tutor-built group showed no change at all.
Same weights, same training data, same students. The only thing that moved was what the model was allowed to give — and when it was allowed to give it.
This is cognitive offloading, not learning
When a model will just solve the problem, a student stops doing the part of the work that actually builds the skill — struggling with it. Practice scores rise because the AI is doing the practicing. The exam is the moment that debt comes due.
The tutor-built model produces the opposite pattern for the plainest reason: it never lets the student skip the reasoning step in the first place, so there's no debt to collect on exam day.
The pattern Talon is built around
This is the same instinct behind every mode in Talon's tutor workspace, not a new one adopted after reading a study. Homework help gives hints and checks reasoning step by step instead of finishing the problem — see our parent's guide to what that should look like. Dialogical learning is built entirely around the same withhold-the-answer principle the trial's better-performing group used: the tutor asks a guiding question so the student has to commit to a position, rather than handing over one to copy.
It extends to how a session ends, too. Before closing out a Socratic conversation, Talon's tutor asks the student to state the core idea in their own words or solve one last check problem unaided — the same reasoning-first habit the trial's tutor-built model used, applied to the moment a session is easiest to rush through.
The question worth asking any AI tutor
Every family evaluating an AI tutor is really answering one question, whether they realize it or not: when the tool doesn't know the answer to a problem it's never seen, does it teach, or does it solve? If nobody can answer that with a specific example, the tool hasn't been evaluated — it's been assumed.
We've written before about the guardrails that should hold up under that question. This trial is the clearest data yet for why they're not a nice-to-have: the same model, pointed at the same students, produced a 145-point swing in outcomes depending on the answer.