The Khan Academy study shows diminishing returns flatten accuracy gains while token costs keep climbing.
Every LLM API call carries a reasoning effort parameter. Most teams never touch it. The default ships at medium or high, and the model thinks longer, generates more tokens, and costs more per call. The assumption is simple: more thinking produces better answers.
That assumption is expensive. And a recent study from Khan Academy puts a number on exactly how expensive.
Scott Frohn tested 26 model-effort-sampling configurations against 900 real student conversations in high school mathematics, measuring both accuracy (Cohen's kappa, a metric that captures how well the model agrees with human scorers) and cost per 1,000 API calls The Impact of LLM Self-Consistency and Reasoning Effort on Automated Scoring Accuracy and Cost, 2026. The result is a cost-accuracy frontier that looks like a hill with a sharp knee: beyond moderate effort, accuracy gains flatten while costs keep climbing.
For anyone running LLM calls at scale, the question is no longer "which model is best." The question is how much reasoning effort your task needs, and whether your current configuration sits on or off that frontier.
The Khan Academy experiment tested models ranging from the cheapest nano-tier to full-size reasoning models, each at multiple effort levels. Out of 26 configurations, five landed on the Pareto frontier, the set of configurations where no other option gives you better accuracy for the same cost or lower cost for the same accuracy. Think of it like a menu where you cross off every dish that is more expensive AND worse-tasting than another dish. What remains is the frontier. Everything else is waste.
Here are the five configurations that survived The Impact of LLM Self-Consistency and Reasoning Effort on Automated Scoring Accuracy and Cost, 2026:
| Model | Reasoning Effort | Cohen's κ | Cost per 1,000 Calls | |---|---|---|---| | GPT-5.4 Nano | Default | 0.750 | $0.27 | | GPT-5.4 Mini | Default | 0.756 | $0.95 | | Gemini 3 Flash Preview | High | 0.771 | $1.91 | | Gemini 3 Flash Preview | Medium | 0.782 | $2.81 | | Gemini 3.1 Pro Preview | Low | 0.794 | $3.88 |
The spread tells the story. Moving from the cheapest configuration to the most accurate one costs 14x more per call. The accuracy gain for that 14x cost increase is 0.044 kappa points, roughly a 6% improvement in human-model agreement.
The configurations that fell off the frontier reveal the pattern. Higher reasoning effort for GPT-5.4 and mid-tier GPT models added cost without moving the accuracy needle The Impact of LLM Self-Consistency and Reasoning Effort on Automated Scoring Accuracy and Cost, 2026. Beyond intermediate effort, each additional reasoning step consumed more tokens but produced answers that were no closer to what a human scorer would give.
Accuracy (κ) 0.80 | ★ Gemini 3.1 Pro (low) | ★ Gemini 3 Flash (medium) 0.78 | | ★ Gemini 3 Flash (high) 0.76 | ★ GPT-5.4 Mini | ★ GPT-5.4 Nano 0.74 | | × GPT-5.4 (high effort) × = off-frontier | × GPT mid-tier (high effort) 0.72 | +------+------+------+------+------→ $0 $1.0 $2.0 $3.0 $4.0 Cost per 1,000 calls
★ = Pareto-optimal (on the frontier) × = Dominated (off the frontier — worse accuracy at equal or higher cost)
The implication for any team running scoring, classification, or evaluation calls at volume: the model doing the most work is not the model giving you the best return per dollar.
Self-consistency is a popular production recipe. Run the same prompt multiple times with temperature sampling turned on, collect the outputs, and pick the majority answer. The logic sounds solid: if three out of five calls agree, the consensus answer is more reliable than any single call.
The Khan Academy data breaks this logic. Temperature sampling itself (introducing randomness into the model's token selection) did improve accuracy compared to deterministic calls, where the model picks the single most likely token every time The Impact of LLM Self-Consistency and Reasoning Effort on Automated Scoring Accuracy and Cost, 2026. But scaling the ensemble from one call to seven calls with majority voting produced no meaningful accuracy gains.
This means the first stochastic call captures the benefit. Every additional parallel call after that multiplies your cost by the number of calls (2x, 3x, 5x, 7x) while the accuracy line stays flat. For a team running 100,000 scoring calls per month at $3 per 1,000, a 5-call ensemble turns a $300 monthly bill into $1,500 with no accuracy improvement to show for it.
The frontier data tells you what is optimal under controlled conditions. The operational problem is that most teams are not running at those conditions and do not know it.
API defaults are the culprit. Without explicitly setting the reasoning effort parameter, teams inherit the provider's default, which is set to medium or high How to Control Reasoning Effort in LLMs: A Practical Guide for SaaS Founders in 2026, 2026. That silent default can add 30 to 50 percent extra token usage and latency per call. And providers change these defaults when they ship model updates, so a configuration that was cost-efficient last month may no longer be.
The problem compounds across model families. OpenAI's reasoning models (GPT-5.5, GPT-5.4 Nano) do not accept a temperature parameter at all. Reasoning effort is controlled via a separate reasoning_effort parameter, and the rest of the sampling behavior is opaque to the user LLM Temperature and Sampling: The Complete 2026 Reference Guide, 2026. Anthropic's Claude 4 Opus and Sonnet default to a moderate-to-high effort mode that can be reduced via a thinking budget. Claude with extended thinking accepts temperature, but the reasoning portion of the output is invariant to the temperature value.
This means every model family has a different knob, a different default, and a different interaction between reasoning effort and sampling. A team that does not audit these parameters on each model update is running on assumptions, not configuration.
| Model | Default Reasoning Effort | Must Set to Low? | Token Overhead if Not Set (%) | Input / Output Pricing (per M tokens) | |---|---|---|---|---| | OpenAI GPT-4o | Medium | Yes, via reasoning_effort: "low" | ~30-40% | Varies by tier | | OpenAI GPT-5.4 Nano | Medium-High (no temperature control) | Yes, via reasoning_effort param | ~30-50% | Lowest tier | | Anthropic Claude 4 Opus | Moderate-High (adaptive thinking) | Reducible via thinking budget | ~30-50% | Premium tier | | Anthropic Claude Sonnet 4.6 | Temperature 1.0, variable top-p | No stable seed available | Variable | Mid tier | | Google Gemini 3.1 Pro | Configurable (low/medium/high) | Yes, explicit param required | ~30-50% | Mid-premium tier | | Google Gemini 3 Flash | Configurable (low/medium/high) | Yes, explicit param required | ~30-40% | Lower-mid tier |
The table makes one thing visible: no two providers handle reasoning effort the same way, and none of them default to the cheapest viable setting.
Model ranking is a distraction for this problem. The Khan Academy frontier shows a consistent directional pattern: low-effort reasoning (a brief deliberation prompt) raised kappa from approximately 0.75 to approximately 0.79 as cost moved from roughly $0.27 to roughly $3.90 per 1,000 calls. Intermediate reasoning levels (kappa around 0.77 to 0.78) sat at roughly $2 to $3 per 1,000 calls The Impact of LLM Self-Consistency and Reasoning Effort on Automated Scoring Accuracy and Cost, 2026.
The question for any team is not "which model scored highest." The question is "what accuracy do I need for this task, and what is the cheapest configuration that gets me there."
A scoring task where kappa of 0.75 is acceptable (the model agrees with human scorers three-quarters of the time on a chance-corrected basis) can run on GPT-5.4 Nano at $0.27 per 1,000 calls. A task that requires kappa of 0.79 needs Gemini 3.1 Pro at low reasoning effort for $3.88 per 1,000 calls. Both are on the frontier. Both are correct choices for their respective accuracy requirements.
The metric that binds the decision is cost-per-correct-answer: total spend on a task family, divided by the number of correct answers Reasoning Effort: Cost vs Quality Benchmarks 2026, 2026. Quality and cost in isolation tell you nothing. The chart that matters is the one where you plot your task's accuracy requirement against the frontier and find the cheapest point that clears it.
The frontier comes from a specific domain: 900 student conversations in high school mathematics, scored against human raters using Cohen's kappa The Impact of LLM Self-Consistency and Reasoning Effort on Automated Scoring Accuracy and Cost, 2026. Kappa is a strong choice for an automated-scoring task because it corrects for chance agreement. But kappa thresholds that are acceptable in educational scoring may not transfer to medical triage, legal review, or code evaluation.
The sample is large enough to establish directional patterns. It is not large enough to guarantee that the exact kappa values hold in other task families. A team running customer support classification at 500,000 calls per month operates at a different error-cost ratio than a team grading math homework.
Separate benchmark work on Olympiad-level math problems highlights the problem from the other end: 30 problems from AIME 2024 showed benchmark saturation where top models exceeded performance ceilings, and all content was in English An Olympiad-Level Math Benchmark for Large Language Models, 2026. A small, English-only benchmark tells you how well a model handles a narrow task, not how the accuracy-cost curve behaves across languages, domains, or production conditions.
The honest takeaway: the frontier's shape (diminishing returns beyond moderate effort, self-consistency providing no accuracy gain) is a strong signal. The exact numbers on the axes need your own calibration experiment. Run your task at three effort levels, measure your accuracy metric, plot the cost, and find your own knee in the curve.
The default reasoning effort on your LLM API calls is not a capability setting. It is a cost setting. And the default is set to spend more than most tasks require.
The frontier from the Khan Academy data gives you the frame: set reasoning effort to the minimum level that meets your task's accuracy bar, skip the self-consistency ensemble, and audit your parameter defaults every time a provider ships a model update. The gap between what you are paying and what you need to pay is sitting in a parameter you never changed.