NOTES
Why benchmark scores don't travel: what a leaderboard number actually tells you about your workload
A vendor's benchmark page says their model is best in class. Point the same model at your own workload and it stumbles on things that look basic: your document layouts, your prompt templates, your edge cases. That gap between the headline number and your experience is not a conspiracy, and it is not incompetence. It's structural. Published scores are measured on someone else's test, in someone else's harness, against someone else's idea of what "good" means. This post is about why that gap exists, and how to make model decisions that don't require taking a leaderboard on faith.
What a benchmark number actually is
Think of a benchmark as a contract between a model and a fixed set of examples. Someone picked the questions, the answer key, and the scoring rule, then ran the model under specific settings: a particular prompt template, decoding parameters, and evaluation code. The resulting number describes how the model behaved on that exact setup.
That makes a score a sample of behavior, not a specification of it. Change the examples, the prompt, or the scoring, and the number moves, often a lot. What gets published is the version the vendor chose to show you, measured in a harness you will never use.
Nobody is lying here. The number is true in a narrow sense; the question is whether that narrow sense includes your workload. Usually it doesn't.
Why the transfer breaks
### Your data is not their data
Benchmark examples come from public corpora: well-formed text, clean formatting, canonical spelling. Your data comes from your own systems: internal documents, customer messages, logs, exports, code with your naming conventions. Models perform differently on distributions they weren't tested on, and no amount of headline accuracy on a public set tells you how yours will behave.
### Your task is not their task
Most public benchmarks are single-turn, well-formed questions with one right answer. Real tasks are messier: multi-step, context-heavy, with judgment calls and formats that must be exact. Models can ace a multiple-choice-style benchmark and still mishandle the parts of your task that the benchmark never exercises: the part where the input is ambiguous, the output has to fit a schema, or the wrong answer is expensive.
### The headline hides the spread
Leaderboard numbers are averages across many subjects or tasks. Your workload is not an average; it is one or a few tasks. Models can look excellent on average while being weak exactly where you need them to be strong, because the average is dominated by categories you don't care about. The number that matters is the one for your slice, and published results rarely break down far enough to show it.
### The harness is theirs
Prompt templates, temperature, output parsing, and the judge used to grade free-form answers all shift results. Vendors tune their evaluation to show the model well. When you plug the same model into your prompts and your parsing, you are running a different experiment, and the results can differ dramatically from the published one.
### The model can change under the same name
Benchmarks run on a specific version of the model. By the time you test it, the served version may have been updated, quantized, or routed differently. Model behavior drifts between releases and sometimes within them. Any published score describes a moment in time, not a guarantee about the model you are calling today.
### Test sets age into training sets
Public benchmarks that have been around for years eventually overlap with what models see during training. When that happens, a high score can reflect familiarity with the test rather than general capability. It is not a scandal, just an inevitability of public datasets. Older benchmarks tell you less than they used to, both about the model's general ability and about your workload in particular.
What to do instead: measure your workload
Skip the hunt for a better public benchmark. Build a small evaluation from your own traffic, run the way you actually use the model.
Start small: sample a few hundred real inputs across the range of what your system handles, the common cases, the messy ones, the edge cases that have caused trouble before. Define what "good" looks like for each output type, not as an abstract score but as the specific things your team checks. Then run the candidate models under your own prompt template, with your own parsing, and grade the outputs by your own criteria.
By comparison, a few hundred examples of your own data are worth more than a public benchmark a hundred times that size, because they test the things you actually need. Perfection is not the bar. An eval is useful when it is representative, versioned, and re-run.
A practical sequence
1. Write down the task. What input does the model receive, what output do you need, and what failure modes cost you the most?
2. Sample real inputs. Pull from production or from realistic test cases, and include the ugly ones.
3. Define good. What does a correct, acceptable, and unacceptable output look like? Be specific enough that two people would grade the same way.
4. Run under your conditions. Your prompt template, your temperature, your output parser, not the vendor's demo.
5. Score by your criteria. Track what matters for your task: accuracy on the fields you need, error types, latency, cost. Look at the spread, not just the average.
6. Version it and re-run. Keep the eval set frozen, and re-run it whenever a model version or a prompt changes. Surprises stop once you have a regression set you trust.
Bottom line
Benchmark numbers are directional. They are a reasonable screen for which models are worth trying, and nothing more. Only the number you measure on your workload transfers, under your conditions, with your definitions of success.
Don't read this as vendor skepticism. It is simply what measurement is: vendors measured their benchmark, and you get to measure yours. Yours is the one that should drive the decision.