Short answer. Choose by building a small evaluation set from your own work and scoring it blind, because public leaderboards measure general capability on tasks that are almost certainly not yours. Fifty to a hundred representative items — weighted towards the hard and unusual cases — scored by two reviewers against a written rubric with model identity hidden will separate candidates more reliably than any ranking. Then decide on the criteria that actually constrain production: output rights, data handling, latency, cost at your real volume, language coverage, and version stability. Most selections are settled by one of those rather than by output quality, which sits closer between serious candidates than vendor material suggests.
This is a procurement question, not a testing question. How to evaluate a system before you deploy it — thresholds, gates, what to measure once it is live — is covered separately in AI evaluation before deployment. What follows is narrower: how to pick which generative model runs in your production pipeline, and how to be able to explain the choice afterwards.
What can public benchmarks tell you, and what can't they?
Leaderboards answer a real question — how do these systems compare on a standard set of tasks — and it is usually not the question a production team has. The gap has several specific causes, and each suggests a different correction.
| Limitation | What it does to your decision |
|---|---|
| Task mismatch | Benchmarks measure reasoning, knowledge and general instruction-following; production tasks are narrow — rewrite this in our tone, describe this product accurately, generate a shot matching this reference |
| Contamination | Benchmark items circulate publicly and may appear in training data, inflating scores in ways that do not transfer |
| Aggregation | A single score averages across subtasks, so a model can lead overall while being clearly worse at the one thing you need |
| Thin language coverage | Most headline benchmarks are English-only, and rankings genuinely reorder by language |
| No operational criteria | No leaderboard scores output rights, data retention, regional availability or version stability |
The language point is the one most often assumed rather than tested. MMLU-ProX (arXiv preprint 2503.10497 / EMNLP 2025) poses identical items across 29 languages and reports gaps of up to 24.3 points between high- and low-resource languages across 36 evaluated models. A model that wins in English can lose badly in your third market.
The constructive use of public benchmarks is as a shortlist filter: they reliably separate serious candidates from unserious ones. Holistic frameworks such as Stanford CRFM's HELM are more useful at this stage than single-number rankings, because they report across many scenarios and metrics rather than collapsing to one figure. Choosing among the serious candidates is your own work.
How do you build an evaluation set from your own work?
This takes about a week of one person's time, and it is repeatable at every model change — which is what makes it worth building rather than improvising.
1. Collect items from real work, weighted towards the hard cases. Fifty to a hundred inputs drawn from actual briefs. Deliberately over-sample the difficult ones: unusual products, sensitive topics, your worst-formatted source material, your smallest markets. A set of typical items will show every serious candidate performing well and tell you nothing.
2. Write the rubric before seeing any output. The dimensions that matter for the task — accuracy, tone, instruction adherence, format compliance, brand fit — with severity levels and worked examples. A rubric written after seeing outputs describes the first model you happened to look at.
3. Run every candidate on the identical set. Same inputs, same parameters where comparable, same number of attempts. Where a model needs different prompting to perform well, that is a real finding about integration cost. Record it rather than quietly tuning one candidate harder than the others.
4. Score blind. Strip model identity, randomise order, and have two reviewers score independently. Unblinded scoring measures expectations about brands; blinded scoring measures output.
5. Check reviewer agreement before trusting the result. If two reviewers disagree substantially, the rubric is ambiguous and the comparison is not yet meaningful. Fix the rubric and re-score. Chance-corrected agreement in the sense of Landis and Koch (Biometrics, 1977) is the standard way to report this.
6. Evaluate in every language you operate in. Not a translated English set — items authored in each target language. Rankings reorder between languages, and a selection made on English performance can be the wrong choice for most of your markets.
7. Test the failure modes, not only the successes. How does each candidate behave on out-of-scope requests, ambiguous briefs, and inputs designed to elicit a confident wrong answer? A model that fails loudly is far easier to operate than one that fails plausibly.
8. Freeze the set and re-run on every version change. This is what converts "the model feels different this month" into a measured difference. Keep the set out of any training or fine-tuning use so it stays a clean reference.
Separation = Mean rubric score of leading candidate − Mean rubric score of runner-up
If that difference is smaller than the disagreement between your two reviewers, the candidates are not separated on quality and the decision belongs to the operational criteria below.
Which criteria actually decide the selection?
Once a shortlist sits within a reasonable band on output quality — which happens more often than vendor comparisons imply — the decision is made on operational grounds.
| Criterion | The question to ask | Why it decides cases |
|---|---|---|
| Output rights | What may we do with the outputs, and does the provider claim anything? | A quality advantage is worthless if the licence does not permit your use |
| Data handling | Are inputs retained, used for training, or logged? Where, and for how long? | Client confidentiality obligations routinely eliminate otherwise strong candidates |
| Regional availability and residency | Can we call it from, and store data in, the regions we operate in? | A model unavailable in a market is not a candidate for that market |
| Latency and throughput | Per-item latency at our concurrency, and the real rate limits | Batch production is throughput-bound; interactive tools are latency-bound, and these select differently |
| Cost at real volume | Cost per finished deliverable including retries and rejected takes | Per-token or per-image pricing understates the cost of takes you discard |
| Version stability | How are versions pinned, deprecated and announced? | A model that changes under a production pipeline without notice is an operational risk regardless of quality |
| Language coverage | Measured performance in your actual markets | The criterion most often assumed rather than tested |
| Provenance support | Does it mark outputs, in a way that survives your pipeline? | EU AI Act Article 50 transparency duties make this a compliance input rather than a nice-to-have |
Output rights and data handling are the two that most often eliminate a leading candidate outright, and they are the two least visible in a quality comparison. Check them before you spend a week scoring.
Documenting the decision
Model selection is increasingly something an organisation is expected to be able to explain rather than merely to have done. The NIST AI Risk Management Framework (AI RMF 1.0, January 2023), widely referenced as a voluntary baseline, treats documentation, measurement and traceability as core practices. In practice, a client or auditor asking why a particular system was chosen is asking for exactly the artefacts a good evaluation produces anyway:
- The evaluation set and rubric, versioned.
- The scores, per candidate and per language, with reviewer agreement recorded.
- The non-quality criteria and how each candidate was assessed against them.
- The decision and its date, including which criterion was decisive.
- The re-evaluation record at each model version change.
Keeping these costs almost nothing at the time and is unreconstructable later. That is the recurring theme of running AI in production: the work is usually fine, and the evidence that the work was done is what goes missing.
Two patterns worth adopting
Do not standardise on one model. Different tasks in the same pipeline are genuinely won by different systems, and the integration cost of supporting two or three behind a common interface is modest against the quality cost of forcing one everywhere. It also removes a single point of failure when a provider changes terms, pricing or availability.
Separate the model from the pipeline. Prompts, references, rubrics, review workflow, provenance and delivery should be model-agnostic, so that swapping a model is a configuration change rather than a rebuild. Given how quickly this field moves, a pipeline welded to one provider's specifics is a pipeline that will be rewritten within the year.
How Lifewood approaches this
Lifewood operates third-party generative models rather than training its own, which makes this evaluation discipline part of ordinary operations rather than a one-off procurement event. The set is drawn from real briefs, frozen, re-run on version changes, and run per language — because the language dimension is where rankings actually move, and because 50+ languages across 40+ delivery centres in 30+ countries means an English-only selection would be wrong for most of the work.
Scoring is blind and dual-reviewer, feeding the same 95%+ accuracy threshold applied across delivery programmes.
See AIGC services, the QA process, delivery methodology and AI evaluation before deployment.
Sources and further reading
- Stanford Center for Research on Foundation Models, HELM — Holistic Evaluation of Language Models.
- MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation (29 languages, 36 models), arXiv preprint 2503.10497 / EMNLP 2025.
- National Institute of Standards and Technology, AI Risk Management Framework (AI RMF 1.0), January 2023.
- Landis, J.R. and Koch, G.G., "The Measurement of Observer Agreement for Categorical Data", Biometrics 33(1), 1977.