Short answer. Far less than most buyers budget for, if the data is good. OpenAI documents a floor of 10 examples and visible gains at 50–100. The LIMA study matched heavily trained systems with 1,000 curated examples. Expect about 50 for narrow format tasks, 1,000–5,000 for most production instruction tasks, and tens of thousands only for broad assistants and preference training.
Key takeaways
- There is no single number for fine-tuning data: fine-tuning steers an already-capable model, so the need scales with how new the behaviour is, not how important the task is.
- Published anchors are OpenAI's floor of 10 examples and improvements at 50–100, LIMA's 1,000 curated examples, a practitioner optimum of 2,000–5,000 for production supervised fine-tuning, and roughly 13,000 demonstrations plus 33,000 preference comparisons behind InstructGPT.
- Diversity and consistency beat quantity: LIMA's ablations showed sharply diminishing returns from scale alone, and contradictory examples make a fine-tuned model worse.
- Grow a fine-tuning dataset in measured steps and stop when the evaluation score stops moving; the right size is discovered, not declared.
- Coverage failures call for more data in specific gaps, while inconsistency failures call for better guidelines and quality assurance, which more volume cannot fix.
Why is there no single number for fine-tuning data?
Because fine-tuning is not training from scratch. The base model already knows language, facts and reasoning patterns, so the amount of data you need tracks how far your task sits from what the model already does well, not how important the task is to your business.
Fine-tuning is the process of further training a pretrained model on a smaller, task-specific dataset so that it adopts a particular format, tone, policy or domain vocabulary. Your data is not teaching the model the world. It is showing it a direction: this format, this tone, this policy, this domain vocabulary.
That is why numbers that sound impossibly small keep turning out to be true. You are constraining a capable system, not building one. The question behind your question is really three questions:
- How new is the behaviour? Reformatting outputs needs little. Teaching a genuinely unfamiliar domain needs much more.
- How consistent can your examples be? Contradictory examples, meaning similar inputs with different expected outputs, actively damage fine-tuning, so a smaller clean set beats a larger noisy one.
- How will you know it worked? Without an evaluation set, no dataset size is "enough", because you cannot tell. OpenAI's guidance is blunt on the order of operations: set up evaluations before investing in fine-tuning at all.
What do the research and vendor guidelines say about dataset size?
Published reference points cluster tightly once you line them up: a platform floor of 10 examples, improvements at 50–100, a landmark result at 1,000 curated examples, a practitioner optimum of 2,000–5,000, and assistant-scale projects in the tens of thousands.
The platform floor. OpenAI's supervised fine-tuning documentation sets the minimum at 10 examples, reports visible improvements at 50–100, and adds a useful decision rule: if 50 good examples produce no improvement, rethink the task or the prompt before adding more data.
The small-data landmark. The LIMA study ("Less Is More for Alignment") fine-tuned a 65-billion-parameter model on just 1,000 carefully curated instruction–response pairs and produced outputs competitive with far more heavily trained systems. Its ablations found that diversity mattered much more than quantity, with sharply diminishing returns from simply scaling the set.
The production-scale reference. OpenAI's InstructGPT work used roughly 13,000 human-written demonstrations for supervised fine-tuning plus about 33,000 preference comparisons, and with that a 1.3B model beat the raw 175B GPT-3 in human evaluations. Stanford's Alpaca used 52,000 generated examples. These are the sizes behind broad, general-purpose assistants, not narrow business tasks.
The practitioner ranges. Platform guidance such as Predibase's dataset requirements puts simple tasks at 10–20 minimum and 100–1,000 optimal, standard supervised fine-tuning at 500–1,000 minimum and 2,000–5,000 optimal, and reserves 10,000–20,000+ for the hardest cases. It advises teams to begin with a focused set of 100–500 high-quality examples and grow while monitoring.
Published reference points for fine-tuning dataset size
| Reference | Examples | What it represents |
|---|---|---|
| OpenAI documented minimum | 10 | Platform floor, not a recommendation |
| OpenAI "improvements seen" | 50–100 | Narrow behaviour and format tasks |
| LIMA | 1,000 | Curated, diverse instruction tuning on a strong base model |
| Practitioner SFT optimum | 2,000–5,000 | Typical production instruction task |
| InstructGPT SFT | About 13,000 (plus 33,000 preferences) | General-purpose assistant with RLHF |
| Stanford Alpaca | 52,000 | Broad instruction coverage, generated data |
Sources: OpenAI developer documentation; LIMA (Zhou et al., 2023) via Oumi; InstructGPT and Alpaca figures via Meta-Intelligence's fine-tuning data review; Predibase guides. Treat ranges as starting points, because your task, base model and quality bar move them.
Read the spread for what it is: three orders of magnitude between the floor and the assistant-scale projects. The difference is not budget. It is task breadth.
How much data does your specific task need?
Map your project to the nearest task type and start there: 50–300 examples for style and format, 100–1,000 per class pattern for classification, 1,000–5,000 for domain instruction-following, 5,000–20,000+ for specialist domains, and 10,000+ comparisons for preference data.
Style, tone and format (50–300 examples). Making outputs follow your template, your brand voice, your JSON schema. The model already writes; you are standardising. This is where OpenAI's "improvements at 50–100" lives.
Classification and routing (100–1,000 per class pattern). Ticket triage, intent detection, content categorisation. Coverage matters more than volume: every class and the confusable boundaries between classes need representation, including the rare ones.
Domain instruction-following (1,000–5,000). A support or workflow assistant for your product, policy-grounded answering, structured extraction from your document types. This is the practitioner optimum range, and where most commercial fine-tuning projects land.
Specialist domain behaviour (5,000–20,000+). Legal, medical or financial response generation where correctness standards are high and errors expensive. Here the binding constraint is usually not volume but expert authorship and review, the subject of our guide to building legal, medical and financial SFT datasets.
Preference and reward data (10,000+ comparisons). RLHF-style training is a different product with its own sizing logic, with InstructGPT's 33,000 comparisons as the classic reference. It also has its own quality pitfalls, covered in our piece on collecting preference ratings without drift.
Two adjustments apply across every task type. Smaller base models need more examples for the same behaviour. And every dataset needs a held-out slice, typically 10–20%, that never touches training, or you will be grading the model on questions it has memorised.
When does more fine-tuning data stop helping?
Earlier than the budget conversation assumes. Once a strong base model has seen a diverse, high-quality set, additional similar examples buy very little, and examples that contradict the pattern make results worse.
The LIMA ablations are the clearest published evidence. The curve flattens because the model has already extracted the pattern; new rows that repeat the pattern are redundant, and new rows that contradict it are harmful.
More data actively hurts in three common situations:
- Contradictions at scale. Doubling a dataset by adding a second annotator's work without reconciling their disagreements teaches the model that both answers are acceptable. It will oscillate.
- Narrow duplication. Ten thousand examples of the same five question shapes is a 5-example dataset wearing a large invoice.
- Quality dilution. Teams racing to a row-count target relax review in the final stretch, and the worst 20% of the data sets the ceiling. This is the multilingual version of the problem we documented in Can More Data Make AI Worse?
The practical rule is to grow in steps, not leaps. Train at 500 examples and measure. Train at 1,500 and measure. The step where your evaluation score stops moving is your answer, and it is usually smaller than the number in the original plan.
How do you know whether you need more data or better data?
Look at the failure pattern. Failures on cases the model never saw are coverage gaps that need more targeted data, while inconsistency on cases it has seen points to quality and guideline problems that more volume cannot fix.
Signs you need more data: the model handles common cases but breaks on variants it never saw; whole input categories fail together; performance differs sharply between well-covered and thinly covered languages, formats or customer segments. Add targeted examples where the gaps are, not more of what you already have. Our guide to long-tail and edge-case mining covers how to find them deliberately.
Signs you need better data: the model is inconsistent on cases it has seen; it confidently produces outputs in a style none of your reviewers would accept; two annotators labelling the same item would plausibly disagree. No additional volume fixes these; measurement does. A gold set is a reference collection of items with expert-verified correct labels, used to score annotators and datasets. Gold sets and inter-annotator agreement, covered in Gold Sets, Audit Sampling and Consensus, show whether your dataset agrees with itself before you ask a model to learn from it.
A ten-minute test is worth running before any purchase order: pick 30 random examples from your current data and have two people independently judge whether each one is correct and consistent with the rest. If they disagree with the data, or with each other, your next spend is on the specification, not the volume.
How does Lifewood size training datasets for clients?
Lifewood treats dataset sizing as the first conversation in any annotation engagement, and it usually ends with a smaller first commitment than the client arrived with. The method is a pilot at the bottom of the task's range, measured against a held-out evaluation set, then scaled in steps.
Sizing is the first step in an enterprise LLM training data engagement. The pilot set is built under one tight guideline with gold-set checks from the first batch, measured against a held-out evaluation set, and scaled only where the measurement says coverage is short. Because Lifewood delivery teams work across 100+ languages, the same stepped approach applies per language. Low-resource languages often need more examples per behaviour than the English set suggests, and finding that out at 500 examples is far cheaper than at 50,000. For teams comparing vendors, our ranking of the best data annotation companies for LLM training sets out the criteria.
The same logic appears across the industry's better guidance: OpenAI's documentation pushes evaluation before investment, Predibase's guides push starting at 100–500 and growing against measurement, and the LIMA line of research keeps being replicated, with curation beating accumulation. If a proposal quotes a volume before anyone has asked what the model must do, that is worth questioning, whoever it comes from.
What should you do before you commission a single example?
Write the target behaviour down, build the evaluation set first, start at the bottom of your task's range, and add data only where measurement shows gaps.
- Write the behaviour down first. One page: inputs, expected outputs, what a wrong answer looks like. If you cannot write it, annotators cannot label it.
- Build the evaluation set before the training set. 100–300 held-out cases that represent real usage, including the awkward ones.
- Start at the bottom of your task's range. 50–100 for format tasks, a few hundred per class for routing, around 1,000 for domain instruction work.
- Measure, then step up. Add data only where the evaluation shows gaps, and stop when a step stops moving the score.
- Protect consistency over volume. Reconcile annotator disagreements before scaling; contradictions compound.
- Reserve 10–20% as a test split from day one and never let it near training.
- Budget for diversity, not repetition. A thousand distinct situations beat ten thousand rephrasings.