The Future of Global Multilingual AI Data Collection

Alongside the GPAI Code of Practice, the European Commission published a template for the public summary of training content, requiring providers to give an overview of the data used to train their models, including the sources it came from. A dataset is therefore judged on two axes rather than one: is it good, and can you prove where it came from. Suppliers who captured consent, contributor metadata and quality decisions as the work happened are in a different position from those planning to reconstruct it after the fact.

The reference point is Shumailov and colleagues' 2024 Nature paper showing that models trained recursively on their own outputs degrade, with measurable model collapse — the progressive loss of rare and unusual cases from a model's outputs as it trains on its own synthetic data — within roughly five to ten generations when trained on synthetic data alone. The failure has a specific shape: the tails of the distribution go first. Rare, unusual and low-frequency cases disappear while average performance still looks acceptable, and only later does output become bland and repetitive.

Two clarifications matter. It is not an argument against synthetic data: the original experiments used purely synthetic data with the human data discarded and no verification, which is not how serious labs operate, and later work has shown collapse is avoidable through verification and by accumulating real data alongside synthetic rather than replacing it. But it is a strong argument for human anchoring, and that argument is sharpest exactly where multilingual work sits, since distribution tails are the whole point of low-resource language collection: regional variants, rare constructions and dialect forms are the things lost first under recursive training. The broader treatment of this trade-off sits in the companion piece on training AI models on AI-generated data.

Kenya's draft Artificial Intelligence and Other Emerging Technologies Policy, published for consultation in 2026, is the clearest signal so far. It targets data annotation, content moderation and AI quality evaluation roles, and proposes a fair-pay reference framework benchmarked against international rates rather than domestic minimums. The gap it addresses is stark: reported earnings of roughly $1.46 to $3.74 an hour for Kenyan data workers against $21 to $27 for equivalent United States roles. The draft also proposes mandatory psychosocial support, written contracts, transparent pay reporting and grievance mechanisms, and Kenya's AI Bill 2026 would add a risk-based framework and a dedicated AI commissioner.

Three consequences follow for anyone commissioning multilingual data. Labour practices become a procurement question rather than a values statement, since buyers subject to EU documentation obligations will increasingly be asked how contributors were treated, not only what was delivered. Price expectations reset, because a quote built on suppressed wages is not a cheaper version of the same service but a different risk profile. And retention becomes strategic: in rare languages, trained contributors are the scarce asset, and fair terms are how a supplier keeps the ability to deliver in that language next year. The mechanics of getting consent and pay right are covered in how data contributors should be consented and paid.

In June 2026 the European Commission selected the EUROPA consortium as winner of the Frontier AI Grand Challenge, a project to build a European open-source frontier AI model in all 24 EU official languages. That is a publicly funded commitment to language coverage that no commercial business case would have produced on its own, sitting alongside the EU's AI Gigafactories programme and national sovereign AI efforts elsewhere. The shift matters for three reasons: public programmes and national institutions are becoming significant commissioners with different requirements from commercial buyers, such as openness, documentation and explicit coverage mandates; coverage can now be funded as a policy objective even where the commercial case fails; and publicly funded datasets tend to be released openly, which raises the floor for everyone working in those languages. The corollary is that languages without a state sponsor or a commercial case remain exposed — public funding is redrawing the map, not flattening it.

Grand View Research valued data collection and labelling at $3.8 billion in 2024 and projects $17.1 billion by 2030, a compound annual growth rate of 28.4%, with Asia Pacific the fastest-growing region and audio among the fastest-growing data types. Underneath that headline, the mix is changing in four ways. Evaluation is becoming a product in its own right, because as models get capable enough that ordinary accuracy checks stop discriminating between them, the valuable work is designing tests that reveal failure, written per language by speakers rather than translated. Preference data — records of which of two model outputs a human judge prefers, used to align a model's behaviour with what a culture considers appropriate — is scarce because it cannot be scraped or translated. Red-teaming is multilingual by necessity, since a safety guardrail holds only where it was trained. And bulk annotation is automating, as routine labelling becomes increasingly machine-assisted with human verification, compressing margins on volume work while raising the premium on judgement. A fuller breakdown of where the money is actually going sits in the economics of multilingual AI data collection.

Roughly 3,000 of the world's 7,000-plus languages have an established writing system, so for predominantly oral languages speech is not one modality among several — it is the only route in. Three developments push the same way: recent open speech systems have shown that deliberate field collection can extend recognition to well over a thousand languages, including hundreds never previously served; audio is among the fastest-growing segments in data collection forecasts; and in many markets voice remains the dominant interface, particularly where literacy in the written standard lags fluency in the spoken language. Speech collection is also structurally harder than text work, requiring physical presence, field conditions rather than studios, biometric-grade consent handling, and transcription decisions that presuppose an orthography the language may not have settled — logistics and governance problems that favour organisations already operating in the relevant regions. The practical side of building this kind of programme is covered in how speech data is collected for low-resource languages.

• Capture provenance from day one: consent, contributor metadata, compensation records and quality decisions recorded as the work happens, since nothing in the regulatory direction of travel suggests this softens.
• Anchor synthetic pipelines in verified human data, accumulating rather than replacing it, and verify before it enters training — this matters most where real data is thinnest.
• Budget for fair labour and expect it to be audited, since the direction is toward benchmarked pay, written contracts and disclosure.
• Invest in evaluation ahead of volume, because per-language evaluation sets written by speakers are becoming the scarce asset.
• Plan speech capability now, since for predominantly spoken languages the constraint is field presence and consent infrastructure, and both take longer to build than a data purchase.

40+ delivery centres across 30+ countries, 100+ languages including underrepresented dialects, and 56,000+ registered contributors are what make field presence a starting condition rather than a mobilisation project — a scale comparable to the providers covered in the leading global multilingual AI data collection companies. Provenance is captured as the work happens — consent, contributor metadata, compensation records, task assignment and quality decisions — rather than assembled at delivery, since the EU timeline above turns that record into an artefact a buyer may have to publish. Contributors are trained and retained rather than sourced per project: 414,120 training hours were delivered to Lifewood's Bangladesh workforce in 2025, which is the mechanism behind being able to deliver in a rare language a second time. Quality is verified against a customer-approved gold set at a 95%+ accuracy SLA, reported per language. This is not legal advice, and obligations differ by jurisdiction and by how data is produced; Lifewood's own AI data services and multilingual data collection pages describe the delivery model in more detail.