Short answer. By measuring three separate things and refusing to let one stand in for the others. Composition asks who is in the dataset and how evenly, using measures such as Shannon entropy, the Simpson index and effective numbers rather than a count of categories. Coverage asks whether the acoustic and linguistic range of real use is present. Outcome parity asks whether the model performs equally well across those groups, which is the only test that matters and the one standard accuracy metrics are least able to detect.
Why isn't "we covered 20 dialects" a diversity measure?
Because it measures richness and ignores evenness, and evenness is where datasets actually fail.
Diversity research, mostly borrowed into machine learning from ecology, separates two ideas. Richness is how many distinct categories are present. Evenness is how balanced they are. A dataset can be rich and severely unbalanced, and the headline number will not show it.
Take two speech datasets, each covering eight regional varieties of a language, each with 1,000 hours. In the first, one variety accounts for 800 hours and the remaining seven share 200. In the second, each variety has 125 hours. Both can honestly claim eight varieties. Only one of them will produce a model that works for all eight.
This is why category counts are the wrong instrument. A supplier claim of "50+ languages" or "20 dialects" tells you the dataset is rich. It says nothing about whether the tail has enough data to matter, and thin per-category volume is exactly the condition under which a category contributes noise instead of capability.
The measurable question is not how many groups are present but how the mass is distributed across them.
What can actually be measured?
Three layers, and confusing them is the most common analytical error in this area.
Composition. Who is in the dataset: region, language variety, age band, gender, and how the volume is distributed across them. This is countable from metadata and is the cheapest layer to measure, provided the metadata exists.
Coverage. Whether the range of real-world conditions is represented: recording environments, device types, background noise, speaking styles, formality registers, spontaneous versus read speech, code-switching. A dataset can be demographically balanced and acoustically monotonous.
Outcome parity. Whether the trained model performs equally well across all of the above. This is the only layer that answers the question people actually care about, and it requires evaluation data broken out by group rather than a single aggregate score.
The layers are not substitutes. Balanced composition does not guarantee balanced outcomes, because some groups are harder for a model for reasons unrelated to sample count. And good aggregate performance says nothing about any of it.
A useful discipline is to state, before collection begins, what the dataset is meant to represent. Diversity is not an absolute property; it is a relationship between a dataset and a target population. Without a stated target, "diverse" is unfalsifiable.
Which metrics do what?
Borrowed mostly from ecology and economics, and each answers a different question. Reporting one alone is usually a choice about what to hide.
Shannon entropy measures uncertainty about which group a randomly chosen sample belongs to. High entropy means the mass is spread; low entropy means one group dominates. It is sensitive to rare categories, which makes it useful for spotting a neglected tail.
The Simpson index and its Gini-Simpson variant measure the probability that two randomly drawn samples come from different groups. The same measure appears as the Herfindahl-Hirschman index in economics and as Gini impurity in machine learning. It is weighted toward common categories, so it is the better instrument for detecting dominance.
Hill numbers and effective numbers convert entropy into an interpretable count: the number of equally common categories that would produce the observed diversity. This is the most communicable of the family. "Eight dialects present, effective diversity 2.4" tells a stakeholder immediately that six of them are close to decorative.
The Vendi Score and similar embedding-based measures assess diversity in a learned representation space rather than over declared categories, which catches variation that no metadata field records.
Coverage and richness estimators address a different question: how much of the true population variation is likely to be missing from the sample.
Because the metrics disagree by design, the practical approach is to report a small fixed set, with effective numbers as the headline figure because it is the hardest to misread.
Why does headline accuracy hide diversity problems?
Because the standard metrics are the ones least sensitive to speaker characteristics. This is measured, not speculative, and it should change how teams report results.
A 2026 study auditing speech recognition performance across speaker demographics found a clear hierarchy of sensitivity among evaluation metrics. Word error rate and character error rate, the two figures almost universally quoted, were least responsive to demographic and acoustic factors, with reported coefficients of determination of 0.040 and 0.012 respectively. The authors concluded that raw lexical error counts are dominated by stochastic noise rather than systematically coupled to a speaker's profile.
Metrics further up the scale behaved differently. Match error rate, word information lost, embedding-weighted error and semantic distance showed materially greater elasticity, capturing demographic variation that word error rate did not.
Semantic distance in particular occupied its own direction in the analysis, encoding information the other measures missed.
The same work names the underlying phenomenon the diversity tax: the extra burden carried by users with marginalised or atypical speech, who adapt their pronunciation or repeatedly correct errors simply to get the same baseline utility as majority-demographic users. That burden is real, and word error rate is poorly equipped to see it.
The operational lesson is uncomfortable but simple. If your quality report leads with a single aggregate word error rate, you have chosen the metric least likely to reveal a diversity problem. Reporting per-group results on semantically sensitive measures is more work and considerably more informative.
What has to be captured at collection time?
Everything you will later want to slice by. Diversity that was not recorded cannot be measured, and it cannot be reconstructed afterwards.
This is the point at which measurement becomes a collection problem rather than an analysis problem. A dataset can only be audited along the dimensions its metadata records.
A workable minimum for speech and text collection:
Speaker attributes. Region and sub-region, language variety, age band, gender, and any second languages relevant to code-switching. Recorded as declared categories with a documented scheme, not free text.
Session attributes. Device type, recording environment, background noise level, prompt type, spontaneous or read, session duration.
Content attributes. Domain, register or formality level, topic, whether code-switching occurred and into which language.
Process attributes. Who transcribed, who reviewed, whether adjudicated, and against which guideline version.
Two constraints apply. Collection of demographic attributes must be consented, purposeful and proportionate, since these are personal data and in some jurisdictions sensitive. And the categories themselves need care, because a scheme designed elsewhere can misrepresent how people in a region actually describe themselves.
Done properly, this metadata is what allows a later question like "does the model underperform for older speakers in one district" to be answerable at all. Lifewood captures this alongside the data as work happens across its delivery network, for the straightforward reason that a dataset whose composition cannot be described is a dataset whose diversity cannot be defended.
How do you run a diversity audit?
Six steps, ideally before delivery rather than after deployment.
State the target population. What should this dataset represent, and according to what source. Without this the audit has no reference point.
Report richness and evenness together. Category counts alongside effective numbers, per dimension. Never counts alone.
Check the tail against a usability floor. For each category, is there enough volume to contribute? Categories below the floor should be reported as present-but-thin rather than counted as coverage.
Measure coverage separately from composition. Device mix, noise conditions and speaking styles need their own distributions.
Evaluate outcomes per group. Using metrics that respond to speaker characteristics, on evaluation sets built by speakers of each variety.
Publish the composition with the dataset. A documented data statement covering how the dataset was assembled, what it represents and what it omits. Under the EU's data governance expectations for higher-risk systems, datasets are expected to reflect the characteristics of the setting where the system will be used, which is a documentation question as much as a collection one.
The recurring theme is that diversity is a claim, and claims need evidence. A supplier who can produce composition tables, effective numbers and per-group results is making a checkable statement. One who offers a list of languages is not.
Key takeaways
- Diversity has two components: richness, meaning how many categories are present, and evenness, meaning how balanced they are. Category counts capture only the first.
- Two datasets with the same eight dialects and the same total hours can be entirely different datasets depending on distribution.
- Measurement has three layers: composition, coverage of real-world conditions, and outcome parity across groups.
- None substitutes for another.
- Shannon entropy is sensitive to rare categories; the Simpson index is weighted toward dominant ones; Hill or effective numbers convert both into an interpretable count.
- Embedding-based measures such as the Vendi Score capture variation that metadata categories never record.
- A 2026 audit of speech recognition found word error rate and character error rate least sensitive to speaker demographics, with reported R² of 0.040 and 0.012.
- Match error rate, word information lost, embedding-weighted error and semantic distance were substantially more responsive to demographic variation.
- The same work names the diversity tax: the extra effort users with atypical speech expend to get the same utility as majority-demographic users.
- Diversity that was not recorded at collection cannot be measured later, so speaker, session, content and process metadata must be captured as work happens.
- Demographic metadata is personal data and must be consented, proportionate and categorised in terms the community recognises.
- An audit states the target population, reports richness with evenness, checks the tail against a usability floor, measures coverage separately, evaluates outcomes per group and publishes the composition.
Sources and further reading
- "Beyond Word Error Rate: Auditing the Diversity Tax in Speech Recognition through Dataset Cartography", arXiv, on metric sensitivity to speaker demographics and the diversity tax
- "Dataset Diversity Metrics and Impact on Classification Models", arXiv, on the metric taxonomy including Shannon entropy, Renyi entropy, Simpson index and Vendi Score
- "Metrics for Dataset Demographic Bias", arXiv, on richness, evenness and representational bias measures
- Emergent Mind, "Entropy and Diversity Metrics", on Hill numbers, Gini-Simpson and effective numbers
- European Commission, AI Act policy page, on data governance expectations for high-risk systems
- Lifewood, company overview and delivery network