Short answer. Speech annotation is the process of labelling audio files so AI models can learn from them. It is not one task but at least four: transcription converts speech to text, diarization assigns each utterance to a speaker, timestamping links text to precise time positions in the audio, and non-speech event annotation marks everything else, including laughter, silence, background noise and coughing. Each is governed by its own specification, and a decision made incorrectly in one layer invalidates the work in the others.
What does each annotation task produce?
Four separate outputs, each necessary for different downstream training tasks.
Transcription, diarization, timestamping and non-speech event labelling are frequently treated as a single job and frequently confused with each other. They are related but distinct, each with its own failure modes and quality criteria.
Transcription converts the spoken words into written text. The output is a text file or a text layer attached to the audio. It is the foundation everything else is built on, because diarization cannot be verified and timestamps cannot be assigned to words that were not first transcribed.
Diarization answers who said what. It identifies speaker turns and assigns each segment to a speaker label, commonly Speaker 1, Speaker 2 and so on, or to named individuals when identity is known. The output is a timeline of speaker-attributed segments rather than a flat text file.
Timestamping links text to time. This can be at utterance level, which records when a speech segment begins and ends, at word level, which records the exact start and end time of each word, or at phoneme level, which records the acoustic events making up each sound. The granularity required depends entirely on the training task.
Non-speech event annotation labels everything in the audio that is not words: silences, laughter, coughing, breathing, background noise, music or unintelligible speech. These labels are needed when a model should understand the full acoustic environment rather than only the words.
What is the difference between verbatim and clean transcription?
The difference is what happens to the material between what the speaker produced and what the transcript contains. Both are legitimate; they serve different models.
This decision is one of the most consequential made in a speech annotation project, and it is frequently under-specified.
Verbatim transcription captures everything: hesitations, filled pauses such as "um" and "uh," false starts, repetitions, self-corrections and non-speech events within the speech. A verbatim transcript might read: "Um, so, so the, uh, delivery was actually, [laughs] quite late." This convention is required when the model needs to learn natural speech patterns, the acoustic correlates of uncertainty, or the social signals carried by non-lexical sounds.
Clean or normalised transcription removes hesitations and repairs and presents the utterance as it would appear in edited prose: "The delivery was quite late." This is appropriate when the model needs to learn meaning and grammatical structure rather than speech characteristics.
Intelligent verbatim sits between them: hesitations are removed but the phrasing is preserved rather than edited into formal prose: "So the delivery was quite late." It is the most common format for business meeting and customer service transcription.
The consequence of choosing incorrectly is significant. A speech recognition model trained on clean transcripts learns to expect clean input and performs poorly on spontaneous speech. A language model trained on verbatim transcripts inherits hesitation patterns it was never meant to reproduce. The choice is set in the specification before collection begins. Changing it after delivery means re-annotating the whole corpus.
What is diarization and why is it hard?
Diarization is identifying who spoke when. It is technically difficult because human annotators and automated tools both struggle with the same edge cases, and in multilingual corpora those edge cases are the norm.
Standard automated diarization uses speaker-change detection followed by clustering of segments that sound like the same person. Accuracy is reasonable when speakers are few, conditions controlled, and one person speaks at a time. Four situations degrade it.
Overlapping speech. Two people speaking simultaneously is normal in conversation and breaks most automatic speaker segmentation models, which assume one speaker per segment. A well-specified diarization task marks overlap regions explicitly rather than arbitrarily assigning them to one speaker.
Channels. Multi-channel recordings, where each speaker is on a separate microphone, make diarization substantially easier and more accurate. Single-channel recordings, which are more common in field collection, require acoustic modelling to separate voices that were recorded together.
Code-switching. In multilingual speech, a speaker may move between languages within a sentence. Acoustic models trained on monolingual data misattribute these segments. Human annotators are required at these points, and they must speak both languages to handle them correctly.
Speaker re-entry. A speaker who has left the conversation and returns after a long pause may not be matched correctly to their earlier segments by an automated tool, especially in longer recordings.
Automated diarization is an effective first pass, not a complete solution. Speaker assignment should be verified by an annotator who listened to the audio.
What is timestamping used for?
For training forced alignment models, subtitle generators and any application where the relationship between spoken words and time is the training signal.
Utterance-level timestamps record when each speech segment begins and ends. Word-level timestamps record the start and end of each individual word, needed for real-time captioning and word-error-rate computation. Phoneme-level timestamps serve acoustic modelling and text-to-speech training.
Forced alignment tools produce word-level timestamps automatically by mapping a transcript to audio using phoneme models, but they are language-specific. Applying an English-trained aligner to a low-resource language produces misaligned timestamps that propagate into any model trained on the result.
Why does this matter for multilingual AI, and what should a spec include?
Because annotation decisions that are trivially standardised in English are genuinely contested in many other languages, and a spec that does not resolve them before collection begins manufactures inconsistency at scale.
Three issues recur.
Orthographic decisions before transcription can begin. Some languages have more than one writing system or competing spelling conventions. A spec that does not commit to one produces a split corpus where the same word is spelled differently by different annotators, and the model learns noise.
Code-switching labels. When a speaker moves from Hindi to English mid-sentence, the transcription convention, the language identification tag and the diarization label all need to be decided in advance. Leaving these to individual annotators produces inconsistency.
Non-speech event inventories. The set of events to label should be listed explicitly. An open-ended instruction to label non-speech events produces very different datasets from two annotators, because one may label breathing and the other may not.
A workable annotation spec states at minimum: the transcription convention; the speaker labelling scheme; the timestamp granularity; the non-speech event inventory; code-switching handling; unintelligible-segment protocol; and how to handle overlapping speech.
Lifewood's speech collection programmes work to exactly this level of specificity, with collection, transcription and review carried out in-language by trained native speakers working to a documented scheme. The downstream effect is a dataset whose composition can be audited.
Key takeaways
- Speech annotation is four distinct tasks: transcription, diarization, timestamping and non-speech event labelling.
- Each produces a different output and serves different training purposes.
- Verbatim transcription preserves hesitations and repairs; clean transcription removes them; intelligent verbatim sits between. The choice is set in the spec and cannot be changed after delivery without re-annotating.
- Diarization assigns speaker labels to turns and is degraded by overlapping speech, single-channel recordings, code-switching and speaker re-entry. Automated tools require human review.
- Utterance-level timestamps locate speech segments; word-level timestamps locate individual words; phoneme-level timestamps serve acoustic modelling and text-to-speech.
- Forced alignment tools trained on one language produce misaligned timestamps for another, requiring native-language models or human validation in multilingual projects.
- Orthographic decisions for languages with competing writing systems must be made before transcription begins, or the corpus encodes inconsistency.
- A complete spec states transcription convention, speaker labelling scheme, timestamp granularity, non-speech event inventory, code-switching protocol, unintelligible-segment handling and overlap convention.