Speech and Audio Annotation: Transcription, Diarization and Timestamping

Transcription, diarization, timestamping and non-speech event labelling are frequently treated as a single job and frequently confused with each other. They are related but distinct, each with its own failure modes and quality criteria.

Transcription converts spoken words into written text, producing a text file or a text layer attached to the audio. It is the foundation everything else is built on, because diarization cannot be verified and timestamps cannot be assigned to words that were not first transcribed.

Diarization answers who said what: it identifies speaker turns and assigns each segment a speaker label — commonly Speaker 1, Speaker 2, or a named individual when identity is known — producing a timeline of speaker-attributed segments rather than a flat text file.

Timestamping links text to time. This can happen at utterance level, which records when a speech segment begins and ends; at word level, which records the exact start and end of each word; or at phoneme level, which records the acoustic events making up each sound. The granularity required depends entirely on the training task, a distinction covered further in multilingual speech data collection services.

Non-speech event annotation labels everything in the audio that is not words: silences, laughter, coughing, breathing, background noise, music or unintelligible speech. These labels matter when a model needs to understand the full acoustic environment, not only the words.

This decision is one of the most consequential made in a speech annotation project, and it is frequently under-specified. Verbatim transcription captures everything: hesitations, filled pauses such as "um" and "uh," false starts, repetitions, self-corrections and non-speech events within the speech. A verbatim transcript might read: "Um, so, so the, uh, delivery was actually, [laughs] quite late." This convention is required when the model needs to learn natural speech patterns, the acoustic correlates of uncertainty, or the social signals carried by non-lexical sounds.

Clean or normalised transcription removes hesitations and repairs, presenting the utterance as it would appear in edited prose: "The delivery was quite late." This is appropriate when the model needs to learn meaning and grammatical structure rather than speech characteristics.

Intelligent verbatim sits between them: hesitations are removed but the phrasing is preserved rather than edited into formal prose — "So the delivery was quite late." It is the most common format for business meeting and customer service transcription.

The consequence of choosing incorrectly is significant. A speech recognition model trained on clean transcripts learns to expect clean input and performs poorly on spontaneous speech. A language model trained on verbatim transcripts inherits hesitation patterns it was never meant to reproduce. The choice belongs in the specification before collection begins, a discipline described in more detail in how a speech data collection programme actually runs; changing it after delivery means re-annotating the whole corpus.

Standard automated diarization uses speaker-change detection followed by clustering of segments that sound like the same person. Accuracy is reasonable when speakers are few, conditions are controlled, and one person speaks at a time. Four situations degrade it.

• Overlapping speech. Two people speaking simultaneously is normal in conversation and breaks most automatic speaker segmentation models, which assume one speaker per segment. A well-specified diarization task marks overlap regions explicitly rather than arbitrarily assigning them to one speaker.
• Channels. Multi-channel recordings, where each speaker is on a separate microphone, make diarization substantially easier and more accurate. Single-channel recordings, more common in field collection, require acoustic modelling to separate voices recorded together.
• Code-switching. In multilingual speech a speaker may move between languages within a sentence, and acoustic models trained on monolingual data misattribute these segments — a pattern examined in collecting speech and text that mixes languages. Human annotators fluent in both languages are required at these points.
• Speaker re-entry. A speaker who has left the conversation and returns after a long pause may not be matched correctly to their earlier segments by an automated tool, especially in longer recordings.

Automated diarization is an effective first pass, not a complete solution. Speaker assignment should be verified by an annotator who listened to the audio, and disagreement between annotators is one signal worth tracking, as discussed in inter-annotator agreement measures.

Utterance-level timestamps record when each speech segment begins and ends. Word-level timestamps record the start and end of each individual word, needed for real-time captioning and word-error-rate computation. Phoneme-level timestamps serve acoustic modelling and text-to-speech training.

Forced alignment tools produce word-level timestamps automatically by mapping a transcript to audio using phoneme models, but they are language-specific. Applying an English-trained aligner to a low-resource language produces misaligned timestamps that propagate into any model trained on the result — one reason speech collection for low-resource languages relies on native-language models or manual validation rather than a single off-the-shelf aligner.

Three issues recur.

• Orthographic decisions before transcription can begin. Some languages have more than one writing system or competing spelling conventions. A spec that does not commit to one produces a split corpus where the same word is spelled differently by different annotators, and the model learns noise.
• Code-switching labels. When a speaker moves from Hindi to English mid-sentence, the transcription convention, the language identification tag and the diarization label all need to be decided in advance. Leaving these to individual annotators produces inconsistency.
• Non-speech event inventories. The set of events to label should be listed explicitly. An open-ended instruction to label non-speech events produces very different datasets from two annotators, because one may label breathing and the other may not.

A workable annotation spec states, at minimum: the transcription convention; the speaker labelling scheme; the timestamp granularity; the non-speech event inventory; code-switching handling; the unintelligible-segment protocol; and how to handle overlapping speech. Voice datasets built for text-to-speech carry an additional layer of casting and studio requirements, covered in studio standards and speaker casting for TTS voice data.

Lifewood's speech collection programmes are built to this level of specificity, with collection, transcription and review carried out in-language by trained native speakers working to a documented scheme, part of a wider multilingual data collection capability spanning 50+ languages. The downstream effect is a dataset whose composition can be audited, which is one reason enterprise buyers also evaluate a provider's low-resource speech data capability before committing volume.