Skip to main content
AI Data

What Does It Take to Collect Sign Language Data for AI?

Lifewood Team6 min read

Short answer. Far more than collecting speech or text. Sign languages are full natural languages with their own grammar, they carry meaning in the hands, face and body at once, and almost no usable parallel data exists because most sign video on the web has no aligned captions. The largest openly available American Sign Language corpus holds a little over 80 hours, which is a rounding error next to speech corpora, and translation quality reflects it.

A hearing person's intuition about this problem is almost always wrong, and the wrongness has a specific shape. People assume sign language is a manual transcription of the surrounding spoken language, that a signer's hands carry the message, and that video of signing is abundant online. None of the three is true, and each false assumption produces a different failure in a dataset.

Sign languages are independent languages. British Sign Language and American Sign Language are not dialects of English or of each other. Grammatical information is carried simultaneously across the hands, the face, the eyes and the upper body, so a model watching only hands is reading a sentence with the inflection removed. And while there is a great deal of sign video on the internet, the overwhelming majority of it cannot train a supervised model, because it has no aligned caption.

Key takeaways

  • Sign languages are full natural languages with their own grammar, not manual encodings of spoken language.
  • Meaning is simultaneous and multi-channel: handshape, movement and location, plus facial expression and body posture carrying grammar, not just affect.
  • The binding constraint is aligned parallel data. Much of the sign language data on the web cannot be used for supervised training because it lacks aligned captions.
  • The largest openly available ASL corpus, How2Sign, holds more than 80 hours of video — against speech corpora measured in tens of thousands of hours.
  • Large broadcast corpora exist, but access is restricted. BOBSL comprises 1,467 hours of BBC content under heavily restricted access terms.
  • Published translation scores have been near the floor: reported BLEU figures around 0.4 to 1.0 on several tracks of the WMT 2022 shared task.
  • Sign video is inherently biometric. The signer's face is part of the linguistic signal, so anonymisation and consent are design constraints rather than afterthoughts.

Quick comparison

CorpusLanguageScaleAccess
How2SignAmerican Sign LanguageMore than 80 hours, multiview, with speech and depthOpenly available
BOBSLBritish Sign Language1,467 hours of BBC broadcastHeavily restricted
WMT-SLT (DSGS)Swiss German Sign LanguageRoughly 17,000 sentencesShared-task terms

Why can you not just scrape sign language video from the web?

Because supervised translation needs the video paired with text that corresponds to it, and almost none of the available video carries that pairing. A major impediment to progress in sign language translation is data scarcity, and specifically that much of the sign language data currently available on the web cannot be used for training supervised models due to the lack of aligned captions.

Think about what alignment means here. It is not enough to know that a ten-minute video corresponds to a ten-minute transcript. A model needs to know which span of video corresponds to which span of text, at roughly sentence granularity, in a language whose sentence boundaries are marked by prosodic and postural cues rather than punctuation. Producing that alignment is skilled annotation work performed by people fluent in the sign language, and it is the reason corpora are measured in tens or hundreds of hours rather than tens of thousands.

There is a second problem that scraping cannot solve. Web video is overwhelmingly interpretation of spoken content rather than native signing, and interpreted language has known register and structure differences from language produced natively. A corpus built entirely from interpreted broadcast is not a sample of how deaf people actually sign to each other.

How much data actually exists?

Very little, by the standards of any other modality. The openly available corpora are small, and the large ones are restricted.

How2Sign is the reference point for open ASL data: a multimodal, multiview continuous ASL dataset comprising a parallel corpus of more than 80 hours of sign language video, with corresponding speech, English transcripts and depth, plus a three-hour subset recorded in a Panoptic studio to enable detailed 3D pose estimation. Its authors describe the absence of large annotated datasets as one of the factors that has hindered progress in the field — the dataset exists as a response to the scarcity it documents.

BOBSL shows the other shape of the problem. It is a large British Sign Language corpus of 1,467 hours of BBC broadcast content, which is an order of magnitude more material than How2Sign. Its access is heavily restricted, which is a reasonable position for broadcast footage of identifiable people and also means it cannot function as a common open benchmark.

For comparison, the WMT 2022 sign language translation shared task worked from Swiss German Sign Language datasets totalling roughly 17,000 sentences, translating DSGS video to German text.

How good is machine sign language translation today?

Poor, and honestly reported as such. Published BLEU scores on several tracks of the WMT 2022 shared task sat near the floor — in the region of 0.4 on some news tracks and around 1.0 on others, with commentary noting that the best system at WMT 2023 achieved approximately 1 BLEU.

It is worth being clear about what a BLEU score near 1 means, because the number is easy to misread as "1 out of 10" or similar. BLEU runs to 100. Competent machine translation between well-resourced spoken languages lands in the 30s and 40s. A score around 1 means the output overlaps the reference translation barely more than chance.

Progress is happening, and it is coming from working around the data constraint rather than through it. One approach applies self-supervised video pretraining on anonymised and unannotated video, then supervised fine-tuning on a smaller curated parallel dataset, reaching state-of-the-art fine-tuned and zero-shot gloss-free performance on How2Sign. The pattern is familiar from low-resource spoken languages: use the abundant unlabelled material to learn representations, and spend the scarce annotation budget where supervision is irreplaceable.

What makes collection different in practice?

Three things change the operational design, and all three come back to the fact that the signer's body is the signal.

The face cannot be blurred away. In speech collection you can anonymise a recording and keep the linguistic content. In sign language the face carries grammar — question marking, negation, intensity, topic — so conventional anonymisation destroys the data. That is why privacy-aware approaches work on anonymisation methods that preserve linguistic content rather than simply obscuring identity, and why consent terms have to be unusually explicit about what identifiable video will be used for.

Capture setup is linguistic, not just technical. Framing has to include the hands, face and upper body together, with enough resolution on the face for expression and enough frame rate for rapid movement. Multiple viewpoints materially improve pose estimation. A single low-framerate camera cropped to the torso will record something that is not recoverable later.

Annotation requires fluency, not training. Alignment, segmentation and glossing must be done by people fluent in the specific sign language, which for many sign languages is a small population. This is the same constraint that governs any genuinely low-resource language programme: the limiting factor is not tooling or throughput but recruiting and retaining qualified native contributors, and paying them properly for skilled linguistic work.

For a deeper treatment of the recruitment side of this, see our note on how to recruit native contributors for African language data, and on the broader economics in low-resource language speech data collection.

Frequently asked questions

No. Sign languages are independent natural languages with their own grammar and vocabulary. American Sign Language and British Sign Language are mutually unintelligible despite both existing alongside English, and neither follows English word order or morphology. A system built on the assumption of word-for-word correspondence will fail on the grammar.

Mostly data volume, with genuine modelling difficulty on top. Speech corpora run to tens of thousands of hours; the largest open sign language corpus is a little over 80 hours. The task is also harder: the model must read several simultaneous channels from video rather than one audio stream, and sentence boundaries are not explicitly marked.

A gloss is a written label for a sign, used as an intermediate representation between video and translated text. Glosses are monotonically aligned to the video and so provide stronger supervision than free text, which makes gloss-based systems easier to train. Gloss-free approaches are more practically appealing, because glossing is expensive expert work, but they give the model less to learn from.

Not for training translation from real signers. Synthesised video can be useful for production and accessibility output, and a study with ASL signers found that videos synthesised from How2Sign could indeed be understood. But a model learning to read human signing needs human signing, with the variation in body, speed, dialect and style that synthesis does not reproduce.

Many hundreds are documented worldwide, and the vast majority have no machine translation support of any kind. Every point about scarcity in this article describes ASL and BSL, which are among the best-resourced sign languages in existence. For most of the others there is no parallel corpus at all.

Sources and further reading

  1. How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language — the more-than-80-hour ASL corpus, its modalities, the Panoptic 3D subset, and the comprehension study with ASL signers
  2. Towards Privacy-Aware Sign Language Translation at Scale — data scarcity, the missing-caption problem, biometric privacy risk, and self-supervised video pretraining
  3. Sign Language Translation from Instructional Videos — translation results and evaluation practice on How2Sign
  4. BBC-Oxford British Sign Language Dataset — Albanie, Varol, Momeni and Bull: the 1,467-hour BSL corpus known as BOBSL, and its access terms

Have an AI or visibility project in mind?

From AI evaluation and human-in-the-loop review to GEO and AEO strategy, our team can help you deploy with confidence and get found in the AI search era.

Talk to our team