Caption Quality Task Force 

The Global Alliance of Speech-to-Text Captioning has launched the Caption Quality Task Force, and we need your help to build our database of captioning samples. This is a multi-year data-gathering and caption-monitoring project. The Task Force will evaluate broadcast captioning quality using the NER (number, edition error and recognition error) model. Data will be gathered across the U.S. with goals of extending that reach globally.

It will take consumers and captioners working together to protect the quality of captioning services available and to also protect the profession that provides these services. There is not one without the other. Left unattended, everyone loses.

Education and awareness, benchmark standards, and service-provider certifications are needed to achieve the goal of providing and protecting equal and effective access.

As this data is collected, it will be evaluated against the WER (Word Error Rate) and NER for accuracy. The results will be organized, stored, and available to various entities for quality orders, legislation, and public education and awareness with regard to equal and effective access accommodations.

Goal: 

Gather two samples of live programming from various time slots on a weekly basis to show consistency in caption quality (e.g, was quality bad due to system failure on one day, or is quality bad daily due to poor-quality service?)

Additionally, we want samples from all markets. Our findings will be posted periodically throughout the study, and reports will be made at each board meeting.

Join us and get involved 

Volunteers should email the Task Force at taskforce@speechtotextcaptioning.org to participate in ongoing region and station monitoring assignments.

 

     Click this link to upload captioning samples (Google account required for upload).

1. When recording a sample, please make note of the following:

• Record 2-5 minutes of live captioned video with corresponding audio
• *Note: If captions are unusable, please submit with or without sound*
•  Include entire video frame (depicting any logos, time, station ID, etc.)

 2. Form will require the following information:

• Date/Time of video/photo 
• Network (ABC, CBS, FOX, NBC, MyNet, etc)
• Station City
• Source of sample (Dish, Facebook, Hulu, network website, etc)

Data-gathering goals:

Potential data collection with the help of 25 people:

1 file month 1 file a week
300 files a year 1,300 a year

Potential data collection with the help of 75 people:

1 file a month 1 file a week 2 files a week
900 files a year 3,900 files a year 7,800 files a year

   File an FCC complaint here. 

   Search Video Programming Distributor (VPD) here.