Time Is the Hardest Label

Video is often treated as a stack of images to be annotated one after another. The difficult part of video is everything that happens between the frames.

One hour of video recorded at thirty frames per second contains 108,000 images.

That arithmetic is the usual starting point for explaining why video annotation is expensive, and it is misleading. If video were only a large number of images, the problem would be one of throughput, and throughput problems yield to money and tooling. Adjacent frames are nearly identical, so most of those 108,000 images carry almost no new information. Label one frame in ten and interpolate, and the volume problem largely goes away.

What remains is harder. A photograph can be labelled by answering “what is here?” A video poses questions that a photograph cannot: is this the same object as before, when did this begin, when did it end, what caused it, what is about to happen. None of these is about a frame. Each is about the relationship between frames, and each turns out to be far more ambiguous than drawing a box.

Four things that do not exist in an image

Identity. In a still image, each object is labelled once. In video, an object must keep the same identifier across every frame in which it appears. That is simple while it stays in view. It becomes a judgement when the object passes behind something and re-emerges, when two similar objects cross paths, or when it leaves the frame and something like it enters a minute later. Annotators have to decide whether this is the same car, and the answer has consequences for any model that learns to track.

Boundaries in time. An action has a start and an end. Where exactly does “picking up a cup” begin: when the hand starts moving, when it touches the cup, or when the cup leaves the table? Studies of temporal annotation have found that different annotators place these boundaries differently, and that models are sensitive to the inconsistency. Sigurdsson and colleagues, analysing action recognition datasets, found considerable ambiguity among people about the temporal extent of everyday activities. Moltisanti and colleagues showed that for interactions with objects in first-person video, labelled start and end times varied enough between annotators to change measured model accuracy, and proposed anchoring boundaries to an explicit, defined event to reduce the variation.

Granularity. The same thirty seconds can be described as “cooking”, “making an omelette”, “cracking an egg” or “moving the right hand toward the bowl”. All are correct. A dataset must choose a level, and annotators must hold to it. The AVA dataset from Google took one approach, defining eighty atomic visual actions, such as standing, holding or talking to, and labelling them for each person at one-second intervals, so that complex activities could be composed from simple parts. Other datasets label whole clips with a single high-level category. These are different tasks producing different models.

Intent and causation. Many valuable video tasks require inferring what cannot be seen. Is this pedestrian about to cross? Is this person shoplifting or putting their own phone in their pocket? Did the worker follow the safety procedure? The label depends on events before and after, on context, and on an inference about a mental state. Two attentive annotators can watch the same clip and disagree in good faith.

What large video datasets reveal

The scale of effort behind research video datasets gives a sense of the difficulty.

Kinetics, one of the standard action-recognition benchmarks, sidestepped temporal annotation by design. Each clip is about ten seconds long and carries one label for the whole clip. This made large-scale collection feasible and limited what could be learned: the label says an action occurs somewhere in the clip, not when.

Something-Something took a different route. Crowd workers were given a template, such as “pushing something from left to right”, and asked to film themselves performing it. The dataset was built to force models to attend to motion: the same objects appear in different actions, so the action cannot be guessed from a single frame. Here the annotation was moved upstream, into the instructions for recording.

Ego4D is the most ambitious example. A consortium of universities and Meta collected 3,670 hours of first-person video from 931 camera wearers in 74 locations across nine countries. The annotation covered narrations of what the wearer was doing, the moments at which objects changed state, who was speaking and to whom, and forecasts of what would happen next. Each layer needed its own guidelines, its own training and its own quality process. The paper describes an annotation effort on a scale few projects have matched.

The differences between these datasets are differences of annotation design. What a video model can do is determined in large part by which temporal questions someone decided to ask and how carefully they defined the answers.

Tools change the cost, not the judgement

Video annotation tooling has improved a great deal, and the gains are real.

The basic technique is keyframe interpolation: an annotator marks an object on selected frames and software fills in the frames between. Vondrick and colleagues studied this in early work on crowdsourced video annotation and reported a finding that still holds. The trade-off between human effort and automation was not simple. Fixed-interval keyframes were inefficient, and the best strategy depended on how the object moved. They also found that getting good results from non-expert annotators took considerable investment in interface and instructions.

More recent tools use learned models to track objects. Meta’s Segment Anything Model 2 extended promptable segmentation to video. In building its dataset, the authors report that annotation with the model in the loop was several times faster than a frame-by-frame approach, and that the model needed roughly a third as many human interactions as earlier methods to segment an object through a video.

These tools address the volume problem and the mechanical part of the identity problem. An annotator clicks once and the object is followed. When the tracker loses the object, drifts onto a neighbour or fails to re-identify it after an occlusion, a person has to notice and correct it. As in the general case discussed in the article on AI-assisted annotation, the human work shifts from drawing to supervising, and what is left is the hard part.

Tools do nothing for the other three problems. No tracker decides where “picking up” begins, which level of description to use, or whether a pedestrian intends to cross. Those remain matters of definition and judgement.

Is detailed temporal labelling still needed?

There is a serious argument that it matters less than it did. Large video-language models are trained on video paired with text, such as narration, subtitles or captions, with no frame-level labels. Self-supervised methods learn from the video itself by predicting masked or future content. If models can learn from raw video and loosely aligned text, perhaps precise temporal annotation is a legacy of an older approach.

For pretraining, that is largely right. General visual representations are increasingly learned without dense labels.

Three needs persist. Evaluation requires ground truth: to know whether a model correctly localised an event in time, someone must have marked where the event was. Fine-tuning for specific applications, such as detecting a defect on a production line, a fall in a care home or a safety violation on a site, requires examples of that specific event, precisely bounded. And the narrations and captions that weakly supervised models learn from are themselves annotations, whose quality and temporal alignment affect what is learned. Ego4D’s narrations were written by people.

So the dense labels are needed in smaller quantity and for more specific purposes, which is the pattern seen across this series.

Designing a video annotation task

Several practices follow from treating time as the central difficulty.

Define events by observable anchors. “The action starts when the hand first contacts the object” can be applied consistently. “The action starts when the person begins” cannot. Each temporal label needs an operational definition.

Fix the granularity and give examples. State the level of description and show boundary cases: what counts as one action, and what as two.

Measure temporal agreement. Agreement on video should be assessed in time, using overlap between annotators’ intervals, as well as on category. Low overlap on boundaries signals an underspecified definition.

Write identity rules. Specify what happens on occlusion, on exit and re-entry, and on merging and splitting. Without rules, each annotator invents their own.

Annotate in passes. Separating the tasks, objects first, then tracks, then events, then attributes, lets each pass have focused guidelines and suitable annotators.

Give annotators the context. Judgements about intent need the preceding and following seconds, sometimes audio, sometimes other camera views. An annotator shown an isolated clip is guessing.

Sample by event, not by time. Uniform sampling of footage yields mostly nothing happening. Use models or metadata to find candidate events and concentrate human effort there.

What this changes