Scale Labs: AI Understands Images Twice as Poorly as Humans
A tool from Scale Labs assessed how well AI can understand images and videos—for example, evaluating situations as a human would. The best model performed nearly twice as poorly as humans.
Scale Labs and the startup Elorian, founded by a former Google DeepMind researcher, developed the benchmark *Humanity's Sixth Sense*. The tool tests how well AI can interpret non-obvious visual cues—for instance, assessing whether a car will fit through a narrow passage or determining who in a room holds informal authority.
The test includes 522 tasks: 288 static scenes and 234 video clips. The questions are designed so that humans solve them intuitively. A control group of humans achieved 93.1% accuracy on the test. The best multimodal model, GPT-6 Astra, scored only 53.6%. GPT-6.1 Sol and Claude Opus 5.5 scored 46.6% and 44.6%, respectively. The median result across the 25 tested models was 30.9%.
An analysis of over 8,500 errors revealed a key issue: 94% of failures were not due to logical mistakes but because the AI failed to notice important visual markers. Only 5% of errors were attributed to weak reasoning. Increasing computational power—up to 4,000 tokens per response—did not improve results and, in some cases, even reduced accuracy. Experiments with cropping frames to simulate gaze focus had minimal effect.
The researchers concluded that scaling logical inference does not address the fundamental limitations of computer vision. Even the most advanced models still cannot perceive scene context as humans do.
