arxiv:2611.11890

Training Vision Transformers to Focus: Emergence of Spatial Attention Head Specialization

Published on Nov 9

· Submitted by

Youyang Ng on Nov 10

Upvote

Authors:

Youyang Ng

Abstract

Two visual focusing losses align a Vision Transformer's multi-head attention with image regions, producing specialized spatial attention heads and improving dense prediction such as zero-shot segmentation without labels.

Generated by deepseek-ai/DeepSeek-V4.1-Flash

Self-supervised learning in Vision Transformer models, such as DINO, enables robust visual perception without requiring labeled data by leveraging large-scale image datasets. However, these models lack a focusing mechanism during training, resulting in perceptual representations that are often ambiguous or overly similar. In this work, we introduce two visual focusing loss functions designed to establish correspondence between the model's attention and specific regions within images. Specifically, we leverage the multi-head attention mechanism within the model to selectively steer the model's focus, leading to the emergence of spatial attention head specialization, characterized by diverse and spatially focused perceptual capabilities without requiring annotations. Through both qualitative and quantitative evaluations, we demonstrate that our method substantially increases the spatial selectivity and diversity of attention heads, improving both interpretability and performance on dense prediction tasks such as zero-shot image segmentation.

View arXiv page View PDF Project page GitHub 2 Add to collection

Community

ngyouyang

Paper author Paper submitter 1 day ago

Vision Transformers are powerful AI models for recognizing images. They process images as if they had many "eyes", each capable of looking at different aspects of an image. However, the ways these eyes see an image can be ambiguous or overly similar, limiting their potential to capture different aspects of the image and making their roles difficult to interpret. We introduce two new training objectives that teach the models what it means to focus. The first encourages them to focus on a small region, while the second encourages them to focus on what remains visible when part of an image is masked. Trained without annotations, our models acquire diverse ways of seeing, with different eyes specializing in different regions. This makes it easier to understand what the models are looking at and improves their ability to identify and outline objects they have not been specifically trained to recognize.