Vision Transformers Need Registers

Vision Transformers Need Registers

Timothée Darcet Affiliation: FAIR, Meta Affiliation: Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK, 38000 Grenoble, France Maxime Oquab Affiliation: FAIR, Meta Julien Mairal Affiliation: Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK, 38000 Grenoble, France Piotr Bojanowski Affiliation: FAIR, Meta

Abstract

摘要

Transformers have recently emerged as a powerful tool for learning visual representations. In this paper, we identify and characterize artifacts in feature maps of both supervised and self-supervised ViT networks. The artifacts correspond to high-norm tokens appearing during inference primarily in low-informative background areas of images, that are repurposed for internal computations. We propose a simple yet effective solution based on providing additional tokens to the input sequence of the Vision Transformer to fill that role. We show that this solution fixes that problem entirely for both supervised and self-supervised models, sets a new state of the art for self-supervised visual models on dense visual prediction tasks, enables object discovery methods with larger models, and most importantly leads to smoother feature maps and attention maps for downstream visual processing.

Transformer 近来成为学习视觉表征的有力工具。本文识别并刻画了有监督与自监督两类 ViT 特征图中的伪影：它们对应推理时主要出现在低信息量背景区域、范数很高的 token，并被模型挪作内部计算之用。我们提出一种简单而有效的做法：在 Vision Transformer 的输入序列中额外加入若干 token，专门承担这一角色。结果表明，该方案能彻底消除有监督与自监督模型上的这一问题，在稠密视觉预测任务上刷新自监督视觉模型的最优成绩，使更大模型也能用于目标发现方法，更重要的是得到更平滑的特征图与注意力图，便于下游视觉处理。

Without registers With registers Input DeiT-III OpenCLIP DINOv2 DeiT-III OpenCLIP DINOv2 Figure 1: Register tokens enable interpretable attention maps in all vision transformers, similar to the original DINO method (Caron et al., 2021). Attention maps are calculated in high resolution for better visualisation. More qualitative results are available in appendix H.

无寄存器 有寄存器 输入 DeiT-III OpenCLIP DINOv2 DeiT-III OpenCLIP DINOv2 图 1：寄存器 token 使各类视觉 Transformer 都能得到可解释的注意力图，与最初的 DINO 方法（Caron 等，2021）类似。为便于可视化，注意力图在高分辨率下计算。更多定性结果见附录 H。

1 Introduction

1 引言

Embedding images into generic features that can serve multiple purposes in computer vision has been a long-standing problem. First methods relied on handcrafted principles, such as SIFT (Lowe, 2004), before the scale of data and deep learning techniques allowed for end-to-end training. Pursuing generic feature embeddings is still relevant today, as collecting valuable annotated data for many specific tasks remains difficult. This difficulty arises because of the required expertise (e.g., medical data, or remote sensing) or the cost at scale. Today, it is common to pretrain a model for a task for which plenty of data is available and extract a subset of the model to use as a feature extractor. Multiple approaches offer this possibility; supervised methods, building on classification or text-image alignment, allow training strong feature models to unlock downstream tasks. Alternatively, self-supervised methods building on the Transformer architecture have attracted significant attention due to their high prediction performance on downstream tasks and the intriguing ability of some models to provide unsupervised segmentations (Caron et al., 2021)

把图像嵌入成可服务多种计算机视觉用途的通用特征，是长期存在的问题。早期方法依赖手工原则，如 SIFT（Lowe，2004）；随后数据规模与深度学习技术使端到端训练成为可能。追求通用特征嵌入在今天仍有意义，因为许多具体任务仍难收集有价值的标注数据——既可能缺专业背景（例如医学或遥感），也可能规模成本过高。如今常见做法是：先在数据充裕的任务上预训练，再取出模型子集作特征提取器。多条路径都能做到这一点：基于分类或图文对齐的有监督方法可训练出强特征模型以解锁下游任务；另一方面，基于 Transformer 架构的自监督方法因其下游预测性能，以及部分模型能给出无监督分割的有趣能力而备受关注（Caron 等，2021）。

In particular, the DINO algorithm is shown to produce models that contain explicit information about the semantic layout of an image. Indeed, qualitative results show that the last attention layer naturally focuses on semantically consistent parts of images and often produces interpretable attention maps. Exploiting these properties, object discovery algorithms such as LOST (Siméoni et al., 2021) build on top of DINO. Such algorithms can detect objects without supervision by gathering information in attention maps. They are effectively unlocking a new frontier in computer vision.

尤其是，DINO 算法被证明能产生显式包含图像语义布局信息的模型。定性结果也表明，最后一层注意力会自然聚焦在语义一致的区域，并常得到可解释的注意力图。LOST（Siméoni 等，2021）等目标发现算法正是建立在 DINO 之上：通过汇聚注意力图中的信息，无需监督即可检测物体，从而开拓计算机视觉的新边界。

DINOv2 (Oquab et al., 2023), a follow-up to DINO, provides features that allow tackling dense prediction tasks. DINOv2 features lead to successful monocular depth estimation and semantic segmentation with a frozen backbone and linear models. Despite the strong performance on dense tasks, we observed that DINOv2 is surprisingly incompatible with LOST. When used to extract features, it delivers disappointing performance, only on par with supervised alternative backbones in this scenario. This suggests that DINOv2 behaves differently than DINO. The investigation described in this work notably exposes the presence of artefacts in the feature maps of DINOv2 that were not present in the first version of this model. These are observable qualitatively using straightforward methods. Also surprisingly, applying the same observations to supervised vision transformers exposes similar artifacts, as shown in Fig. 2. This suggests that DINO is, in fact, an exception, while DINOv2 models match the baseline behavior of vision transformers.

DINO 的后续工作 DINOv2（Oquab 等，2023）提供了足以应对稠密预测任务的特征：在冻结骨干与线性头的设定下，即可做好单目深度估计与语义分割。尽管稠密任务表现很强，我们却惊讶地发现 DINOv2 与 LOST 几乎不兼容——用它提取特征时效果令人失望，大致只与有监督骨干相当。这说明 DINOv2 的行为与 DINO 不同。本文的调查尤其揭示了 DINOv2 特征图中存在初代模型没有的伪影，用简单方法即可定性观察到。更意外的是，把同样观察套到有监督视觉 Transformer 上，也会看到类似伪影，如图 2 所示。这意味着 DINO 其实是例外，而 DINOv2 更接近视觉 Transformer 的基线行为。

Input DeiT-III-B DeiT-III-L OpenCLIP-B OpenCLIP-L DINO-B DINOv2-g Figure 2: Illustration of artifacts observed in the attention maps of modern vision transformers. We consider ViTs trained with label supervision (DeiT-III), text-supervision (OpenCLIP) or self-supervision (DINO and DINOv2). Interestingly, all models but DINO exhibit peaky outlier values in the attention maps. The goal of this work is to understand and mitigate this phenomenon.

输入 DeiT-III-B DeiT-III-L OpenCLIP-B OpenCLIP-L DINO-B DINOv2-g 图 2：现代视觉 Transformer 注意力图中伪影的示意。我们考察了标签监督（DeiT-III）、文本监督（OpenCLIP）与自监督（DINO、DINOv2）训练的 ViT。有趣的是，除 DINO 外，所有模型的注意力图都出现尖峰离群值。本文旨在理解并缓解这一现象。

In this work, we set out to better understand this phenomenon and develop methods to detect these artifacts. We observe that they are tokens with roughly 10x higher norm at the output and correspond to a small fraction of the total sequence (around 2%). We also show that these tokens appear around the middle layers of the vision transformer, and that they only appear after a sufficiently long training of a sufficiently big transformer. In particular, we show that these outlier tokens appear in patches similar to their neighbors, meaning patches that convey little additional information.

本文着手更清楚地理解这一现象，并发展检测这些伪影的方法。我们观察到：它们是输出端范数大约高出 10 倍的 token，只占全序列很小一部分（约 2%）；大约出现在视觉 Transformer 的中层；且只有模型足够大、训练足够久才会出现。尤其是，这些离群 token 出现在与邻域相似的图像块上，也就是额外信息很少的位置。

As part of our investigation, we evaluate the outlier tokens with simple linear models to understand the information they contain. We observe that, compared to non-outlier tokens, they hold less information about their original position in the image or the original pixels in their patch. This observation suggests that the model discards the local information contained in these patches during inference. On the other hand, learning an image classifier on outlier patches yields significantly stronger accuracy than doing so on the other patches, suggesting that they contain global information about the image. We propose the following interpretation to these elements: the model learns to recognize patches containing little useful information, and recycle the corresponding tokens to aggregate global image information while discarding spatial information.

作为调查的一部分，我们用简单线性模型评估离群 token 所含信息。相对非离群 token，它们关于自身在图像中位置、以及对应图像块原始像素的信息更少——说明推理时模型丢弃了这些块的局部信息。另一方面，在离群块上学习图像分类器，准确率显著高于其他块，说明它们携带了图像的全局信息。我们据此解释：模型学会识别信息量很少的图像块，并回收相应 token，在丢弃空间信息的同时聚合全局图像信息。

This interpretation is consistent with an inner mechanism in transformer models that allows performing computations within a restricted set of tokens. In order to test this hypothesis, we append additional tokens - that we call registers - to the token sequence, independent of the input image. We train several models with and without this modification and observe that the outlier tokens disappear from the sequence entirely. As a result, the performance of the models increases in dense prediction tasks, and the resulting feature maps are significantly smoother. These smooth feature maps enable object discovery methods like LOST mentioned above with the updated models.

这一解释与 Transformer 中可在受限 token 集合内完成计算的内部机制一致。为检验该假设，我们在 token 序列末尾追加与输入图像无关的额外 token——称为寄存器（registers）。在有无这一改动的若干模型上训练后，我们观察到离群 token 从序列中完全消失；稠密预测任务性能上升，特征图显著更平滑；上述 LOST 一类目标发现方法也可在更新后的模型上重新奏效。

2 Problem Formulation

2 问题形式化

As shown in Fig. 2, most modern vision transformers exhibit artifacts in the attention maps. The unsupervised DINO backbone (Caron et al., 2021) has been previously praised for the quality of local features and interpretability of attention maps. Surprisingly, the outputs of the subsequent DINOv2 models have been shown to hold good local information but exhibit undesirable artifacts in attention maps. In this section, we propose to study why and when these artifacts appear. While this work focuses on alleviating artefacts in all vision transformers, we focus our analysis on DINOv2.

如图 2 所示，多数现代视觉 Transformer 的注意力图都带有伪影。无监督的 DINO 骨干（Caron 等，2021）曾因局部特征质量与注意力图可解释性而广受赞誉。令人惊讶的是，后续 DINOv2 模型的输出虽仍保有良好局部信息，注意力图却出现不良伪影。本节讨论这些伪影 为何 以及 何时 出现。尽管目标是缓解所有视觉 Transformer 中的伪影，分析将聚焦于 DINOv2。

2.1 Artifacts in the local features of DINOv2

2.1 DINOv2 局部特征中的伪影

Artifacts are high-norm outlier tokens. We want to find a quantitative way of characterizing artefacts that appear in the local features. We observe that an important difference between “artifact” patches and other patches is the norm of their token embedding at the output of the model. In Fig. 3 (left), we compare the norm of local features for a DINO and DINOv2 model given a reference image. We clearly see that the norm of artifact patches is much higher than the norm of other patches. We also plot the distribution of feature norms over a small dataset of images in Fig. 3 (right), which is clearly bimodal, allowing us to choose a simple criterion for the rest of this section: tokens with norm higher than 150 will be considered as “high-norm” tokens, and we will study their properties relative to regular tokens. This hand-picked cutoff value can vary across models. In the rest of this work, we use “high-norm” and “outlier” interchangeably.

伪影是高范数离群 token。 我们希望定量刻画局部特征中的伪影。观察到「伪影」块与其他块的重要差别在于模型输出端 token 嵌入的范数。图 3（左）比较了同一参考图像上 DINO 与 DINOv2 局部特征的范数：伪影块的范数明显高于其他块。图 3（右）在小规模图像集上画出特征范数分布，呈明显双峰，于是本节其余部分采用简单判据：范数高于 150 的 token 视为「高范数」token，并相对常规 token 研究其性质。该手工阈值可随模型变化。后文中「高范数」与「离群」可互换使用。

Outliers appear during the training of large models. We make several additional observations about the conditions in which these outlier patches appear during the training of DINOv2. This analysis is illustrated in Fig. 4. First, these high-norm patches seem to differentiate themselves from other patches around layer 15 of this 40-layer ViT (Fig. 4(a)). Second, when looking at the distribution of norms along training of DINOv2, we see that these outliers only appear after one third of training (Fig. 4(b)). Finally, when analyzing more closely models of different size (Tiny, Small, Base, Large, Huge and giant), we see that only the three largest models exhibit outliers (Fig. 4(c)).

离群点在大模型训练过程中出现。 我们还补充观察了 DINOv2 训练中这些离群块出现的条件，见图 4。首先，在这 40 层 ViT 中，高范数块大约从第 15 层开始与其他块分化（图 4(a)）。其次，沿 DINOv2 训练过程看范数分布，离群点大约在训练进行到三分之一之后才出现（图 4(b)）。最后，细看不同规模（Tiny、Small、Base、Large、Huge、giant）时，只有最大的三个模型出现离群点（图 4(c)）。

Figure 3: Comparison of local feature norms for DINO ViT-B/16 and DINOv2 ViT-g/14. We observe that DINOv2 has a few outlier patches, whereas DINO does not present these artifacts. For DINOv2, although most patch tokens have a norm between 0 and 100, a small proportion of tokens have a very high norm. We measure the proportion of tokens with norm larger than 150 at 2.37%.

图 3：DINO ViT-B/16 与 DINOv2 ViT-g/14 局部特征范数对比。DINOv2 有少量离群块，而 DINO 没有这些伪影。对 DINOv2，多数图像块 token 的范数在 0 到 100 之间，但仍有一小部分范数很高。我们测得范数大于 150 的 token 占比为 2.37%。

High-norm tokens appear where patch information is redundant. To verify this, we measure the cosine similarity between high-norm tokens and their 4 neighbors right after the patch embedding layer (at the beginning of the vision transformer). We illustrate the density plot in Fig. 5(a). We observe that high-norm tokens appear on patches that are very similar to their neighbors. This suggests that these patches contrain redundant information and that the model could discard their information without hurting the quality of the image representation. This matches qualitative observations (see Fig. 2) that they often appear in uniform, background areas.

高范数 token 出现在图像块信息冗余之处。 为验证这一点，我们在图像块嵌入层之后（视觉 Transformer 入口）测量高范数 token 与其 4 个邻居的余弦相似度，密度见图 5(a)。高范数 token 出现在与邻居非常相似的图像块上——说明这些块信息冗余，丢弃其信息未必损害图像表征质量。这与定性观察一致（见图 2）：它们常出现在均匀的背景区域。

(a) Norms along layers.

(a) 各层范数。

High-norm tokens hold little local information. In order to better understand the nature of these tokens, we propose to probe the patch embeddings for different types of information. For that we consider two different tasks: position prediction and pixel reconstruction. For each of these tasks, we train a linear model on top of the patch embeddings, and measure the performance of this model. We compare the performance achieved with high-norm tokens and with other tokens, to see if high-norm tokens contain different information than “normal” tokens.

高范数 token 几乎不含局部信息。 为更好理解这些 token 的性质，我们探测图像块嵌入所含的不同类型信息，考虑两项任务：位置预测与像素重建。对每项任务，在图像块嵌入上训练线性模型并度量性能；再比较高范数 token 与其他 token，看前者是否携带不同信息。

• Position prediction. We train a linear model to predict the position of each patch token in the image, and measure its accuracy. We note that this position information was injected in the tokens before the first ViT layer in the form of absolute position embeddings. We observe that high-norm tokens have much lower accuracy than the other tokens (Fig. 5(b)), suggesting they contain less information about their position in the image. • Pixel reconstruction. We train a linear model to predict the pixel values of the image from the patch embeddings, and measure the accuracy of this model. We observe again that high-norm tokens achieve much lower accuracy than other tokens (Fig. 5(b)). This suggests that high-norm tokens contain less information to reconstruct the image than the others.

• 位置预测 。 训练线性模型预测每个图像块 token 在图像中的位置并测准确率。注意该位置信息在第一层 ViT 之前以绝对位置嵌入形式注入。我们观察到高范数 token 的准确率远低于其他 token（图 5(b)），说明它们关于自身位置的信息更少。 • 像素重建 。 训练线性模型由图像块嵌入预测图像像素值并测准确率。同样，高范数 token 远低于其他 token（图 5(b)），说明其重建图像的信息更少。

\downarrow

	position prediction
	top-1 acc	avg. distance $\downarrow$	L2 error $\downarrow$
normal	41.7	0.79	18.38
outlier	22.8	5.09	25.23

\downarrow

	位置预测
	top-1 准确率	平均距离 $\downarrow$	L2 误差 $\downarrow$
正常	41.7	0.79	18.38
离群	22.8	5.09	25.23

Artifacts hold global information. In order to evaluate how much global information is gathered in the high-norm tokens, we propose to evaluate them on standard image representation learning benchmarks. For each image in a classification dataset, we forward it through DINOv2-g and extract the patch embeddings. From those, we choose a single token at random, either high-norm or normal. This token is then considered as the image representation. We then train a logistic regression classifier to predict the image class from this representation, and measure the accuracy. We observe that the high-norm tokens have a much higher accuracy than the other tokens (Table 1). This suggests that outlier tokens contain more global information than other patch tokens.

伪影携带全局信息。 为评估高范数 token 中汇聚了多少全局信息，我们在标准图像表征学习基准上评估它们。对分类数据集中的每张图，经 DINOv2-g 前向得到图像块嵌入，再随机取一个 token——或高范数或正常——作为图像表征，用逻辑回归预测类别并测准确率。高范数 token 的准确率远高于其他 token（表 1），说明离群 token 比其他图像块 token 含有更多全局信息。

IN1k P205 Airc. CF10 CF100 CUB Cal101 Cars DTD Flow. Food Pets SUN VOC [CLS] 86.0 66.4 87.3 99.4 94.5 91.3 96.9 91.5 85.2 99.7 94.7 96.9 78.6 89.1 normal 65.8 53.1 17.1 97.1 81.3 18.6 73.2 10.8 63.1 59.5 74.2 47.8 37.7 70.8 outlier 69.0 55.1 79.1 99.3 93.7 84.9 97.6 85.2 84.9 99.6 93.5 94.1 78.5 89.7 Table 1: Image classification via linear probing on normal and outlier patch tokens. We also report the accuracy of classifiers learnt on the class token. We see that outlier tokens have a much higher accuracy than regular ones, suggesting they are effectively storing global image information.

	IN1k	P205	Airc.	CF10	CF100	CUB	Cal101	Cars	DTD	Flow.	Food	Pets	SUN	VOC
[CLS]	86.0	66.4	87.3	99.4	94.5	91.3	96.9	91.5	85.2	99.7	94.7	96.9	78.6	89.1
normal	65.8	53.1	17.1	97.1	81.3	18.6	73.2	10.8	63.1	59.5	74.2	47.8	37.7	70.8
outlier	69.0	55.1	79.1	99.3	93.7	84.9	97.6	85.2	84.9	99.6	93.5	94.1	78.5	89.7

IN1k P205 Airc. CF10 CF100 CUB Cal101 Cars DTD Flow. Food Pets SUN VOC [CLS] 86.0 66.4 87.3 99.4 94.5 91.3 96.9 91.5 85.2 99.7 94.7 96.9 78.6 89.1 正常 65.8 53.1 17.1 97.1 81.3 18.6 73.2 10.8 63.1 59.5 74.2 47.8 37.7 70.8 离群 69.0 55.1 79.1 99.3 93.7 84.9 97.6 85.2 84.9 99.6 93.5 94.1 78.5 89.7 表 1：在正常与离群图像块 token 上做线性探测的图像分类。同时报告在类别 token 上学习的分类器准确率。离群 token 准确率远高于常规 token，说明它们确实在存储全局图像信息。

	IN1k	P205	Airc.	CF10	CF100	CUB	Cal101	Cars	DTD	Flow.	Food	Pets	SUN	VOC
[CLS]	86.0	66.4	87.3	99.4	94.5	91.3	96.9	91.5	85.2	99.7	94.7	96.9	78.6	89.1
正常	65.8	53.1	17.1	97.1	81.3	18.6	73.2	10.8	63.1	59.5	74.2	47.8	37.7	70.8
离群	69.0	55.1	79.1	99.3	93.7	84.9	97.6	85.2	84.9	99.6	93.5	94.1	78.5	89.7

2.2 Hypothesis and remediation

2.2 假设与修复

Having made these observations, we make the following hypothesis: large, sufficiently trained models learn to recognize redundant tokens, and to use them as places to store, process and retrieve global information. Furthermore, we posit that while this behavior is not bad in itself, the fact that it happens inside the patch tokens is undesirable. Indeed, it leads the model to discard local patch information (Tab. 5(b)), possibly incurring decreased performance on dense prediction tasks.

基于上述观察，我们提出假设： 足够大 、 训练足够充分 的模型会学会识别 冗余 token，并把它们当作 存储 、 处理 与 取回 全局信息的场所。进一步认为：这一行为本身未必有害，但发生在图像块 token 内部却不可取——它会让模型丢弃局部块信息（表 5(b)），可能损害稠密预测性能。

We therefore propose a simple fix to this issue: we explicitly add new tokens to the sequence, that the model can learn to use as registers. We add these tokens after the patch embedding layer, with a learnable value, similarly to the [CLS] token. At the end of the vision transformer, these tokens are discarded, and the [CLS] token and patch tokens are used as image representations, as usual. This mechanism was first proposed in Memory Transformers (Burtsev et al., 2020), improving translation tasks in NLP. Interestingly, we show here that this mechanism admits a natural justification for vision transformers, fixing an interpretability and performance issue that was present otherwise.

因此我们提出简单修复：显式向序列加入新 token，供模型当作寄存器使用。这些 token 加在图像块嵌入层之后，取值可学习，形式类似 [CLS] token。视觉 Transformer 末端丢弃它们，仍用 [CLS] 与图像块 token 作图像表征。该机制最早由 Memory Transformers（Burtsev 等，2020）提出，用于改进 NLP 翻译任务。有趣的是，我们表明它对视觉 Transformer 有自然理由：修复原本存在的可解释性与性能问题。

We note that we have not been able to fully determine which aspects of the training led to the appearance of artifacts in different models. The pretraining paradigm seems to play a role, as OpenCLIP and DeiT-III exhibit outliers both at size B and L (Fig. 2). However, the model size and training length also play important parts, as observed in Fig. 4.

需要指出，我们尚未完全厘清训练中哪些因素导致不同模型出现伪影。预训练范式似乎有影响：OpenCLIP 与 DeiT-III 在 B、L 规模上都有离群点（图 2）。但模型规模与训练时长同样重要，如图 4 所示。

N

N

3 Experiments

3 实验

In this section, we validate the proposed solution by training vision transformers with additional [reg] register tokens. We evaluate the effectiveness of our approach by a quantitative and qualitative analysis. We then ablate the number of registers used for training, to check that they do not cause a performance regression, evaluate an unsupervised object discovery method atop our features and finally provide a qualitative analysis of the patterns learnt by the registers.

本节通过训练带额外 [reg] 寄存器 token 的视觉 Transformer 验证所提方案。我们用定量与定性分析评估有效性，再消融训练所用寄存器数量以确认不会造成性能回退，在特征之上评估无监督目标发现方法，最后定性分析寄存器学到的模式。

3.1 Training algorithms and data

3.1 训练算法与数据

As the proposed solution is a simple architectural change, we can easily apply it to any training procedure. We try it on three different state-of-the-art training methods for supervised, text-supervised, and unsupervised learning, shortly described below.

所提方案只是简单的架构改动，易于套到任意训练流程。我们在有监督、文本监督与无监督学习的三种前沿训练方法上试用，简述如下。

DEIT-III (Touvron et al., 2022) is a simple and robust supervised training recipe for classification with ViTs on ImageNet-1k and ImageNet-22k. We choose this method as an example of label-supervised training as it is simple, uses the base ViT architecture, achieves strong classification results, and is easy to reproduce and modify with our improvements. We run this method on the ImageNet-22k dataset, using the ViT-B settings, as provided in the official repository [^1].

DEIT-III （Touvron 等，2022）是在 ImageNet-1k 与 ImageNet-22k 上用 ViT 做分类的简单稳健有监督配方。我们选它作为标签监督的代表：设定简单、使用基础 ViT 架构、分类结果强，且易于复现并用我们的改进改写。在 ImageNet-22k 上按官方仓库提供的 ViT-B 设定运行[^1]。

OpenCLIP (Ilharco et al., 2021) is a strong training method for producing text-image aligned models, following the original CLIP work. We chose this method as an example of text-supervised training because it is open-source, uses the base ViT architecture, and is easy to reproduce and modify with our improvements. We run the OpenCLIP method on a text-image-aligned corpus based on Shutterstock that includes only licensed image and text data. We use a ViT-B/16 image encoder, as proposed in the official repository [^2].

OpenCLIP （Ilharco 等，2021）是沿 CLIP 原工作得到强图文对齐模型的训练方法。我们选它作为文本监督代表：开源、使用基础 ViT、易于复现与修改。数据为基于 Shutterstock、仅含授权图文的对齐语料，图像编码器为官方仓库建议的 ViT-B/16[^2]。

DINOv2 (Oquab et al., 2023) is a self-supervised method for learning visual features, following the DINO work. We apply our changes to this method as it is the main focus of our study. We run this method on ImageNet-22k with the ViT-L configuration. We use the official repository [^3].

DINOv2 （Oquab 等，2023）是沿 DINO 学习视觉特征的自监督方法，也是本研究的主要对象。我们在 ImageNet-22k 上以 ViT-L 配置运行，使用官方仓库[^3]。

3.2 Evaluation of the proposed solution

3.2 对所提方案的评估

As shown in Fig. 1, we get rid of the artifacts by training models with additional register tokens. In the appendix, we provide additional qualitative results for more images in Fig. 19. In order to quantitatively measure this effect, for each model, we probe the norm of features at the output of the model. We report these norms for all three algorithms with and without registers in Fig. 7. We see that when training with registers, models do not exhibit large-norm tokens at the output, which confirms the initial qualitative assessment.

如图 1 所示，用额外寄存器 token 训练即可消除伪影。附录图 19 给出更多图像的定性结果。为定量衡量，我们对每个模型探测输出端特征范数，并在图 7 中报告三种算法有无寄存器时的范数：带寄存器训练时，输出不再出现大范数 token，与最初定性判断一致。

Figure 7: Effect of register tokens on the distribution of output norms on DINOv2, OpenCLIP and DeiT-III. Using register tokens effectively removes the norm outliers that were present previously.

图 7：寄存器 token 对 DINOv2、OpenCLIP 与 DeiT-III 输出范数分布的影响。使用寄存器可有效去除原先存在的范数离群点。

Performance regression. In the previous section, we have shown that the proposed approach removes artifacts from local feature maps. In this experiment, we want to check that the use of register tokens does not affect the representation quality of those features. We run linear probing on ImageNet classification, ADE20k Segmentation, and NYUd monocular depth estimation. We follow the experimental protocol outlined in Oquab et al. (2023). We summarize the performance of the models described in Sec. 3.1 with and without register tokens in Table 2(a). We see that when using registers, models do not lose performance and sometimes even work better. For completeness, we also provided the zero-shot classification performance on ImageNet for OpenCLIP (Table 2(b)), which remains unchanged. Please note that the absolute performance of our OpenCLIP reproduction is lower due to the data source we used.

性能是否回退 。 上一节表明所提方法能去除局部特征图中的伪影。本实验检查寄存器是否影响这些特征的表征质量。我们在 ImageNet 分类、ADE20k 分割与 NYUd 单目深度估计上做线性探测，协议遵循 Oquab 等（2023）。表 2(a) 汇总第 3.1 节模型有无寄存器时的表现：使用寄存器不会掉点，有时甚至更好。完整性起见，表 2(b) 还给出 OpenCLIP 在 ImageNet 上的零样本分类，几乎不变。请注意，因所用数据源不同，我们复现的 OpenCLIP 绝对性能偏低。

\downarrow

	ImageNet	ADE20k	NYUd
	Top-1	mIoU	rmse $\downarrow$
DeiT-III	84.7	38.9	0.511
DeiT-III+reg	84.7	39.1	0.512
OpenCLIP	78.2	26.6	0.702
OpenCLIP+reg	78.1	26.7	0.661
DINOv2	84.3	46.6	0.378
DINOv2+reg	84.8	47.9	0.366

		ImageNet
		Top-1
OpenCLIP		59.9
OpenCLIP+reg		60.1

\downarrow

	ImageNet	ADE20k	NYUd
	Top-1	mIoU	rmse $\downarrow$
DeiT-III	84.7	38.9	0.511
DeiT-III+reg	84.7	39.1	0.512
OpenCLIP	78.2	26.6	0.702
OpenCLIP+reg	78.1	26.7	0.661
DINOv2	84.3	46.6	0.378
DINOv2+reg	84.8	47.9	0.366

		ImageNet
		Top-1
OpenCLIP		59.9
OpenCLIP+reg		60.1

4

4

Input 0 [reg] 1 [reg] 2 [reg] 4 [reg] 8 [reg] 16 [reg] Figure 8: Ablation of the the number of register tokens used with a DINOv2 model. (top) : qualitative visualization of artifacts appearing as a function of number of registers. (bottom) : performance on three tasks (ImageNet, ADE-20k and NYUd) as a function of number of registers used. While one register is sufficient to remove artefacts, using more leads to improved downstream performance.

输入 0 [reg] 1 [reg] 2 [reg] 4 [reg] 8 [reg] 16 [reg] 图 8：DINOv2 模型所用寄存器 token 数量的消融。 （上） ：伪影随寄存器数量变化的定性可视化。 （下） ：三项任务（ImageNet、ADE-20k、NYUd）随寄存器数量的性能。一个寄存器足以去除伪影，更多则进一步提升下游表现。

3.3 Object discovery

3.3 目标发现

61.9

61.9

VOC 2007 VOC 2012 COCO 20k DeiT-III 11.7 13.1 10.7 DeiT-III+reg 27.1 32.7 25.1 OpenCLIP 38.8 44.3 31.0 OpenCLIP+reg 37.1 42.0 27.9 DINOv2 35.3 40.2 26.9 DINOv2+reg 55.4 60.0 42.0 Table 3: Unsupervised Object Discovery using LOST (Siméoni et al., 2021) on models with and without registers. We evaluated three types of models trained with various amounts of supervision on VOC 2007, 2012 and COCO. We measure performance using corloc. We observe that adding register tokens makes all models significantly more viable for usage in object discovery.

	VOC 2007	VOC 2012	COCO 20k
DeiT-III	11.7	13.1	10.7
DeiT-III+reg	27.1	32.7	25.1
OpenCLIP	38.8	44.3	31.0
OpenCLIP+reg	37.1	42.0	27.9
DINOv2	35.3	40.2	26.9
DINOv2+reg	55.4	60.0	42.0

VOC 2007 VOC 2012 COCO 20k DeiT-III 11.7 13.1 10.7 DeiT-III+reg 27.1 32.7 25.1 OpenCLIP 38.8 44.3 31.0 OpenCLIP+reg 37.1 42.0 27.9 DINOv2 35.3 40.2 26.9 DINOv2+reg 55.4 60.0 42.0 表 3：在有无寄存器的模型上用 LOST（Siméoni 等，2021）做无监督目标发现。我们评估了三种监督程度的模型在 VOC 2007、2012 与 COCO 上的表现，指标为 corloc。加入寄存器 token 后，所有模型在目标发现上的可用性都明显提高。

	VOC 2007	VOC 2012	COCO 20k
DeiT-III	11.7	13.1	10.7
DeiT-III+reg	27.1	32.7	25.1
OpenCLIP	38.8	44.3	31.0
OpenCLIP+reg	37.1	42.0	27.9
DINOv2	35.3	40.2	26.9
DINOv2+reg	55.4	60.0	42.0

3.4 Qualitative evaluation of registers

3.4 寄存器的定性评估

In this final experiment, we qualitatively probe for the behavior of register tokens. We want to verify if they all exhibit similar attention patterns or whether a differentiation automatically emerges. To this end, we plot the attention maps of the class and register tokens to patch tokens. The result of this visualization is shown in Fig. 9. We see that registers do not have a completely aligned behavior. Some selected registers exhibit interesting attention patterns, attending to the different objects in the scene. While nothing enforced this behavior, their activations had some natural diversity. We leave the study of the regularization of registers for future work.

最后一组实验定性探测寄存器 token 的行为：它们是都呈现相似注意力模式，还是会自动分化？我们绘制类别 token 与寄存器 token 对图像块 token 的注意力图，见图 9。寄存器行为并不完全一致——部分寄存器出现有趣模式，会关注场景中不同物体。尽管训练并未强制这一点，它们的激活仍带有一定自然多样性。寄存器的正则化研究留作未来工作。

Input [CLS] [reg0] [reg6] [reg8] [reg 12] Figure 9: Comparison of the attention maps of the [CLS] and register tokens. Register tokens sometimes attend to different parts of the feature map, similarly to slot attention (Locatello et al., 2020). This behaviour was never required from the model, and emerged naturally from training.

输入 [CLS] [reg0] [reg6] [reg8] [reg 12] 图 9：[CLS] 与寄存器 token 注意力图对比。寄存器有时会关注特征图的不同部分，类似 slot attention（Locatello 等，2020）。这一行为从未被要求，而是训练中自然涌现。

4 Related Work

4 相关工作

用预训练模型提取特征 。 自在 ImageNet-1k（Russakovsky 等，2015）上预训练的 AlexNet（Krizhevsky 等，2012）CNN 起，用预训练网络提取视觉特征经久不衰。更新近的模型用现代架构升级同一设定，如 ResNet（例如 DETR，Carion 等，2020）乃至 Vision Transformer。因 Transformer 易在训练时处理多模态，现成骨干常在标签监督（如 ImageNet-22k 上的 DeiT-III，Touvron 等，2022）或文本监督（如 CLIP，Radford 等，2021）下训练，成为强 视觉基础模型 ，随规模扩展良好，并在检测（Carion 等，2020）与分割（Zheng 等，2021；Kirillov 等，2023）等任务上表现优异。此时监督依赖标签或文本对齐形式的标注；数据集偏差（Torralba & Efros，2011）刻画不清，却驱动学习并塑造模型。另一路径是不用监督，让模型通过需要理解图像内容的代理任务从数据中学习（Doersch 等，2015）。这一自监督范式在多种 ViT 方法中展开：MAE（He 等，2022）训练模型重建图像被遮挡区域的像素，再微调应对新任务；自蒸馏一族（He 等，2020；Caron 等，2021；Zhou 等，2022）则展示了冻结骨干上的强性能，使任务专用下游模型对域移更稳健。本文分析聚焦自监督学习，尤其是对学习局部特征特别有效的 DINOv2（Oquab 等，2023）。我们表明：尽管基准分数优秀，DINOv2 特征仍有不良伪影；在学习过程中纠正它们还能进一步抬升基准表现。更令人惊讶的是，DINOv2 建立在并无伪影迹象的 DINO（Caron 等，2021）之上。随后我们在 DeiT-III 与 OpenCLIP 上验证，纠正技术对有监督范式同样成立。

Transformer 中的额外 token 。 用特殊 token 扩展 Transformer 序列因 BERT（Devlin 等，2019）而流行。但多数做法加入新 token 是为了：向网络提供新信息（如 BERT 的 [SEP]）；让网络在输入上多花计算（如 AdaTape 的 tape tokens，Xue 等，2023）；或在这些 token 中汇聚信息，并把其输出当作模型输出——分类用 BERT/ViT 的 [CLS]（Dosovitskiy 等，2021）；生成用 BERT/BEiT 的 [MASK]（Bao 等，2021）；检测用 DETR 的 object queries（Carion 等，2020）、YOLOS 的 detection tokens（Fang 等，2021）与 ViDT（Song 等，2021）；或多模态信息在解码前的累积，如 Perceiver 的 latent token 数组（Jaegle 等，2021；Jaegle 等，2022）。与这些工作不同，我们加入的 token 不提供信息，其输出也不作任何用途——它们只是寄存器，供模型在前向过程中学习存储与取回信息。更接近的是 Memory Transformer（Burtsev 等，2020）：用加入序列的记忆 token 改进 Transformer，提升翻译表现；后续 Bulatov 等（2022）处理复杂的复制-重复-反转任务；Sandler 等（2022）把该线扩展到视觉微调，但观察到此类 token 跨任务迁移不佳。相比之下，我们不做微调，而是在预训练阶段使用额外 token，以改进面向所有下游任务的特征。更重要的是，第 2 节给出新洞见：记忆 token 所实现的机制已在视觉 Transformer 中自然出现；我们的研究表明， 这类 token 不是创造、而是隔离这一既有行为 ，从而避免附带副作用。

Attention maps of vision transformers. Visualising the attention map from [CLS] token to patch tokens was popularized in DINO (Caron et al., 2021). It was shown there that the attention maps of DINO were clean of artifacts, as opposed to the attention maps of previous vision transformers. Other works have since reported interesting attention maps using various techniques: by modifying the optimisation procedure (Chen et al., 2022), by steering the attention scores towards useful image parts (Shi et al., 2023), by modifying the architecture of the transformer layers (Yu et al., 2024), or by introducing a learnable pooling to produce the [CLS] token (Psomas et al., 2023).

视觉 Transformer 的注意力图 。 从 [CLS] 到图像块 token 的注意力可视化因 DINO（Caron 等，2021）而流行；文中表明 DINO 的注意力图干净无伪影，与此前视觉 Transformer 不同。此后多项工作用不同技术报告有趣注意力图：改优化过程（Chen 等，2022）、把注意力分数导向有用图像区域（Shi 等，2023）、改 Transformer 层架构（Yu 等，2024）、或引入可学习池化以产生 [CLS]（Psomas 等，2023）。

5 Conclusion

5 结论

In this work, we exposed artifacts in the feature maps of DINOv2 models, and found this phenomenon to be present in multiple existing popular models. We have described a simple method to detect these artifacts by observing that they correspond to tokens with an outlier norm value at the output of the Transformer model. Studying their location, we have proposed an interpretation that models naturally recycle tokens from low-informative areas and repurpose them into a different role for inference. Following this interpretation, we have proposed a simple fix, consisting of appending additional tokens to the input sequence that are not used as outputs, and have found that this entirely removes the artifacts, improving the performance in dense prediction and object discovery. Moreover, we have shown that the proposed solution also removes the same artifacts present in supervised models such as DeiT-III and OpenCLIP, confirming the generality of our solution.

本文揭示了 DINOv2 特征图中的伪影，并发现多种流行模型也存在这一现象。我们描述了简单检测方法：它们对应 Transformer 输出端范数离群的 token。研究其位置后，我们提出解释：模型自然回收低信息区域的 token，并在推理中改派用途。据此提出简单修复——向输入序列追加不作输出的额外 token——发现这能彻底消除伪影，并改进稠密预测与目标发现。我们还表明该方案同样去除 DeiT-III、OpenCLIP 等有监督模型中的同类伪影，印证了通用性。

Acknowledgments We thank Hu Xu, Oriane Siméoni, Mido Assran and Armand Joulin for their insightful discussions and help during the course of this work. We thank Pyrrhus for posing for fig 8. Julien Mairal was supported by ANR 3IA MIAI@Grenoble Alpes (ANR-19-P3IA-0003) and by ERC grant number 101087696 (APHELEIA project).

致谢 感谢 Hu Xu、Oriane Siméoni、Mido Assran 与 Armand Joulin 在本研究过程中富有洞见的讨论与帮助。感谢 Pyrrhus 为图 8 摆拍。Julien Mairal 得到 ANR 3IA MIAI@Grenoble Alpes（ANR-19-P3IA-0003）与 ERC 资助号 101087696（APHELEIA 项目）的支持。

References

参考文献

Bao et al. (2021) Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. In ICLR , 2021. Bulatov et al. (2022) Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. Recurrent memory transformer. In NeurIPS , 2022. Burtsev et al. (2020) Mikhail S Burtsev, Yuri Kuratov, Anton Peganov, and Grigory V Sapunov. Memory transformer. arXiv preprint arXiv:2006.11527 , 2020. Carion et al. (2020) Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV , 2020. Caron et al. (2021) Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV , 2021. Chen et al. (2022) Xiangning Chen, Cho-Jui Hsieh, and Boqing Gong. When vision transformers outperform resnets without pre-training or strong data augmentations. In ICLR , 2022. Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL , 2019. Doersch et al. (2015) Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In ICCV , 2015. Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2021. Fang et al. (2021) Yuxin Fang, Bencheng Liao, Xinggang Wang, Jiemin Fang, Jiyang Qi, Rui Wu, Jianwei Niu, and Wenyu Liu. You only look at one sequence: Rethinking transformer in vision through object detection. In NeurIPS , 2021.

Psomas et al. (2023) Bill Psomas, Ioannis Kakogeorgiou, Konstantinos Karantzalos, and Yannis Avrithis. Keep it simpool: Who said supervised transformers suffer from attention deficit? In ICCV , 2023. Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021. Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. IJCV , 2015. Sandler et al. (2022) Mark Sandler, Andrey Zhmoginov, Max Vladymyrov, and Andrew Jackson. Fine-tuning image transformers using learnable memory. In CVPR , 2022. Shi et al. (2023) Baifeng Shi, Siyu Gai, Trevor Darrell, and Xin Wang. Toast: Transfer learning via attention steering, 2023. Siméoni et al. (2021) Oriane Siméoni, Gilles Puy, Huy V Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet, and Jean Ponce. Localizing objects with self-supervised transformers and no labels. In BMVC , 2021. Song et al. (2021) Hwanjun Song, Deqing Sun, Sanghyuk Chun, Varun Jampani, Dongyoon Han, Byeongho Heo, Wonjae Kim, and Ming-Hsuan Yang. Vidt: An efficient and effective fully transformer-based object detector. In ICLR , 2021. Torralba & Efros (2011) Antonio Torralba and Alexei A. Efros. Unbiased look at dataset bias. In CVPR , 2011. Touvron et al. (2022) Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit iii: Revenge of the vit. In ECCV , 2022. Wang et al. (2023) Xudong Wang, Rohit Girdhar, Stella X Yu, and Ishan Misra. Cut and learn for unsupervised object detection and instance segmentation. In CVPR , 2023.

Xue et al. (2023) Fuzhao Xue, Valerii Likhosherstov, Anurag Arnab, Neil Houlsby, Mostafa Dehghani, and Yang You. Adaptive computation with elastic input sequence. In ICML , 2023. Yu et al. (2024) Yaodong Yu, Tianzhe Chu, Shengbang Tong, Ziyang Wu, Druv Pai, Sam Buchanan, and Yi Ma. Emergence of segmentation with minimalistic white-box transformers. In CPAL , 2024. Zheng et al. (2021) Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In CVPR , 2021. Zhou et al. (2022) Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. In ICLR , 2022.

Appendix A Interpolation artifacts and outlier position distribution

附录 A 插值伪影与离群位置分布

Figure 10: Feature norms along locations: proportion of tokens with norm larger than the cutoff value at a given location. Left: official DINOv2 model (no antialiasing), right: our models (with antialiasing). At some positions, more than 20% of tokens have a high norm.

图 10：各位置上的特征范数：给定位置处范数超过阈值的 token 比例。左：官方 DINOv2 模型（无抗锯齿）；右：我们的模型（有抗锯齿）。部分位置超过 20% 的 token 具有高范数。

16\times 16\rightarrow 7\times 7

16\times 16\rightarrow 7\times 7

We plot in Figure 10 (left) the proportion of outlier tokens, characterized by a norm larger than the cutoff value defined manually, following the distribution of norms shown in Fig. 3 (main text). We make two observations:

图 10（左）画出离群 token 比例，其判据为范数大于按图 3（正文）范数分布手工设定的阈值。我们有两点观察：

16\times 16

16\times 16

Second, the outliers tend to appear in areas closer to the border of the feature map rather than in the center. Our interpretation is that the base model tends to recycle tokens in low-informative areas to use as registers: pictures produced by people tend to be object-centric, and in this case the border areas often correspond to background, which contains less information than the center.

其次，离群点更易出现在特征图靠近边界而非中心的区域。我们的解释是：基座模型倾向于回收低信息区域的 token 当作寄存器；人们拍摄的照片多为以物体为中心，边界常对应背景，信息量少于中心。

Appendix B Complexity analysis

附录 B 复杂度分析

Figure 12: Increase in model parameter and FLOP count when adding different numbers of registers. Adding registers can increase model FLOP count by up to 6% for 16 registers. However, in the more common case of using 4 registers, that we use in most of our experiments, this increase is below 2%. In all cases, the increase in model parameters is negligible.

图 12：加入不同数量寄存器时模型参数量与 FLOP 的增幅。16 个寄存器最多可使 FLOP 增加约 6%；更常见的、我们多数实验使用的 4 个寄存器情形下，增幅低于 2%。无论哪种情况，参数量增幅都可忽略。

n=4

n=4

Appendix C Analysis of LOST performance

附录 C LOST 性能分析

The results presented in Sec. 3.3 show that adding registers allows us to obtain better object discovery performance with DINOv2 models. The conclusions for the two other models studied in this work could be more crisp. In order to understand why this is so, we qualitatively study the impact of removing artifacts on the intermediate computations in the LOST algorithm. We show the intermediate outputs of LOST for all models on a given input image in Fig. 13.

第 3.3 节结果表明，加入寄存器可提升 DINOv2 上的目标发现表现；对其余两个模型，结论则不那么干脆。为理解原因，我们定性研究去除伪影对 LOST 算法中间计算的影响，并在图 13 给出同一输入图像上各模型的 LOST 中间输出。

Adding registers improves the scores and the resulting seed expansion for DeiT-III and DINOv2. This observation is coherent with the improved numbers reported in Table 3. For OpenCLIP, however, the LOST algorithm seems robust to the type of outliers observed in the local features. Adding registers does remove artifacts (as clearly shown in Fig. 20) but does not have much impact on the LOST score. It is also worth noting that OpenCLIP, with or without registers, provides comparable performance to DINOv2 without registers and DeiT-III with registers. The qualitative assessment is coherent with the numbers reported in Table 3.

加入寄存器改善了 DeiT-III 与 DINOv2 的分数及所得种子扩展，与表 3 中提升的数字一致。对 OpenCLIP，LOST 似乎对局部特征中所见类型的离群点较为稳健：加入寄存器确实去除伪影（图 20 清楚显示），但对 LOST 分数影响不大。也值得注意：无论有无寄存器，OpenCLIP 的表现都与无寄存器的 DINOv2、有寄存器的 DeiT-III 大致相当。定性判断与表 3 一致。

Figure 13: Illustration of the intermediate computations in the LOST algorithm for all models. Adding registers drastically improves the look of all intermediate steps for DeiT-III and DINOv2. The difference is less striking for the OpenCLIP model.

图 13：各模型在 LOST 算法中中间计算的示意。加入寄存器显著改善 DeiT-III 与 DINOv2 各中间步骤的观感；对 OpenCLIP 差别不那么醒目。

A surprising observation is that despite the existence of high-norm patches in the output of OpenCLIP models without registers (as seen in Fig. 7), the seed expansion score in Fig. 13 looks smooth. In the LOST experiment with OpenCLIP models, we do not use the features directly, but the values from the computation of attention maps. In Fig. 14, we show the seed expansion score for OpenCLIP models with and without registers for keys, queries and values. We see that artifacts are clearly visible as spots in the background for keys and queries, for the model without registers. As soon as registers are used, the LOST score is focusing on the object, with a smoother score for values. We qualitatively observe that for the OpenCLIP model, the value projection filters out the outliers even without registers. This means that the outliers appear to live in the null space of the value projection layer; the investigation for this phenomenon is left for future work.

一个意外观察是：尽管无寄存器的 OpenCLIP 输出存在高范数块（见图 7），图 13 中的种子扩展分数却看起来平滑。在 OpenCLIP 的 LOST 实验中，我们不用特征本身，而用注意力图计算中的 values。图 14 给出有无寄存器时，分别用 keys、queries、values 的 OpenCLIP 种子扩展分数：无寄存器时，keys 与 queries 的背景中清晰可见斑点状伪影；一旦使用寄存器，LOST 分数聚焦物体，values 上的分数也更平滑。定性看，OpenCLIP 的 value 投影即使无寄存器也会滤掉离群点——亦即离群点似乎落在 value 投影层的零空间中；对该现象的调查留作未来工作。

Figure 14: Illustration of the seed expansion score in LOST for an OpenCLIP model with and without registers for the three types of features considered: keys, queries, and values. The score is qualitatively improved across all features, with fewer artifacts appearing. Interestingly, the seed expansion map computed using values does not exhibit artifacts with nor without registers.

图 14：有无寄存器的 OpenCLIP 模型上，LOST 对三类特征（keys、queries、values）的种子扩展分数示意。各类特征上分数都定性改善，伪影更少。有趣的是，用 values 计算的种子扩展图无论有无寄存器都不显伪影。

Appendix D Behavior of models trained with registers

附录 D 带寄存器训练的模型行为

In order to better understand the phenomenon at hand, we examine the question of to what extent did the register tokens ”replace” the high-norm tokens and took on the same role.

为更好理解手头现象，我们考察：寄存器 token 在多大程度上「取代」了高范数 token，并承担相同角色。

D.1 Norms

D.1 范数

(a) DINOv2 - no register

(a) DINOv2 - 无寄存器

In Fig. 15 we compare the distribution of token norms for a model with or without registers. This figure is similar to Fig. 7 but with a finer granularity, as we also plot the norm distribution of individual register tokens and [CLS] tokens. We observe the following: with registers, the norms of patch tokens do not contain outliers anymore, and the high-norm tokens are entirely contained in the set of registers. As a result, we conclude that the behavior leading to high-norm outliers in the model is effectively absorbed in the registers.

图 15 比较有无寄存器时 token 范数的分布。该图类似图 7，但粒度更细：我们也画出各个寄存器 token 与 [CLS] token 的范数分布。观察到：有寄存器时，图像块 token 的范数不再含离群点，高范数 token 完全落在寄存器集合中。因此，导致高范数离群的行为被有效吸收进寄存器。

An additional interesting observation is that the norms of the registers appear to be quantized, compared to the previous outliers; we leave the investigation of this phenomenon for future work.

另一个有趣观察是：相对原先的离群点，寄存器的范数呈现量子化模样；对该现象的调查留作未来工作。

D.2 Information held by tokens

D.2 Token 所含信息

We report on table 4 the linear probing performance of models trained with and without registers, when using different tokens as representations. We evaluate on the aircrafts dataset, as it showed clear conclusions in the similar table 1. We observe that adding a register does not significantly modify the scores obtained with the [CLS] or patch tokens. However, the outlier patches are removed, and their behavior is transferred to the newly added register.

表 4 报告有无寄存器训练的模型在以不同 token 作表征时的线性探测表现。评估在 aircrafts 数据集上进行，因类似表 1 中结论清晰。观察到：加入寄存器不会显著改变 [CLS] 或图像块 token 上的分数；但离群块被去除，其行为转移到新加的寄存器上。

top-1 accuracy #registers [CLS] normal patch outlier patch register 0 84.6 15.5 73.3 N/A 1 85.2 14.5 N/A 71.1 Table 4: Linear probing of models with and without registers on the Aircraft dataset, using various tokens as representation. We observe that the behavior of the outlier tokens, aggregating global information, is absorbed into the register.

	top-1 accuracy
#registers	[CLS]	normal patch	outlier patch	register
0	84.6	15.5	73.3	N/A
1	85.2	14.5	N/A	71.1

top-1 准确率 #寄存器 [CLS] 正常图像块 离群图像块 寄存器 0 84.6 15.5 73.3 N/A 1 85.2 14.5 N/A 71.1 表 4：有无寄存器的模型在 Aircraft 数据集上、以不同 token 作表征的线性探测。我们观察到离群 token 聚合全局信息的行为被吸收进寄存器。

	top-1 准确率
#寄存器	[CLS]	正常图像块	离群图像块	寄存器
0	84.6	15.5	73.3	N/A
1	85.2	14.5	N/A	71.1

We further conduct an evaluation of the local information contained in the patch tokens of a model trained with and without registers (table 5). We observe that the non-outliers patches, in both cases, hold similar local information, confirming that the registers only remove the outlier behavior, without significantly modifying the information held by the other patches.

我们进一步评估有无寄存器训练的模型中图像块 token 所含局部信息（表 5）。两种情形下非离群块持有的局部信息相近，确认寄存器只去除离群行为，而不显著改变其他块所持信息。

\downarrow

		position prediction	reconstruction
#registers	patches considered	top-1 acc	L2 error $\downarrow$
0	non-outliers	66.3	15.9
4	non-outliers (ie all)	65.8	16.0

\downarrow

		位置预测	重建
#寄存器	所考虑图像块	top-1 准确率	L2 误差 $\downarrow$
0	非离群	66.3	15.9
4	非离群（即全部）	65.8	16.0

D.3 Positional focus

D.3 位置关注

(a) [CLS]

In Fig. 16 we display the positional focus for the class token and the 4 registers of a DINOv2+reg model. We produce these plots by running the model on a random subset of ImageNet-22k, and averaging the attention maps for the corresponding tokens at the last layer. We note that ImageNet-22k contains mostly object-centric images rather than scenes, which explains why the average attention maps correspond to centered blobs.

图 16 展示 DINOv2+reg 模型中类别 token 与 4 个寄存器的位置关注。做法是：在 ImageNet-22k 的随机子集上运行模型，并对最后一层相应 token 的注意力图取平均。需注意 ImageNet-22k 多为以物体为中心的图像而非场景，因此平均注意力图呈现居中团块。

We make several observations. First, the attention maps for registers can be different of each other; for example, register 3 tends to focus on border areas, while the other registers tend to focus on more centered areas. Register 2 tends to focus slightly more on the upper areas of images that others. This is consistent with Fig. 9, where we show registers focusing on different large areas of the image, suggesting some level of specialization.

有几点观察。首先，各寄存器的注意力图可以彼此不同：例如寄存器 3 倾向关注边界，其余更偏中心；寄存器 2 比其他略更关注图像上部。这与图 9 一致——寄存器关注图像中不同大片区域，暗示一定程度的分工。

Second, by comparing the register maps to the [CLS] token map and to a patch token map, we observe that registers produce maps with a large support area, very similarly to the [CLS] token, and very different of a typical patch token which is more localized. As the [CLS] token is known to carry global information (as proven by the linear probing classification performance): this suggests that registers also carry global information.

其次，把寄存器图与 [CLS] 图、以及某个图像块 token 的图对比：寄存器产生大支撑区域的图，与 [CLS] 非常相似，而与更局部化的典型图像块 token 很不同。已知 [CLS] 携带全局信息（线性探测分类表现可证），这说明寄存器也携带全局信息。

Appendix E Masked autoencoders

附录 E 掩码自编码器

Masked Autoencoding (He et al., 2022) is another common way of pretraining self-supervised models. We observe in Fig. 17 that there are no artifacts in the maps produced by MAE: our hypothesis is that the absence of artifacts is due to the training procedure using only a local loss on the patch tokens, rather than an objective involving global aggregation of information. However, we also note that the performance of MAE models is very low for self-supervised representation learning (75% linear probing performance on ImageNet classification for ViT-Large), preventing it from being used as is, and making fine-tuning a requirement.

掩码自编码（He 等，2022）是另一常见自监督预训练方式。图 17 显示 MAE 产生的图中没有伪影：我们的假设是，这源于训练过程只在图像块 token 上使用局部损失，而非涉及全局信息聚合的目标。但也要指出，MAE 作为自监督表征学习的性能很低（ViT-Large 在 ImageNet 分类线性探测约 75%），难以直接使用，微调几乎是必需。

Figure 17: First three principal components of the output feature map of a ViT-Large Masked Autoencoder.

图 17：ViT-Large 掩码自编码器输出特征图的前三个主成分。

Appendix F Behavior per attention head

附录 F 各注意力头的行为

In this section, we investigate whether the artifacts appear only on the attention maps for specific heads of the last vision transformer block, or for all of them. We show in Fig. 18 the input image along with the attention maps for different heads. We observe that the artifacts appear for all attention heads, despite heads focusing on different areas of the object. We still observe that some heads focus more on artifacts than others.

本节考察伪影是只出现在最后视觉 Transformer 块的特定注意力头上，还是所有头都有。图 18 给出输入图像与各头注意力图：尽管各头关注物体不同区域，伪影仍出现在所有注意力头上；我们仍观察到有些头比其他头更关注伪影。

Figure 18: Attention maps of the [CLS] token to the patch tokens, shown here separately per attention head. We produce these maps with a DINOv2-L model trained without registers.

图 18：[CLS] token 对图像块 token 的注意力图，按注意力头分别展示。这些图由无寄存器训练的 DINOv2-L 产生。

Appendix G Variance on token information probing

附录 G Token 信息探测的方差

The results presented in table 1 are obtained by taking a random patch token, either normal or outlier. However, the choice of this token adds a significant source of variance in the evaluation. For thoroughness, we report in table 6 the standard deviation of the scores obtained relative to this choice.

表 1 的结果通过随机取一个正常或离群的图像块 token 得到。但该选择会给评估带来显著方差。为求完备，表 6 报告相对这一选择的分数标准差。

dataset Airc. CF10 CF100 CUB Cal101 Cars DTD token normal 17.1\pm0.5 97.1\pm0.1 81.3\pm0.3 18.6\pm0.6 73.2\pm1.3 10.8\pm0.3 63.1\pm0.8 outlier 79.1\pm0.5 99.3\pm0.0 93.7\pm0.3 84.9\pm2.1 97.6\pm0.7 85.2\pm0.9 84.9\pm0.9 [CLS] 87.3 99.4 94.5 91.3 96.9 91.5 85.2 dataset Flow. Food IN1k P205 Pets SUN VOC token normal 59.5\pm1.2 74.2\pm0.3 65.8\pm0.1 53.1\pm0.3 47.8\pm0.5 37.7\pm0.3 70.8\pm0.5 outlier 99.6\pm0.0 93.5\pm0.2 69.0\pm0.7 55.1\pm1.0 94.1\pm0.2 78.5\pm0.2 89.7\pm0.1 [CLS] 99.7 94.7 86.0 66.4 96.9 78.6 89.1 Table 6: Image classification via linear probing on normal and outlier patch tokens. As we select the patch tokens randomly among the set of eligible tokens, this adds a source of variability. We report the standard deviation of this variability in grey along with the scores. This table is a detailed view of table 1.

dataset	Airc.	CF10	CF100	CUB	Cal101	Cars	DTD
token
normal	17.1±0.5	97.1±0.1	81.3±0.3	18.6±0.6	73.2±1.3	10.8±0.3	63.1±0.8
outlier	79.1±0.5	99.3±0.0	93.7±0.3	84.9±2.1	97.6±0.7	85.2±0.9	84.9±0.9
[CLS]	87.3	99.4	94.5	91.3	96.9	91.5	85.2
dataset	Flow.	Food	IN1k	P205	Pets	SUN	VOC
token
normal	59.5±1.2	74.2±0.3	65.8±0.1	53.1±0.3	47.8±0.5	37.7±0.3	70.8±0.5
outlier	99.6±0.0	93.5±0.2	69.0±0.7	55.1±1.0	94.1±0.2	78.5±0.2	89.7±0.1
[CLS]	99.7	94.7	86.0	66.4	96.9	78.6	89.1

数据集 Airc. CF10 CF100 CUB Cal101 Cars DTD token 正常 17.1\pm0.5 97.1\pm0.1 81.3\pm0.3 18.6\pm0.6 73.2\pm1.3 10.8\pm0.3 63.1\pm0.8 离群 79.1\pm0.5 99.3\pm0.0 93.7\pm0.3 84.9\pm2.1 97.6\pm0.7 85.2\pm0.9 84.9\pm0.9 [CLS] 87.3 99.4 94.5 91.3 96.9 91.5 85.2 数据集 Flow. Food IN1k P205 Pets SUN VOC token 正常 59.5\pm1.2 74.2\pm0.3 65.8\pm0.1 53.1\pm0.3 47.8\pm0.5 37.7\pm0.3 70.8\pm0.5 离群 99.6\pm0.0 93.5\pm0.2 69.0\pm0.7 55.1\pm1.0 94.1\pm0.2 78.5\pm0.2 89.7\pm0.1 [CLS] 99.7 94.7 86.0 66.4 96.9 78.6 89.1 表 6：在正常与离群图像块 token 上做线性探测的图像分类。由于在合格 token 集合中随机选取，引入了可变性；我们在分数旁用灰色报告该可变性的标准差。本表是表 1 的细化视图。

数据集	Airc.	CF10	CF100	CUB	Cal101	Cars	DTD
token
正常	17.1±0.5	97.1±0.1	81.3±0.3	18.6±0.6	73.2±1.3	10.8±0.3	63.1±0.8
离群	79.1±0.5	99.3±0.0	93.7±0.3	84.9±2.1	97.6±0.7	85.2±0.9	84.9±0.9
[CLS]	87.3	99.4	94.5	91.3	96.9	91.5	85.2
数据集	Flow.	Food	IN1k	P205	Pets	SUN	VOC
token
正常	59.5±1.2	74.2±0.3	65.8±0.1	53.1±0.3	47.8±0.5	37.7±0.3	70.8±0.5
离群	99.6±0.0	93.5±0.2	69.0±0.7	55.1±1.0	94.1±0.2	78.5±0.2	89.7±0.1
[CLS]	99.7	94.7	86.0	66.4	96.9	78.6	89.1

Appendix H Qualitative Results

附录 H 定性结果

We trained three popular models: DeiT-III, OpenCLIP, DINOv2 with and without the introduction of register tokens. We observe in Fig. 19 the attention maps in the last layer of the Vision Transformer, for all three cases. We see that our approach provides much cleaner attention maps, with considerably fewer artifacts, explaining the improvement on the downstream object discovery task mentioned in Sec. 3.3. The feature maps are also visibly improved, as shown in Fig. 20. Finally, we also show the norm of the patch tokens in Fig. 21, and confirm that in all three models, artifact patches correspond to norm outliers.

我们训练了三种流行模型 DeiT-III、OpenCLIP、DINOv2，并比较引入寄存器 token 与否。图 19 观察视觉 Transformer 最后一层的注意力图：我们的方法给出干净得多的注意力图，伪影少得多，从而解释第 3.3 节下游目标发现任务的提升。特征图同样明显改善，见图 20。图 21 还展示图像块 token 的范数，确认三种模型中伪影块都对应范数离群点。

Without registers With registers Input DeiT-III OpenCLIP DINOv2 DeiT-III OpenCLIP DINOv2 Figure 19: Attention maps of models trained without and with registers on various images.

无寄存器 有寄存器 输入 DeiT-III OpenCLIP DINOv2 DeiT-III OpenCLIP DINOv2 图 19：有无寄存器训练的模型在多张图像上的注意力图。

[-3\sigma,+3\sigma]

[-3\sigma,+3\sigma]

Without registers With registers Input DeiT-III OpenCLIP DINOv2 DeiT-III OpenCLIP DINOv2 Figure 21: Maps of token norms for models trained without and with registers on various images. The norm outliers are very visible for models trained without registers.

无寄存器 有寄存器 输入 DeiT-III OpenCLIP DINOv2 DeiT-III OpenCLIP DINOv2 图 21：有无寄存器训练的模型在多张图像上的 token 范数图。无寄存器时范数离群点非常醒目。