Vision Tower Bench

Choosing a vision tower for a text-only LLM

We probed eleven frozen vision towers across layers. No tower wins everywhere, spatial information often peaks before the final layer, and projectors can help or hurt.

Finding 1No tower wins every capabilitySee the evidence ↓Finding 2Geometry peaks before the final layerSee the evidence ↓Finding 3Projectors can help or hurtSee the evidence ↓

PaperRepo 🤗Towers

Version

Why this exists

Adding vision to a base LLM means choosing a tower, layer, and projector. Full-model benchmarks do not isolate those choices, so we probe frozen features before multimodal training.

What the capability axes mean

Recognition

Can a probe identify the object, with plenty of labels and with almost none?

Geometry

Can it recover depth and surface orientation from a single image?

Cross-view

Can it match points across views or object instances?

Occlusion

How much recognition survives when half the image is hidden?

Capability Profiles

Each column measures a separate capability. Close values may be unresolved.

Tower	Few-label1% top-1 ↑	Full-label100% top-1 ↑	Depthd1 ↑	Normalsdeg ↓	Cross-viewNAVI@2cm ↑	Occlusion50% kept ↑
DINOv2	0.369	0.908	0.702	18.85	0.539	52%
SigLIP2	0.656	0.917	0.670	23.90	0.401	49%
Muse Glimmer	0.433	0.921	0.659	24.30	0.214	52%
GLM 5.3 Flash	0.675	0.919	0.643	24.35	0.382	45%
Nemotron Omni	0.523	0.903	0.641	24.39	0.419	46%
Kimi K2.6	0.345	0.886	0.659	25.51	0.365	28%
Qwen3.8	0.245	0.879	0.674	24.24	0.378	30%
DeepSeek V4.1	0.245	0.874	0.654	25.27	0.392	35%
Gemma 4	0.316	0.874	0.555	27.25	0.304	30%
Kimi K3	0.219	0.837	0.655	27.03	0.333	20%
MiniMax M3	0.048	0.342	0.542	31.48	0.235	18%

Muse Glimmer, GLM 5.3 Flash, and SigLIP2 are unresolved at full labels.
Depth and normals use the best matched tower layer. NAVI uses the final tower layer.
↑ higher is better, ↓ lower is better.

Method, in short

Towers stay frozen. We tap eight depths plus merged and projected stages. Features cache once; only the readouts train. Recognition uses ImageNet-100, geometry uses DIODE and KITTI, and correspondence uses NAVI, ScanNet and SPair.

Heads are reduced to 512 dimensions with train-split PCA. Each cell selects a learning rate on validation and runs three seeds.

Figure 1

How the bench taps a model

Results

Finding 1

No tower wins every capability

DINOv2 leads the spatial columns. GLM 5.3 Flash leads with 1% labels. Muse Glimmer is nominally first with full labels, but the top three are unresolved.

Figure 2

Recognition rankings change with label budget

Finding 2

Geometry peaks before the final layer

Geometry peaks early in 43 of 44 arms. Recognition reaches the final layer in 17 of 22 attention arms; MiniMax M3 is the clearest exception.

Figure 3

Geometry peaks early

Stars mark each tower's depth peak in the matched arm.

Finding 3

Projectors can help or hurt

On DIODE, 25 of 36 comparisons favor the Projector, seven favor the Tower, and four are unresolved. Results also reverse across KITTI and correspondence.

Projector	DIODE				KITTI	NAVI	ScanNet	SPair
Projector	Depth (M)	Depth (R)	Normals (M)	Normals (R)	KITTI	NAVI	ScanNet	SPair
Muse Glimmer	+.026	+.093	+0.79°	+2.89°	−.004	+.106	+.032	+.133
Kimi K2.6	+.017	+.020	+2.20°	+3.11°	−.005	+.057	+.008	+.115
Kimi K3	+.078	+.027	+2.50°	+2.83°	+.001	+.059	+.021	−.014
Qwen3.8	+.058	+.160	+4.31°	+4.44°	−.002	+.019	−.019	+.022
Gemma 4	+.101	+.056	+2.72°	+2.31°	+.026	+.042	−.002	+.012
GLM 5.3 Flash	+.038	−.072	+0.97°	+0.10°	−.002	+.037	−.003	+.015
DeepSeek V4.1	+.080	−.026	+3.33°	−0.09°	−.009	+.012	−.033	−.023
Nemotron Omni	−.009	−.050	+0.37°	−0.47°	−.011	+.002	−.021	−.011
MiniMax M3	−.094	−.116	−3.15°	+0.50°	−.098	−.009	−.025	−.013

Figure 4

Projector advantage over the final tower layer

—Positive favors the Projector.
—Normals report error removed.
—M = capacity-matched to 512 dimensions.
—R = raw token width.

Projector better, interval excludes zero
Tower better, interval excludes zero
Unresolved

How to choose

Pick the tower first, then the layer, then the projector. If there is no dominant requirement, start with SigLIP2. It is strong on many tasks, with no weak axis. Regardless, a base LLM will need a projector.

Geometry peaks before the final layer in 43 of 44 arms, so tap an intermediate layer for spatial work and use Figure 3 to place it. The final layer is reasonable for recognition, except on MiniMax M3 and Qwen3.8.

If the priority is	Start with
Geometry or point correspondence	DINOv2. Tapped at 0.75 to 0.875 relative depth.
Recognition with few labels	GLM 5.3 Flash, SigLIP2. No cross-model bootstrap, so test both.
Recognition with abundant labels	Muse Glimmer, GLM 5.3 Flash, SigLIP2. Unresolved, so choose accordingly.
Reusing a native projector	Muse Glimmer, Gemma 4. Shortlist, then test on the target domain.
Fully fine-tuning the tower	Use the results to pick candidates.

What v2 does not establish

—These are frozen-representation results, not evaluations of a working VLM.
—MiniMax M3 rises to 0.40 with full-grid tokens but remains last.
—Near-ties, attention pooling, and the small geometry test sets remain unresolved.

Representation explorer

What the features look like

Qualitative PCA of frozen patch tokens. Visible structure is not a benchmark score.

source · 448²

DINOv2 · tower · 32 x 32

DINOv2 segments its subject

The eagle separates as one solid region. First on all three correspondence sets, fourth on full-label recognition.

NAVI recall@2cm0.539 · rank 1 of 11

recognition top-10.908 · rank 4 of 11

1% labels top-10.369 · rank 5 of 11

Changelog

Sept 2026

Vision Towers for Gemma 4, MiniMax M3 and GLM 5.3 Flash published on Hugging Face.

Sept 2026

Vision Towers for Deepseek V4.1 Flash and Nemotron 3 Nano Omni published on Hugging Face.

Sept 2026

Vision Tower Bench v2 adds five Towers and expands the correspondence and robustness results.

Sept 2026

Vision Tower Bench v1 released, covering six Towers on geometry, semantic and correspondence.

Sept 2026

Vision Towers for Kimi K2.6, Kimi K3, Muse Glimmer and Qwen3.8 published on Hugging Face.