Dense Prediction Transformers (DPT)

Vision Transformers (ViTs) originally achieved state-of-the-art performance on image-level classification tasks by processing images as sequences of non-overlapping patches. However, applying transformers to dense prediction tasks—such as monocular depth estimation, semantic segmentation, and surface normal prediction—presents unique challenges. These tasks require pixel-level resolution and spatially fine-grained localization, whereas standard ViTs output a sequence of 1D token representations with a downsampled spatial stride (typically $16 \times 16$ pixels).

The Dense Prediction Transformer (DPT) architecture, introduced by Ranftl et al. (2021), successfully bridges this gap. By replacing convolutional backbones with a transformer encoder coupled to a novel reassemble-and-fusion decoder, DPT provides global receptive field reasoning at every stage without losing high-resolution spatial fidelity.

Why CNNs Struggle with Global Context

In conventional Convolutional Neural Networks (CNNs) like ResNet or U-Net:

• Early layers possess small, local receptive fields that grow linearly with depth.
• Long-range spatial relationships (e.g., establishing depth consistency across an entire room or road) require deep stacking and aggressive spatial downsampling (strided convolutions and pooling).
• Downsampling destroys fine spatial details, which decoders must struggle to reconstruct through skip connections.

In contrast, a Vision Transformer encoder maintains a global receptive field from the very first self-attention layer. Every image patch can directly attend to every other patch, making it inherently suited for global depth and scene reasoning.

CNN Receptive Field:
Layer 1: [■] (Local 3x3)
Layer 2: [■■■] (Intermediate)
Layer 5: [■■■■■■■] (Global, but degraded spatial resolution)

Vision Transformer (DPT Encoder):
Layer 1: [■■■■■■■■■■■■■■■■] (Full Global Context via Self-Attention from Step 1)

DPT Architecture Overview

The DPT model consists of three primary stages:

1. Transformer Encoder: Processes $16 \times 16$ non-overlapping image patches through $L$ standard transformer blocks with multi-head self-attention.
2. Reassemble Blocks: Extracts token representations from intermediate transformer layers (e.g., layers $l \in {3, 6, 9, 12}$) and restructures them into multi-scale 2D feature maps.
3. Refine & Fusion Blocks: Progressively merges the multi-scale representations using residual convolutional blocks to output a full-resolution pixel prediction map.

Input Image (H x W x 3)
         │
         ▼
   [ ViT Encoder ] ───► Tokens at Layer l=3, 6, 9, 12
         │
         ├─► Layer 12 Tokens ──► [ Reassemble s=32 ] ──► [ Fusion ] ──┐
         ├─► Layer  9 Tokens ──► [ Reassemble s=16 ] ──► [ Fusion ] ──┼─► [ Output Head ] ──► Full Resolution (H x W)
         ├─► Layer  6 Tokens ──► [ Reassemble s=8  ] ──► [ Fusion ] ──┤    (Depth / Segmentation)
         └─► Layer  3 Tokens ──► [ Reassemble s=4  ] ──► [ Fusion ] ──┘

The Reassemble Operation

The core innovation of DPT is the Reassemble Block, which converts a flat set of $N_p = \frac{H}{16} \times \frac{W}{16}$ tokens with feature dimension $D$ back into a 3D feature tensor of shape $H’ \times W’ \times C’$ at different spatial scales $s \in {4, 8, 16, 32}$.

The Reassemble operation consists of three sub-steps:

1. Readout Token Handling

The class token (if present) is either discarded or projected and added back to all spatial tokens:

$$\mathbf{t}{\text{spatial}} = \text{Project}(\mathbf{t}{1:N}) + \mathbf{t}_{\text{cls}}$$

2. Spatial Reshaping

The 1D sequence of tokens is unflattened into a 2D spatial grid:

$$\mathbf{X}{\text{2D}} = \text{Reshape}\left(\mathbf{t}{\text{spatial}}, \frac{H}{16}, \frac{W}{16}, D\right)$$

3. Spatial Rescaling (Projection & Sampling)

A $1 \times 1$ convolution projects feature dimension $D$ to intermediate channel size $\hat{C}$. Next, spatial resolution is scaled by factor $\frac{16}{s}$ using strided convolutions (to downsample) or transposed convolutions / bilinear interpolation (to upsample):

$$\text{Reassemble}{s}(\mathbf{t}) = \text{Rescale}{s}\left(\text{Conv}{1 \times 1}(\mathbf{X}{\text{2D}})\right)$$

• For $s = 4$: upsampled by $4\times$ to produce high-resolution, low-level semantic features ($\frac{H}{4} \times \frac{W}{4}$).
• For $s = 32$: downsampled by $2\times$ to produce low-resolution, high-level semantic features ($\frac{H}{32} \times \frac{W}{32}$).

Residual Fusion Blocks

Feature maps from successive scales are merged bottom-up using Residual ConvUnits (RCU) and Feature Fusion Blocks:

Feature Map at Scale s (Low Res)  ──► [ Bilinear Upsample 2x ] ──► [ RCU ] ──┐
                                                                              ├──► (+) ──► [ RCU ] ──► Next Scale
Feature Map from Reassemble (s/2) ───────────────────────────────► [ RCU ] ──┘

Each Feature Fusion Block:

1. Upsamples the coarser feature map by a factor of 2.
2. Passes both features through Residual ConvUnits.
3. Element-wise adds the features and passes the result through an additional RCU.

Finally, an output prediction head upsamples the fused features to the original image dimensions $H \times W$ and predicts continuous depth values or per-pixel class logits.

Performance Comparison on Monocular Depth

DPT provides substantially sharper object boundaries, cleaner depth discontinuities, and improved global geometric consistency compared to CNN-based baselines like ResNet-50 and standard FPNs:

Key Takeaways

• DPT demonstrates that pure and hybrid Vision Transformers can outperform deep convolutional networks on dense, pixel-level prediction problems.
• By retaining a global receptive field throughout all layers, DPT avoids the spatial distortion and global context loss common in CNN downsampling pipelines.
• The modular Reassemble and Fusion design allows arbitrary transformer backbones (ViT-Base, ViT-Large, BEiT, Swin) to be plugged directly into dense visual estimation tasks.