Image Super-Resolution and Restoration

Single Image Super-Resolution (SISR) is the computer vision task of reconstructing a high-resolution (HR) image from a degraded, low-resolution (LR) observation. SISR is inherently an ill-posed inverse problem: for every low-resolution patch, there exist infinitely many plausible high-resolution configurations that downsample to the identical LR signal.

Deep learning has revolutionized super-resolution and image restoration by learning rich generative image priors from vast photographic datasets.

Degradation Model

The forward degradation process mapping a high-resolution ground truth image $\mathbf{I}{\text{HR}}$ to a low-resolution input $\mathbf{I}{\text{LR}}$ is mathematically formulated as:

$$\mathbf{I}{\text{LR}} = (\mathbf{I}{\text{HR}} \otimes \mathbf{k}) \downarrow_s + \mathbf{n}$$

where:

• $\otimes$ denotes spatial convolution with a blur kernel $\mathbf{k}$ (e.g., Gaussian point-spread function).
• $\downarrow_s$ represents spatial downsampling by scale factor $s \in {2\times, 4\times, 8\times}$.
• $\mathbf{n}$ is additive noise (sensor noise, compression artifacts, JPEG quantization).

The goal of a super-resolution model $f_\theta$ parameterized by weights $\theta$ is to invert this process: $\hat{\mathbf{I}}{\text{HR}} = f\theta(\mathbf{I}_{\text{LR}})$.

Evolution of Super-Resolution Architectures

+-----------------------------------------------------------------------------------+
| 1. Early CNNs (SRCNN, VDSR)                                                       |
|    Bicubic Upsample First -> Deep Convolutional Feature Extraction               |
+-----------------------------------------------------------------------------------+
                                         │
                                         ▼
+-----------------------------------------------------------------------------------+
| 2. Sub-Pixel Convolution (ESPCN, EDSR, RCAN)                                      |
|    Process in Low-Res Space -> Pixel Shuffle Upsampling Layer at the Very End    |
+-----------------------------------------------------------------------------------+
                                         │
                                         ▼
+-----------------------------------------------------------------------------------+
| 3. Adversarial Methods (SRGAN, ESRGAN)                                            |
|    Perceptual Loss (VGG Features) + Relativistic GAN -> High-Frequency Textures   |
+-----------------------------------------------------------------------------------+
                                         │
                                         ▼
+-----------------------------------------------------------------------------------+
| 4. Transformer & Diffusion Models (SwinIR, HAT, StableSR)                         |
|    Shifted Window Self-Attention / Stochastic Denoising Diffusion Restoration     |
+-----------------------------------------------------------------------------------+

Core Deep Learning Paradigms

1. Sub-Pixel Convolution (PixelShuffle)

Early models (like SRCNN) pre-upsampled the LR image using bicubic interpolation before passing it through convolutions. This introduced significant computational redundancy.

ESPCN (Shi et al., 2016) introduced the Sub-Pixel Convolutional Layer (commonly known as PixelShuffle), which extracts features entirely in the compact low-resolution space and rearranges channels into spatial dimensions at the final layer:

$$\mathcal{PS}(T){c, y, x} = T{c \cdot s^2 + \bmod(y, s) \cdot s + \bmod(x, s),, \lfloor y/s \rfloor,, \lfloor x/s \rfloor}$$

Transforming a tensor of shape $(B, C \cdot s^2, H, W)$ into $(B, C, sH, sW)$ with zero computational overhead compared to deconvolution.

2. Deep Residual Channel Attention (RCAN)

To train networks with hundreds of layers without gradient degradation, RCAN introduced residual-in-residual architectures with Channel Attention (CA) mechanisms:

• Channel attention computes global average pooling across spatial dimensions to capture channel-wise statistics.
• A gating mechanism learns inter-channel dependencies, dynamically amplifying high-frequency edge channels while attenuating flat, low-frequency regions.

3. Perceptual and Adversarial Super-Resolution (ESRGAN)

Minimizing pixel-level Mean Squared Error ($L_2$ loss) or Mean Absolute Error ($L_1$ loss) produces high Peak Signal-to-Noise Ratio (PSNR) values, but leads to overly smooth, plastic-like textures because the model averages out plausible high-frequency details.

Enhanced SRGAN (ESRGAN) resolves this through:

• Residual-in-Residual Dense Blocks (RRDB): Removes batch normalization layers to prevent color shifting and halo artifacts.
• Relativistic Average GAN (RaGAN): Evaluates the probability that real data is more realistic than generated data:

$$D_{\text{Ra}}(x_{\text{real}}, x_{\text{fake}}) = \sigma\left(C(x_{\text{real}}) - \mathbb{E}[C(x_{\text{fake}})]\right)$$

• Perceptual Loss: Evaluates $L_1$ distance in deep feature space extracted before activation layers of a pretrained VGG-19 network.

4. SwinIR: Image Restoration using Swin Transformers

SwinIR adapts shifted-window vision transformers (Swin) for image restoration:

• Local window self-attention computes dependencies within non-overlapping $8 \times 8$ pixel patches.
• Shifted windowing enables cross-window connections across successive layers.
• Residual Swin Transformer Blocks (RSTB) combine local attention with global residual shortcuts, capturing both fine textural details and long-range content context.

Evaluation Metrics

Pixel Fidelity Metrics (Favor Smoothness):
• PSNR (Peak Signal-to-Noise Ratio): Logarithmic ratio of peak signal to mean squared error.
• SSIM (Structural Similarity Index): Measures luminance, contrast, and structural consistency.

Perceptual Quality Metrics (Favor Realistic Textures):
• LPIPS (Learned Perceptual Image Patch Similarity): Distance between deep VGG/AlexNet feature activations.
• NIQE / PI (Natural Image Quality Evaluator / Perceptual Index): No-reference perceptual realism score.

The Perception-Distortion Tradeoff demonstrates mathematically that as perceptual quality improves (lower LPIPS, sharper photorealistic textures), pure pixel-matching distortion (higher PSNR) must inevitably degrade.

Summary & Key Takeaways

• PixelShuffle (sub-pixel convolution) enabled real-time super-resolution by processing feature extractions exclusively at low resolution.
• While $L_1 / L_2$ losses maximize PSNR, adversarial and perceptual losses (ESRGAN) are essential for generating sharp, realistic textures and fine hairs.
• Modern transformer backbones (SwinIR, HAT) and diffusion models (StableSR) represent the state of the art, synthesizing plausible high-resolution details in natural scenes and medical diagnostics.