Gradient Clipping: Preventing Exploding Gradients

Gradient clipping prevents training instability by limiting the magnitude of gradients during backpropagation. This is essential for RNNs, deep networks, and training with large learning rates.

The Problem

In deep or recurrent networks, gradients can grow exponentially through many layers, causing numerical instability and divergence.

Norm Clipping

Scale gradients so their norm doesn’t exceed a threshold:

class GradientClipping:
    def __init__(self, max_norm=1.0, norm_type=2):
        self.max_norm = max_norm
        self.norm_type = norm_type
    
    def clip_gradients(self, parameters):
        total_norm = 0.0
        for p in parameters:
            if p.grad is not None:
                param_norm = p.grad.data.norm(self.norm_type)
                total_norm += param_norm.item() ** self.norm_type
        
        total_norm = total_norm ** (1.0 / self.norm_type)
        clip_coef = self.max_norm / (total_norm + 1e-6)
        
        if clip_coef < 1:
            for p in parameters:
                if p.grad is not None:
                    p.grad.data.mul_(clip_coef)
        
        return total_norm


# PyTorch built-in
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

Value Clipping

Clip individual gradient values to a range:

def clip_grad_value(parameters, min_val=-1, max_val=1):
    for p in parameters:
        if p.grad is not None:
            p.grad.data.clamp_(min_val, max_val)


# PyTorch
for p in model.parameters():
    if p.grad is not None:
        p.grad.data.clamp_(min=-1, max=1)

Adaptive Gradient Clipping (AGC)

Clip gradients based on the ratio to parameter norms:

class AdaptiveGradientClipping:
    def __init__(self, eps=1e-3, clip_factor=0.01):
        self.eps = eps
        self.clip_factor = clip_factor
    
    def clip_gradients(self, parameters):
        for p in parameters:
            if p.grad is None:
                continue
            
            param_norm = p.data.norm(float('inf'))
            grad_norm = p.grad.data.norm(float('inf'))
            
            # Clip if gradient norm is too large relative to parameter
            clip_coef = self.clip_factor * (param_norm + self.eps) / (
                grad_norm + self.eps
            )
            
            if clip_coef < 1:
                p.grad.data.mul_(clip_coef)


# AGC is particularly useful for fine-tuning large pretrained models

Gradient Accumulation

When memory limits prevent large batches:

class GradientAccumulator:
    def __init__(self, model, accumulation_steps=4):
        self.model = model
        self.accumulation_steps = accumulation_steps
        self.step = 0
    
    def step(self, loss):
        loss = loss / self.accumulation_steps
        loss.backward()
        
        self.step += 1
        if self.step % self.accumulation_steps == 0:
            torch.nn.utils.clip_grad_norm_(self.model.parameters(), 1.0)
            self.optimizer.step()
            self.optimizer.zero_grad()


# Accumulates gradients over 4 forward passes before updating

Practical Guidelines

Gradient clipping and gradient penalty are complementary; clipping limits instantaneous spikes while penalty penalizes large gradients throughout training.