In-Context Learning: Theory and Mechanisms

What Is In-Context Learning?

In-context learning (ICL) is the ability of large language models to adapt their behavior based on examples provided in the input prompt — without any gradient updates to their parameters. Given a few input-output demonstration pairs followed by a new query, the model infers the implicit task and applies it to the new input.

Example of a few-shot ICL prompt:

Translate English to French:
  sea otter → loutre de mer
  peppermint → menthe poivrée
  plush giraffe → girafe en peluche
  cheese →

The model correctly outputs “fromage” — not because it was fine-tuned on translation, but because it recognized and applied the demonstrated pattern purely from context.

ICL was observed at scale in GPT-3 (Brown et al., 2020) and has since become a central paradigm for applying large language models. Explaining why it works has become one of the most important theoretical questions in modern AI.

Empirical Characteristics

Scaling and Emergence

ICL improves dramatically with model scale. Small models (< 1B parameters) show weak ICL ability; models beyond ~10B parameters exhibit much stronger ICL. This emergent behavior suggests ICL relies on capabilities that only appear above a certain scale threshold.

Sensitivity to Demonstration Format

ICL is sensitive to:

• Label correctness: Randomly shuffled labels hurt but do not eliminate ICL — suggesting the format matters more than the specific labels.
• Input distribution: Demonstrations drawn from the task distribution help, even if labels are wrong.
• Ordering: The order of demonstrations can affect accuracy by several percentage points.
• Format: Adding separators, instruction prefixes, or few-shot templates significantly affects performance.

Task Coverage During Pretraining

ICL generalizes better to tasks that are well-represented in pretraining data. Rare tasks or novel formats yield weaker ICL, suggesting pretraining coverage is a key predictor of ICL competence.

Theoretical Perspectives

ICL as Implicit Bayesian Inference

One influential interpretation (Xie et al., 2021) frames ICL as implicit Bayesian inference over a latent concept:

$$p_\theta(y | x, C) \approx \int p(y | x, z) p(z | C) dz$$

Where:

• $C = {(x_1, y_1), \ldots, (x_k, y_k)}$ are the demonstrations.
• $z$ is a latent concept or task variable.
• $p(z | C)$ is the posterior over tasks given demonstrations.

The model, through pretraining on a mixture of tasks, learns to approximate this Bayesian inference implicitly. The demonstrations update its implicit posterior over which task is being asked.

Key assumptions of this view:

1. Pretraining data is a mixture of many tasks, each described by a latent concept.
2. Demonstrations are drawn i.i.d. from a single task.
3. The model has learned to distinguish between tasks during pretraining.

Under these assumptions, ICL is provably equivalent to Bayes-optimal posterior inference in the limit of infinite model capacity.

ICL as Gradient Descent in Forward Pass

Another theoretical perspective (Akyürek et al., 2022; Von Oswald et al., 2023) argues that Transformer attention layers implement implicit gradient descent on the demonstration examples during the forward pass.

For a linear attention transformer with $\ell$ layers, the key-value pairs in context act as a training set, and attention outputs approximate:

$$\hat{W}\ell = W\ell^{\text{init}} - \eta \sum_{(x_i, y_i) \in C} \nabla_W \mathcal{L}(W_\ell x_i, y_i)$$

This is equivalent to one step of gradient descent on the linear regression loss, with the demonstrations serving as the training data. Notably, the gradient is computed within the forward pass without any external optimizer.

Formal construction: A single attention head with linear activations computes:

$$\text{Attn}(Q, K, V) = \text{softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)V$$

With appropriate weight matrices, this implements:

$$\Delta \hat{y} = -\eta K^\top (K K^\top)^{-1} (y_\text{demo} - \hat{y}_\text{demo}) \cdot q$$

Which is an update in the direction of the residual on demonstrations — i.e., gradient descent.

ICL as Algorithm Selection

A third perspective (Garg et al., 2022) views ICL as algorithm selection from a hypothesis class learned during pretraining. The model has seen many algorithms during pretraining (sorting, regression, classification) and identifies which algorithm is appropriate for the demonstrated examples, then applies it to the query.

This view explains why ICL generalizes poorly to novel algorithmic structures not seen in pretraining: the model cannot select an algorithm it has not internalized.

Information-Theoretic View

The mutual information between demonstrations $C$ and the task label $y$ quantifies the value of ICL:

$$I(y; C | x) = H(y | x) - H(y | x, C)$$

ICL is valuable when demonstrations substantially reduce uncertainty about $y$. This is highest when:

• The model has high prior uncertainty about the task.
• Demonstrations are informative about the task structure.
• The format of demonstrations precisely specifies the required transformation.

The Role of Pretraining

Task Diversity Hypothesis

ICL ability correlates with the diversity of tasks in pretraining data. A model trained on a narrow corpus (e.g., only news articles) develops weak ICL, while a model trained on diverse web text develops strong ICL across many domains.

Formally, if pretraining data is drawn from a distribution $p_\text{meta}(\mathcal{T})$ over tasks $\mathcal{T}$, ICL competence on a new task $\mathcal{T}\text{new}$ scales with $p\text{meta}(\mathcal{T}_\text{new})$ — the “meta-coverage” of the new task.

Induction Heads

Mechanistic interpretability research (Olsson et al., 2022) identified induction heads — attention heads that implement a simple ICL operation:

1. Find the most recent token in context that matches the current query token.
2. Copy the token that followed that match in context.

Formally: if the sequence contains $\ldots [A][B] \ldots [A]$, an induction head predicts $[B]$ at position $[A]$.

Induction heads emerge at a phase transition during training on sequences of ~2 layers, and their appearance correlates with a sudden improvement in ICL ability. They implement the simplest possible form of ICL — copying patterns — and serve as the foundation for more complex ICL behavior in deeper layers.

In-Context Learning and In-Weights Learning

A key distinction:

What ICL Can and Cannot Learn

What ICL Can Learn

• Input-output mappings: Arbitrary functions from inputs to labels, given enough demonstrations.
• Format transformations: Translating, reformatting, restructuring data.
• Classification with new label names: Assigning outputs to novel label strings.
• Simple algorithms: Sorting, arithmetic operations, pattern matching.
• Domain-specific terminology: Adapting to technical jargon or specialized vocabulary.

What ICL Struggles With

• Learning new knowledge: ICL cannot inject facts not present in pretraining. If the model doesn’t know the capital of a fictional country, demonstrations won’t help.
• Very long dependencies: ICL degrades with many demonstrations if the task is complex and the model’s context window saturates.
• Novel algorithms: Tasks requiring algorithmic structures absent from pretraining (e.g., novel formal grammars) are poorly learned in context.
• Consistent rule following: Models can fail to consistently apply a demonstrated rule across all examples in a long context.

Improving ICL Performance

Demonstration Selection

Not all demonstrations are equally informative. Retrieval-based selection chooses demonstrations most similar to the query using dense retrieval:

$$C^* = \text{topk}_{(x_i, y_i) \in \mathcal{D}} \text{sim}(f(x_i), f(x^*))$$

Where $f$ is a text encoder. This outperforms random selection, especially in low-shot settings.

Chain-of-Thought Demonstrations

Including intermediate reasoning steps in demonstrations dramatically improves ICL on multi-step reasoning tasks:

Q: Roger has 5 tennis balls. He buys 2 more cans with 3 balls each. How many tennis balls does he have now?
A: Roger started with 5 balls. 2 cans × 3 balls = 6 new balls. 5 + 6 = 11. The answer is 11.

The chain-of-thought format teaches the model to reason step-by-step before answering, improving accuracy on math, logic, and commonsense tasks.

Calibrating ICL

ICL predictions can be miscalibrated if the model has a prior bias toward certain labels. Contextual calibration (Zhao et al., 2021) estimates the model’s label bias on content-free inputs (e.g., “N/A”) and applies a learned affine correction to prediction probabilities.

Self-Generated Demonstrations

When labeled examples are unavailable, models can generate their own demonstrations using zero-shot prompting, then use them as few-shot examples (bootstrapping). This “self-ICL” approach often outperforms zero-shot prompting.

ICL in Long-Context Models

As context windows grow (1M+ tokens in Gemini 1.5, Claude 3), ICL scales to many more demonstrations. This enables:

• Many-shot ICL: Hundreds to thousands of examples outperform few-shot, approaching fine-tuning accuracy for some tasks.
• In-context document understanding: The entire document becomes the context, enabling ICL over book-length inputs.
• In-context retrieval augmentation: Embedding retrieved passages directly in context as demonstration-like context.

However, long-context models can struggle with the “lost in the middle” problem: information in the middle of very long contexts is less reliably utilized than information at the beginning or end.

Open Questions

• What is the true mechanism of ICL? Bayesian inference, gradient descent, and algorithm selection are all partial explanations — a unified theory is still lacking.
• Why does label correctness matter less than format? The empirical finding that wrong labels hurt but don’t prevent ICL remains theoretically puzzling.
• What determines the ICL-to-fine-tuning gap? When is fine-tuning strictly superior to ICL, and can longer context compensate?
• How does ICL interact with instruction tuning? RLHF and instruction fine-tuning change how models process demonstrations; the interaction is not fully understood.

Summary

In-context learning is a remarkable emergent capability of large language models, enabling task adaptation at inference time without any parameter updates. Theoretical frameworks — Bayesian inference, implicit gradient descent, algorithm selection — each capture important aspects of why ICL works, but a complete mechanistic understanding remains an open research frontier. Empirically, ICL performance depends strongly on model scale, pretraining diversity, demonstration quality, and format. As context windows grow and models improve, the boundary between ICL and fine-tuning continues to blur, making ICL theory increasingly central to understanding and deploying modern language models.