Deep Reinforcement Learning - Combining Deep Learning with RL

Deep Reinforcement Learning combines the pattern recognition power of deep neural networks with the decision-making capability of reinforcement learning. This enables agents to learn from high-dimensional observations like images and video.

The Challenge: High-Dimensional Observations

Traditional RL assumes discrete, manageable state spaces. But many real problems have huge state spaces:

Image Input:

• 84×84×3 RGB image = 21,168 dimensions
• Practically infinite state space
• Traditional Q-learning infeasible

Solution: Use neural networks to approximate value functions and policies

Deep Q-Network (DQN)

Breakthrough that combined deep learning with Q-learning.

Architecture

Input: Image (84×84×3)
    ↓
Conv Layer (32 filters, 8×8)
    ↓
Conv Layer (64 filters, 4×4)
    ↓
Conv Layer (64 filters, 3×3)
    ↓
Flatten
    ↓
Dense (512 neurons)
    ↓
Output: Q-values for 18 actions

Purpose: Learn function Q(s,a) from images

Key Innovations

1. Experience Replay

Problem: Consecutive experiences highly correlated

Solution: Store experiences, sample random mini-batches

Memory = []
for each experience (s, a, r, s'):
    Memory.append((s, a, r, s'))
    
Batch = random sample from Memory
Train on batch

Benefit:

• Decorrelates data
• Reuses experiences
• More sample efficient

2. Target Network

Problem: Using same network for predictions and targets creates instability

Solution: Maintain two networks:

• Q-network: Updated frequently (current)
• Target network: Updated slowly (stable target)

Update step:
    Q(s,a) target = R + γ max_a' Q_target(s', a')
    Update Q-network to match target
    Every N steps: Copy Q-network to Q_target

Benefit: Stable training, less divergence

DQN Algorithm

Initialize Q-network and target network
Memory = empty
for episode in 1 to N:
    state = initial state
    for step in 1 to max_steps:
        action = ε-greedy(Q-network, state)
        reward, next_state = environment(action)
        Store (state, action, reward, next_state) in memory
        
        Batch = sample from memory
        Targets = reward + γ max_a' Q_target(next_state, a')
        Loss = MSE(Q-network(state, action), targets)
        Update Q-network
        
        state = next_state
        
        Every N steps:
            Copy Q-network weights to target network

Atari Success

Historic Breakthrough (2013):

• Trained single DQN on 49 Atari games
• Human-level performance on many games
• Surpassed human on majority of games

Impact:

• Validated deep RL approach
• Sparked massive interest
• Became foundational technique

Improvements to DQN

Double DQN

Problem: DQN overestimates Q-values

Solution: Use two networks for action selection and evaluation

Q(s,a) target = R + γ Q_target(s', argmax_a Q(s', a))
                       └─ Use Q to select ─┘ └ Use target to evaluate ┘

Dueling Networks

Separate value and advantage streams

Input
    ↓
Shared layers
    ├─→ Value stream → V(s)
    └─→ Advantage stream → A(s,a)
    ↓
Combine: Q(s,a) = V(s) + A(s,a) - mean(A)

Benefit: Learn value and advantages separately, more stable

Prioritized Experience Replay

Not all experiences equally important

Calculate TD-error for each experience
Experiences with high error: Learn more from
Sample with probability ∝ TD-error

Benefit: Focus on surprising experiences, faster learning

Policy Gradient Methods for Deep RL

REINFORCE with Neural Networks

Network outputs policy π(a|s)
    ↓
Actor takes action sampled from policy
    ↓
Receives reward
    ↓
∇ J(θ) ∝ ∇ log π(a|s) * return
Update network weights

Characteristics:

• Policy network directly
• High variance (single trajectory)
• Converges slowly

Actor-Critic Architecture

Combine policy gradient (actor) and value estimation (critic)

Actor Network: π(a|s)
    - What action to take
    - Policy network

Critic Network: V(s)
    - How good is state
    - Value network

Algorithm:

for each step:
    state ← current
    action ← sample from actor
    reward ← take action
    next_state ← observe
    
    TD-error = reward + γ V(next_state) - V(state)
    
    Actor loss = -log π(action|state) * TD-error
    Critic loss = (TD-error)²
    
    Update actor to increase loss
    Update critic to minimize loss

Advantages:

• Lower variance than REINFORCE
• More sample efficient
• Stable learning

A3C (Asynchronous Advantage Actor-Critic)

Parallel training for efficiency

Multiple workers parallel:
    Worker 1: Generate experience, update gradients
    Worker 2: Generate experience, update gradients
    Worker 3: Generate experience, update gradients
    ...
Central network: Aggregate updates

Benefit:

• Parallelizable
• Faster training
• Better exploration

PPO (Proximal Policy Optimization)

Simple, effective policy gradient method

Key Idea: Limit policy changes to prevent instability

New_loss = min(ratio * advantage, clip(ratio, 1-ε, 1+ε) * advantage)

Where ratio = π_new(a|s) / π_old(a|s)

Benefit:

• Stable training
• Easy to implement
• Good performance
• Widely used

Multi-Agent RL

Multiple agents learning simultaneously

Challenges

• Non-stationary environments (other agents change)
• Credit assignment (who caused success/failure?)
• Cooperation vs competition
• Communication

Approaches

Independent Learners:

• Each agent learns independently
• Assumes stationary environment
• Simple but may not converge

Centralized Training, Decentralized Execution:

• Training: Central coordinator sees all
• Execution: Each agent independent
• Better coordination, scalable

Communication: Agents learn to communicate

Exploration Strategies

Exploration essential for finding good policies

Random Actions

ε-greedy: With probability ε, random action
Problems: Inefficient, undirected

Curiosity-Driven

Explore states with high prediction error

Error = |predicted next state - actual|
High error = interesting state
Explore interesting states

Count-Based

Track state visitation

Reward += 1 / (count[state] + 1)
Bonus for visiting new states

Real-World Applications

Game Playing

• Atari: DQN mastery
• Chess: AlphaZero (novel moves, defeated Stockfish)
• Go: AlphaGo (defeated Lee Sedol)
• StarCraft: AlphaStar
• Dota 2: OpenAI Five

Robotics

Manipulation:

• Learning to grasp objects
• Stacking blocks
• Assembly tasks

Navigation:

• Robot path planning
• Obstacle avoidance
• Autonomy

Control:

• Quadrotor flight
• Bipedal walking
• Mechanical optimization

Autonomous Driving

• Lane keeping
• Traffic navigation
• Collision avoidance
• Decision making under uncertainty

Resource Optimization

• Data center cooling (saved 40% energy)
• Traffic light coordination
• Power grid management
• Healthcare resource allocation

Training Challenges

Sample Efficiency

Challenge: Needs millions of interactions

Solutions:

• Imitation learning: Learn from demonstrations
• Model-based: Learn environment model
• Transfer learning: Reuse knowledge

Exploration vs Exploitation

Too much exploration: Slow convergence Too little: Stuck in local optima

Solutions:

• Decay exploration over time
• Curiosity-driven
• Ensemble methods

Credit Assignment

Challenge: Which actions caused reward?

Solutions:

• Discount factor (γ)
• Value functions (baseline)
• Eligibility traces

Sim-to-Real Gap

Simulation differs from reality

Solutions:

• Domain randomization: Vary simulator
• Transfer learning: Adapt to real world
• Meta-learning: Learn to adapt

Tools and Frameworks

OpenAI Gym

Standardized environments:

• Classic control
• Atari
• MuJoCo robotics
• Custom environments

Stable Baselines3

Implementations of algorithms:

• DQN, PPO, A3C, SAC
• Production quality
• Easy to use

RLlib

Distributed RL framework:

• Multi-agent support
• Scales to many GPUs
• Production ready

Conclusion

Deep Reinforcement Learning combines neural networks with RL algorithms, enabling learning from high-dimensional observations. DQN pioneered the approach; policy gradient methods like PPO provide alternatives. Key innovations—experience replay, target networks, actor-critic methods—make training stable and efficient. While challenged by sample efficiency and exploration, deep RL has achieved superhuman performance in games, robotics, and autonomous systems. As techniques improve and computational resources increase, deep RL will enable increasingly complex autonomous agents.