Object Detection Architectures - YOLO, R-CNN, and Beyond

Object detection—identifying objects and their locations in images—is fundamental to many AI applications. Modern architectures have made this task efficient and accurate. This post explores the major approaches.

The Object Detection Problem

Task: For each object in image:

1. Identify what it is (classification)
2. Identify where it is (localization)

Output: Bounding box + class + confidence

Example:
Dog at [50, 100, 200, 300], confidence 95%
Person at [400, 80, 550, 500], confidence 98%

R-CNN Family

The Region-based CNN family pioneered modern detection.

R-CNN (Original)

Process:

1. Generate ~2000 region proposals (using selective search)
2. Extract features from each region (CNN)
3. Classify region (SVM)
4. Refine bounding box (regression)

Architecture:

Image
  ↓
Region Proposals (2000 regions)
  ↓ (For each region)
CNN Feature Extraction
  ↓
SVM Classification
  ↓
Bounding Box Refinement

Pros: First successful deep learning detector

Cons:

• Slow (test time: 47 seconds/image)
• Complex pipeline
• Many separate components

Fast R-CNN

Innovation: Extract features once, then region-based classification

Architecture:

Image
  ↓
CNN to extract features (entire image)
  ↓
Region Proposals on feature map
  ↓ (For each region)
ROI Pooling (fixed size)
  ↓
Classification + Box Refinement

Speedup: 10x faster than R-CNN

Faster R-CNN

Innovation: Learn region proposals (Region Proposal Network)

Architecture:

Image
  ↓
CNN backbone
  ↓
Region Proposal Network (RPN)
  ↓ (Learnable regions)
ROI Pooling
  ↓
Classification + Box Refinement

Process:

• Anchor boxes at different scales/aspect ratios
• Network predicts objectness scores
• Learns good regions automatically

Speed: 5 fps on GPU

Accuracy: High (state-of-the-art at time)

Current Status: Still used as baseline; reference implementation

Mask R-CNN

Extension: Add instance segmentation

Added Component: Mask branch alongside classification

Faster R-CNN outputs:
- Bounding box
- Class

Mask R-CNN adds:
- Pixel-level mask for each object

Applications:

• Separate instances (3 dogs → 3 masks)
• Detailed object shape
• Pose estimation (with keypoint branch)

YOLO Family

Single-shot detection: Predict everything at once.

YOLO (You Only Look Once)

Innovation: Treat detection as regression problem

Grid-Based Approach:

Divide image into S×S grid (e.g., 7×7)
Each cell predicts:
- Bounding box(es)
- Confidence score
- Class probabilities

Process:

1. Divide image into grid
2. Each cell predicts boxes and classes
3. Apply non-maximum suppression (remove duplicates)
4. Output detected objects

Advantages:

• Speed: 45 fps (real-time)
• Global context: Sees entire image
• **Simple:**Straightforward pipeline

Disadvantages:

• Accuracy: Lower than R-CNN methods
• Small objects: Struggles with many small objects
• Close objects: Struggles with objects close together
• Aspect ratios: Struggles with unusual aspect ratios

YOLOv2

Improvements:

• Batch normalization
• Anchor boxes (like Faster R-CNN RPN)
• Multi-scale training
• Better network architecture

Accuracy: Better than YOLOv1

Speed: Still real-time

YOLOv3 and Beyond

Enhancements:

• Multi-scale predictions
• Better backbone (Darknet-53)
• Skip connections
• Improved class prediction

Current: YOLOv8, YOLOv9, YOLOv10

• Better accuracy-speed tradeoff
• Improvements in architecture and training

SSD (Single Shot Detector)

Approach: Multi-scale feature maps for detection

Architecture:

Input
  ↓
Feature Pyramid (multiple scales)
  ↓
Convolutional Predictors
  ↓
Non-maximum Suppression
  ↓
Detections

Process:

• Extract features at different scales
• Each scale handles different object sizes
• Parallel predictions at all scales

Advantages:

• Multi-scale handling (good for different sizes)
• Reasonable accuracy-speed tradeoff

Status: Reference architecture but surpassed by YOLO v3+

RetinaNet

Innovation: Focal loss to handle class imbalance

Problem: Most predictions are negative (background)

Focal Loss:

Reduces loss for easy negatives
Focuses on hard positives

Architecture:

• Feature pyramid network (FPN)
• Focal loss during training
• Anchor-based prediction

Advantage: Better balance of accuracy and speed

Anchor vs Anchor-Free Approaches

Anchor-Based

Predict offsets from predefined boxes.

Anchor: [100, 100, 200, 200]
Prediction: [+10, +5, +20, +15]
Final box: [110, 105, 220, 215]

Pros: Stable predictions, good inductive bias

Cons: Design depends on data distribution

Anchor-Free

Directly predict object center and size.

Center: [150, 150]
Size: [100, 100]

Examples: CenterNet, FCOS

Pros: Simpler, fewer hyperparameters

Cons: Harder to optimize

Modern Approaches

Vision Transformer (ViT) for Detection

Apply transformer architecture to detection.

Image patches → Transformer → Detection head

Advantages:

• Non-local attention (see entire image)
• Good for large-scale detection
• Parallelizable

Disadvantage: Computationally expensive

One-Stage vs Two-Stage

Evaluation Metrics

IoU (Intersection over Union)

Measure overlap between predicted and ground truth boxes.

IoU = Intersection Area / Union Area

Example:

Overlap:     Correct (IoU=0.8)   Incorrect (IoU=0.2)
██████       ██████              ██████
██████       ██████              ██

mAP (mean Average Precision)

• Calculate precision-recall curve for each class
• Calculate area under curve (AP)
• Average across all classes (mAP)

Example:

mAP@0.5: Average precision at IoU=0.5 threshold
mAP@0.75: Average precision at IoU=0.75 threshold

Practical Considerations

Real-Time vs Accuracy

Real-time required (≥30 fps):

• Use YOLO, MobileNet-based
• Lighter models
• Optimize for speed

High accuracy important:

• Use Faster R-CNN, RetinaNet
• Heavier models
• Optimize for mAP

Small Object Detection

Challenges:

• Few pixels to detect on
• Limited information

Solutions:

• Feature pyramid networks
• Smaller stride in backbone
• Data augmentation (focus on small objects)
• Specialized architectures (FCOS, CornerNet)

Resource Constraints

Edge devices, mobile, embedded:

• Use efficient backbones (MobileNet, ShuffleNet)
• Quantization (int8 instead of float32)
• Pruning (remove unimportant weights)
• Knowledge distillation (smaller teacher model)

Implementation Frameworks

OpenCV

C++ and Python library:

• Pre-trained models
• Easy integration
• Not bleeding edge

PyTorch

Deep learning framework:

• Flexible
• Custom architectures
• Research standard

TensorFlow/Keras

Higher-level framework:

• Simpler to use
• Good for prototyping
• Production deployments

Detectron2 (Meta/Facebook)

Research platform:

• Reference implementations
• Easy customization
• Used in research

Conclusion

Modern object detection architectures balance accuracy and speed. Two-stage detectors (Faster R-CNN, RetinaNet) prioritize accuracy; single-stage detectors (YOLO, SSD) prioritize speed. Each has tradeoffs suited to different applications. Understanding these architectures, their innovations, and evaluation metrics enables choosing the right approach for your problem. Whether you need real-time detection for autonomous driving or high-accuracy detection for medical imaging, there’s an architecture suited to your requirements.