Computer Vision - Teaching Machines to See

Computer Vision enables machines to interpret visual information from images and videos. From medical diagnosis to autonomous driving, computer vision has transformed countless domains. This post explores the field’s fundamentals and applications.

Why Computer Vision is Hard

Visual understanding appears effortless to humans but is complex for machines.

Challenges:

Variability:

• Same object from different angles
• Different lighting conditions
• Various scales (close vs far)
• Partial occlusion (hidden parts)
• Deformable objects

Interpretation:

• What defines an object?
• Context matters
• Spatial relationships
• Abstract concepts

Scale:

• Images contain millions of pixels
• Naïve approach: Process each pixel independently (infeasible)
• Need intelligent feature extraction

Image Representation

Pixel Structure

Grayscale Image:

• 2D array of values (0-255)
• 0 = black, 255 = white

Color Image:

• 3D array: height × width × channels
• Channels: Red, Green, Blue (RGB)
• Each pixel: R, G, B values

Example:

256×256 RGB image = 256 × 256 × 3 = 196,608 values
High resolution image = millions of values

Image Channels

RGB:

• Red, Green, Blue channels
• Most common for color images

Grayscale:

• Single intensity channel
• Used when color not important

HSV:

• Hue (color), Saturation, Value
• Sometimes better for certain tasks

Classical Computer Vision

Before deep learning, computer vision used hand-crafted features.

Feature Extraction

Edge Detection: Find boundaries between regions

Original Image:
█████ ░░░░░
█████ ░░░░░

Edge Detection:
████░
█░░░░
░░░░░

Corner Detection: Find intersection points

SIFT (Scale-Invariant Feature Transform):

• Extract distinctive features
• Scale invariant
• Rotation invariant

Algorithms

Template Matching:

• Search for known pattern in image
• Simple but limited

Histogram of Oriented Gradients (HOG):

• Capture edge directions
• Used for person detection
• Hand-crafted features

BRIEF, ORB:

• Binary descriptors
• Fast matching
• Lower accuracy

Deep Learning for Computer Vision

CNNs revolutionized computer vision by learning features automatically.

CNN Advantages

• Automatic Feature Learning: No need to hand-craft features
• Hierarchical Features: Low-level edges → high-level objects
• Translation Invariance: Object recognized anywhere in image
• Weight Sharing: Same filter across image (efficiency)

Landmark Architectures

LeNet (1998):

• First successful CNN
• 2 conv layers for digit recognition

AlexNet (2012):

• Deep CNN breakthrough
• Won ImageNet competition by huge margin
• 8 layers, 60M parameters

VGGNet (2014):

• Showed depth matters
• 16-19 layers
• Uniform 3×3 filters

ResNet (2015):

• Residual connections
• 152+ layers practical
• Skip connections prevent degradation

MobileNet (2017):

• Efficient architecture
• Mobile/edge deployment
• Depthwise separable convolutions

Core Computer Vision Tasks

Classification

Task: What is in the image?

Output: Single label (dog, cat, car)

Metric: Accuracy

Input: Image
Model: CNN
Output: "Cat" (99% confidence)

Object Detection

Task: What objects are in image and where?

Output: Bounding boxes + classes + confidence

Input: Image
Model: YOLO, Faster R-CNN
Output: 
- Dog at [10, 20, 100, 150]
- Cat at [200, 50, 300, 200]

Key Algorithms:

• R-CNN: Region-based CNN
• YOLO: Real-time detection
• SSD: Single Shot Detector

Semantic Segmentation

Task: Classify each pixel

Output: Pixel-level labels

Input: Image
Model: U-Net, FCN
Output: Each pixel labeled (road, car, person, etc.)

Applications:

• Medical imaging
• Autonomous driving
• Satellite imagery

Instance Segmentation

Task: Identify individual object instances

Output: Separate mask per object

Input: Image with 3 dogs
Output: 3 separate masks for 3 dogs
(vs semantic: all dogs in single mask)

Key Algorithm: Mask R-CNN

Pose Estimation

Task: Detect body keypoints (joints)

Output: 2D or 3D positions of keypoints

Input: Person in image
Output: Positions of head, shoulders, elbows, wrists, etc.

Applications:

• Activity recognition
• Sports analytics
• VR/AR

Optical Flow

Task: Estimate motion between frames

Output: Motion vectors for each pixel

Frame 1: Person on left
Frame 2: Person on right
Motion: Vector pointing right

Applications:

• Video compression
• Motion estimation
• Action recognition

Object Detection in Detail

Two-Stage Detectors

Process:

1. Generate region proposals (potential object locations)
2. Classify regions and refine boxes

Examples: R-CNN, Faster R-CNN, Mask R-CNN

Pros: Higher accuracy Cons: Slower

Single-Stage Detectors

Process:

1. Predict classes and boxes directly from image
2. No separate proposal stage

Examples: YOLO, SSD, RetinaNet

Pros: Faster, real-time Cons: Lower accuracy

YOLO (You Only Look Once)

Key Innovation: Treat detection as regression problem

Input: Divide image into grid
Each cell predicts:
- Bounding box coordinates
- Confidence score
- Class probabilities
Output: Non-maximum suppression (remove duplicates)

Speed: Real-time (30+ fps) Accuracy: Lower than two-stage

Practical Applications

Autonomous Driving

Vision Tasks:

• Object detection: Cars, pedestrians, signs
• Lane detection
• Traffic light recognition
• Obstacle avoidance

Challenges:

• Real-time performance
• Varied lighting/weather
• Safety critical

Medical Imaging

Applications:

• Tumor detection
• Disease classification
• Image reconstruction

Benefits:

• Faster diagnosis
• Consistency
• Catch early problems

Challenges:

• Limited training data
• Privacy concerns
• Regulatory requirements

Face Recognition

Steps:

1. Face detection: Locate face in image
2. Face alignment: Normalize pose
3. Feature extraction: Generate face embedding
4. Matching: Compare embeddings

Applications:

• Security
• Personal device unlock
• Photo organization

Challenges:

• Privacy concerns
• Bias (varies by ethnicity)
• Spoofing (photos, masks)

Retail Analytics

Uses:

• Customer counting
• Queue analysis
• Product placement
• Shelf monitoring

Agriculture

Applications:

• Crop health monitoring
• Yield prediction
• Weed detection
• Disease identification

Preprocessing and Data Augmentation

Preprocessing

Normalization:

• Resize to standard size
• Scale pixel values to [0, 1] or [-1, 1]
• Subtract mean, divide by std

Augmentation (training):

• Rotation: Model invariant to small rotations
• Flipping: Horizontal flip for most objects
• Cropping: Robustness to framing
• Color jittering: Lighting variations
• Blur: Focus variations

Why Augmentation Matters

Increases effective training data size and robustness.

Transfer Learning in Vision

Pre-trained models (ImageNet) accelerate development:

Pre-trained on ImageNet (1M images)
    ↓
Fine-tune on your task (1000 images)
    ↓
Achieves good performance with less data

Benefit: Leverage features learned from diverse images

Challenges and Limitations

Robustness

Domain Shift: Model trained on one type of image fails on different type

Adversarial Examples: Small perturbations fool model

Mitigation: Data augmentation, adversarial training

Bias

Models can have racial, gender, or other biases from training data

Interpretability

Hard to understand why model makes decisions

Tools:

• Saliency maps: Highlight important pixels
• Attention visualization: Show where model focuses
• Grad-CAM: Gradient-based visualization

Conclusion

Computer vision enables machines to interpret visual information. Classical methods used hand-crafted features; deep learning learns them automatically. CNNs excel at classification, and specialized architectures handle detection, segmentation, and other tasks. From autonomous driving to medical imaging, computer vision applications continue expanding. Understanding the fundamentals—CNNs, transfer learning, and task-specific architectures—enables building effective vision systems for real-world problems.