Neural Networks Fundamentals - From Biological Inspiration to Computation

Neural networks are the foundation of modern deep learning. Inspired by biological brains, they’ve become remarkably powerful tools for solving complex problems. This post explores the fundamentals.

Biological Inspiration

The brain contains ~86 billion neurons connected by ~100 trillion synapses. Each neuron:

Receives signals from other neurons
Processes signals
Fires (activates) if signal exceeds threshold
Sends signals to other neurons

Artificial neural networks mimic this architecture.

The Artificial Neuron (Perceptron)

Basic Structure

A single artificial neuron has:

Inputs: x₁, x₂, …, xₙ
Weights: w₁, w₂, …, wₙ (strengthen/weaken inputs)
Bias: b (shift threshold)
Activation Function: f (introduce non-linearity)
Output: y

Mathematical Operation:

z = w₁x₁ + w₂x₂ + ... + wₙxₙ + b
y = f(z)

The Neuron Process

x₁ ─────→ w₁ ─────→
x₂ ─────→ w₂ ─────→ [Σ] ──→ [f] ──→ y
x₃ ─────→ w₃ ─────→
           b ─────→

Multiply inputs by weights
Sum the weighted inputs plus bias
Apply activation function
Produce output

Intuition: Classification Example

Predicting if a customer will buy based on age and income:

y = f(0.1 × age + 0.2 × income - 5)

Weight 0.2 > 0.1: Income more important than age
Bias -5: Threshold adjustment
f: Non-linear transformation

Activation Functions

Activation functions introduce non-linearity, enabling networks to learn complex patterns.

ReLU (Rectified Linear Unit)

Formula: f(z) = max(0, z)

Characteristics:

Output 0 if input negative
Output input if positive
Simple and fast
Leads to sparse activations

When to Use:

Default choice for hidden layers
Most modern networks use ReLU
Easier to train than older activations

        |
    /   |
   /    |
  /_____|_____
         |

Sigmoid

Formula: f(z) = 1 / (1 + e^(-z))

Characteristics:

Output range: 0 to 1
S-shaped curve
Smooth gradient
Prone to vanishing gradients

When to Use:

Binary classification output layer
Historical choice (less common now)
Interpretable as probability

  1 |_____
    |    /
  0 |___/

Tanh (Hyperbolic Tangent)

Formula: f(z) = (e^z - e^(-z)) / (e^z + e^(-z))

Characteristics:

Output range: -1 to 1
Centered around zero
Stronger gradient than sigmoid
Still has vanishing gradient issues

When to Use:

Sometimes better than sigmoid
ReLU usually preferred now
Rarely used in modern architectures

Softmax

Formula: f(z_i) = e^(z_i) / Σ e^(z_j)

Characteristics:

Output probability distribution (sum to 1)
Generalizes sigmoid to multi-class
Only used in output layer

When to Use:

Multi-class classification output layer
Need probability outputs

Neural Network Architecture

Single Layer (Perceptron)

Components:

Input layer
One hidden layer
Output layer

Limitations:

Can only learn linearly separable patterns
Can’t solve XOR problem

Multi-Layer Networks (Deep Learning)

Components:

Input layer: Receives raw data
Hidden layers: Learn representations
Output layer: Produces predictions

Key Principle:

Each layer transforms input to higher-level representation
Early layers: Simple features
Middle layers: Complex patterns
Final layer: Task-specific predictions

Example Architecture

Input Layer    Hidden Layer 1    Hidden Layer 2    Output Layer
                                                    
x₁ ─→                                           ┌→ y₁
x₂ ─→           ┌→ h₁₁ ─→ ┌→ h₂₁ ─→ ┌→
x₃ ─→─→─→─→─→─→┤          │        ├→─→─→─→─→┤→ y₂
x₄ ─→           └→ h₁₂ ─→ └→ h₂₂ ─→ └→
x₅ ─→                                           └→ y₃

Universal Approximation

Theorem: A neural network with one hidden layer containing enough neurons can approximate any continuous function.

Implication: Deep networks aren’t theoretically necessary for function approximation, but:

Deeper networks need fewer neurons
More efficient representation
Better generalization
Practical advantages

Forward Pass: Computing Output

Given inputs, compute output through network.

Process:

Layer 1:  z₁ = W₁x + b₁,  h₁ = f(z₁)
Layer 2:  z₂ = W₂h₁ + b₂,  h₂ = f(z₂)
Output:   z₃ = W₃h₂ + b₃,  y = f(z₃)

Matrix Operations:

W: Weight matrices (large if many neurons)
More layers: More matrix multiplications
Deep networks: Many operations per prediction

Backpropagation: Learning by Error

Backpropagation efficiently computes how much each weight contributed to error.

The Process

Forward Pass: Compute predictions
Compute Error: Compare to targets
Backward Pass: Compute gradient for each weight
Update: Adjust weights to reduce error

Gradient Computation

Chain Rule: Apply in reverse through layers

dL/dw = dL/dz × dz/dw

For each weight, how much did it contribute to final loss?

Why Backprop is Revolutionary

Before Backprop:

Computing gradients for deep networks was intractable
Neural networks limited to 1-2 layers
Training was very slow

After Backprop (1986):

Efficient gradient computation for any depth
Deep networks became practical
Enabled modern deep learning

Loss Functions

Loss functions measure prediction error.

Mean Squared Error (MSE)

Formula: MSE = (1/n) Σ(y_actual - y_pred)²

Use: Regression tasks

Cross-Entropy

Formula: CE = -Σ y_actual × log(y_pred)

Use: Classification tasks

Training Process

Initialize: Set weights to small random values
Forward Pass: Compute predictions for batch
Compute Loss: Calculate error
Backward Pass: Compute gradients
Update Weights: Adjust by learning rate × gradient
Repeat: For multiple epochs over data

Challenges in Neural Network Training

Vanishing Gradients

Problem: Gradients become very small in deep networks

Why: Chain rule multiplies many small numbers

Solution: ReLU, batch normalization, residual connections

Exploding Gradients

Problem: Gradients become very large

Why: Large weights multiply together

Solution: Gradient clipping, careful weight initialization

Slow Convergence

Problem: Takes many iterations to train

Solution: Adaptive learning rates (Adam), better initialization

Practical Tips

Start Small: Simple network on small data
Normalize Inputs: Mean 0, std 1
Monitor Loss: Should decrease smoothly
Validate Regularly: Check on validation set
Use Standard Architectures: Don’t invent new designs
Tune Learning Rate: Critical hyperparameter
Batch Normalization: Speeds training, improves stability

Conclusion

Neural networks learn by adjusting weights to minimize prediction error. The key innovation—backpropagation—enables efficient training of deep networks. Understanding neurons, activation functions, and backpropagation provides foundation for modern deep learning. While training neural networks involves challenges, these are well-understood and practical solutions exist. The combination of architectural depth and backpropagation enables learning of increasingly complex patterns from data.