Natural Language Processing - Teaching Machines to Understand Language

Natural Language Processing (NLP) is the field dedicated to teaching computers to understand and generate human language. From sentiment analysis to machine translation, NLP powers many modern AI applications.

Why NLP is Hard

Human language is complex:

Ambiguity:

“I saw the man with the telescope”
- Who has the telescope? (Multiple interpretations)

Context Dependency:

“The bank approved my loan”
- “bank” = financial institution
“I sat on the river bank”
- “bank” = shore

Idioms and Figurative Language:

“It’s raining cats and dogs”
- Not literally about animals

Evolving Language:

New words, slang, misspellings
Cultural and regional variations

Sarcasm and Implied Meaning:

“Oh great, another meeting” (probably negative)

Text Representation

Before processing, convert text to numbers.

Tokenization

Split text into units (tokens).

Word Tokenization:

"Hello, world!" → ["Hello", ",", "world", "!"]

Subword Tokenization (BPE):

"Unbelievable" → ["Un", "believ", "able"]
Useful for:
- Handling rare words
- Out-of-vocabulary words
- Multiple languages

Character Tokenization:

"Hello" → ["H", "e", "l", "l", "o"]
Fine-grained but loses word structure

Bag of Words

Simplest representation: Count word occurrences.

Example:

Document: "The cat sat on the mat"
Vocabulary: {the, cat, sat, on, mat, dog, ...}
Representation: [2, 1, 1, 1, 1, 0, ...]
              (the appears twice, cat once, etc.)

Limitations:

Loses word order (sequence information)
Loses context
Treats all words equally

TF-IDF (Term Frequency-Inverse Document Frequency)

Weight words by importance.

Intuition: Common words (the, a, is) less important than rare words

Formula:

TF-IDF(word) = Frequency(word) × log(Total Documents / Documents with word)

Effect:

Common words: Lower weight
Rare words: Higher weight

Word Embeddings: Word2Vec

Represent each word as dense vector (usually 300 dims).

Key Idea: Semantically similar words have similar vectors

Training: Predict surrounding words from target word

Example:

Vector for "king": [0.2, 0.5, -0.1, ..., 0.3]
Vector for "queen": [0.25, 0.48, -0.08, ..., 0.32]
Similar → geometrically close

Interesting Properties:

Vector("king") - Vector("man") + Vector("woman") ≈ Vector("queen")

Contextual Embeddings: ELMo, BERT

Same word, different meanings depending on context.

Example:

"The bank approved my loan"    → "bank" embedding = finance-related
"I sat on the river bank"      → "bank" embedding = geography-related

How It Works:

Process entire sentence (full context)
Generate word embeddings based on context
Same word gets different embeddings in different contexts

Advantage: Better captures word meaning

NLP Tasks

Sentiment Analysis

Task: Determine if text expresses positive, negative, or neutral sentiment

Application:

Product reviews
Social media monitoring
Customer feedback
Brand reputation

Approaches:

Rule-based: Manually define sentiment words
ML-based: Train classifier on labeled reviews
Deep Learning: Neural networks process text

Named Entity Recognition (NER)

Task: Identify and classify named entities (people, places, organizations)

Example:

Input: "Apple CEO Tim Cook visited Paris yesterday"
Output: 
  - "Apple" (Organization)
  - "Tim Cook" (Person)
  - "Paris" (Location)

Applications:

Information extraction
Question answering
Content organization

Machine Translation

Task: Translate text from one language to another

Example:

Input (French): "Bonjour, comment allez-vous?"
Output (English): "Hello, how are you?"

Challenges:

Word order differs by language
Idioms don’t translate literally
Context matters for ambiguous words

Modern Approach: Encoder-Decoder transformers with attention

Text Classification

Task: Assign text to predefined categories

Examples:

Spam detection
Topic classification
Intent identification (for chatbots)

Question Answering

Task: Find answer to question in text

Example:

Question: "What is the capital of France?"
Context: "Paris is the capital and largest city of France"
Answer: "Paris"

Approaches:

Span-based: Find start and end of answer
Generative: Generate answer from scratch

Text Summarization

Task: Compress text to shorter summary

Types:

Extractive: Select important sentences
Abstractive: Generate new sentences capturing essence

Language Modeling

Core Task: Predict next word given previous words

Formula:

P(next_word | previous_words) = ?

N-gram Models

Idea: Predict based on last N words

Example (3-gram):

"The quick brown ___"
Based on "brown" and previous word, predict next word

Limitation: Limited context window

Neural Language Models

Idea: Use neural networks to capture longer dependencies

Process:

Embed previous words
Process through layers
Predict next word probability

Result: Better predictions with longer context

Pre-training and Fine-tuning

Modern NLP uses transfer learning.

Pre-training

Objective: Language understanding on massive corpus

Methods:

Next word prediction (GPT style)
Masked language modeling (BERT style)
- Hide word, predict it
- Learn from full context

Data: Billions of words from internet, books, etc.

Duration: Weeks on powerful hardware

Result: General language understanding

Fine-tuning

Process:

Take pre-trained model
Replace task-specific layer
Train on task-specific data
Much faster, needs less data

Impact: Democratizes NLP - anyone can build good models

Common Architectures

RNN-Based

Pros:

Sequential processing captures order
Works with variable length

Cons:

Slow (can’t parallelize)
Vanishing gradient problems

Transformer-Based

Pros:

Parallelizable
Captures long dependencies
Better performance
Faster training

Cons:

Higher computational cost
O(n²) memory with sequence length

Dominates modern NLP

Pre-trained Models

BERT (Encoder)

Bidirectional understanding
Fine-tune for classification, NER
Good for understanding tasks

GPT (Decoder)

Autoregressive generation
Fine-tune for text generation
Good for generation tasks

T5 (Encoder-Decoder)

Unified text-to-text framework
Fine-tune for any NLP task
Flexible architecture

Challenges in NLP

Ambiguity

Solutions:

Larger models better at disambiguation
Incorporate context
Knowledge bases

Out-of-Vocabulary Words

Solutions:

Subword tokenization (BPE)
Character-level models
Transfer learning from large corpora

Low-Resource Languages

Challenge: Limited training data

Solutions:

Transfer learning from high-resource languages
Cross-lingual models
Synthetic data generation

Bias in Language Models

Issue: Models encode societal biases

Example:

"man is to computer programmer as woman is to ___"
→ Model might generate: "nurse", "teacher" (reflecting stereotypes)

Solutions:

Audit for bias
Debiasing techniques
Diverse training data

Practical NLP Workflow

Define Task: What problem to solve?
Choose Model: Pre-trained model suitable?
Prepare Data: Collect, clean, tokenize
Fine-tune: Train on task data
Evaluate: Performance metrics
Deploy: Make available to users
Monitor: Track performance, bias

Conclusion

NLP teaches machines to understand human language. From tokenization to contextual embeddings, techniques have advanced dramatically. Pre-trained transformer models capture broad language understanding. Fine-tuning on specific tasks achieves excellent performance with minimal data. Modern NLP powers chatbots, translation, content analysis, and creative applications. As models scale and improve, NLP capabilities continue to expand, enabling increasingly sophisticated language understanding and generation.