Deep Learning Interview Map

Master Navigation Document | 5-min read to 3-day deep-dive

Table of Contents

Main Narrative (5-Stage Deep Dive)

• Problem-Evolution Narrative
  1. Stage 1: Why Neural Networks?
  2. Stage 2: Why Convolutional Neural Networks (CNNs)?
  3. Stage 3: CNN Architecture Evolution (LeNet → EfficientNet)
  4. Stage 4: Computer Vision Task-Specific Evolution (Detection & Segmentation)
  5. Stage 5: Generative, Attention & Sequence Models Evolution

Quick Reference & Study Tools

6. Architecture Family Tree
7. Key Innovation Timeline
8. Parameter Counting Cookbook
9. Elevator Answers Bank
10. When to Use What
11. Common Interview Gotchas
12. Deep-Dive Links
13. Quick Review Checklist

Problem-Evolution Narrative

Stage 1: Why Neural Networks?

Elevator Answer:

Neural networks solve non-linear decision boundaries and eliminate manual feature engineering by composing non-linear transformations. Each layer learns increasingly abstract representations end-to-end via backpropagation. The cost: data-hungry, computationally expensive, and difficult to train deeply.

Gap Filled: ✅ Non-linearity, ✅ Feature learning
New Problem Introduced: ❌ Computationally expensive on high-dimensional inputs (images) | ❌ Parameter explosion

Stage 2: Why Convolutional Neural Networks (CNNs)?

Key Insight: Images have three inductive biases CNNs exploit:

1. Locality: Neighboring pixels are correlated; distant pixels rarely interact directly
2. Stationarity: Patterns (edges, textures) occur anywhere; the same filter detects them everywhere
3. Compositionality: Complex features are built from simple ones hierarchically (pixels → edges → shapes → objects)

Fully-connected networks ignore all three. CNNs leverage them.

Convolution Fundamentals

What is Convolution? Convolution is sliding a small weight matrix (kernel) across the image, computing dot products at each position. Instead of connecting every pixel to every neuron (millions of parameters), we use one small kernel everywhere (parameter sharing).

Kernel Matrix (Filter) A kernel is a learnable small matrix (e.g., 3×3, 5×5) that detects local patterns:

1
2
3
4
5
6
7
8
9

Example 3×3 kernel:
┌───────────────┐
│ w₁₁  w₁₂  w₁₃ │
│ w₂₁  w₂₂  w₂₃ │
│ w₃₁  w₃₂  w₃₃ │
└───────────────┘

Convolution operation at position (i,j):
output[i,j] = Σ kernel[k,l] × image[i+k, j+l] + bias

Example: 3×3 edge detector kernel applied to 224×224×3 image:

• 9 parameters per kernel (3×3)
• Used 50,000+ times across spatial positions (due to stationarity)
• Detects edges everywhere in the image with the same filter

What is Pooling? Pooling reduces spatial dimensions (e.g., 224×224 → 112×112) by summarizing local regions:

• Max Pooling: Take max value in 2×2 window → extracts strongest features
• Average Pooling: Take average in 2×2 window → smoother, regularized version

Why Pooling Helps Position Invariance:

• Pooling makes small spatial shifts irrelevant (shift 1 pixel → max pooling output often same)
• Reduces parameters and computation (224→112→56→28 spatial dims)
• Creates translation invariance: if an edge shifts slightly, max pooling still captures it

CNN Core Concepts (before table):

CNN vs. Fully-Connected: Problem-Solution

Key Gains: ✅ Parameter sharing: 150K→9 parameters for first layer (16K× reduction)
✅ Sparse connections: Only local neighborhoods; respects image structure
✅ Translation invariance: Pooling + stacking make detection position-agnostic
✅ Hierarchical features: Composable abstractions (edges→shapes→objects)

Trade-offs: ❌ Limited receptive field in early layers: 3×3 kernel sees 3×3 region only; need depth to see larger context
❌ Boundary handling: Image edges need padding strategy (zero-pad, reflect, etc.)
❌ Still requires depth: Must stack layers to capture global structure (compositionality doesn’t come free)

Elevator Answer:

A fully-connected network would assign unique weights to each pixel (150K+ parameters for first layer). CNNs use parameter sharing: one 3×3 kernel (9 weights) slides across the image, reusing those weights 50,000+ times. This exploits stationarity (edges look the same everywhere) and locality (pixels interact primarily with neighbors). Sparse connections reduce parameters 16,000×. Pooling provides translation invariance (shift object slightly, pooling still detects it). Stacking layers enables compositionality: early layers learn edges, middle layers learn shapes, deep layers learn objects. Cost: need depth to capture global context; early layers see only 3×3 neighborhoods.

Gap Filled: ✅ Parameter efficiency (sharing), ✅ Spatial locality (sparse connections), ✅ Position invariance (pooling), ✅ Hierarchical features (compositionality), ✅ Scale to large images
New Problem Introduced: ❌ Early layers have limited receptive field (3×3 sees only 3×3) | ❌ Need depth for global context | ❌ Vanishing gradients in very deep networks

Stage 3: CNN Architecture Evolution (LeNet → EfficientNet)

Key Design Questions: How do we train deeper networks? How do we fit images on GPUs? How do we reuse low-level features? How do we scale architectures optimally?

1. LeNet-5 (1998)

Significance: LeNet-5 proved convolutional architectures work for handwritten digits. Established CNN foundation: convolution → pooling → dense. Limitation: Tanh saturates; MSE suboptimal for classification.

2. AlexNet (2012)

Key Innovations: ReLU (non-saturating, enables deeper nets), Dropout (regularization), GPU parallelization, data augmentation. Won ImageNet 2012 with 84.7% top-5—revived deep learning.

3. VGG-16 (2014)

Key Concept: 3×3 Stacking — Two 3×3 convolutions = one 5×5 receptive field, but fewer parameters and more non-linearity (two ReLU). Showed depth > width. 92.7% top-5 accuracy.

4. ResNet-34 (2015)

Key Concept: Skip Connections (Residual Blocks) — y = F(x) + x instead of y = F(x). Network learns residuals, not full transformations. During backprop, gradient flows through skip path (includes +1 term), preventing vanishing gradients. Enables 152-layer training. 8× deeper than VGG with fewer params (21.8M vs. 138M).

5. Inception-v3 (2015)

Key Concepts:

• Inception Module: Parallel 1×1, 3×3, 5×5 convolutions capture multi-scale features simultaneously.
• 1×1 Convolutions (Bottlenecks): Pointwise operation mixes channels without spatial change. Used to reduce dims before expensive 3×3/5×5. Example: reduce 192→128 before 3×3 = 3× compute reduction. Became standard in efficient architectures.

93.7% top-5 with 27M params—clever design beats raw scale.

6. DenseNet-121 (2016)

Key Concept: Dense Connections vs. Skip Connections — ResNet adds: y = F(x) + x. DenseNet concatenates: each layer sees ALL previous layers. Benefits: (1) Stronger gradient flow—gradients reach all earlier layers, (2) Feature reuse—low-level features reused by all, (3) 7.98M params vs. 138M VGG (17× reduction) for competitive accuracy.

7. MobileNet-v1 (2017)

Key Concept: Depthwise-Separable Convolutions — Decouple spatial + channel operations for 9× param reduction.

How it works:

1. Depthwise (3×3 per channel, NO channel mixing): Each of 128 channels processed independently by its own 3×3 filter. Params = 3²×128 = 1,152
2. Pointwise (1×1 conv, YES channel mixing): Combine all 128 channels via 1×1 to create 256 output channels. Params = 1×1×128×256 = 32,768

Why it works: Spatial patterns (edges, textures) are often channel-local → depthwise learns them cheaply. Cross-channel learning happens in pointwise (1×1 is fast since it’s just weighted sums). Total: 1,152 + 32,768 = 33,920 params vs. standard 294,912 → 9× reduction.

Why no channel mixing in depthwise? Spatial operations (detecting edges) don’t need to see “red + blue channels together”—each channel’s edges are independent. Mixing channels would waste compute.

Why channel mixing in pointwise? After spatial features are extracted per-channel, we need to combine them (e.g., “if red edge AND blue edge both present → important feature”). That’s what 1×1 does efficiently.

Where: MobileNet (mobile inference, 4.24M params), EfficientNet, all modern efficient models.

8. EfficientNet-B0 (2019)

Key Concept: Compound Scaling (Depth × Width × Resolution) — Previous: scale one dimension. EfficientNet: scale all jointly via NAS.

Formula: Depth: d’ = d × α^φ | Width: w’ = w × β^φ | Resolution: r’ = r × γ^φ

φ=0 (B0) to φ=7 (B7). Optimal ratio α:β:γ discovered via NAS. Result: B0 (5.3M) = ResNet-50 accuracy with 5× fewer params. B7 (66M) surpasses SOTA with 2× fewer params than ResNet-50.

Evolution Summary

Elevator Answer: CNN evolution solves three problems: (1) Vanishing gradients (ResNet skip connections), (2) Parameter explosion (ResNet bottlenecks, MobileNet depthwise-sep, DenseNet concatenation), (3) Efficiency (EfficientNet compound scaling). Key: ReLU (non-saturating), BN (stable gradients), skip connections (gradient flow), 1×1 convolutions (bottleneck), depthwise-separable (9× param reduction), compound scaling (depth×width×res jointly). Result: AlexNet 60M → EfficientNet 5.3M, same accuracy, 100× faster mobile.

New Problem (Stage 3→4): ❌ Classification-only | ❌ Doesn’t locate objects | ❌ Needs task-specific heads

Stage 4: Computer Vision Task-Specific Evolution (Detection & Segmentation)

Key Design Questions: How do we localize objects? How do we assign class to every pixel? How do we separate instances? How do we do it in real-time?

Understanding the Tasks

Object Detection: Predict bounding boxes (x, y, width, height) + class label for each object in an image. Example: “car at (100, 50, 200, 300), person at (400, 80, 500, 400)”. Used in autonomous driving, surveillance, retail.

Object Localization: Find a single object’s location + classify it. Simpler than detection (one object assumed). Example: localize and classify the main subject in a photo.

Semantic Segmentation: Assign a class label to every pixel in the image. All pixels of class “car” are labeled “car”; all “road” pixels are labeled “road”. No distinction between individual cars—just the category. Used in autonomous driving (road/non-road), medical imaging (tumor/healthy).

Instance Segmentation: Like semantic segmentation, but distinguish individual objects. Car #1 pixels labeled separately from Car #2 pixels. Used for crowd counting, multi-object tracking.

A. Object Detection: R-CNN Family (Region-Based)

Concept: Find candidate regions, then classify + refine each region. Two-stage process: propose → classify.

1. R-CNN (2014)

Key Concept: Selective Search — Hand-crafted proposal algorithm that generates ~2000 candidate regions per image:

1. Start with oversegmentation: Segment image into tiny regions (100s) via bottom-up hierarchical clustering
2. Iteratively merge similar regions using: color similarity, texture similarity (SIFT), size preference (merge smaller first), shape fit (containment)
3. Region hierarchy: Results in nested proposals across scales (small objects → large objects)
4. Output: ~2000 bounding box proposals ranked by likelihood

Each proposal is then warped to 224×224, passed through AlexNet feature extractor, and classified via SVM. Bbox regressor refines coordinates per region. Bottleneck: 50s/image due to ~2000 separate CNN forward passes. Proved region-based detection paradigm but inspired faster alternatives (RPN in Faster R-CNN).

2. Fast R-CNN (2015)

Key Concept: RoI Pooling — Instead of warping each region to fixed size, extract region from feature map and max-pool to fixed output (e.g., 7×7). Enables single CNN pass on full image, then region pooling. Shared backbone dramatically speeds up inference. Multi-task loss learns classification + bbox refinement simultaneously.

3. Faster R-CNN (2015)

Key Concept: Region Proposal Network (RPN) — Step-by-Step

Problem to solve: Fast R-CNN still uses Selective Search (hand-crafted, slow, not trained end-to-end). Can we learn proposals directly from the data?

Solution: RPN is a small neural network that slides over the backbone feature map:

1. Slide a 3×3 window across the feature map (e.g., 13×13 feature map from ResNet)
2. At each position, place 9 anchor boxes (3 scales × 3 aspect ratios: 128×128, 256×256, 512×512; 1:1, 1:2, 2:1)
3. For each anchor, predict:
   • Classification: Is this anchor foreground (object) or background (no object)? (2 outputs: softmax)
   • Bbox regression: Adjust anchor coords (Δx, Δy, Δw, Δh) to fit actual object (4 outputs: regression)
4. Output: ~2000 region proposals (filtered by confidence, NMS removes near-duplicates)

Example: 13×13 feature map × 9 anchors = 1,521 anchors. After filtering by confidence + NMS → ~2000 proposals

Loss function: RPN trained jointly with detection head via multi-task loss:

• L_RPN_cls (classification: object/background)
• L_RPN_bbox (bbox regression: refine anchor to proposal)
• L_detection_cls (detection: classify proposal)
• L_detection_bbox (detection: refine proposal to final box)

Result: End-to-end learnable proposals. Eliminates hand-crafted Selective Search. Proposals learned from data via gradient descent.

Evolution Summary: R-CNN → Fast R-CNN → Faster R-CNN

Internalization Flow:

1. R-CNN problem: CNN is expensive; running it ~2000 times per image is slow
2. Fast R-CNN insight: Extract features once on full image, then pool regions from the feature map (RoI Pooling)
3. Fast R-CNN problem: Selective Search proposals are still hand-crafted and not learned from data
4. Faster R-CNN insight: Add RPN—a learned network that slides over features to generate proposals. Train RPN + detection jointly via multi-task loss.
5. Result: Fully end-to-end learnable detection pipeline

4. Faster R-CNN + FPN (2017)

Key Concept: Feature Pyramid Network (FPN) — Multi-scale detection via feature reuse.

The Problem: Deeper layers (stride 32) have semantic features but low resolution (miss small objects). Shallow layers (stride 4) have fine details but weak semantics. Standard RPN on single layer misses scale variation.

FPN Solution: Build a pyramid of semantically-rich multi-scale features.

1. Bottom-up pathway: Standard backbone (ResNet) produces feature maps at multiple strides (C2, C3, C4, C5 = stride 4, 8, 16, 32)
2. Top-down pathway: Start from C5 (coarsest, most semantic); upsample 2× + lateral connection (1×1 conv) from lower level; repeat
3. Output: P2, P3, P4, P5 (all semantically rich + spatially appropriate for their scale)
4. RPN at each level: Apply RPN to each P level with scale-specific anchors (P2 detects tiny objects, P5 detects large)

Why it matters: Before FPN, small objects were hard because single-layer RPN used stride-32 features (5×5 for 160×160 obj). With FPN, P2 (stride 4) gives 40×40 for same object (16× more spatial info). Result: small object mAP jumps from ~10% to ~20% on COCO. Every modern detector uses FPN or similar multi-scale principle.

Prominence: FPN is foundational. Faster R-CNN + FPN (2017) became the standard baseline for detection for 5+ years. Understanding multi-scale is essential for any detection system.

B. Object Detection: YOLO Family (Single-Stage)

Concept: Divide image into grid; predict class + box offsets per cell. One-stage, end-to-end, fast but slightly less accurate.

5. YOLO v1 (2015)

Key Concept: Grid-Based Single-Stage — Divide image into 7×7 grid. Each cell predicts bounding boxes + class probabilities independently.

Output tensor breakdown (7×7×30):

• Grid: 7×7 = 49 cells
• Per-cell predictions: 30 values
  • Box 1: (x, y, w, h, confidence) = 5 values
  • Box 2: (x, y, w, h, confidence) = 5 values
  • Class probabilities: 20 classes (PASCAL VOC) = 20 values
  • Total per cell: 5 + 5 + 20 = 30 values
• Final output: 7×7 grid × 30 values/cell = 7×7×30 tensor (48,400 values total)

Why this design? Each grid cell is responsible for detecting objects whose center falls in that cell. 2 boxes per cell allow multiple objects in one cell. Loss combines bbox regression (MSE), confidence (MSE), and classification (MSE). No region proposals; unified, fast pipeline. Trade-off: lower accuracy than Faster R-CNN but 45 fps enables real-time applications.

6. YOLO v2 (2016)

Key Concept: Anchor Boxes & Multi-Scale — YOLO v2 adds anchor boxes (similar to RPN concept): predefined box sizes per grid cell. Network learns offsets from anchor, not absolute coords. Enables better small object detection. Batch normalization stabilizes training. Multi-scale training (randomly resize input) improves robustness to different object scales. Result: accuracy parity with Faster R-CNN while maintaining speed.

7. YOLO v3 (2018)

Key Concept: Multi-Scale Predictions — v3 predicts at 3 scales simultaneously (13×13 for large objects, 52×52 for small). Similar to FPN but single-stage. IoU loss (intersection-over-union) geometrically better than MSE for bounding boxes. Result: significantly better small object detection (52×52 grid can detect 4-pixel objects).

8. YOLO v5/v8 (2020+)

Key Concept: Production Optimization — v5/v8 focused on engineering excellence: CSPDarknet reduces redundancy, Focus layer, PANet for multi-scale fusion, mosaic augmentation (4× effective batch diversity), GIoU/DIoU loss (geometry-aware). Adaptive anchors learn from data. Result: state-of-the-art speed-accuracy balance; production-ready; open-source.

C. Object Detection: Summary

Detection Evolution: R-CNN (region proposals, slow) → Fast R-CNN (shared features, pooling) → Faster R-CNN (learnable RPN) || YOLO v1 (grid, real-time) → v2 (anchors, BN) → v3 (multi-scale) → v5 (production-optimized).

D. Key Detection Concepts

Anchor Boxes: Pre-defined box templates (3 scales × 3 aspect ratios = 9 anchors per location). RPN predicts offsets from anchors (Δx, Δy, Δw, Δh) instead of absolute coordinates. Enables multi-scale detection without retraining. Example: anchors = [(128,128), (256,256), (512,512)] × [1:1, 1:2, 2:1 aspect ratio].

RPN (Region Proposal Network): Lightweight 3×3 conv applied to each position on feature map. Outputs: (1) classification score (object vs. background), (2) bbox offsets (4 values per anchor). Generates ~6000 raw proposals; post-processing (NMS) reduces to ~2000. Replaces hand-crafted Selective Search; learned from data.

RoI Pooling vs. RoI Align: Both extract fixed-size features (e.g., 7×7) from variable-size regions. Pooling quantizes coordinates (aligns to grid); may miss alignment. Align uses bilinear sampling (continuous interpolation); preserves precision. Mask R-CNN uses Align for better instance masks.

NMS (Non-Maximum Suppression): Post-processing step. Given overlapping box predictions: (1) keep box with highest confidence, (2) discard boxes with IoU > threshold (typically 0.5). Removes duplicate detections from multiple anchors. Greedy algorithm; keeps best, suppresses similar.

IoU (Intersection over Union): Metric for bounding box overlap. IoU = Area(intersection) / Area(union). Used in: (1) NMS (discard if IoU > 0.5), (2) matching predictions to ground truth, (3) mAP metric (mAP@0.5 = mAP evaluated at IoU threshold 0.5).

E. Upsampling Techniques (Critical for Segmentation)

Problem: Convolutional networks downsample (pooling, stride=2) to extract features; segmentation needs to recover spatial resolution. How do we go from coarse feature maps back to original image size?

Solution: Two main approaches with different trade-offs.

1. Bilinear Interpolation (Non-Learnable)

Concept: Estimate missing pixel values using weighted average of four nearest neighbors.

Math: For position (x, y) between integer coordinates:

• Find 4 surrounding pixels: (⌊x⌋, ⌊y⌋), (⌊x⌋+1, ⌊y⌋), (⌊x⌋, ⌊y⌋+1), (⌊x⌋+1, ⌋y⌋+1)
• Weight by distance: closer pixels weighted higher
• Output = weighted sum of 4 neighbors

Example: Upsample 2×2 feature map to 4×4 image

1
2
3
4
5

Input (2×2):        Output (4×4):
[1  2]              [1    1.33  1.67  2]
[3  4]              [1.67 2.33  3     3.33]
                    [2.33 3     3.67  4]
                    [3    3.33  3.67  4]

Properties:

• ✅ Fast (simple interpolation; no learnable parameters)
• ✅ Smooth transitions between pixels
• ❌ Not learnable; can’t adapt to data
• ❌ No feature refinement (just geometric scaling)
• ✅ Used in: FCN (simple upsampling), DeepLab (lightweight decoder)

2. Transposed Convolution (Learnable, “Deconvolution”)

Concept: Learnable upsampling via “inverse convolution.” Think of it as: regular convolution maps N pixels → 1 output. Transposed convolution maps 1 input → N outputs.

Math: Regular conv with stride=2 downsamples (e.g., 4×4 → 2×2). Transposed conv with stride=2 upsamples (e.g., 2×2 → 4×4).

Mechanics:

1. Take input feature map (2×2)
2. Place each element in a sparse grid (stride=2)
3. Convolve with learnable kernel (e.g., 3×3)
4. Sum overlapping regions

Example: Upsample 2×2 → 4×4 with 3×3 kernel (stride=2, padding=1)

1
2
3
4
5

Input:              Sparse grid:        After 3×3 conv:
[a  b]              [a  0  b  0]        [learnable
[c  d]              [0  0  0  0]         output]
                    [c  0  d  0]
                    [0  0  0  0]

Properties:

• ✅ Learnable (kernel weights trained via backprop)
• ✅ Can refine features during upsampling
• ✅ Adapts to data (learns what details to restore)
• ❌ Can produce checkerboard artifacts (overlapping regions)
• ✅ Used in: U-Net (decoder), Mask R-CNN (mask head), VAE decoders

Checkerboard Artifact: Overlapping regions can produce “checkerboard” patterns. Mitigated by careful kernel initialization or resize-convolution trick (bilinear upsample + 1×1 conv).

3. Comparison: Bilinear vs. Transposed Convolution

Practical Rule:

• Use bilinear when: (1) Memory/compute critical, (2) Simple geometric upsampling sufficient, (3) Post-processing step
• Use transposed conv when: (1) Feature refinement important, (2) Learned upsampling needed, (3) Part of differentiable pipeline

4. Hybrid Approach: Resize-Convolution

Problem: Transposed convolution can produce artifacts.

Solution: Bilinear upsample + learnable 1×1 convolution.

1

Input → Bilinear Upsample (2×) → 1×1 Conv (learnable refinement) → Output

Advantages:

• Geometric upsampling (bilinear; smooth)
• • learnable refinement (1×1 conv; feature adaptation)
• Avoids checkerboard artifacts
• Used in modern architectures (DeepLab decoder, some GAN generators)

When You See These in Architectures:

• FCN: “Bilinear upsample × 32” = 32× geometric upsampling; simple, fast
• U-Net: “Transposed conv 3×3, stride 2” = learnable 2× upsampling per decoder level
• Mask R-CNN: “4× bilinear upsample” on mask head = geometric upsampling; sufficient for instance masks
• DeepLab v3+: “4× bilinear upsample + skip connection” = combines fast upsampling + spatial detail from encoder

F. Segmentation Architectures (Using Upsampling Techniques)

Semantic Segmentation: Per-pixel classification. Input: image. Output: segmentation map (H×W×C where C = num classes).

Instance Segmentation: Per-pixel classification + instance ID. Combines detection (bounding boxes) + segmentation (masks per box).

1. FCN (2015)

Key Concept: Fully-convolutional architecture (no FC layers). Backbone extracts features, then upsample back to input resolution via bilinear interpolation. Skip connections from coarser layers help with fine details. Output: segmentation map (class per pixel). Simple but outputs coarse predictions (stride 32).

2. U-Net (2015)

Key Concept: Encoder-Decoder + Skip Concatenation — Encoder downsamples with pooling (learns features, loses spatial info). Decoder upsamples with transposed convolutions (recovers spatial resolution). Skip connections concatenate (not add) encoder outputs with decoder upsamples—preserves fine-grained spatial details. Medical imaging standard; works well with small datasets.

3. DeepLab v3+ (2018)

Key Concept: Atrous Convolution (Dilated Conv) — Regular convolution samples adjacent pixels. Atrous conv samples with gaps (dilation rate d): pixel at distance d. Maintains spatial resolution while expanding receptive field without parameters. ASPP module: multiple dilations (1, 6, 12, 18) capture multi-scale context. Result: fine-grained segmentation without 8× upsampling overhead.

4. Mask R-CNN (2017)

Key Concept: Instance Segmentation Pipeline — Extend Faster R-CNN with a mask branch. For each detected region: (1) extract features via RoI Align (bilinear sampling, no quantization), (2) pass to FCN mask head (4 conv + upsample), (3) output binary mask per instance. Multi-task loss: detection + mask. Separates individual objects unlike semantic segmentation.

G. Segmentation: Summary

H. Key Segmentation Concepts

Atrous/Dilated Convolution: Standard conv samples adjacent pixels (dilation=1). Dilated conv samples with gaps (dilation=d). Expands receptive field without adding parameters. Example: 3×3 kernel with dilation=2 sees 5×5 neighborhood. Used in DeepLab for efficient large receptive fields without stride-based downsampling.

ASPP (Atrous Spatial Pyramid Pooling): Multiple dilations applied in parallel (dilations 1, 6, 12, 18) on same input feature map. Captures multi-scale context at same resolution (unlike FPN pyramid which changes stride). Output: concatenate all branches; fuse via 1×1 conv. Used in DeepLab v3+ to build rich context at coarse resolution before 4× upsampling.

Transposed Convolution: Learnable upsampling (inverse of strided conv). Kernel size K, stride S → upsamples (H, W) → (S×H, S×W). Learns what details to restore during decoding. Used in U-Net decoder and Mask R-CNN mask head. Can produce checkerboard artifacts; mitigated by careful initialization or resize-convolution (bilinear + 1×1 conv).

Skip Connections: Connect encoder outputs directly to corresponding decoder inputs (concatenate, not add). Preserves fine-grained spatial details lost during downsampling. Example: U-Net concatenates each encoder level to corresponding decoder level. Enables precise localization.

Encoder-Decoder Structure: Downsampling path (encoder; extract features, lose spatial info) + upsampling path (decoder; recover resolution, restore details via upsampling + skip connections). Universal for segmentation; enables end-to-end training of spatial tasks.

Stage 5: Generative, Attention & Sequence Models Evolution

Key Design Questions: How do we model data distribution p(x)? How do we generate new samples? How do we capture long-range dependencies? How do we learn similarity metrics?

Understanding the Need for Stage 5

Problem from Stages 1-4: Classification models (CNNs, RNNs) learn p(y|x)—predict labels given inputs. They’re discriminative: good at supervised tasks but can’t:

• Generate new realistic samples
• Interpolate between data points
• Model complex data distributions
• Understand global structure of data

Solution: Three complementary paradigms emerge:

1. Generative Models — Learn p(x): model the data distribution itself

• Use case: Generate new images (image synthesis), face generation, style transfer, data augmentation, anomaly detection
• Paradigm 1 (VAE): Probabilistic; encode x → latent z; decode z → x’. Interpretable; can interpolate
• Paradigm 2 (GAN): Adversarial; Generator tries to fool Discriminator. Sharper images but training unstable

2. Attention & Transformers — Capture long-range dependencies without CNN localization or RNN sequential bottlenecks

• CNN problem: Receptive field grows slowly with layers (1→3→5→7…). Hard to see entire image in early layers
• RNN problem: Sequential bottleneck; can’t parallelize; information bottleneck in hidden state h_t
• Solution: All-pairs attention (query attends to all keys); parallelizable; captures any-to-any dependencies in 1 layer
• Use case: Language modeling (GPT, BERT), machine translation, image classification (Vision Transformer)

3. Sequence Models — Explicitly model temporal/sequential structure

• RNN: Hidden state h_t carries forward sequentially; one timestep at a time
• LSTM/GRU: Gated hidden state (forget/update gates); fixes vanishing gradients; can remember 100+ timesteps
• Use case: Text generation, speech recognition, time series forecasting, machine translation (seq2seq uses both Attention + Sequence)

A. Generative Models Evolution (Autoencoder Paradigm)

Core Concept: Encode data x into a latent bottleneck z; add probabilistic constraint z ~ N(0,I); decode z back to x. Minimizes reconstruction error + KL divergence (probability regularization).

1. Autoencoder (AE) (baseline, ~2006)

Significance: Proves encoder-decoder architecture works for unsupervised learning; proves we can compress data via bottleneck. But no principled way to generate new samples.

2. Variational Autoencoder (VAE) (2013)

Key Concept: Reparameterization trick lets gradients flow through sampling operation (z = μ + σ⊙ε). KL term forces z toward standard normal N(0,I), enabling generation. Trade-off: reconstruction vs. regularity.

3. VAE Variants: Evolution & Trade-offs (β-VAE, VQ-VAE, Hierarchical)

Why Variants? Standard VAE trades reconstruction quality for regularity (KL term). Variants address specific needs.

High-Level Evolution: Standard VAE → β-VAE (disentangle) OR VQ-VAE (discrete) OR Hierarchical (multi-scale). Choose variant based on your goal: interpretability, sample quality, or structured generation.

Interview Takeaway: You don’t need to memorize each variant deeply. Key: understand the core VAE principle (reconstruct + KL), then variants are just loss tweaks or architectural changes for specific trade-offs.

B. Generative Models Evolution (Adversarial Paradigm)

Core Concept: Generator G(z) creates fake samples; Discriminator D(x) distinguishes real vs. fake. Adversarial game: G tries to fool D; D tries to catch G. Result: sharper samples but training instability until stabilization tricks (DCGAN, WGAN, Spectral Norm, StyleGAN).

1. Vanilla GAN (2014)

Problem: JS divergence saturates when distributions don’t overlap; D becomes too confident → G receives zero gradient → training collapses.

Vanilla GAN Loss Function Breakdown:

1

min_G max_D [log D(x) + log(1 - D(G(z)))]

Discriminator’s perspective (max_D: wants to maximize this term):

• log D(x): Real data x should be classified as real (D(x) → 1) → log(1) = 0 (best); log(0.5) = -0.301 (bad)
• log(1 - D(G(z))): Fake data G(z) should be classified as fake (D(G(z)) → 0) → log(1) = 0 (best); log(0.5) = -0.301 (bad)
• Interpretation: D wants both terms large (=0). Strategy: D(x) → 1 and D(G(z)) → 0

Generator’s perspective (min_G: wants to minimize this term):

• G doesn’t have direct control over log D(x) (real data is fixed)
• G minimizes log(1 - D(G(z))): wants D(G(z)) → 1 (fake data classified as real)
• Alternative phrasing: Instead of min log(1 - D(G(z))), practitioners often use max log D(G(z)) (same effect, better gradient signal)

Intuition:

• D’s game: “I get better when I correctly identify real vs. fake”
• G’s game: “I get better when D mistakes my fake for real”
• Nash equilibrium: When P_G = P_data (perfect generator) → D can’t distinguish → D(x) = D(G(z)) = 0.5 for all x, z

Why it’s unstable:

• When P_G is far from P_data, D becomes very confident (D(G(z)) ≈ 0 for all G(z))
• log(1 - D(G(z))) ≈ log(1) ≈ 0 → gradient ≈ 0 → G gets NO signal to improve
• D wins completely; G can’t learn (mode collapse, training collapse)

2. DCGAN (Deep Convolutional GAN) (2015)

Deconvolution (Transposed Convolution): Inverse of strided convolution — upsamples spatial dimensions while mixing channels. Example: 4×4 input → 3×3 kernel with stride=2 → 9×9 output (learnable upsampling).

Key Concept: Batch Normalization stabilizes gradient flow. Stride convolutions + LeakyReLU provide better signal propagation. Result: reproducible, stable training.

3. Wasserstein GAN (WGAN) (2017)

Key Concept: Wasserstein distance provides gradients even when distributions don’t overlap. Loss is meaningful throughout training (not just 0 or log(2)).

4. Spectral Normalization GAN (2018)

Key Concept: Spectral norm is the largest singular value of weight matrix. Constraining it bounds gradient magnitude → stable training. Simpler than WGAN gradient penalty.

5. Conditional GAN (cGAN) (2014)

Key Concept: Condition both G and D on auxiliary info (class, attributes). Enables controllable generation.

6. Pix2Pix (cGAN for Image-to-Image Translation) (2016)

Key Concept: U-Net + PatchGAN + L1 loss = stable, structured translation. Paired data required but realistic outputs.

7. CycleGAN (Unpaired Image Translation) (2017)

Key Concept: Cycle consistency (x_A → G(x_A) → x_A) is self-supervision. Without paired data, forces semantic consistency during translation.

8. StyleGAN (2018)

Key Concept: Separate style (applied via AdaIN) from structure (constant input + coarse layers). Result: disentangled generation; can edit hairstyle (layer 3-5) independent of face shape (layers 1-2).

9. StyleGAN2 (2019)

Key Concept: Path length penalty ensures interpolations are smooth. R1 gradient penalty (on D) is better than weight clipping (simpler, more stable).

GAN Evolution Summary: Vanilla (unstable, mode collapse) → DCGAN (BN helps) → WGAN (Wasserstein loss + stable gradients) → Spectral Norm (controls Lipschitz) → conditional variants (Pix2Pix paired, CycleGAN unpaired) → StyleGAN (disentangled style) → StyleGAN2 (polished, production-ready).

C. Attention & Sequence Models Evolution

VAE Evolution Logic:

1. AE → VAE: Add KL constraint on z; force z ~ N(0,I); enables interpolation and generation
2. VAE → β-VAE: Weight KL by β>1; trade reconstruction for disentanglement; separate factors of variation
3. VAE → VQ-VAE: Discrete codebook instead of continuous; sharper samples but less interpretable
4. VAE → Hierarchical: Multi-scale latent structure; different levels capture different abstractions

B. Generative Models Evolution (Adversarial Paradigm)

GAN Evolution Logic:

1. Vanilla GAN → DCGAN: Architecture guidelines (stride convs, BN, structured networks); training stabilization
2. DCGAN → WGAN: Replace JS divergence with Wasserstein distance; smoother gradients; less mode collapse
3. WGAN → Spectral Norm: Stabilize discriminator via weight normalization; improve diversity
4. Vanilla/DCGAN → Conditional: Add class conditioning; controllable generation
5. Conditional → Pix2Pix: Paired image translation; L1 reconstruction + adversarial
6. Pix2Pix → CycleGAN: Unpaired translation; cycle consistency loss enables style transfer without data
7. Conditional → StyleGAN: Disentangle style from content; mapping network (W space); adaptive instance normalization; exceptional quality
8. StyleGAN → StyleGAN2: Remove artifacts; progressive training; path length regularization

C. Attention & Sequence Models Evolution

Core Concept Evolution: RNNs process sequentially (slow). LSTMs add gated memory (faster gradient flow). Attention removes the sequential bottleneck (parallelizable). Transformers eliminate recurrence entirely (foundation for LLMs).

1. RNN (Recurrent Neural Network) (~1986)

Loss Function & Training:

For sequence classification (entire sequence → one label):

• Process entire sequence: x_1 → x_2 → … → x_T
• Use final hidden state h_T for classification
• Loss: L = CrossEntropy(softmax(W × h_T), y_true)

For sequence-to-sequence (each timestep predicts next token, e.g., language modeling or machine translation):

• Process x_1 → h_1 → predict y_1
• Process x_2 → h_2 → predict y_2
• …
• Process x_T → h_T → predict y_T
• Loss: L = ∑_{t=1}^T CrossEntropy(softmax(W × h_t), y_t)

Example: Language model predicting next word

1
2
3
4
5
6
7
8
9

Input sequence:  [The, cat, sat, on, the]
Target:          [cat, sat, on, the, <EOS>]

At t=1: h_1 from "The" → predict "cat" → L_1 = -log P("cat")
At t=2: h_2 from "The, cat" → predict "sat" → L_2 = -log P("sat")
...
At t=5: h_5 from "The, cat, sat, on, the" → predict "<EOS>" → L_5 = -log P("<EOS>")

Total loss: L = (L_1 + L_2 + L_3 + L_4 + L_5) / 5

Backpropagation Through Time (BPTT):

Goal: Adjust weights W_h, W_x so that each h_t predicts the next token well.

Forward pass (left to right):

1
2
3
4
5

x_1 → [h_1 = tanh(W_h×h_0 + W_x×x_1)] → output y_1, loss L_1
      ↓
x_2 → [h_2 = tanh(W_h×h_1 + W_x×x_2)] → output y_2, loss L_2
      ↓
x_3 → [h_3 = tanh(W_h×h_2 + W_x×x_3)] → output y_3, loss L_3

Backward pass (right to left; “through time”):

1
2
3
4
5
6
7
8

∂L/∂W_h = ∂L_3/∂W_h + ∂L_2/∂W_h + ∂L_1/∂W_h

where ∂L_3/∂W_h depends on: h_3, h_2, h_1, h_0 (chains backward)

∂L_3/∂W_h = ∂L_3/∂y_3 × ∂y_3/∂h_3 × ∂h_3/∂W_h 
           + ∂L_3/∂y_3 × ∂y_3/∂h_3 × ∂h_3/∂h_2 × ∂h_2/∂W_h
           + ∂L_3/∂y_3 × ∂y_3/∂h_3 × ∂h_3/∂h_2 × ∂h_2/∂h_1 × ∂h_1/∂W_h
           + ... (chains all the way back to h_0)

Problem: Vanishing Gradients

The chain rule multiplies: ∂h_3/∂h_2 × ∂h_2/∂h_1 × ∂h_1/∂h_0

Since ∂h_t/∂h_{t-1} = tanh’(·) × W_h ≈ 0.1-0.9 (tanh’ peaks at 0.25), the product shrinks:

• 0.5 × 0.5 × 0.5 = 0.125 (already small after 3 steps)
• 0.5^7 ≈ 0.0078 (nearly vanishes after 7 steps)

Result: Gradients reaching early timesteps are nearly zero → weights don’t update → RNN can’t learn long-term dependencies.

How RNNs Know When to END (Sequence Termination):

RNNs don’t automatically know when to stop. Three strategies:

Strategy 1: Fixed Length

• Always predict exactly T timesteps (e.g., T=100 words)
• Simple; used in fixed-length sequence tasks
• Inefficient: wastes computation on padding

Strategy 2: EOS (End-Of-Sequence) Token

• Add special token <EOS> to vocabulary
• During training: include <EOS> in target sequence (e.g., [“hello”, “world”, “”])
• Loss includes predicting <EOS> at end: L_T = -log P("")
• Network learns: “when done, predict ”
• During generation:
  • Generate y_1, y_2, … until model predicts <EOS>
  • Stop immediately (variable-length output)

Example:

1
2
3
4
5
6
7
8

Target: ["The", "cat", "sat", "<EOS>"]
During generation:
  t=1: sample "The" (argmax or random sample)
  t=2: sample "cat"
  t=3: sample "sat"
  t=4: sample "<EOS>" → STOP (model decided to end)
  
Result: "The cat sat" (3 words, not padded to fixed length)

Strategy 3: Maximum Length (Inference Fallback)

• Set max_length=100; stop after 100 predictions even if <EOS> not generated
• Safety mechanism; prevents infinite loops
• Used as backup if model gets stuck

Why This Matters for Interviews:

• ✅ Loss function: Per-timestep cross-entropy; why each position matters
• ✅ BPTT gradient chains: Why early timesteps get vanishing gradients
• ✅ EOS token: How RNNs learn to generate variable-length sequences
• ✅ Vanishing gradients problem: Motivates LSTM (next architecture)

2. LSTM (Long Short-Term Memory) (1997) — Intuitive Explanation

Intuition (Forget the Math): Think of C_t as a “conveyor belt” running through time. At each step:

1. Forget gate decides: “What should I drop off this conveyor?” (multiply by ~0-1)
2. Input gate decides: “What new information should I add?” (multiply by ~0-1)
3. Output gate decides: “What should I output to the next step?” (multiply by ~0-1)

The crucial insight: Addition (+) instead of multiplication (×). A conveyor belt carrying information forward via addition prevents gradients from vanishing. Information can persist unchanged (forget gate ≈ 1) or be erased (forget gate ≈ 0), but the path through addition keeps gradients alive.

Example: Sentiment analysis “The movie was great but the plot was confusing”

• Forget gate: “drop the ‘great’” when you see “but” (learn this via backprop)
• Input gate: “add the ‘confusing’” when you see it
• Result: Final sentiment ≈ negative (correctly ignored the early “great”)

3. GRU (Gated Recurrent Unit) (2014) — Simplified LSTM

Intuition: GRU is LSTM’s “lite” version. Instead of “what to forget” + “what to add” + “what to output”, GRU just asks:

• Update gate: “How much should I update? Keep 90% of h_{t-1}, replace 10% with h̃_t” (linear blend)
• Reset gate: “Should I reset the hidden state before computing the candidate?” (optional)

Practical: LSTM if you have compute/data; GRU if training time is tight.

4. Transformer (Self-Attention Architecture) (2017) — Comprehensive Fundamentals

The Core Insight: Forget about recurrence. Instead of processing x_1 → x_2 → … → x_T sequentially, process all T tokens at once, and let each token attend to every other token to build context.

Problem Solved: RNNs are slow (sequential) and lose information (hidden state bottleneck). Transformers are fast (parallel) and have no bottleneck (direct attention to all history).

Loss Function (depends on task):

• Language Modeling (GPT): Cross-entropy per token. Predict next token given all previous tokens (causal masking). Loss = -log P(y_t | y_1…y_{t-1})
• Machine Translation (seq2seq): Cross-entropy per token. Encoder processes source; Decoder predicts target tokens. Loss = ∑t -log P(y_t | x, y_1…y{t-1})
• Classification (BERT): Cross-entropy over final [CLS] token or token sequence. Loss = -log P(class | input tokens)
• Masked Language Model (BERT): Cross-entropy on masked tokens only. Randomly mask 15% of tokens; predict masked tokens from context. Loss = -log P(masked_token | unmasked context)

Key concept: Unlike GANs (adversarial loss) or VAE (reconstruction + KL), Transformer loss is task-specific (usually cross-entropy). The “learning” happens via attention—weights adjust to predict the target correctly.

Section 4.1: What is Self-Attention?

Example: Sentence “The cat sat on the mat”

• When processing “sat”: Q = “I need subject/object info”; scan all Keys (Q·K) to find “cat” and “mat”; pull their Values (information)
• Attention = weighted combination: high weight on “cat” and “mat”, low on “the”

Formula:

1
2
3
4
5
6

Attention(Q, K, V) = softmax(Q·K^T / √d) × V

- Q·K^T: Compare query to all keys (matrix multiplication) → similarity scores
- / √d: Normalize by dimension (prevents huge values)
- softmax: Convert scores to probabilities (0-100%, sum to 1)
- × V: Weight and sum all values using those probabilities

Result: Each token gets a custom weighted combination of all tokens’ values. The weighting is learned (via training).

Section 4.2: Multi-Head Self-Attention (Multiple Representation Subspaces)

Problem: One attention head sees the entire embedding space. Multiple heads = richer representations.

Intuition: Like having 8 specialists looking at the data from different angles, then combining their insights.

Section 4.3: Position Encoding (How Transformers Know Token Order)

Problem: Self-attention is “position-agnostic”. “The cat sat” vs. “sat the cat” look identical (same tokens, different positions).

Solution: Add position information via sinusoidal encoding:

1
2

PE(position, 2i)   = sin(position / 10000^{2i/d})
PE(position, 2i+1) = cos(position / 10000^{2i/d})

• Position 0: [sin(0), cos(0), sin(0), cos(0), …]
• Position 1: [sin(1/10000^0), cos(1/10000^0), sin(1/10000^2), …]
• Position T: [sin(T/1), cos(T/1), …]

Result: Each position has a unique vector. Add this to token embeddings before attention. Network learns “earlier tokens have these patterns, later tokens have those patterns”.

Section 4.4: Transformer Block (Encoder + Decoder Structure)

Encoder (process input; build understanding):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

Input tokens → Embedding + Position Encoding
           ↓
        Multi-Head Self-Attention (each token attends to all)
           ↓
        Add & Norm (residual + layer normalization)
           ↓
        Feed-Forward (Dense → ReLU → Dense, per token)
           ↓
        Add & Norm (residual + layer normalization)
           ↓
        Output (repeat layer 6-24 times)

Key Components:

• Multi-Head Self-Attention: Build context by attending to all tokens
• Feed-Forward: Per-token dense layer (not recurrent; parallelizable)
• Residual Connections (Add): y = layer(x) + x (helps gradient flow)
• Layer Norm: Stabilizes training; applied before each sub-layer (pre-norm)

Decoder (generate output; one token at a time):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

Previous output tokens → Embedding + Position Encoding
           ↓
        Multi-Head Self-Attention (CAUSAL: can only attend to past tokens)
           ↓
        Add & Norm
           ↓
        Multi-Head Cross-Attention (attend to Encoder output)
           ↓
        Add & Norm
           ↓
        Feed-Forward
           ↓
        Add & Norm
           ↓
        Output (Dense + Softmax → next token)

Key Difference: Decoder uses causal masking (future tokens set to -∞ before softmax) + cross-attention to encoder.

Self-Attention vs Cross-Attention Explained:

Intuition: Self-attention = “understand relationships within decoder sequence”. Cross-attention = “align decoder with encoder”. Combined, they enable: (1) coherent output (self-attention), (2) relevant to input (cross-attention).

Section 4.5: Why Transformers Scale to Billions of Parameters

Scaling Laws: Transformer performance scales predictably with model size (# params) and dataset size. Double the model → ~3% accuracy gain. RNNs hit a wall; Transformers keep improving.

Section 4.6: Parallelization Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

RNN (Sequential):
t=1: x_1 → h_1 (wait for previous h_0, done)
t=2: x_2 → h_2 (wait for h_1, done)
t=3: x_3 → h_3 (wait for h_2, done)
Total: 3 sequential steps

Transformer (Parallel):
t=1,2,3: x_1, x_2, x_3 all processed simultaneously
- Embed all 3 tokens
- Compute attention all at once (one matrix operation: Q×K^T for all pairs)
- Compute FF all at once (batch operation)
Total: 1 "step" (GPU computes matrix ops on all T tokens)

Speedup: O(T) sequential steps → O(1) parallel steps (in terms of GPU cycles).

Key Concepts Summary:

• Self-Attention: Each token attends to all others; learns what’s important
• Multi-Head: Different heads specialize (subject-verb, long-range, punctuation)
• Position Encoding: Sinusoidal signals tell network position information
• Residual + Norm: Gradient flow + training stability
• Parallel Architecture: No recurrence = all tokens at once = GPUs happy
• Scalability: Scales to billions of parameters; improves with more data

5. Vision Transformer (ViT) (2020)

Key Concept: Apply Transformer to vision (no convolutions). Proves attention mechanism is sufficient without convolutional inductive biases. Paradigm shift: Architecture-agnostic learning.

6. Masked Self-Attention (Causal) (~2019)

Key Concept: Causal masking enforces autoregressive constraint (no information leakage from future). Enables parallel training of language models.

Sequence/Attention Evolution Summary: RNN (sequential) → LSTM (gated memory) → Transformer (pure attention) → ViT (vision). Each step traded sequential processing for parallelization/global context.

D. Metric Learning & Siamese Networks

Core Concept: Learn embeddings where similar instances cluster; dissimilar instances spread apart. Enable one-shot, zero-shot, few-shot learning.

Key Concepts:

• Triplet loss: d(a, p) - d(a, n) + m ≤ 0 (margin ensures separation)
• Hard negative mining: Select negatives that are hard to distinguish (closest to positive); improves convergence
• Few-shot learning: Use prototypical networks to classify with minimal examples (5 or 10 per class)

Stage 5 Summary: Why Each Paradigm?

Elevator Answer:

Stage 1-4 Gap: Classification models learn p(y|x)—they predict labels but can’t generate new samples, understand data distribution, or capture long-range context efficiently.

Stage 5 Solution (Three complementary paradigms):

1. Generative Models (VAE & GAN):

   • VAE: Encode x → z ~ N(0,I); decode z → x’. Probabilistic, interpretable, can interpolate. ELBO loss balances reconstruction + KL regularity. β-VAE (β>1) forces disentanglement. Trade-off: blurry outputs.
   • GAN: Generator vs. Discriminator adversarial game. Sharper samples than VAE. Unstable training (JS divergence saturates). Fixed by: DCGAN (BN), WGAN (Wasserstein distance), Spectral Norm (Lipschitz constraint), StyleGAN (disentangled style).

2. Attention & Transformers (Replace sequential bottleneck):

   • RNN/LSTM/GRU: Process sequentially; LSTM fixes vanishing gradients via gated memory (C_t = f⊙C_{t-1} + i⊙C̃_t).
   • Attention: Query-key-value mechanism. Multi-head: each head specializes (syntax, semantics, position). O(T²) but captures any-to-any dependencies in 1 layer.
   • Transformer: No recurrence; pure multi-head self-attention + position encoding. Fully parallelizable. Foundation for LLMs (BERT, GPT, T5).
   • Vision Transformer: Apply Transformer to images (patches); outperforms CNNs at scale; no inductive bias needed.

3. Metric Learning (Few-shot, zero-shot):

   • Siamese networks + Triplet loss: Learn embeddings where d(anchor, same-class) < d(anchor, different-class) + margin.
   • Enables one-shot (1 example), few-shot (5-10 examples), zero-shot (no training examples) learning.

Result: VAE/GAN for generation & synthesis | Transformers for language & vision at scale | Metric learning for few-shot & retrieval.

Paradigm Trade-offs:

• VAE: Interpretable but blurry
• GAN: Sharp but training unstable (until stabilization tricks)
• LSTM: Sequential; proven; slower on GPUs
• Transformer: Parallelizable; scales infinitely; but O(T²) memory; no local inductive bias
• Metric Learning: Sample-efficient; requires careful mining strategies

New Frontiers Beyond Stage 5: ❌ Diffusion models (iterative refinement; beats GAN on quality) | ❌ Contrastive learning (SimCLR, MoCo; self-supervised) | ❌ Vision-Language models (CLIP; multimodal) | ❌ Multimodal fusion (text+image+audio)

Architecture Family Tree

Deep Learning

Discriminative: Learn p(y|x)—predict labels given inputs; supervised, task-specific.
Generative: Learn p(x)—model data distribution; can synthesize new samples; unsupervised or self-supervised.

1. DISCRIMINATIVE (Supervised)

• Neural Networks (NN)

  • Multilayer Perceptron (MLP)
  • Activation Functions (ReLU, Sigmoid, Tanh)
  • Backpropagation
  • Optimization (SGD, Adam, etc.)

• Convolutional Neural Networks (CNN)

  • LeNet-5 (1998)

  • AlexNet (2012)

  • VGG (2014)

  • ResNet (2015)

  • DenseNet (2016)

  • Inception (2015)

  • MobileNet (2017)

  • EfficientNet (2019)

  • Task-Specific CNN Variants

    • YOLO (Real-time Object Detection)
    • R-CNN / Fast R-CNN / Faster R-CNN (Region-based Detection)
    • U-Net (Semantic Segmentation)
    • Mask R-CNN (Instance Segmentation)

• Recurrent Neural Networks (RNN)

  • LSTM (Long Short-Term Memory)
  • GRU (Gated Recurrent Unit)
  • Conv-LSTM (Convolutional + Recurrent)

2. GENERATIVE (Unsupervised)

• Variational Autoencoders (VAE)

  • β-VAE (Disentangled Representations)
  • β-TCVAE (Factorized VAE)
  • Hierarchical VAE
  • Vector-Quantized VAE (VQ-VAE)

• Generative Adversarial Networks (GAN)

  • DCGAN (Deep Convolutional GAN)
  • Pix2Pix (Conditional GAN)
  • StyleGAN (Style-based Generator)
  • CycleGAN (Unpaired Image-to-Image)

3. ATTENTION-BASED (Context)

• Transformer Architecture
  • Self-Attention
  • Multi-Head Attention
  • Cross-Attention
  • Vision Transformer (ViT)
  • BERT (Bidirectional Encoder)

4. METRIC LEARNING

• Siamese Networks
  • Contrastive Learning
  • Triplet Loss

Key Innovation Timeline

Parameter Counting Cookbook

Formula Reference

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

Conv Layer: 
  Params = (kernel_h × kernel_w × in_channels + bias) × out_channels
  Example: Conv(out_channels=32, kernel=3×3, in_channels=3)
           = (3 × 3 × 3 + 1) × 32 = 28 × 32 = 896 params

Dense Layer:
  Params = (in_features + bias) × out_features
  Example: Dense(in=256, out=10)
           = (256 + 1) × 10 = 2,570 params

Batch Normalization (per channel):
  Params = 2 × num_channels (weight + bias; no running stats count as "trainable")

Dropout:
  Params = 0 (no learnable parameters)

Pooling:
  Params = 0 (no learnable parameters)

Worked Example

Problem: Count trainable + non-trainable params in:

1
2
3
4
5
6
7

Input(224×224×3) 
  → Conv(36 filters, 2×2, no_padding, stride=1)
  → BatchNorm()
  → Dropout(0.2)
  → Conv(7 filters, 2×2, no_padding, stride=1)
  → Flatten()
  → Dense(softmax)

Step 1: Conv Layer 1

• Input: 224×224×3
• Output: (224-2+1) × (224-2+1) × 36 = 223×223×36 (no padding, 2×2 kernel)
• Params: (2×2×3 + 1) × 36 = 13 × 36 = 468 trainable
• Non-trainable: 0

Step 2: BatchNorm

• Input channels: 36
• Params: 2×36 = 72 trainable (γ weight, β bias; running mean/var are not trainable)
• Non-trainable: 2×36 = 72 (running mean, variance)

Step 3: Dropout

• Params: 0

Step 4: Conv Layer 2

• Input: 223×223×36
• Output: (223-2+1) × (223-2+1) × 7 = 222×222×7
• Params: (2×2×36 + 1) × 7 = 145 × 7 = 1,015 trainable
• Non-trainable: 0

Step 5: Flatten

• Input: 222×222×7 = 345,468 neurons
• Output: 1D vector of 345,468 units
• Params: 0

Step 6: Dense (Softmax)

• Assuming 10 classes
• Params: (345,468 + 1) × 10 = 3,454,690 trainable
• Non-trainable: 0

Total:

Key Insight: 99.9% of parameters are in the final Dense layer (after Flatten). This is why:

1. Early CNN layers are cheap (weight sharing)
2. Flattening kills efficiency (explodes parameter count)
3. Global Average Pooling or Fully-Convolutional architectures are preferred (reduce flattened size)

Elevator Answers Bank

Use these as interview blueprints. Practice saying them aloud in 2-3 minutes.

Q: What are skip connections? Why do they help?

Answer:

A skip connection takes the input to a layer and adds it directly to the output, bypassing the layer’s learned transformation. In ResNet: output = Conv(input) + input instead of just output = Conv(input).

Why it helps: Backpropagation multiplies gradients across layers. With many layers, these products shrink toward zero (vanishing gradient). Skip connections create a “shortcut path” where gradients flow untouched—gradient = 1 along the skip path. This means even if Conv(input) gradients vanish, the skip path preserves the signal. Result: we can train 100+ layer networks.

Secondary benefit: The network learns residuals (differences) rather than full reconstructions. It’s often easier to learn f(x) = y - x than g(x) = y directly.

Q: How would you build a model to count the number of people in a room?

Answer:

This is a regression + localization task masquerading as counting. Two approaches:

Approach 1: Density Map (Best for crowded scenes)

• Input: Image
• Output: Heatmap same spatial size, pixel values = local crowd density
• Loss: MSE between predicted density and ground-truth density map
• Count: Integrate (sum) the heatmap
• Why: Handles occlusion; robust to scale/pose variation
• Architecture: U-Net encoder-decoder (upsampling via transposed conv)

Approach 2: Direct Regression (Simple scenes)

• Input: Image
• Network: CNN → Global Average Pooling → Dense → scalar output (count)
• Loss: MSE or Poisson regression (counts are discrete, Poisson models variance better)
• Why: Simple; no annotation overhead for dense labels
• Limitation: Fails in occlusion; assumes fixed camera

In interview: “I’d start with Approach 1 (density map + U-Net) because it’s more interpretable—I can visualize where the model thinks people are—and generalizes better to dense/occluded scenes. Loss function choice matters: MSE works, but Poisson regression or negative log-likelihood is theoretically better for count data.”

Q: What’s the difference between VAE and GAN?

Answer:

Both learn to generate images, but via fundamentally different approaches:

Aspect	VAE	GAN
Goal	Learn data distribution + interpretable latent space	Learn to generate realistic samples
Mechanism	Encode image → latent z with constraint (KL); decode z → reconstruct	Generator creates fake; Discriminator classifies real vs. fake; play min-max game
Loss	Reconstruction + KL divergence (explicit, interpretable)	Adversarial (implicit; min-max formulation)
Latent space	Continuous, axis-aligned Gaussian; interpolation works	No guarantee; may have holes/discontinuities
Sample quality	Blurry, but stable; training is straightforward	Sharp, photo-realistic but training is unstable (mode collapse)
Inference	Deterministic decoder; fast	Generator only; no encoder
Best use	Anomaly detection, controlled generation, interpolation	Photo-realistic synthesis, style transfer, super-resolution

Key tradeoff: VAEs are principled and interpretable but produce blurry outputs. GANs are empirically better at realism but harder to train and interpret.

In interview: “VAE is a full probabilistic model; GAN is an adversarial game. VAEs are better when you need interpretability and stable training (anomaly detection). GANs are better when you need realism (face generation). Modern practice uses hybrid: StyleGAN (GAN-based but with disentangled latent code like VAE).”

Q: Compute trainable parameters for [architecture]

Use the Parameter Counting Cookbook above. During interview:

Answer:

“I’ll break it down layer by layer:

• Conv layer: (kernel_h × kernel_w × in_channels + 1) × out_channels
• Dense layer: (in_features + 1) × out_features
• BN: 2 × num_channels (weight + bias)
• Dropout/Pooling: zero params

Let me trace through [their architecture]: [Show calculation]

The key insight: Early CNN layers are cheap due to weight sharing. Flattening explodes parameter count. This is why modern architectures use Global Average Pooling or Fully-Convolutional designs.”

Q: Explain backpropagation in a 2-layer network

Answer:

Forward pass:

• Input x → hidden layer (y = W1·x + b1) → activation (a = ReLU(y)) → output (ŷ = W2·a + b2)

Loss: L = (ŷ - y_true)²

Backward pass (chain rule):

• dL/dW2 = dL/dŷ · dŷ/dW2 = 2(ŷ - y_true) · a ← gradient of loss w.r.t. W2
• dL/da = dL/dŷ · dŷ/da = 2(ŷ - y_true) · W2 ← backprop to activation
• dL/dW1 = dL/da · da/dy · dy/dW1 = [2(ŷ - y_true) · W2 · 1{y>0}] · x ← ReLU derivative
• Update: W2_new = W2 - α · dL/dW2 (similarly for W1, b1, b2)

Key insight: Gradients are products of derivatives chained backward. If each derivative < 1, the product shrinks (vanishing gradient). If > 1, it explodes. This is why weight initialization and activation choice matter."

Q: Why is ReLU better than sigmoid in deep networks?

Answer:

Sigmoid: σ(z) = 1/(1+e^-z); derivative max = 0.25

• Gradient always < 0.25; multiply through 100 layers → ~10^-50 (vanishing)
• Computationally expensive (exponential)

ReLU: max(0, z); derivative = 1 (if z>0) or 0 (if z<0)

• Gradient = 1 for active neurons; no exponential shrinkage
• Simple: one comparison + max
• Problem: “Dead ReLU”—once a neuron fires z<0, gradient=0 forever

Modern fix: Leaky ReLU (0.01·z if z<0) ensures gradient always nonzero

In interview: “ReLU is the standard because its gradient is linear (not exponential decay like sigmoid), preventing vanishing gradients in deep networks. Trade-off: Dead ReLU neurons, which Leaky ReLU fixes.”

Q: Explain Batch Normalization

Answer:

Batch Normalization (BN) normalizes each layer’s input to have zero mean and unit variance within a batch:

1. Normalize: z_norm = (z - μ_batch) / √(σ_batch² + ε)
2. Scale/shift: z_out = γ·z_norm + β (learnable parameters)

Why it helps:

• Internal Covariate Shift: As earlier layers’ weights change, later layers see wildly different distributions. BN keeps distributions stable.
• Allows higher learning rates: Gradients are more stable; don’t explode/vanish.
• Acts as regularizer: Noise from small batch sizes acts like dropout.

During training: Use batch statistics (μ, σ from current batch)
During inference: Use running statistics (exponential moving average computed during training)

Trade-off: Depends on batch size. Tiny batches (batch=1) → high noise; BN struggles. Large batches → stable but may not fit in memory. Layer Normalization (normalize across features, not batch) fixes this for some cases."

Q: What’s the difference between semantic and instance segmentation?

Answer:

Task	Goal	Output	Use Case
Semantic Segmentation	Classify each pixel	Heatmap per class; all instances of a class have same label	Medical imaging (tumor vs. healthy), scene parsing
Instance Segmentation	Classify + separate individuals	Mask per object; each car is separate even if both are “cars”	Autonomous driving, object counting, crowd density

Architectures:

• Semantic: U-Net, DeepLab (atrous convolution for receptive field)
• Instance: Mask R-CNN (Faster R-CNN + FCN head per region)

In interview: “Semantic answers ‘what is each pixel?’; instance answers ‘what is each pixel and which object does it belong to?’ Mask R-CNN is the go-to for instance: it detects boxes (Faster R-CNN), then segments within each box.”

Q: How does YOLO differ from R-CNN?

Answer:

Aspect	R-CNN / Faster R-CNN	YOLO
Approach	Region-based (find regions, then classify)	Single-stage (classify grid cells directly)
Pipeline	1. Region proposals (RPN/Selective Search) 2. Classify each region 3. Refine boxes	1. Divide image into grid 2. Predict class + box offset per cell 3. Done
Speed	~25 fps (Faster R-CNN)	45+ fps (YOLO-v1)
Accuracy	Higher mAP (better localization)	Slightly lower (trades accuracy for speed)
Small objects	Better (RPN focuses on candidates)	Worse (grid cells too coarse for tiny objects)
Use case	High-accuracy applications (medical imaging)	Real-time applications (autonomous driving, surveillance)

Evolution: YOLO-v1 (grid cells) → YOLO-v3 (multi-scale predictions) → YOLO-v5 (CSPDarknet backbone, improved NMS)

In interview: “YOLO trades accuracy for speed via a single forward pass. It’s unified end-to-end but struggles with small/clustered objects. Faster R-CNN is region-based: RPN proposes regions, then we classify—better for accuracy-critical tasks.”

Q: What is transfer learning? When would you use it?

Answer:

Transfer learning: Train on large dataset (ImageNet), then fine-tune on your small dataset.

Stages:

1. Pre-trained backbone: Load weights trained on ImageNet (e.g., ResNet-50)
2. Freeze or fine-tune:
   • Freeze early layers (learn low-level features like edges)
   • Fine-tune later layers (learn task-specific patterns)
3. Add task-specific head: Replace last Dense layer; train from scratch

When to use:

• ✅ Small dataset (<10K images); pre-training reduces overfitting
• ✅ Similar domain (natural images); ImageNet features transfer well
• ✅ Limited compute; no need to train from scratch (3-7 days → 1-2 hours)

When NOT to use:

• ❌ Large dataset (>1M images); train from scratch often better
• ❌ Highly specialized domain (e.g., medical ultrasound); ImageNet features don’t apply
• ❌ Frozen backbone + new head underperforms; need some fine-tuning

In interview: “Transfer learning is Occam’s Razor for deep learning: if someone already trained ImageNet, why retrain? Use their weights + adapt. Only train from scratch if your domain is far from ImageNet or you have massive data.”

When to Use What

Classification Task

Detection Task

Segmentation Task

Generative Task

Sequence/Video Task

Common Interview Gotchas

1. “How deep should my network be?”

Common wrong answer: “Deeper = better; always use 152 layers.”
Better answer: “Depends on data + compute. Generalization error ~ O(1/n + capacity) where n=dataset size, capacity=model complexity. Deep networks fit data better but overfit on small datasets. Use validation loss to pick depth. On ImageNet (1.3M images), 50-152 layers is standard. On <10K images, 18-34 layers is safer.”

2. “Why did you use 3×3 convolutions instead of 5×5?”

Common wrong answer: “3×3 is smaller; fewer parameters.”
Better answer: “3×3 stacking has two benefits: (1) receptive field grows depth-wise (two 3×3 = one 5×5 in field size but nonlinear), (2) parameter reduction: 5×5 has 25 weights per channel; two 3×3 have 18. Also, 3×3 is hardware-optimized on GPUs. Trade-off: needs 2× forward passes instead of 1. VGG popularized this.”

3. “Why does batch normalization hurt at test time?”

Common wrong answer: “It doesn’t; use batch norm everywhere.”
Better answer: “BN uses running statistics at test time (computed during training), not batch statistics. If train/test distributions shift (domain adaptation), running stats become stale. Solution: use Layer Norm (normalizes features, not batch) or update BN stats with test data.”

4. “Can I just fine-tune the last layer if I have 10K images?”

Common wrong answer: “Yes, transfer learning only needs last-layer training.”
Better answer: “Depends on domain similarity. If domain is close to ImageNet (natural images), fine-tune head only. If domain is far (e.g., medical ultrasound), unfreeze 2-3 layers and use low learning rate. Rule of thumb: freeze if data<1K; fine-tune if data>10K.”

5. “Why does my GAN’s generator collapse to one image (mode collapse)?”

Common wrong answer: “Use more layers.”
Better answer: “Mode collapse = generator learns to fool discriminator by producing one highly-convincing sample, ignoring others. Solutions: (1) Wasserstein GAN (smoother loss), (2) spectral normalization (stabilize discriminator), (3) progressive training (grow complexity), (4) ensemble of generators, (5) better architecture (StyleGAN uses style-mixing). GANs are adversarial; equilibrium is hard to reach.”

6. “How do I decide between MSE and cross-entropy loss?”

Common wrong answer: “MSE for regression, cross-entropy for classification.”
Better answer: “More nuanced: (1) MSE assumes Gaussian noise around predictions; unbounded outputs. (2) Cross-entropy assumes categorical distribution; bounded (softmax). (3) For regression, MSE is standard unless outliers are heavy—use Huber loss. (4) For classification with imbalanced classes, weighted cross-entropy. (5) For counts (people, cars), Poisson regression is theoretically better than MSE.”

Deep-Dive Links

Each section below is a separate document with full details. Read sequentially or jump as needed:

• 01_Neural_Networks_Foundations.md — Forward/backward pass, activation functions, weight initialization, optimizers, hyperparameter tuning
• 02_CNNs_and_Convolution.md — Convolution mechanics, pooling, BN, Dropout, data augmentation, parameter counting
• 03_SOTA_Architectures.md — LeNet → AlexNet → VGG → ResNet → DenseNet → EfficientNet; skip connections, bottleneck blocks, depthwise-separable
• 04_Computer_Vision_Tasks.md — Detection (YOLO, R-CNN), Segmentation (U-Net, DeepLab), Localization, Instance vs. Semantic
• 05_Generative_and_Attention.md — VAE, GAN, Attention, Siamese, Sequence models (LSTM, GRU), Transformers

Quick Review Checklist

• Narrative: Can you explain “why CNN after NN” in 1 minute?
• Timeline: Can you list 5 major architectures + innovations?
• Parameter counting: Can you compute params for a 3-layer network by hand?
• Elevator answers: Can you answer “What are skip connections?” without notes?
• Task selection: Given a problem (counting people, real-time detection), can you pick an architecture?
• Gotchas: Do you know the pitfalls (mode collapse, vanishing gradients, BN at test time)?

Status: ✅ Master map complete. Ready for sequential deep-dives (01 → 05) or ad-hoc jumps.

Next: Proceed to 01_Neural_Networks_Foundations.md or ask for refinement.