Deep learning, built from the University of Nebraska CSCE 479/879 course. Where the machine learning section covers learning from data in general, this section is about neural networks with depth, the model class that learns its own representations instead of relying on hand-engineered features. Each note links back to the machine-learning fundamental it stands on. The history of how these models took over is in The Deep Learning Revolution.
Foundations
- Artificial Neural Networks - the perceptron to the multilayer network.
- Backpropagation - how a deep network computes its gradients.
- Activation Functions - the nonlinearity that makes depth mean something.
Vision: convolutional networks
- Convolutional Neural Networks - shared local filters that exploit spatial structure.
- Feature Attribution and Saliency - gradients with respect to the input, revealing which pixels drove a prediction.
- Pooling and CNN Architectures - downsampling and the LeNet to ResNet arc.
Sequences and representations
- Recurrent Neural Networks - networks with state over sequences, and LSTM/GRU.
- Embeddings - learned dense vectors whose geometry encodes meaning.
- Tokenization and Subword Units - how text becomes the integers a model actually sees, and what BPE buys.
- Attention and Transformers - every position attends to every other, no recurrence, the architecture behind large language models.
- Self-Supervised Learning and Pretraining - inventing labels from unlabeled data, and the pretrain-then-finetune pattern.
Graphs
- Graph Neural Networks - message passing over graph structure, and permutation invariance.
Generative models
- Autoencoders - encoder, bottleneck, decoder, and the variational version.
- Generative Adversarial Networks - generator versus discriminator.
- Diffusion Models - reverse a noising process to generate.
Reinforcement learning
- Reinforcement Learning - agents, rewards, and Markov decision processes.
- Deep Reinforcement Learning - neural function approximation, DQN, and policy gradients.
Training deep networks
- Vanishing and Exploding Gradients - why depth stalled training, and the initialization and normalization that fixed it.
- Weight Initialization - the variance-preserving argument behind Xavier and He.
- Normalization: Batch and Layer - what each normalizes over, and why transformers reach for layer norm.
- Faster Optimizers and Learning Rate Scheduling - momentum through Adam, and scheduling the learning rate they share.
- Transfer Learning and Reusing Pretrained Layers - reuse a pretrained network instead of training from scratch.
- Regularization in Deep Learning - dropout, batch norm, weight decay, early stopping.
- Meta-Learning - learning to learn across tasks.
Reading notes
- Géron Catch-Up: Chapters 1, 4, 10 - the through-line across Hands-On ML chapters 1, 4, and 10.
Connects to
The fundamentals every note here builds on are in Machine Learning. The linear algebra a network runs on is in Linear Algebra Fundamentals, and the governance questions deep learning forced open are in AI Governance. The probabilistic foundations, maximum likelihood, cross-entropy, and variance, are in Statistics, and the systems that make training at scale possible, parallel and distributed computation across GPUs, are in Systems (see processes and threads and distributed consensus).