Deep Learning using R

Session 4 – 02:00:00 – Convolutional Neural Networks
This session introduces convolutional neural networks (CNNs), the dominant architecture for image data and spatial data more generally. We begin by explaining the limitations of MLPs for images: treating an image as a flat vector loses spatial structure, and fully-connected layers require prohibitive numbers of parameters for high-resolution images. Convolutional layers are introduced as a solution: local receptive fields, parameter sharing through filters/kernels, and translation invariance that allows learning features independent of position. The convolution operation is explained in detail: sliding filters across the input, computing dot products to create feature maps, and stacking multiple filters to detect different patterns. Pooling layers are covered for dimensionality reduction and local invariance: max pooling taking the maximum value in each region, average pooling, and stride as an alternative. CNN architectures are discussed: stacking convolutional layers to build hierarchical representations, increasing the number of filters in deeper layers, and typical patterns (conv-relu-pool). Implementation in torch is covered using nn_conv2d (2D convolution), nn_max_pool2d (pooling), and nn_batch_norm2d (batch normalization for stable training). We apply CNNs to image classification tasks using real-world image datasets, demonstrating how CNNs exploit spatial structure for superior performance. Visualizing learned features and filters provides intuition about what CNNs learn in early vs deep layers. Applications beyond computer vision are discussed, including any data with spatial or local structure.

Session 5 – 02:00:00 – Introduction to Language Models and Transformers
This session introduces language models and the transformer architecture that powers modern large language models. We begin with the language modeling task: predicting the next word or token given previous context, and how this simple objective enables learning rich representations of language. The historical progression from recurrent neural networks (RNNs) to transformers is sketched, motivating the attention mechanism as a more effective way to model long-range dependencies. Tokenization methods are covered: character-level (splitting text into characters), byte-pair encoding (BPE learning subword units from data), and WordPiece tokenization used by modern LLMs. Embeddings are introduced: representing discrete tokens as continuous vectors that capture semantic relationships. The self-attention mechanism is explained in detail: the query-key-value formulation, computing attention scores between all pairs of tokens, and using attention weights to aggregate information. Multi-head attention extends this by learning multiple attention patterns in parallel. Positional encodings are covered as the method for incorporating word order into the model. Causal masking is explained for autoregressive generation: preventing the model from attending to future tokens during training. Transformer blocks combine attention with position-wise feedforward networks. The GPT architecture is introduced as a decoder-only transformer using causal masking for unidirectional language modeling. This session provides the conceptual foundation for implementing transformers in the next session.

Session 6 – 02:00:00 – Implementing and Using GPT Models
This session combines from-scratch implementation with practical approaches to working with transformer models in R. We begin by implementing a minimal GPT from scratch in torch, walking through each component: token embeddings, positional encodings, masked self-attention, multi-head attention, feedforward layers, and stacking transformer blocks. Training a character-level language model on a small text corpus (such as Shakespeare) demonstrates the complete pipeline. Text generation is covered in detail: autoregressive sampling one token at a time, temperature scaling to control randomness, top-k sampling (restricting to k most likely tokens), and top-p/nucleus sampling (restricting to cumulative probability p). Having understood transformers from first principles, we discuss practical approaches to using pre-trained models in R, including options for bridging to Python libraries when needed using reticulate. We discuss the landscape of large language models: GPT-4, Claude, LLaMA, and the growing open-source ecosystem. Scale effects and emergent capabilities are explained: how larger models develop abilities not present in smaller models. Current applications are surveyed: text generation, classification, question answering, summarization, and embeddings for semantic search. The session concludes with understanding the capabilities and limitations of LLMs, including hallucination, reasoning abilities, and appropriate use cases. Participants leave with both the foundational understanding from building transformers from scratch and knowledge of how to integrate deep learning into R-based workflows.