Build & Train a GLM-5.3-Flash Model From Scratch with Python

This freeCodeCamp.org upload, credited to the creator @vukrosic, builds a scaled-down language model patterned on GLM-5.3-Flash using Python. It runs about 44 minutes. The code, a slide deck and written reports sit in the glm-5.3-flash-from-scratch repository on GitHub, so you can follow along file by file.

The video starts by stating what gets built and spends a short section on the job of a modern AI researcher and on framing research questions. Tokenization comes next, with an argument for a small byte-level vocabulary. Embeddings and a high-level pass through the transformer follow, then an overview of the GLM-5.3 architecture and its specifications.

Most of the first half is a code walkthrough. It takes the model one part at a time: how tokens become embeddings, manifold constrained hyperconnections (which the description ties to DeepSeek-style residuals), the output projection with weight tying, and RMSNorm. Positional encodings get a RoPE versus NoPE comparison, and a sparse attention indexer is introduced alongside them. Linear attention, described as state-space memory, is set against sparse attention. Mixture of experts with shared experts closes the language side, and a vision section adds patch embeddings and 2D RoPE so the model can read images.

Training takes the second half. Pre-training covers the loss, AdamW, and experiments on data diversity, interleaving and curricula. Post-training uses reinforcement learning: designing reward functions, Group Relative Policy Optimization (GRPO), parameter-efficient updates that freeze layers, and reading results for task gains and regressions. Experiments on group size, temperature and seeds come last, followed by advice for people aiming at AI research work.

The repository README fills in the scale. Its model has 25,730,592 parameters over 12 layers, against 320B total parameters for the released GLM-5.3-Flash. It reads raw UTF-8 bytes as a 260-token vocabulary, routes each token to the top 2 of 8 experts, and has a 192-token context. The README calls the project an educational implementation rather than a reproduction, lists NVIDIA CUDA, Apple Silicon MPS and CPU as supported, and describes its RL step as RLOO policy-gradient post-training with an executable reward. On its synthetic confirmation set, greedy pass@1 for the three trained families went from 0/24 to 16/24, and the README reports that the untrained square and list_sum families regressed.