But what is cross-entropy? | Compression is Intelligence Part 2

This is Part 2 of the Compression is Intelligence series from 3Blue1Brown, and it runs just under 34 minutes. The description states its subject in one line: where the loss function for training LLMs comes from. Part 1 is linked in the description, and since the second chapter is a recap, you can start here, though Part 1 gives fuller context.

The video opens at 0:00 with language trees and zipping, an idea it comes back to later. At 3:02 it recaps optimal codes from the first part. Cross-entropy itself is defined at 5:20, and intuition and examples follow from 8:26.

At 12:59 the video applies cross-entropy to the language trees from the opening. The section on pre-training LLMs starts at 14:55 and runs close to six minutes. From 20:38 the question is what makes this loss function the best choice. For background on Lagrange multipliers, the description links lessons from the author's Khan Academy days, and the credits list Lagrange multiplier scenes among the animations.

Distillation is the topic at 26:13. A segment on 3b1b Talent, which the description links as job opportunities for this audience, runs from 30:12. The final chapter, from 31:35, covers KL divergence.

The animations are made mostly with manim, a custom Python library. The credits also name a NanoGPT animation and a 3D black-box model.