How Does Tokenization Work?

Tokenization splits text into smaller units called tokens (like words or subwords) so that models can process and learn from language efficiently.