https://josedavidbaena.com Personal website and blog where I document thoughts, ideas, and interests in software engineering, web performance, and open source technologies. en-us josedab@gmail.com (José David Baena) josedab@gmail.com (José David Baena) Mon, 24 Nov 2025 00:00:00 GMT https://josedavidbaena.com/blog/nanochat/tokenizer-design-choices-bpe-vocabulary https://josedavidbaena.com/blog/nanochat/tokenizer-design-choices-bpe-vocabulary Your tokenizer decides what your model sees. 32K BPE vocab. 3.5 tokens per word. 10M tokens/sec in Rust. These choices compound at every training step. Mon, 24 Nov 2025 00:00:00 GMT josedab@gmail.com (José David Baena) nanochattokenizationbpetiktokenpractical-guide