José David Baena – Distributed Systems Engineer https://josedavidbaena.com Production notes and source-backed analysis on distributed systems, messaging infrastructure, open-source internals, and model engineering. en-us josedab@gmail.com (José David Baena) josedab@gmail.com (José David Baena) Wed, 12 Aug 2026 00:00:00 GMT https://josedavidbaena.com/blog/distilled-engineering/dark-knowledge-soft-targets Dark Knowledge: What a Teacher's Wrong Answers Are Actually Worth https://josedavidbaena.com/blog/distilled-engineering/dark-knowledge-soft-targets Learn what soft targets preserve beyond hard labels, how temperature and KL direction change the signal, and when top-k logit storage loses too much. Wed, 12 Aug 2026 00:00:00 GMT josedab@gmail.com (José David Baena) model-distillationknowledge-distillationllmmachine-learningai-engineering https://josedavidbaena.com/blog/kimi-k3/02-architecture-kda-attnres-experts Inside Kimi K3: How KDA, AttnRes, and 896 Experts Work https://josedavidbaena.com/blog/kimi-k3/02-architecture-kda-attnres-experts Trace Kimi K3's 69 KDA layers, 24 gated MLA layers, Attention Residuals, and 896 experts without overstating what released artifacts prove. Tue, 28 Jul 2026 00:00:00 GMT josedab@gmail.com (José David Baena) kimi-k3architectureattentionmixture-of-expertsmachine-learning https://josedavidbaena.com/blog/nanochat/building-chatgpt-for-100-dollars Build Your Own ChatGPT for $100 https://josedavidbaena.com/blog/nanochat/building-chatgpt-for-100-dollars Train a complete ChatGPT-like system from scratch for $100: tokenizer, pretraining, SFT, RL, deployment—the full LLM pipeline for the price of dinner. Tue, 14 Oct 2025 00:00:00 GMT josedab@gmail.com (José David Baena) machine-learningaitutorialllmdeep-learningpytorch https://josedavidbaena.com/blog/tiny-language-models/tiny-llm-case-studies-production Tiny LLM Deployment Patterns: Architecture Blueprints from Published Benchmarks https://josedavidbaena.com/blog/tiny-language-models/tiny-llm-case-studies-production Deployment patterns for tiny LLMs in healthcare, legal, manufacturing, and edge—grounded in published benchmarks from Microsoft, Apple, and MLPerf. Mon, 13 Oct 2025 00:00:00 GMT josedab@gmail.com (José David Baena) machine-learningdeployment-patternsproductionbenchmarksarchitecture https://josedavidbaena.com/blog/tiny-language-models/tiny-llm-edge-deployment-guide Edge Device Deployment: Running Tiny LLMs on Raspberry Pi, Mobile, and IoT https://josedavidbaena.com/blog/tiny-language-models/tiny-llm-edge-deployment-guide 2 tokens/sec naive. 28 tokens/sec optimized. Same Raspberry Pi 4. INT4 quantization, llama.cpp, and ARM NEON turn cloud models into edge deployments. Sat, 04 Oct 2025 00:00:00 GMT josedab@gmail.com (José David Baena) machine-learningedge-computingdeploymentraspberry-pimobileoptimization https://josedavidbaena.com/blog/tiny-language-models/tiny-llm-fine-tuning-domain-adaptation Fine-Tuning Tiny Models: LoRA, QLoRA, and Domain Adaptation Strategies https://josedavidbaena.com/blog/tiny-language-models/tiny-llm-fine-tuning-domain-adaptation Domain accuracy 8% to 68%. Cost $23 instead of $15,000. LoRA trains 0.1% of params. QLoRA adds 4-bit quantization. Six hours: generalist to expert. Wed, 01 Oct 2025 00:00:00 GMT josedab@gmail.com (José David Baena) machine-learningfine-tuningloraqloradomain-adaptationtutorial https://josedavidbaena.com/blog/tiny-language-models/quantization-aware-training-tutorial Quantization-Aware Training: INT8/INT4 Models That Maintain Quality https://josedavidbaena.com/blog/tiny-language-models/quantization-aware-training-tutorial Naive INT8 drops MMLU 24%. QAT drops it 2.4%. Train with fake quantization so the model learns resilience. GPTQ, mixed-precision, deployment recipes. Sat, 27 Sep 2025 00:00:00 GMT josedab@gmail.com (José David Baena) machine-learningquantizationtrainingoptimizationtutorial https://josedavidbaena.com/blog/tiny-language-models/knowledge-distillation-complete-tutorial Knowledge Distillation: How to Train a 1.5B Model That Matches Your 7B https://josedavidbaena.com/blog/tiny-language-models/knowledge-distillation-complete-tutorial Llama-7B to 1.5B. 90% quality at 1/5 the size. 3× inference speed. From temperature-scaled softmax to production deployment—with working PyTorch code. Tue, 23 Sep 2025 00:00:00 GMT josedab@gmail.com (José David Baena) machine-learningdistillationtrainingtutorialpytorch https://josedavidbaena.com/blog/tiny-language-models/tiny-llm-architecture-comparison Tiny LLM Architecture Comparison: TinyLlama vs Phi-2 vs Gemma vs MobileLLM https://josedavidbaena.com/blog/tiny-language-models/tiny-llm-architecture-comparison Seven tiny models, one decision. Phi-2 wins on reasoning (56.7% MMLU). MobileLLM on speed (120 tok/s). Qwen on multilingual. Match constraints to model. Sat, 20 Sep 2025 00:00:00 GMT josedab@gmail.com (José David Baena) machine-learningllmarchitecturebenchmarkscomparison https://josedavidbaena.com/blog/tiny-language-models/efficient-attention-mechanisms-tiny-models Efficient Attention Mechanisms for Tiny Language Models https://josedavidbaena.com/blog/tiny-language-models/efficient-attention-mechanisms-tiny-models Attention burns 50% of inference time and 75% of memory. MQA shrinks KV cache 4×. Flash Attention fuses kernels for 2–4× speedup. GQA splits the gap. Fri, 19 Sep 2025 00:00:00 GMT josedab@gmail.com (José David Baena) machine-learningattentiontransformersoptimizationtutorial https://josedavidbaena.com/blog/tiny-language-models/model-compression-techniques-comprehensive-guide Model Compression: 14GB to 450MB While Keeping 90% Quality https://josedavidbaena.com/blog/tiny-language-models/model-compression-techniques-comprehensive-guide 14GB to 450MB. 12 to 60 tokens/sec. 90% quality retained. Four techniques—distillation, quantization, pruning, LoRA—that compose. PyTorch code included. Mon, 15 Sep 2025 00:00:00 GMT josedab@gmail.com (José David Baena) machine-learningmodel-compressiondistillationquantizationpruningloratutorial https://josedavidbaena.com/blog/tiny-language-models/tiny-language-models-math-foundations Mathematical Foundations of Model Compression: Theory Behind Tiny LLMs https://josedavidbaena.com/blog/tiny-language-models/tiny-language-models-math-foundations Quantization cuts size 75% at 2% accuracy loss. Distillation transfers soft labels. Pruning exploits the Lottery Ticket Hypothesis. The math behind it. Wed, 10 Sep 2025 00:00:00 GMT josedab@gmail.com (José David Baena) machine-learningdeep-learningmodel-compressionquantizationdistillationmathematics https://josedavidbaena.com/blog/tiny-language-models/tiny-language-models-complete-guide Tiny Language Models: How 1.3B Parameters Can Beat 7B on Reasoning https://josedavidbaena.com/blog/tiny-language-models/tiny-language-models-complete-guide Microsoft Phi-1.5 (1.3B) matches models 5× larger on reasoning. TinyLlama hit ~1T tokens for 3 epochs. Data quality beats size—and that changes things. Fri, 05 Sep 2025 00:00:00 GMT josedab@gmail.com (José David Baena) machine-learningaitiny-llmefficient-modelsedge-computingmobile-ai