We ran a large-scale distillation attack on the Kimi K3 technical report by reading it in parallel at the Hugging Face Journal Club :)
https://t.co/C5H7Ox8Jty
Our main takeaway is that there doesn't appear to be any "secret sauce" behind frontier performance.
Instead, K3 is a demonstration of how many difficult algorithmic and infrastructure decisions have to work together (distillation, RL scheduling, environments, quantisation, MoE sharding etc) to push a model to the frontier.
Here’s what we learned:
• Frontier post-training increasingly looks like expert training followed by distillation. K3 trains specialists across three domains and three reasoning-effort levels, then distils the resulting nine experts back into a single checkpoint using multi-teacher OPD
• Reasoning effort is treated as a trainable capability. Token budgets are estimated from the SFT model, and a stage-wise curriculum anneals from long to short rollouts to produce low-, high- and max-effort experts.
• Partial rollouts keep expensive RL infrastructure busy. Completed trajectories can trigger immediate updates, while unfinished ones are carried into later iterations and reprioritised through a priority queue. Per-token corrections help control the resulting off-policy behaviour.
• The reward model is itself an agent. Rather than relying only on fixed or binary rewards, it generates task-specific rubrics and scores responses against them on the fly.
• The environment interface is designed for composition. A unified, white-box abstraction allows the same training system to operate across different agent harnesses like Codex, CC, Hermes instead of specialising around a single one.
• The systems work is as interesting as the algorithmic work. GPUs dynamically switch between training and inference, rollout concurrency responds to KV-cache pressure, and reference models can live on CPU until needed.
• Quantisation is part of training—not merely deployment. K3 uses QAT for weights and activations, while matching the trainer and inference server’s quantisation schemes to reduce train–inference mismatch.
• The chat template is novel and introduces an extensible token markup language is intended to accommodate new tools and modalities without repeatedly redesigning the template.
• Synthetic task generation is becoming increasingly autonomous. Agents traverse a knowledge graph and search the web to generate training tasks at scale, although the report leaves some details of their integration underspecified. (This part was very cool)