In the previous post I complained at length about pipeline parallelism. This post pays off that debt: without PP, where do the parameters go, how do we split long contexts, and what about cross-node communication?

Before getting into it, let’s make some assumptions and set some constraints to shrink the design space. Otherwise, if we keep bolting on every feature, we end up with another Megatron, and at that point we might as well just use Megatron.

Constraints

• Support trillion-scale parameter counts (1T+ params)
• Support one-million-token context length
• NVL72 cluster
• No reliance on CPU offload for model state or activations.

Recently, the center of my work shifted from building inference systems to building post-training systems. Watching code that was clean and crisp in the inference engine turn into twists and turns inside the training framework, the furrow between my brows grew deeper. It runs slowly, it is inflexible to use, it is riddled with traps, and it is hard to maintain. This training framework was meant to be a lean system, yet it was drifting toward Megatron-level complexity.

My colleagues and I attempted refactors several times, but there was always some grime in the way of making the code elegant. Gradually, more and more clues pointed in the same direction: pipeline parallelism (PP).

Last week, our team summarized some recent progress we made on point-to-point communication for LLM systems and posted a paper on arXiv. We also open-sourced the code on GitHub.

We built an RDMA communication library based on the idea of Unordered Reliable Datagram (URD) semantics. It runs on both AWS EFA and NVIDIA ConnectX. We applied this library to three scenarios: KvCache transfer in disaggregated inference, model-parameter updates in RL post-training, and MoE communication. The MoE kernel actually runs slightly faster than DeepEP on ConnectX-7 during decode, and on EFA we achieved the first actually-usable performance as well.

In this post, I want to share the backstory — the motivation, the design decisions, and some fun debugging moments along the way. If you want the full technical details, check the paper, source code, and linked blog posts at the end.

In the previous blog post, I walked through how we achieved cross-machine RL weight updates in just 2 seconds. This post is a quick follow-up with a few extra details:

1. For Kimi-K2 (1T params), with 256 GPUs in BF16 training and 128 GPUs in FP8 inference, weight updates take less than 1.3 seconds.
2. The pipeline for parameter updates has been tuned a bit more, adding two parallelizable steps: H2D memcpy and a global communication barrier.
3. I ran a PyTorch Profiler trace to get a visual breakdown of the update pipeline and see exactly where time is being spent.
4. Added a few figures for easier intuition.

I just spent the past two weeks getting cross-machine parameter updates for Qwen3-235B (BF16 training, FP8 inference) to run in just 2 seconds (128 GPUs for training, 32 GPUs for inference). Instead of writing a “here’s the solution” kind of post, I want to share my exploration process and thoughts along the way. I’ll post a shorter, polished version on the company blog in a few days.