Kimi K2.7

Supported on main with the Megatron backend. This recipe trains Kimi-K2.7-Code, a 1T-parameter DeepSeek-V3-style MoE (61 layers, 384 routed experts, MLA), with DAPO-style GRPO, LoRA and fake-INT4 QAT, at the full 262,144-token context on 2x8 B300.

K2.5 and K2.6 share K2.7's architecture (KimiK25ForConditionalGeneration), so the same script should work, but we've only run it end to end on K2.7. It only works with the INT4 QAT checkpoints. Training is text-only and full fine-tuning is not supported.

How it works

• vLLM serves the INT4 release as-is (compressed-tensors W4A16, ~595 GB), one engine per node. The LoRA adapter is hot-loaded from disk after every step (merge_lora=false). The INT4 base weights never change.
• The trainer holds BF16 masters dequantized from the INT4 release. LoRA goes on linear_proj and linear_fc1/linear_fc2 (dense MLP, shared expert and all routed experts, which are ~96% of the parameters).
• Fake-INT4 QAT (trainer.policy.model.fake_int4_qat) quantizes the frozen expert GEMMs onto the same INT4 grid in the forward pass. For Kimi's QAT checkpoints this makes the trainer's experts bit-exact with vLLM's. TIS corrects the remaining mismatch from LoRA.
• Text-only (language_model_only=true on trainer and vLLM). The vision tower stays frozen in vLLM and is never built in Megatron.
• Expert LoRA rank is scaled down (normalize_moe_lora=true): routed experts get rank rank // moe_router_topk, so the adapter stays small enough to export and hot-load every step.

Setup

Every node needs the repo, the venv, the HF cache and the BF16 masters at the same paths.

1. Install the megatron extra. The lockfile is CUDA 13 (torch 2.13, Transformer Engine 2.16, all CUDA libs from pip), which B300 needs, with driver R580+.

   uv sync --extra megatron

2. Dequantize the INT4 release to BF16 masters on every node, or on a shared filesystem (~595 GB in, ~2.1 TB out). The script checks that re-quantizing reproduces the release bit-for-bit.

   uv run --isolated examples/train/megatron/dequantize_compressed_tensors_int4.py \
       --input <path-to-Kimi-K2.7-Code-snapshot> \
       --output /data/skyrl/models/Kimi-K2.7-Code-BF16

   For RTN-quantized checkpoints (scale_divisor=7.5), use the original BF16 release as masters instead. The script will fail if you try to dequantize them.

3. Prepare the data on the head node: bash examples/train/algorithms/dapo/prepare_dapo_data.sh

4. Start Ray on both nodes. Export the env block at the bottom of the run script (LD_LIBRARY_PATH, CUDNN_PATH, SKYRL_LD_LIBRARY_PATH_EXPORT=1, UV_PROJECT_ENVIRONMENT) and your NCCL settings (NCCL_SOCKET_IFNAME, NCCL_IB_HCA, GLOO_SOCKET_IFNAME) in the shell that runs ray start on every node. Ray workers inherit the raylet's env, and only LD_LIBRARY_PATH and SKYRL_* / UV_* / HF_* are forwarded.

   export RAY_RUNTIME_ENV_HOOK=ray._private.runtime_env.uv_runtime_env_hook.hook
   # head
   uv run --no-sync --extra megatron ray start --head --port=6379 --num-gpus=8
   # worker
   uv run --no-sync --extra megatron ray start --address=<head-ip>:6379 --num-gpus=8

Running

Run on the head node:

bash examples/train/megatron/run_megatron_dapo_kimi_k2.7_code_lora_int4_qat.sh

Key settings (2 nodes x 8 GPUs, colocated):

# vLLM: one engine per node serving the INT4 release, TP=8
NUM_INFERENCE_ENGINES=2
INFERENCE_ENGINE_TENSOR_PARALLEL_SIZE=8

# Megatron: TP=2 x CP=8 (DP=1), experts sharded EP=16 (24 experts/GPU/layer,
# ~127 GB of frozen BF16 masters + ~12 GB non-expert per GPU)
MEGATRON_TP=2  MEGATRON_CP=8  MEGATRON_EP=16  MEGATRON_ETP=1  MEGATRON_PP=1

# Full 262,144-token context
MAX_PROMPT_LENGTH=$((1024 * 8))
MAX_RESPONSE_LENGTH=$((262144 - MAX_PROMPT_LENGTH))

# INT4-served actor: trainer loads BF16 masters, fake-quantizes experts
trainer.policy.model.path="moonshotai/Kimi-K2.7-Code"
trainer.policy.model.fake_int4_qat.enabled=true
trainer.policy.model.fake_int4_qat.scale_divisor=7.0   # Kimi QAT convention
trainer.policy.model.fake_int4_qat.q_min=-7
trainer.policy.model.fake_int4_qat.bf16_base_path=/data/skyrl/models/Kimi-K2.7-Code-BF16
trainer.policy.megatron_config.lora_config.merge_lora=false
trainer.policy.megatron_config.lora_config.normalize_moe_lora=true
trainer.policy.language_model_only=true
generator.inference_engine.language_model_only=true
trainer.flash_attn=false        # MLA + sample packing needs TE cuDNN fused attention

To smoke-test a fresh cluster, shrink the run:

LOGGER=console MAX_RESPONSE_LENGTH=2048 TRAIN_BATCH_SIZE=8 MINI_BATCH_SIZE=8 \
N_SAMPLES_PER_PROMPT=4 \
bash examples/train/megatron/run_megatron_dapo_kimi_k2.7_code_lora_int4_qat.sh \
    trainer.max_training_steps=2

QAT_MODE=off keeps the INT4 engine but turns off fake-quant and TIS, to show the uncorrected mismatch (same toggle as the Qwen3.6 INT4 script).

Notes

• Adapter sync: with merge_lora=false, adapters are written once per node, so no shared filesystem is needed.
• Offload: the trainer's frozen BF16 masters (~1.1 TB per node) are offloaded while vLLM generates, to mmaps under SKYRL_FROZEN_OFFLOAD_DIR (default /data/skyrl/frozen-offload). Point it at a fast local disk, or set it to 0 to use pinned host RAM.
• cuDNN on B300: if the host has a system CUDA/cuDNN, the venv's pip libraries must come first on LD_LIBRARY_PATH and CUDNN_PATH must point at the pip cuDNN (the script does both). Otherwise TE fails with CUDNN_STATUS_SUBLIBRARY_LOADING_FAILED. A system cuDNN 9.x that differs from the pip one can still cause CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH. Remove it, or install a matching pip nvidia-cudnn-cu13.
• Startup time: loading 595 GB of INT4 weights plus CUDA graph capture can exceed the default 600s health check on a cold cache. The script raises SKYRL_WAIT_UNTIL_INFERENCE_SERVER_HEALTHY_TIMEOUT_S and VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS.
• MLA prefill: the script sets attention_config.mla_prefill_backend=FLASHINFER. The default FA4 prefill doesn't work with the locked nvidia-cutlass-dsl.
• LoRA targets skip the MLA q/kv projections: mcore names them differently, and vLLM's MLA weight absorption doesn't apply adapters to kv_b_proj.
• Checkpoints save only the LoRA adapters and optimizer state. The base is reloaded from bf16_base_path on resume. Exported adapters point at the INT4 release, so you can serve them against the public checkpoint.