SkyRL
Examples

LoRA Training in SkyRL

LoRA training is supported with the fsdp and megatron backends.

This guide demonstrates how to use Low-Rank Adaptation (LoRA) for efficient reinforcement learning training in SkyRL. LoRA allows you to train large language models with significantly reduced memory footprint and computational requirements by only updating a small set of parameters.

What is LoRA?

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique that freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture.

Configuration

LoRA can be configured for both policy and critic models in the training configuration. Here's how to set it up:

trainer:
  policy:
    model:
      path: "Qwen/Qwen2.5-1.5B-Instruct"
      lora:
        rank: 32              # LoRA rank (higher = more parameters)
        alpha: 32             # LoRA scaling parameter
        dropout: 0            # LoRA dropout rate
        lora_sync_path: "/tmp/skyrl_lora_sync"  # Path for adapter sync
        target_modules: "all-linear"  # Apply to all linear layers OR
        # specify specific modules as a list
        exclude_modules: null  # Modules to exclude from LoRA
        train_attn: true      # Apply LoRA to the attention projections
        train_mlp: true       # Apply LoRA to the MLP projections
  critic:
    model:
      path: "Qwen/Qwen2.5-1.5B-Instruct"
      lora:
        rank: 32
        alpha: 32
        dropout: 0
        target_modules: "all-linear"
        exclude_modules: null

Key Parameters

  • rank: The rank of the low-rank decomposition. Higher values mean more trainable parameters but also more memory usage. Common values are 8, 16, 32, or 64.

  • alpha: The scaling parameter for LoRA. Often set equal to rank, but can be tuned independently.

  • dropout: Dropout rate applied to LoRA layers. Helps with regularization.

  • lora_sync_path: Directory path where LoRA adapters are saved and synchronized between training and inference engines.

  • target_modules: Specifies which modules to apply LoRA to. By default, LoRA is applied to all linear layers in the model - you can customize which modules to target by providing a list of specific module names.

  • exclude_modules: Specifies which modules to exclude from LoRA application. By default, no modules are excluded.

  • train_attn / train_mlp: Whether to apply LoRA to the attention and MLP projections. Both default to true. Setting one to false narrows target_modules: "all-linear" to the other group; at least one must stay true. Narrowing:

    • requires target_modules: "all-linear" — with an explicit list, select the modules there;
    • on Megatron, keeps the attention or MLP half of Megatron's "all-linear" module list; the MLP half includes routed MoE experts. It raises for vision-language models that build a vision tower, and for attention-only on multi-latent attention models (DeepSeek-V3, Kimi K2, GLM-4.7/5.3-Flash);
    • on FSDP, needs Llama/Qwen-style module names (self_attn, mlp), is not supported for vision-language models, and does not reach routed MoE experts, which transformers stores as fused nn.Parameter tensors; shared experts and dense MLP layers are covered.

    See Tinker limitations for the module lists.

Running LoRA Training

For text-only training of a vision-language model (for example, Qwen3.5) with FSDP, you can set trainer.policy.language_model_only=true to load only its language model. LoRA export restores the enclosing language_model namespace so vLLM can match the adapter weights to its inference modules. This follows the same namespace convention as full-weight synchronization; support for other VLM architectures depends on their inference-model naming.

Here's a complete example of running LoRA training on GSM8K:

Dataset Preparation

First, prepare your dataset:

uv run examples/train/gsm8k/gsm8k_dataset.py --output_dir $HOME/data/gsm8k

Training Script

Create a training script with LoRA configuration:

#!/bin/bash
set -x

DATA_DIR="$HOME/data/gsm8k"
NUM_GPUS=4
LOGGER="wandb"  # change to "console" to print to stdout
INFERENCE_BACKEND="vllm"

CUDA_VISIBLE_DEVICES=0,1,2,3 uv run --isolated --extra fsdp -m skyrl.train.entrypoints.main_base \
  data.train_data="['$DATA_DIR/train.parquet']" \
  data.val_data="['$DATA_DIR/validation.parquet']" \
  trainer.algorithm.advantage_estimator="grpo" \
  trainer.policy.model.path="Qwen/Qwen2.5-0.5B-Instruct" \
  trainer.policy.model.lora.rank=32 \
  trainer.policy.model.lora.alpha=32 \
  trainer.policy.model.lora.lora_sync_path="/tmp/skyrl_lora_sync" \
  trainer.strategy=fsdp \
  trainer.placement.colocate_all=true \
  trainer.placement.policy_num_gpus_per_node=$NUM_GPUS \
  trainer.placement.ref_num_gpus_per_node=$NUM_GPUS \
  generator.inference_engine.num_engines=$NUM_GPUS \
  generator.inference_engine.tensor_parallel_size=1 \
  trainer.train_batch_size=128 \
  trainer.policy_mini_batch_size=128 \
  trainer.micro_forward_batch_size_per_gpu=64 \
  trainer.micro_train_batch_size_per_gpu=64 \
  trainer.ckpt_interval=10 \
  generator.sampling_params.max_generate_length=1024 \
  trainer.policy.optimizer_config.lr=3.0e-5 \
  trainer.algorithm.use_kl_loss=true \
  generator.inference_engine.backend=$INFERENCE_BACKEND \
  generator.batched=true \
  environment.env_class=gsm8k \
  generator.n_samples_per_prompt=4 \
  trainer.logger="$LOGGER" \
  trainer.project_name="gsm8k_0.5b_lora" \
  trainer.run_name="gsm8k_0.5b_lora_test" \
  trainer.ckpt_path="$HOME/ckpts/gsm8k_0.5b_lora_ckpt"

Launch Training

Set up your WandB API key and run the training:

export WANDB_API_KEY=your_wandb_api_key
bash examples/train/lora/run_qwen2_5_0.5b_gsm8k_grpo_lora.sh

Configuration Tips

We recommend looking at the blog LoRA Without Regret for more details and practical guidance on setting LoRA hyperparameters. Here are the key takeaways:

  1. Learning rate: Use roughly 10× higher learning rate for LoRA than for full fine-tuning; LoRA is more tolerant of large learning rates, especially for shorter runs.

  2. Rank: Choose a rank large enough to match dataset complexity (e.g., 32–64 for most RL fine-tuning tasks). LoRA performs best when not capacity-limited.

  3. Layer coverage: Apply LoRA to all layers, particularly MLP/MoE layers — attention-only LoRA tends to underperform.

  4. Memory (FSDP): With LoRA the base model is never updated, so it is stored in bf16 rather than as fp32 master weights (set trainer.bf16=false to keep it in fp32). On a single node, trainer.policy.fsdp_config.fsdp_size=1 keeps a full copy of the model on each GPU (no weight sharding, gradient all-reduce only, like DDP), which avoids re-gathering the base weights on every forward and backward pass.

Initializing from a PEFT Adapter (Resuming a Tinker Run)

trainer.policy.model.lora.init_adapter_path (Megatron) starts training from the weights of an existing HuggingFace PEFT LoRA adapter (adapter_config.json + adapter_model.safetensors, local or cloud path). The policy's adapters are built to match the adapter file, so the initial policy computes exactly what the adapter computed; the optimizer state and LR schedule start fresh. Use it to continue a run that was trained with Tinker, to pick up a SkyRL lora_adapter checkpoint export in a new run, or to start RL from an adapter trained with peft.

From Tinker

Tinker only lets you download sampler checkpoints (the LoRA weights; the optimizer state stays on their side), and the download is in Tinker's internal key layout. Convert it to a standard PEFT adapter with the tinker-cookbook, then point SkyRL at the result:

tinker checkpoint download tinker://<run-id>/sampler_weights/<name> --output ./tinker_ckpt
uv run --with tinker-cookbook python -c "
from tinker_cookbook import weights
weights.build_lora_adapter(base_model='Qwen/Qwen3.6-27B', adapter_path='./tinker_ckpt/<dir>', output_path='./peft_adapter')
"

bash examples/train/megatron/run_megatron_dapo_qwen3.6_27b_lora.sh \
  trainer.policy.model.lora.rank=32 \
  trainer.policy.model.lora.alpha=32 \
  trainer.policy.model.lora.init_adapter_path=./peft_adapter

lora.rank and lora.alpha must equal the adapter's r and lora_alpha (Tinker uses lora_alpha=32); SkyRL fails fast with the adapter's values otherwise. With the default target_modules="all-linear" the adapted modules are exactly those in the adapter, which for Tinker includes lm_head (train_unembed); an explicit target_modules list must cover every adapted module, and the extra ones start as fresh zero adapters.

How the layouts are reconciled:

  • Separate vs. fused projections. Tinker and peft train one adapter per HF projection (q_proj, k_proj, v_proj, each with its own lora_A), while Megatron gives the fused linear_qkv a single adapter. SkyRL builds the fused module at the sum of the constituent ranks with a block-diagonal lora_B (rank 96 for three rank-32 adapters), which computes the same function as the three separate adapters. The same applies to GatedDeltaNet in_proj (in_proj_qkv/z/b/a) and linear_fc1 (gate_proj/up_proj). The trainable parameterization after import is a superset of the original (the off-diagonal blocks of the widened lora_B can move), so the training trajectory is not bit-identical to continuing in Tinker, but the starting point is.
  • Tinker's GatedDeltaNet keys. The converter leaves linear_attn.in_proj_q/k/v as three modules that do not exist in HF's Qwen3.5/3.6 (in_proj_qkv is one linear). SkyRL folds them into one in_proj_qkv adapter of rank 3r, recorded through rank_pattern / alpha_pattern so the scale is unchanged.
  • PEFT patterns. rank_pattern, alpha_pattern and use_rslora are honored per module. For adapter-only weight sync (merge_lora=false) the per-module scales are folded into lora_B before publishing, since vLLM applies a single lora_alpha / r; set trainer.policy.model.lora.max_lora_rank to a vLLM-accepted value at least as large as the widest module (Megatron fused projections run at a multiple of rank). Megatron's default merge_lora=true needs no such setting.

Resuming with trainer.resume_mode from a SkyRL checkpoint saved by such a run keeps init_adapter_path set: the adapter file defines the module ranks, and the checkpoint then overrides the adapter weights, optimizer and schedule.

Not supported yet: the FSDP backend, MoE expert adapters (dense models and attention/MLP adapters only), DoRA, modules_to_save, lora_type=canonical_lora, and trainer.fused_lm_head_logprob=true together with an adapter on lm_head (the fused head reads the base output_layer.weight directly).

tests/backends/skyrl_train/gpu/gpu_ci/megatron/test_lora_init_adapter.py checks the import: a policy initialized from a Tinker-shaped adapter (separate q/k/v, in_proj_q/k/v, lm_head) scores fixed sequences within bf16 tolerance of HuggingFace + peft on the same adapter.

Current Limitations

SkyRL's LoRA implementation has the following current limitations:

  1. Disk-based synchronization: LoRA adapters are saved to disk and reloaded rather than synchronized in-memory.

  2. Single adapter per model: Currently, only one LoRA adapter can be active per model at a time.

These limitations are being addressed in future releases, with plans for in-memory synchronization and improved adapter management.

On this page