LoRA Training in SkyRL
LoRA training is supported with the fsdp and megatron backends.
This guide demonstrates how to use Low-Rank Adaptation (LoRA) for efficient reinforcement learning training in SkyRL. LoRA allows you to train large language models with significantly reduced memory footprint and computational requirements by only updating a small set of parameters.
What is LoRA?
LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique that freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture.
Configuration
LoRA can be configured for both policy and critic models in the training configuration. Here's how to set it up:
trainer:
policy:
model:
path: "Qwen/Qwen2.5-1.5B-Instruct"
lora:
rank: 32 # LoRA rank (higher = more parameters)
alpha: 32 # LoRA scaling parameter
dropout: 0 # LoRA dropout rate
lora_sync_path: "/tmp/skyrl_lora_sync" # Path for adapter sync
target_modules: "all-linear" # Apply to all linear layers OR
# specify specific modules as a list
exclude_modules: null # Modules to exclude from LoRA
train_attn: true # Apply LoRA to the attention projections
train_mlp: true # Apply LoRA to the MLP projections
critic:
model:
path: "Qwen/Qwen2.5-1.5B-Instruct"
lora:
rank: 32
alpha: 32
dropout: 0
target_modules: "all-linear"
exclude_modules: nullKey Parameters
-
rank: The rank of the low-rank decomposition. Higher values mean more trainable parameters but also more memory usage. Common values are 8, 16, 32, or 64. -
alpha: The scaling parameter for LoRA. Often set equal to rank, but can be tuned independently. -
dropout: Dropout rate applied to LoRA layers. Helps with regularization. -
lora_sync_path: Directory path where LoRA adapters are saved and synchronized between training and inference engines. -
target_modules: Specifies which modules to apply LoRA to. By default, LoRA is applied to all linear layers in the model - you can customize which modules to target by providing a list of specific module names. -
exclude_modules: Specifies which modules to exclude from LoRA application. By default, no modules are excluded. -
train_attn/train_mlp: Whether to apply LoRA to the attention and MLP projections. Both default totrue. Setting one tofalsenarrowstarget_modules: "all-linear"to the other group; at least one must staytrue. Narrowing:- requires
target_modules: "all-linear"— with an explicit list, select the modules there; - on Megatron, keeps the attention or MLP half of Megatron's
"all-linear"module list; the MLP half includes routed MoE experts. It raises for vision-language models that build a vision tower, and for attention-only on multi-latent attention models (DeepSeek-V3, Kimi K2, GLM-4.7/5.3-Flash); - on FSDP, needs Llama/Qwen-style module names (
self_attn,mlp), is not supported for vision-language models, and does not reach routed MoE experts, whichtransformersstores as fusednn.Parametertensors; shared experts and dense MLP layers are covered.
See Tinker limitations for the module lists.
- requires
Running LoRA Training
For text-only training of a vision-language model (for example, Qwen3.5) with FSDP,
you can set trainer.policy.language_model_only=true to load only its language
model. LoRA export restores the enclosing language_model namespace so vLLM
can match the adapter weights to its inference modules.
This follows the same namespace convention as full-weight synchronization;
support for other VLM architectures depends on their inference-model naming.
Here's a complete example of running LoRA training on GSM8K:
Dataset Preparation
First, prepare your dataset:
uv run examples/train/gsm8k/gsm8k_dataset.py --output_dir $HOME/data/gsm8kTraining Script
Create a training script with LoRA configuration:
#!/bin/bash
set -x
DATA_DIR="$HOME/data/gsm8k"
NUM_GPUS=4
LOGGER="wandb" # change to "console" to print to stdout
INFERENCE_BACKEND="vllm"
CUDA_VISIBLE_DEVICES=0,1,2,3 uv run --isolated --extra fsdp -m skyrl.train.entrypoints.main_base \
data.train_data="['$DATA_DIR/train.parquet']" \
data.val_data="['$DATA_DIR/validation.parquet']" \
trainer.algorithm.advantage_estimator="grpo" \
trainer.policy.model.path="Qwen/Qwen2.5-0.5B-Instruct" \
trainer.policy.model.lora.rank=32 \
trainer.policy.model.lora.alpha=32 \
trainer.policy.model.lora.lora_sync_path="/tmp/skyrl_lora_sync" \
trainer.strategy=fsdp \
trainer.placement.colocate_all=true \
trainer.placement.policy_num_gpus_per_node=$NUM_GPUS \
trainer.placement.ref_num_gpus_per_node=$NUM_GPUS \
generator.inference_engine.num_engines=$NUM_GPUS \
generator.inference_engine.tensor_parallel_size=1 \
trainer.train_batch_size=128 \
trainer.policy_mini_batch_size=128 \
trainer.micro_forward_batch_size_per_gpu=64 \
trainer.micro_train_batch_size_per_gpu=64 \
trainer.ckpt_interval=10 \
generator.sampling_params.max_generate_length=1024 \
trainer.policy.optimizer_config.lr=3.0e-5 \
trainer.algorithm.use_kl_loss=true \
generator.inference_engine.backend=$INFERENCE_BACKEND \
generator.batched=true \
environment.env_class=gsm8k \
generator.n_samples_per_prompt=4 \
trainer.logger="$LOGGER" \
trainer.project_name="gsm8k_0.5b_lora" \
trainer.run_name="gsm8k_0.5b_lora_test" \
trainer.ckpt_path="$HOME/ckpts/gsm8k_0.5b_lora_ckpt"Launch Training
Set up your WandB API key and run the training:
export WANDB_API_KEY=your_wandb_api_key
bash examples/train/lora/run_qwen2_5_0.5b_gsm8k_grpo_lora.shConfiguration Tips
We recommend looking at the blog LoRA Without Regret for more details and practical guidance on setting LoRA hyperparameters. Here are the key takeaways:
-
Learning rate: Use roughly 10× higher learning rate for LoRA than for full fine-tuning; LoRA is more tolerant of large learning rates, especially for shorter runs.
-
Rank: Choose a rank large enough to match dataset complexity (e.g., 32–64 for most RL fine-tuning tasks). LoRA performs best when not capacity-limited.
-
Layer coverage: Apply LoRA to all layers, particularly MLP/MoE layers — attention-only LoRA tends to underperform.
-
Memory (FSDP): With LoRA the base model is never updated, so it is stored in bf16 rather than as fp32 master weights (set
trainer.bf16=falseto keep it in fp32). On a single node,trainer.policy.fsdp_config.fsdp_size=1keeps a full copy of the model on each GPU (no weight sharding, gradient all-reduce only, like DDP), which avoids re-gathering the base weights on every forward and backward pass.
Initializing from a PEFT Adapter (Resuming a Tinker Run)
trainer.policy.model.lora.init_adapter_path (Megatron) starts training from the weights of an existing HuggingFace PEFT
LoRA adapter (adapter_config.json + adapter_model.safetensors, local or cloud path). The policy's adapters
are built to match the adapter file, so the initial policy computes exactly what the adapter computed; the
optimizer state and LR schedule start fresh. Use it to continue a run that was trained with
Tinker, to pick up a SkyRL lora_adapter checkpoint export in a new
run, or to start RL from an adapter trained with peft.
From Tinker
Tinker only lets you download sampler checkpoints (the LoRA weights; the optimizer state stays on their side), and the download is in Tinker's internal key layout. Convert it to a standard PEFT adapter with the tinker-cookbook, then point SkyRL at the result:
tinker checkpoint download tinker://<run-id>/sampler_weights/<name> --output ./tinker_ckpt
uv run --with tinker-cookbook python -c "
from tinker_cookbook import weights
weights.build_lora_adapter(base_model='Qwen/Qwen3.6-27B', adapter_path='./tinker_ckpt/<dir>', output_path='./peft_adapter')
"
bash examples/train/megatron/run_megatron_dapo_qwen3.6_27b_lora.sh \
trainer.policy.model.lora.rank=32 \
trainer.policy.model.lora.alpha=32 \
trainer.policy.model.lora.init_adapter_path=./peft_adapterlora.rank and lora.alpha must equal the adapter's r and lora_alpha (Tinker uses lora_alpha=32);
SkyRL fails fast with the adapter's values otherwise. With the default target_modules="all-linear" the
adapted modules are exactly those in the adapter, which for Tinker includes lm_head (train_unembed); an
explicit target_modules list must cover every adapted module, and the extra ones start as fresh zero
adapters.
How the layouts are reconciled:
- Separate vs. fused projections. Tinker and
pefttrain one adapter per HF projection (q_proj,k_proj,v_proj, each with its ownlora_A), while Megatron gives the fusedlinear_qkva single adapter. SkyRL builds the fused module at the sum of the constituent ranks with a block-diagonallora_B(rank 96 for three rank-32 adapters), which computes the same function as the three separate adapters. The same applies to GatedDeltaNetin_proj(in_proj_qkv/z/b/a) andlinear_fc1(gate_proj/up_proj). The trainable parameterization after import is a superset of the original (the off-diagonal blocks of the widenedlora_Bcan move), so the training trajectory is not bit-identical to continuing in Tinker, but the starting point is. - Tinker's GatedDeltaNet keys. The converter leaves
linear_attn.in_proj_q/k/vas three modules that do not exist in HF's Qwen3.5/3.6 (in_proj_qkvis one linear). SkyRL folds them into onein_proj_qkvadapter of rank3r, recorded throughrank_pattern/alpha_patternso the scale is unchanged. - PEFT patterns.
rank_pattern,alpha_patternanduse_rsloraare honored per module. For adapter-only weight sync (merge_lora=false) the per-module scales are folded intolora_Bbefore publishing, since vLLM applies a singlelora_alpha / r; settrainer.policy.model.lora.max_lora_rankto a vLLM-accepted value at least as large as the widest module (Megatron fused projections run at a multiple ofrank). Megatron's defaultmerge_lora=trueneeds no such setting.
Resuming with trainer.resume_mode from a SkyRL checkpoint saved by such a run keeps init_adapter_path set:
the adapter file defines the module ranks, and the checkpoint then overrides the adapter weights, optimizer
and schedule.
Not supported yet: the FSDP backend, MoE expert adapters (dense models and attention/MLP adapters only), DoRA,
modules_to_save, lora_type=canonical_lora, and trainer.fused_lm_head_logprob=true together with an
adapter on lm_head (the fused head reads the base output_layer.weight directly).
tests/backends/skyrl_train/gpu/gpu_ci/megatron/test_lora_init_adapter.py checks the import: a policy initialized from a Tinker-shaped adapter (separate q/k/v, in_proj_q/k/v, lm_head)
scores fixed sequences within bf16 tolerance of HuggingFace + peft on the same adapter.
Current Limitations
SkyRL's LoRA implementation has the following current limitations:
-
Disk-based synchronization: LoRA adapters are saved to disk and reloaded rather than synchronized in-memory.
-
Single adapter per model: Currently, only one LoRA adapter can be active per model at a time.
These limitations are being addressed in future releases, with plans for in-memory synchronization and improved adapter management.