Remote Inference Server

This example uses the fsdp backend.

This example shows how to run GRPO training on GSM8K using a standalone inference engine instance behind an HTTP server.

Using remote inference servers is especially useful when you're iterating on just the generation code (ex: the agent loop) or on the trainer and want to avoid startup/teardown of the vLLM servers each time.

In this example, we will be using 4 GPUs for training (policy and ref models) and 4 GPUs for inference (inference engine) on a single node.

Prerequisites

First, make sure you are familiar with the standard setup process for running GRPO training. See Quick Start Guide for more details.

Starting the Remote Inference Server

We use the serve entrypoint (skyrl.train.entrypoints.serve) to bring up a standalone inference deployment: a vLLM server group fronted by the SkyRL router. It exposes all the standard endpoints in vLLM (including /v1/chat/completions and /v1/completions) plus the control-plane endpoints used for sleep/wake and weight updates in SkyRL.

To launch the deployment, run the following command (the full script is at examples/train/remote_inference_server/run_vllm_server.sh):

uv run --isolated --extra fsdp -m skyrl.train.entrypoints.serve \
    # model and tensor parallel size
    trainer.policy.model.path="Qwen/Qwen2.5-1.5B-Instruct" \
    generator.inference_engine.tensor_parallel_size=4 \

    # one engine, non-colocated (training lives on separate GPUs)
    generator.inference_engine.num_engines=1 \
    trainer.placement.colocate_all=false \

    # vllm performance related settings
    generator.inference_engine.gpu_memory_utilization=0.9 \
    generator.inference_engine.max_num_batched_tokens=8192 \
    generator.inference_engine.max_num_seqs=1024

On startup the serve entrypoint logs the proxy_url (data plane) and server_urls (control plane) of the deployment. You pass those to the training run below.

Starting Training

Now that we've started our remote inference deployment, we can start a training run!

To start training, we need to set up our training script. Point the trainer at the proxy_url and server_urls logged by the serve entrypoint. You can find a complete example in examples/train/remote_inference_server/run_remote.sh:

uv run --isolated --extra fsdp -m skyrl.train.entrypoints.main_base \
    # Setup for training against an external inference deployment
    generator.inference_engine.run_engines_locally=False \
    generator.inference_engine.external_proxy_url="http://127.0.0.1:8000" \
    generator.inference_engine.external_server_urls="['http://127.0.0.1:8001']" \

    # sampling parameters for generation
    generator.sampling_params.temperature=0.6 \
    generator.sampling_params.top_p=0.95 \

    # Data setup
    data.train_data="['$HOME/data/gsm8k/train.parquet']" \
    data.val_data="['$HOME/data/gsm8k/validation.parquet']" \

    # Policy model - make sure this is the same model used to launch the inference engine server
    trainer.policy.model.path="Qwen/Qwen2.5-1.5B-Instruct" \
    trainer.algorithm.advantage_estimator="grpo" \

    # Whether or not to colocate all models on the same set of GPUs - we set it to false here,
    # but you can colocate even with a standalone inference engine!
    trainer.placement.colocate_all=False \

    # Model placement arguments for policy and ref models - make sure that the total number of gpus 
    # used for training and inference is maximized
    trainer.placement.policy_num_gpus_per_node=4 \
    trainer.placement.ref_num_gpus_per_node=4 \

    # Training batch size and mini/micro batch sizes for logprobs + training passes
    trainer.train_batch_size=64 \
    trainer.policy_mini_batch_size=64 \
    trainer.micro_forward_batch_size_per_gpu=20 \
    trainer.micro_train_batch_size_per_gpu=20 \

    # Evaluation
    trainer.eval_batch_size=1024 \
    trainer.eval_before_train=true \
    trainer.eval_interval=5 \

    ... # Other parameters (see `examples/train/remote_inference_server/run_remote.sh` for more)

Launching Your Training Run

You're done setting up! Now let's get our training run started!

export WANDB_API_KEY=your_wandb_api_key
bash examples/train/remote_inference_server/run_remote.sh

What's Next?

Now that you've set up training with a remote inference server, you might want to explore ways of speeding up training:

Async Training: Asynchronous off-by-one training in < 100 lines of code!