Applications submit inference requests. Controllers can also use workflow signals such as tool waits or service deadlines when available.
Observe → decide → actuate → verify
RIDEControl
An open control plane for LLM inference.
Decide which requests enter the engine, how many model replicas to run, and which inference state to keep. Deploy independent controllers and inspect their status and logs through one interface.
System boundary
Engines execute. The control plane decides.
RIDE is not another inference engine. It is the policy, coordination, and evidence layer that turns workload intent and live telemetry into bounded actions on an engine fleet.
Independent MODs choose actions. The shared RIDE SDK and runtime manage their lifecycle, status, and logs. The Ray executor runs components as Ray Jobs on an existing cluster.
vLLM-HUST, vLLM, SGLang, and compatible serving backends execute prefill, decode, cache, and communication paths.
The data plane remains independently measurable and replaceable.
How controllers work
Workload, topology, engine, model, queue, and resource signals.
Admission, routing, placement, budgets, state, quality, and recovery.
Apply decisions through versioned telemetry and control interfaces.
Correctness, latency, goodput, resource use, and fallback status.
Research decision areas
Workflow & SLO
Schedule DAG stages, admit work, allocate capacity, and protect fairness along the critical path.
ADMISSION · DEADLINE · FAIRNESSRouting & placement
Choose a backend, cache holder, model replica, adapter, or continuation target from live fleet state.
SELECT · PLACE · RECOVERState & KV
Predict reuse, materialize or recompute state, select tiers, and account for transfer and survival horizons.
REUSE · MATERIALIZE · TIERQuality & economics
Trade quality, latency, cost, safety, and service risk without hiding failures behind aggregate throughput.
QUALITY · COST · GUARDRAILRIDE MOD catalog
Controllers for inference serving
A MOD packages a control policy as a component. RIDE gives each component the same management interface so its policy can evolve independently.
Request Throttling Controller
Decides which agent requests to admit and which to queue. For multi-turn workloads with tool waits, it limits contention for capacity and gives active workflows a chance to reuse their inference state.
Lower batch completion time in one matched multi-turn replay (353.74 s → 270.80 s). The engine configuration was held fixed across the two runs. This result covers one replay; task accuracy and repeatability remain unverified.
Instance Controller
Studies when to add or remove inference replicas using live load and replica readiness. Evaluation compares latency and service capacity against fixed replicas under a comparable accelerator budget.
KV Controller
Studies which inference state to retain, move, or recompute. Policies use reuse and memory pressure signals, while the engine or KV backend carries out the requested operations.
Research portfolio
RIDE Lab also hosts independent research projects, each with its own owner and results.
Loading the portfolio…
Quickstart
Put a controller in front of a model
Start with an existing Ray cluster and a reachable model endpoint. This public demo is a standalone MOD that limits concurrent non-streaming requests. The graduated Request Throttling Controller above uses a different policy.
1. Install RIDE and connect to Ray
Run on your management host. Replace the example Ray Dashboard address with your cluster’s address. Install the demo on the management host. Eligible Ray workers also need the package, either preinstalled or supplied through Ray runtime_env. Leave the management process running.
git clone https://github.com/vLLM-HUST/ride.git
cd ride
python -m pip install '.[ray]'
python -m pip install 'git+https://github.com/vLLM-HUST/ride-controller-demo.git'
ride start --ray-address http://ray-head:8265
2. Start the MOD and inspect it
Open another terminal on the same management host. Set the backend to your model endpoint. The controller endpoint must point to the Ray node that will run it. On multi-node clusters, use a Ray resource constraint to select that node.
ride start controller demo --name demo-main \
--backend http://model-host:8000 \
--endpoint http://controller-host:9000 --port 9000 --executor ray
ride describe controller demo-main
ride logs controller demo-main
Send non-streaming POST requests to /v1/chat/completions at the controller endpoint. Use RIDE for component status and logs, and the Ray Dashboard to inspect the underlying Job. Use ride stop controller demo-main to stop the demo. The model keeps running.
curl -fsS http://controller-host:9000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"your-served-model","messages":[{"role":"user","content":"Hello"}],"stream":false}'
Technical homes
One stack. Explicit ownership.
RIDE provides the control-plane architecture while each project retains the repository and organization that best match its technical ownership.