Observe → decide → actuate → verify

RIDEControl

An open control plane for LLM inference.

Decide which requests enter the engine, how many model replicas to run, and which inference state to keep. Deploy independent controllers and inspect their status and logs through one interface.

System boundary

Engines execute. The control plane decides.

RIDE is not another inference engine. It is the policy, coordination, and evidence layer that turns workload intent and live telemetry into bounded actions on an engine fleet.

03 / INTENTAgents, workflows, users

Applications submit inference requests. Controllers can also use workflow signals such as tool waits or service deadlines when available.

02 / RIDEInference control plane

Independent MODs choose actions. The shared RIDE SDK and runtime manage their lifecycle, status, and logs. The Ray executor runs components as Ray Jobs on an existing cluster.

01 / EXECUTELLM inference engines

vLLM-HUST, vLLM, SGLang, and compatible serving backends execute prefill, decode, cache, and communication paths.

00 / RESOURCEAccelerators, memory, network

The data plane remains independently measurable and replaceable.

How controllers work

01 / OBSERVEBuild live context

Workload, topology, engine, model, queue, and resource signals.

02 / DECIDEChoose an action

Admission, routing, placement, budgets, state, quality, and recovery.

03 / ACTUATEUse engine connectors

Apply decisions through versioned telemetry and control interfaces.

04 / REPORTReport outcomes

Correctness, latency, goodput, resource use, and fallback status.

Research decision areas
WF

Workflow & SLO

Schedule DAG stages, admit work, allocate capacity, and protect fairness along the critical path.

ADMISSION · DEADLINE · FAIRNESS
RP

Routing & placement

Choose a backend, cache holder, model replica, adapter, or continuation target from live fleet state.

SELECT · PLACE · RECOVER
KV

State & KV

Predict reuse, materialize or recompute state, select tiers, and account for transfer and survival horizons.

REUSE · MATERIALIZE · TIER
QE

Quality & economics

Trade quality, latency, cost, safety, and service risk without hiding failures behind aggregate throughput.

QUALITY · COST · GUARDRAIL

RIDE MOD catalog

Controllers for inference serving

A MOD packages a control policy as a component. RIDE gives each component the same management interface so its policy can evolve independently.

Graduated

Request Throttling Controller

Decides which agent requests to admit and which to queue. For multi-turn workloads with tool waits, it limits contention for capacity and gives active workflows a chance to reuse their inference state.

23.45%

Lower batch completion time in one matched multi-turn replay (353.74 s → 270.80 s). The engine configuration was held fixed across the two runs. This result covers one replay; task accuracy and repeatability remain unverified.

Incubating

Instance Controller

Studies when to add or remove inference replicas using live load and replica readiness. Evaluation compares latency and service capacity against fixed replicas under a comparable accelerator budget.

Incubating

KV Controller

Studies which inference state to retain, move, or recompute. Policies use reuse and memory pressure signals, while the engine or KV backend carries out the requested operations.

Research portfolio

RIDE Lab also hosts independent research projects, each with its own owner and results.

Loading the portfolio…

Quickstart

Put a controller in front of a model

Start with an existing Ray cluster and a reachable model endpoint. This public demo is a standalone MOD that limits concurrent non-streaming requests. The graduated Request Throttling Controller above uses a different policy.

1. Install RIDE and connect to Ray

Run on your management host. Replace the example Ray Dashboard address with your cluster’s address. Install the demo on the management host. Eligible Ray workers also need the package, either preinstalled or supplied through Ray runtime_env. Leave the management process running.

git clone https://github.com/vLLM-HUST/ride.git
cd ride
python -m pip install '.[ray]'
python -m pip install 'git+https://github.com/vLLM-HUST/ride-controller-demo.git'
ride start --ray-address http://ray-head:8265

2. Start the MOD and inspect it

Open another terminal on the same management host. Set the backend to your model endpoint. The controller endpoint must point to the Ray node that will run it. On multi-node clusters, use a Ray resource constraint to select that node.

ride start controller demo --name demo-main \
  --backend http://model-host:8000 \
  --endpoint http://controller-host:9000 --port 9000 --executor ray
ride describe controller demo-main
ride logs controller demo-main

Send non-streaming POST requests to /v1/chat/completions at the controller endpoint. Use RIDE for component status and logs, and the Ray Dashboard to inspect the underlying Job. Use ride stop controller demo-main to stop the demo. The model keeps running.

curl -fsS http://controller-host:9000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"your-served-model","messages":[{"role":"user","content":"Hello"}],"stream":false}'

Technical homes

One stack. Explicit ownership.

RIDE provides the control-plane architecture while each project retains the repository and organization that best match its technical ownership.