From Ollama to vLLM on an RTX 5090

dashaun@dashaun.com (DaShaun Carter) — Tue, 22 Sep 2026 12:09:27 -0500

From Ollama to vLLM on an RTX 5090 Link to heading

I run a couple of DGX Sparks with DeepSeek via vLLM and Ray. The RTX 5090 box (kintsugi) was still on Ollama. That split stopped making sense once my agent workflows outgrew what Ollama does well.

Why I left Ollama Link to heading

Ollama is fine for one user and one chat. My multi-agent workflows fire dozens of concurrent requests, and Ollama serializes them behind a single model slot. Queues form, latency climbs, and agents stall waiting on each other.

vLLM does continuous batching. Requests arrive, get packed into the same forward pass, and everyone gets a turn. Same GPU, but the pipeline stays busy.

The box before Link to heading

kintsugi ran Ollama with eleven models: a few Qwen sizes, Llama 3.x, Phi-4, Granite, an embedding model, and two FLUX diffusion models. The FLUX models moved to a 4090 / M4 because vLLM does not serve diffusion image models. It serves LLMs.

Picking a model Link to heading

For a 32 GB card the sweet spot is a MoE with sparse active params: fast because only a slice of the network runs per token, small enough to quantize into the card. That pointed at Qwen3-Coder-30B-A3B (30B total, 3.3B active).

Then the quantization decision.

I serve one model, no swap logic. vLLM keeps weights in VRAM for the life of the process, so unlike Ollama there is no load and unload. With one model that is exactly what I want.

How it runs Link to heading

The repo is self-contained: a compose file, a setup script, and a systemd unit. sudo ./scripts/setup.sh installs Docker and the NVIDIA Container Toolkit if missing, wires the nvidia runtime, installs the unit, and starts the stack. The unit plus restart: unless-stopped mean the box comes back serving after a reboot.

git clone https://github.com/dashaun/rtx5090-vllm
cd rtx5090-vllm
sudo ./scripts/setup.sh
curl localhost:8000/v1/models

What bit us Link to heading

Everything looked simple until it ran. Four problems, each with a clear cause once we found it.

nvidia-smi failed with a driver/library mismatch. The driver had been updated, the kernel module was stale, and the box had been up for ten days. Fix: reboot.
docker compose up died with “unknown or invalid runtime name: nvidia”. The box runs rootless Docker as the user context, but the systemd unit talks to the system daemon. The daemon had a daemon.json declaring the nvidia runtime that it was not loading. Fix: restart Docker and target the system daemon. Rootless Docker is not supported by this setup, and that is now documented.
vLLM crashed on startup: the model config says compressed-tensors, and I had forced --quantization awq. The “AWQ” in the repo name lied. Fix: delete the flag and let vLLM read the config.
A few days in I found 17 GB of model weights inside a directory literally named ~. Compose does not expand ~ in volume paths, so the cache mount created a literal directory. Fix: mount ./.cache/huggingface, which Compose resolves relative to the project. The cache is gitignored, so the repo stays clean.

Decisions that stuck Link to heading

System Docker, not rootless, for the service. The systemd unit runs compose as root, and rootless Docker ignores /etc/docker/daemon.json.
compressed-tensors INT4 over a gated AWQ build and an FP8 build that does not fit.
One model, no swap. A second model means a second compose service, not Ollama-style load and unload.
Model cache in the repo under .cache, so git pull plus a restart updates the stack without re-downloading weights.

Where it landed Link to heading

Public repo: https://github.com/dashaun/rtx5090-vllm

An OpenAI-compatible endpoint at http://kintsugi:8000/v1 serving qwen3-coder. Point Spring AI or your favorite agent harness at it. Ollama is uninstalled and its ~94 GB of models are gone from the box.

If you have outgrown Ollama, clone it and take it for a spin. The setup script is the whole story: one command, reboot safe.

Ollama on DaShaun