From Ollama to vLLM on an RTX 5090

Published 5-minute read

From Ollama to vLLM on an RTX 5090 Link to heading

I run a couple of DGX Sparks with DeepSeek via vLLM and Ray. The RTX 5090 box (kintsugi) was still on Ollama. That split stopped making sense once my agent workflows outgrew what Ollama does well.

Why I left Ollama Link to heading

Ollama is fine for one user and one chat. My multi-agent workflows fire dozens of concurrent requests, and Ollama serializes them behind a single model slot. Queues form, latency climbs, and agents stall waiting on each other.

vLLM does continuous batching. Requests arrive, get packed into the same forward pass, and everyone gets a turn. Same GPU, but the pipeline stays busy.

Multi-agent load Ollama req 1 req 2 req 3 req 4 queue grows model slot one at a time 1 / N served vLLM req 1 req 2 req 3 in parallel engine batch 3 / 3 concurrent

The box before Link to heading

kintsugi ran Ollama with eleven models: a few Qwen sizes, Llama 3.x, Phi-4, Granite, an embedding model, and two FLUX diffusion models. The FLUX models moved to a 4090 / M4 because vLLM does not serve diffusion image models. It serves LLMs.

Picking a model Link to heading

For a 32 GB card the sweet spot is a MoE with sparse active params: fast because only a slice of the network runs per token, small enough to quantize into the card. That pointed at Qwen3-Coder-30B-A3B (30B total, 3.3B active).

Then the quantization decision.

Qwen3-Coder-30B-A3B on 32 GB 30B total · 3.3B active · MoE Official AWQ gated · needs HF token and license ✗ Official FP8 ~30.5 GB weights · no KV cache headroom in 32 GB ✗ cyankiwi INT4 open · ~16 GB · auto-detected (compressed-tensors) chosen

I serve one model, no swap logic. vLLM keeps weights in VRAM for the life of the process, so unlike Ollama there is no load and unload. With one model that is exactly what I want.

How it runs Link to heading

The repo is self-contained: a compose file, a setup script, and a systemd unit. sudo ./scripts/setup.sh installs Docker and the NVIDIA Container Toolkit if missing, wires the nvidia runtime, installs the unit, and starts the stack. The unit plus restart: unless-stopped mean the box comes back serving after a reboot.

git clone https://github.com/dashaun/rtx5090-vllm
cd rtx5090-vllm
sudo ./scripts/setup.sh
curl localhost:8000/v1/models
Agents Agent Cline Agent LangChain Agent Qwen Code vLLM · Docker API server POST /v1/chat/completions Engine continuous batching qwen3-coder 30B-A3B · INT4 · 16 GB RTX 5090 32 GB tokens back to agents kintsugi · Ubuntu · systemd restarts it on boot

What bit us Link to heading

Everything looked simple until it ran. Four problems, each with a clear cause once we found it.

1 nvidia-smi fails cause: driver updated, kernel module stale, box up 10 days fix: reboot 2 unknown or invalid runtime name: nvidia cause: rootless vs system Docker; daemon.json not reloaded fix: target system daemon, systemctl restart docker 3 compressed-tensors does not match awq cause: model is INT4 compressed-tensors despite the AWQ name fix: drop --quantization awq, let vLLM read the config 4 17 GB of weights in a literal ~ dir cause: Compose does not expand ~ in volume paths fix: mount ./.cache/huggingface relative to the project

  1. nvidia-smi failed with a driver/library mismatch. The driver had been updated, the kernel module was stale, and the box had been up for ten days. Fix: reboot.

  2. docker compose up died with “unknown or invalid runtime name: nvidia”. The box runs rootless Docker as the user context, but the systemd unit talks to the system daemon. The daemon had a daemon.json declaring the nvidia runtime that it was not loading. Fix: restart Docker and target the system daemon. Rootless Docker is not supported by this setup, and that is now documented.

  3. vLLM crashed on startup: the model config says compressed-tensors, and I had forced --quantization awq. The “AWQ” in the repo name lied. Fix: delete the flag and let vLLM read the config.

  4. A few days in I found 17 GB of model weights inside a directory literally named ~. Compose does not expand ~ in volume paths, so the cache mount created a literal directory. Fix: mount ./.cache/huggingface, which Compose resolves relative to the project. The cache is gitignored, so the repo stays clean.

Decisions that stuck Link to heading

  • System Docker, not rootless, for the service. The systemd unit runs compose as root, and rootless Docker ignores /etc/docker/daemon.json.
  • compressed-tensors INT4 over a gated AWQ build and an FP8 build that does not fit.
  • One model, no swap. A second model means a second compose service, not Ollama-style load and unload.
  • Model cache in the repo under .cache, so git pull plus a restart updates the stack without re-downloading weights.

Where it landed Link to heading

Public repo: https://github.com/dashaun/rtx5090-vllm

An OpenAI-compatible endpoint at http://kintsugi:8000/v1 serving qwen3-coder. Point Spring AI or your favorite agent harness at it. Ollama is uninstalled and its ~94 GB of models are gone from the box.

If you have outgrown Ollama, clone it and take it for a spin. The setup script is the whole story: one command, reboot safe.