Docker Model Runner, standalone edition (`dmr`)

My favourite tool for serving local models is Docker Model Runner. Until now, apart from the Linux version, you had to go through Docker Desktop to use it. That's not a problem for me 😉, but I know there are cases where users can't install Docker Desktop (by choice, because of company policy, ...). Luckily, it's now possible to enjoy a "standalone" version of Docker Model Runner.

This blog post explains how to install and use dmr (the standalone version of Docker Model Runner): a single binary that bundles both the inference daemon and the full model-management CLI (run, pull, ls, rm, ps, ...).

✋ Key point: dmr has no dependency on Docker Desktop nor on a running Docker engine. It's just an HTTP process. It therefore fits any macOS, Linux or Windows machine, with or without Docker installed.

1. Is a release binary provided?

Yes. The docker/model-runner repository publishes precompiled archives on every tag of the form dmr-vX.Y.Z.

The current release is dmr-v0.1.0, which provides the following archives:

The binary is statically compiled. It cross-compiles cleanly to all the targets above.

2. Installation via direct download

I've only tested the macOS version so far. Feel free to give me feedback if you try it on Linux or Windows.

macOS (Apple Silicon / arm64)

curl -fsSL -o dmr.tar.gz \
  https://github.com/docker/model-runner/releases/download/dmr-v0.1.0/dmr-darwin-arm64.tar.gz
tar xzf dmr.tar.gz
sudo mv dmr /usr/local/bin/
dmr version

Linux x86-64

curl -fsSL -o dmr.tar.gz \
  https://github.com/docker/model-runner/releases/download/dmr-v0.1.0/dmr-linux-amd64.tar.gz
tar xzf dmr.tar.gz
sudo mv dmr /usr/local/bin/
dmr version

For Linux ARM64, replace dmr-linux-amd64.tar.gz with dmr-linux-arm64.tar.gz.

Windows x86-64 (PowerShell)

Invoke-WebRequest -Uri "https://github.com/docker/model-runner/releases/download/dmr-v0.1.0/dmr-windows-amd64.zip" -OutFile dmr.zip
Expand-Archive dmr.zip -DestinationPath .
.\dmr.exe version

3. Usage

dmr works in two steps:

• you start the daemon (dmr serve),
• then you send it commands with the CLI.

3.1 Start the daemon

# Starts the daemon in the foreground (default TCP port 12434)
dmr serve

Leave that terminal open (or launch it in the background with dmr serve &).

Useful options (see dmr serve --help):

Example with a custom port and models folder:

dmr serve --port 13434 --models-path ./models

3.2 Manage and run models

In another terminal (the CLI talks to the daemon on http://localhost:12434 automatically):

dmr pull hf.co/Menlo/Jan-nano-gguf:Q4_K_M         # download a model
dmr run hf.co/Menlo/Jan-nano-gguf:Q4_K_M "Hello"  # run a single prompt
dmr ls                                            # list local models
dmr ps                                            # list running models
dmr rm hf.co/Menlo/Jan-nano-gguf:Q4_K_M           # remove a local model

Interactive chat mode (no prompt as argument):

dmr run hf.co/Menlo/Jan-nano-gguf:Q4_K_M
> Who is Jean-Luc Picard?
...
> /bye

The standalone CLI has full parity with the docker model plugin: run, pull, push, tag, inspect, logs, bench, configure, etc. Only the commands that manage the docker-model-runner container (install-runner, start-runner, stop-runner, ...) are hidden, because dmr serve replaces them: there is no container to manage.

4. HTTP API (OpenAI-compatible)

dmr exposes an OpenAI-compatible REST API. Once dmr serve is running (port 12434 by default):

# List models
curl http://localhost:12434/models

# Download a model
curl http://localhost:12434/models/create -X POST -d '{"from": "ai/gemma3"}'

# Chat completion
curl http://localhost:12434/engines/llama.cpp/v1/chat/completions -X POST -d '{
  "model": "ai/gemma3",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Hello, how are you?"}
  ]
}'

# Metrics (Prometheus format)
curl http://localhost:12434/metrics

5. Important caveat: the llama.cpp backend

dmr bundles the daemon and the CLI, but not the llama-server inference engine itself. How it obtains it depends on the platform:

• macOS and Windows: the llama-server binary is downloaded automatically on demand from Docker Hub (image docker/docker-model-backend-llamacpp), directly through the registry's HTTP API — no Docker engine required. It is cached in ~/.docker/model-runner/llama.cpp/bin. A network connection is therefore needed on first launch.
• Linux: there is no automatic download. The backend is normally provided bundled (as is the case in the official model-runner container image). In standalone mode on Linux, you must therefore provide the llama-server binary yourself:
  • either by placing it in the expected install directory (~/.docker/model-runner/llama.cpp/bin/),
  • or by pointing dmr at an existing binary via the LLAMA_SERVER_PATH environment variable.

Useful environment variables

6. "Quick start" recap

# 1. Grab the binary (Linux amd64 here)
curl -fsSL -o dmr.tar.gz \
  https://github.com/docker/model-runner/releases/download/dmr-v0.1.0/dmr-linux-amd64.tar.gz
tar xzf dmr.tar.gz && sudo mv dmr /usr/local/bin/

# 2. Start the daemon (in the background)
dmr serve &

# 3. Download then run a model
dmr pull ai/gemma3
dmr run  ai/gemma3 "Hello!"

References

• Standalone release: https://github.com/docker/model-runner/releases/tag/dmr-v0.1.0
• Standalone binary source: cmd/dmr
• Packaging / distribution: packaging/README.md
• CLI documentation (docker model): cmd/cli/README.md
• Official DMR documentation: https://docs.docker.com/ai/model-runner/get-started/