A mini code agent with Docker Agent + Docker Model Runner - Part 1

Foreword

At Docker, there are 2 open source projects I use every day:

• Docker Model runner (DMR): a local LLM model server behind an OpenAI-compatible API (but not only).
• Docker Agent: a TUI and a code agent orchestrator that can connect to various providers (OpenAI, Anthropic, etc.) at once, but also to Docker Model Runner.

Important: if for one reason or another you don't have the ability to install Docker (Desktop or Engine), Docker Agent is actually a standalone binary that doesn't need Docker to run (but it also exists as a Docker Desktop plugin). And in the same way, Docker Model Runner exists in a "standalone" version (without Docker), which I talk about in this blog post: "Docker Model Runner, standalone edition".

To install Docker Agent, head over here: docker agent installation

Introduction: a new series of blog posts

Today, I'm starting a series of blog posts (in 6 steps) about building a mini code agent with Docker Agent and Docker Model Runner. The goal is to show how you can build a local code agent, running on a small LLM (here Mellum2) that can execute shell commands (our 1st step in the series).

Mellum2 ?

Mellum2-12B-A2.5B-Instruct is a small code-specialized MoE: it only activates around 2.5B parameters per token while keeping 12B "in reserve", which gives it a good perf/latency ratio on "modest" machines, especially in Q4_K_M.

MoE stands for "Mixture-of-Experts": the model is a Mixture-of-Experts with 64 MLP experts (MultiLayer Perceptron ¹) of which 8 are activated per token, i.e. about 2.5B parameters actually used at each step. So you get the capacity of a 12B (expert specialization) but an inference cost close to a dense ~2.5B, hence a good quality/consumption trade-off.

It's as if the model had 64 specialized "sub-networks", and for each token the router picks only a few of them, instead of lighting up the whole network like a classic dense model.

Why Docker Agent ?

The small model trap: the context window

A small local model is appealing: local, free, private, the code doesn't leave for a third party. But it has a narrow context window (often 8k to 32k tokens): that's the amount of text it can "see" and process at once. And it doesn't forgive vagueness in instructions.

Concrete consequence: if you plug a tool designed for large cloud models, like Claude Code, onto a small local model, it can become unusable.

For instance, I tried the experiment (on a MacBook Air M4 with 32 GB RAM): I launched Claude Code on a small local model (jan-code-4b-gguf) and asked it a simple question ("I want a hello world in Go").

ANTHROPIC_BASE_URL=http://localhost:12434 \
  claude --model huggingface.co/janhq/jan-code-4b-gguf:Q4_K_M

the completion took 5 minutes and a few seconds (versus a few seconds with a purpose-built agent, ... another one of my side projects). The reason: Claude Code injects huge system instructions, a big tool catalog, the entire conversation history... And the small model buckles under the context.

The recipe is simple: reduce the information sent to the model. That's exactly what Docker Agent enables: you start "from scratch" on the system-instructions side. Nothing is imposed, you write your own instruction tailored for the small model, and you only add the useful tools, step by step.

But let's get to it.

Part 01 — The foundation: a dmr agent + a single shell tool

This first step sets up the bare minimum: a code agent running on a small local LLM (Mellum2, served by Docker Model Runner) with a single tool, shell.

The idea: the agent loop

A code agent relies on a very simple mechanism: the agent loop.

1. You give the model a tool catalog (here, just one: shell).
2. The model decides on its own when to call a tool, and with which arguments.
3. You execute the tool and send back the result; it continues, possibly with several tools in a row, until it produces a final answer.

✋ Important: the model never executes anything itself. It just "asks" for a tool call; it's the program using it (the agent, here Docker Agent) that executes it and sends back the result.

You could write this loop by hand (in Go, in JavaScript, in any language). With Docker Agent, you don't write it: you declare the agent in a YAML file, and it's Docker Agent that runs the loop for you.

The files for this step are available here: 01-socle-dmr-shell

File Roles:

• local.agent.yaml: the base agent, with the instruct model used outside a sandbox or a container (DMR endpoint: http://localhost:12434/engines/v1)
• local.agent-thinking.yaml: the same agent, with the thinking model used outside a sandbox or a container (DMR endpoint: http://localhost:12434/engines/v1)
• agent.yaml: the base agent, with the instruct model launched in a container or a sandbox, the DMR endpoint changes: http://host.docker.internal:12434/engines/v1
• agent-thinking.yaml: the same agent, with the thinking model launched in a container or a sandbox, the DMR endpoint changes: http://host.docker.internal:12434/engines/v1
• spec.yaml: an sbx kit if you want to launch the agent in a sbx sandbox, see Launching via sbx

• sbx is a tool developed by Docker to run AI coding agents in isolated microVM sandboxes.
• each step folder (each blog post) contains the same spec.yaml kit: this makes each exercise self-contained for launching via sbx, and you'll get a ready-to-go sandbox with Docker Agent good to use.
• you can perfectly well use docker-agent directly, without sbx.

The configuration, line by line with agent.yaml

agents:
  root: # every docker-agent config has an entry agent named "root"
    model: mellum # reference to a model defined below
    description: ... # what this agent is for (useful in multi-agent, step 06)
    instruction: | # the "system prompt": who it is, how to behave
      You are a code agent ...
    toolsets:
      - type: shell # the only tool: run shell commands

models:
  mellum:
    provider: dmr # the provider: Docker Model Runner
    model: huggingface.co/jetbrains/mellum2-12b-a2.5b-instruct-gguf-q4_k_m:Q4_K_M
    base_url: http://host.docker.internal:12434/engines/v1

Note, I use base_url: http://host.docker.internal:12434/engines/v1 because I'm working in a sandbox. If you trust yourself enough 😈 to run the agent directly on your host machine, you can replace it with base_url: http://localhost:12434/engines/v1 (DMR always listens on port 12434 of the host machine).

Two blocks are enough:

• agents: describes who works. root is the default agent; its instruction is the system prompt, its description summarizes it, and toolsets lists what it can do.
• models: describes with which engine. Here a local model via the dmr provider.

Why start with shell and a single tool

On its own, the shell tool opens up the whole environment: ls, grep, find, git, cat, go, cargo... The model read massive amounts of command lines during training; giving it shell means giving it hundreds of tools at once, without writing a schema for each. 💡 And when faced with a command it doesn't know, it can document itself all on its own (tool --help) as the loop goes.

We deliberately keep a single tool at this step: we'll add reading/writing files (filesystem) in step 02 (next blog post), reasoning and memory in step 03 (another upcoming blog post), etc.

The engine: the dmr provider

Docker Model Runner (DMR) serves models locally behind an OpenAI-compatible API (but also Anthropic and Ollama). docker-agent connects to it via the dmr provider: no API key needed, everything runs on the machine.

base_url: localhost or host.docker.internal ?

DMR always listens on the host machine's port 12434. What changes is where you call it from:

• From the host machine: the program runs on the host, so localhost = the host and you'll use the following endpoint: http://localhost:12434/engines/v1.
• From a container or a sandbox: localhost refers to the container itself, not the host; a request to localhost:12434 would look for DMR "inside" the container, where it isn't. Docker then provides a special name, host.docker.internal, which resolves to the host machine from inside the container, so you'll use this endpoint: http://host.docker.internal:12434/engines/v1.

In other words, host.docker.internal is the "bridge" that lets a container reach a service running on its host.

Prerequisites

# Pull the model once
docker model pull huggingface.co/jetbrains/mellum2-12b-a2.5b-instruct-gguf-q4_k_m:Q4_K_M

# Or if you use the standalone version of DMR:
dmr model pull huggingface.co/jetbrains/mellum2-12b-a2.5b-instruct-gguf-q4_k_m:Q4_K_M

# Check that the endpoint responds
curl http://localhost:12434/engines/v1/models

⚠️ The agent loop only triggers if the model can produce well-formed tool calls (tool_calls). Mellum2-instruct does it, and that's what makes this step possible.

How does the model "know" to call shell ?

Let's go through the agent loop in detail; it's docker-agent that wires it:

1. The tools are sent to the model on every request. The type: shell from the YAML is serialized into an OpenAI tool schema (name, description, parameters) added to the tools field of the call.
2. The model was trained for function calling: it recognizes this tools field and, when relevant, replies not with text but with a structured call request.
3. The model asks, it doesn't execute. It's docker-agent that actually runs the command, captures the output, sends it back to the model, then relaunches the model, until the final answer. That's the agent loop.

user   : "how many .go files?"
model  : (tool_calls) shell({"cmd": "find . -name '*.go' | wc -l"})  # the model ASKS
tool   : "128"                                                       # docker-agent EXECUTES and replies
model  : "There are 128 .go files."                                  # final answer

Launching the agent

We move into the lesson's folder, then launch docker-agent with local.agent.yaml by its name (the agent then works in this folder):

cd 01-socle-dmr-shell

# Interactive TUI
docker agent run local.agent.yaml

# Without TUI, a one-shot question
docker agent run --exec local.agent.yaml "List the files in this folder and count them."

# Validate the config without executing anything or calling the model
docker agent run --dry-run local.agent.yaml

✋ Important: we worked with local.agent.yaml to launch the agent directly from the host machine. This is not a good practice from a security standpoint: the model can execute shell commands on your machine. In the next section, I explain how to use sbx to launch the agent in an isolated sandbox with docker-agent pre-installed, which is much safer. So read this blog post all the way through before launching the agent on your host machine 😉.

With the interactive TUI

In interactive mode (so we launched the agent with docker agent run local.agent.yaml or with sbx using sbx run docker-agent --kit . -- run agent.yaml), you get the following TUI (Text User Interface):

Type your prompt:

The agent asks you for permission to execute code (create the go file, and compile it):

The agent gives you a summary of what it did:

Tool approval: --safety and --yolo

shell = maximum power = maximum risk. By default, docker-agent asks for confirmation before each command. The --safety flag adjusts this behavior:

✋ Beware in non-interactive mode (--exec), there's no terminal to confirm.

Launching via sbx (Docker Sandboxes)

You can also run docker-agent in a sandbox with sbx, without installing docker-agent. We move into the step's folder (each folder contains its agent.yaml and its spec.yaml), then:

cd 01-socle-dmr-shell
sbx run docker-agent --kit . -- run agent.yaml

Let's break down the command:

More information on sbx kits: kits

Sub-step: the model that "thinks"

Mellum2 comes in two variants:

agent-thinking.yaml changes only one thing: the model it points to. Once launched, the agent first displays its reasoning, then calls shell, then concludes.

Don't forget to load the model:

# Pull the model once
docker model pull huggingface.co/jetbrains/mellum2-12b-a2.5b-thinking-gguf-q4_k_m:Q4_K_M

# Or if you use the standalone version of DMR:
dmr model pull huggingface.co/jetbrains/mellum2-12b-a2.5b-thinking-gguf-q4_k_m:Q4_K_M

With the interactive TUI

We launched the agent with docker agent run localhost.agent-thinking.yaml or with sbx using sbx run docker-agent --kit . -- run agent-thinking.yaml

You can follow the model's "thinking mechanism" in the TUI before it gives you the final result. The model first emits a reasoning (thinking tokens, <think>... style), then calls shell, then concludes:

How docker-agent handles "thinking"

• The thinking model natively emits its reasoning (thinking tokens, <think>... style). docker-agent separates it from the final answer and displays it as a reasoning trace.
• On DMR's llama.cpp backend, you can bound this reasoning with thinking_budget, which is passed to the engine as --reasoning-budget:
  yaml

  models:
    mellum_thinking:
      provider: dmr
      model: huggingface.co/jetbrains/mellum2-12b-a2.5b-thinking-gguf-q4_k_m:Q4_K_M
      thinking_budget:
        adaptive # minimal | low | medium | high | adaptive (unlimited)
        # none or 0 → completely cuts off reasoning
  

instruct + think, or thinking model: two paths to the same goal

Explicit reasoning improves the reliability of decisions (which tool, which arguments), at the cost of more tokens and more latency, and these tokens consume the context window, already narrow on a small model. Hence two strategies:

• thinking model (here): reasoning is always there, native;
• instruct model + think tool (to be seen in an upcoming blog post with step 03): you add a reasoning scratchpad on demand to a model that doesn't have one.

The docs for docker-agent's think tool say it clearly: it's made for models without native reasoning capability; if your model already knows how to think (like the thinking variant), the tool is superfluous.

Test prompts

From the lesson's folder, launch the agent in one-shot and watch the loop (call shell → result → answer):

cd 01-socle-dmr-shell

docker-agent run --exec --yolo localhost.agent.yaml "List the files in this folder and count them."
docker-agent run --exec --yolo localhost.agent.yaml "Summarize in three points what the agent.yaml file contains."

With the thinking model, a prompt that asks for reasoning makes the thinking visible before the tool call:

docker agent run --exec --yolo localhost.agent-thinking.yaml "Is there a README.md in this folder? Think, then check with shell."

Takeaways

• A docker-agent agent is two YAML blocks: agents: (who) and models: (with which engine).
• The dmr provider plugs in a small local LLM; in a container/sandbox, aim at host.docker.internal.
• A single toolset: shell is enough to get started, the model decides when to call it, docker-agent executes it.
• --safety/--yolo handle tool approval; --dry-run validates without launching anything.
• The thinking model reasons natively (adjustable via thinking_budget); otherwise, the think tool will provide this service to an instruct model.

In the next blog post we'll see how to read and write code (filesystem + todo).