Docker Agent + llama.cpp: a local code agent in 5 minutes 🤖💙🦙

My favourite code agent is still Docker Agent, especially when I work with "local" LLMs, be they "big" (gemma-4-26B-A4B-it) when I'm on my work laptop, or more modest (Mellum2-12B-A2.5B-Instruct) when I'm on my personal Mac Book Air.

A big advantage of Docker Agent is being able to connect to various model providers, remote ones (like Anthropic, OpenAI, OVH's AI endpoints, MistralAI, ...) or local ones (like Ollama, Docker Model Runner, llama.cpp, ...)

Today, the one I'm interested in is llama.cpp, because when a new model in GGUF format shows up, llama.cpp is generally the first to be updated to handle the model's specifics.

Kronk is very reactive too, I had written an article about it that I'll have to refresh

But let's get back to llama.cpp, which we'll have to install and start.

Prerequisites

Llama.cpp

Installing it is very simple, a single curl command is enough (otherwise there are other options, I'll let you refer to the llama.cpp website).

curl -LsSf https://llama.app/install.sh | sh

Then all you have to do is start llama in serve mode (so, in API mode) so that it can be used by a code agent:

llama serve -hf JetBrains/Mellum2-12B-A2.5B-Instruct-GGUF-Q4_K_M:Q4_K_M

If the model isn't present on your machine, llama.cpp will download it (from Hugging Face, hence the -hf flag):

And then the model is served on http://localhost:8080

You can run a few checks to verify:

# Server health
curl http://localhost:8080/health
# {"status":"ok"}

# Exact name of the exposed model
curl http://localhost:8080/v1/models

# First chat completion
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "JetBrains/Mellum2-12B-A2.5B-Instruct-GGUF-Q4_K_M:Q4_K_M",
    "messages": [{"role": "user", "content": "Say hello in one short sentence."}],
    "max_tokens": 800
  }'

# {"choices":[{"finish_reason":"stop","index":0,"message":{"role":"assistant","content":"Hello!"}}], ...

Docker Agent

There are several ways to get Docker Agent. The simplest one is to have a recent version of Docker Desktop, and in that case all you need to type (to check) is:

docker agent version

In that case, Docker Agent is a Docker Desktop plugin.

But you can install Docker Agent in a "standalone" version (you don't need docker or Docker Desktop to make it work), on Mac or Linux:

brew install docker-agent

You can also download the latest release from https://github.com/docker/docker-agent/releases, and you'll find a Windows version there too.

And this time you'll use the docker-agent command instead of docker agent:

docker-agent version

So all that's left is to write the configuration of our agent.

Creating a configuration for Docker Agent

In a folder of your choice, create an agent.yaml file with the content below:

providers:
  llamacpp:
    api_type: openai_chatcompletions
    base_url: http://localhost:8080/v1
    # If you work from a container
    #base_url: http://host.docker.internal:8080/v1

models:
  mellum2:
    provider: llamacpp
    model: JetBrains/Mellum2-12B-A2.5B-Instruct-GGUF-Q4_K_M:Q4_K_M
    #max_tokens: 8192
    temperature: 0.7
    provider_opts:
      context_size: 262144

agents:
  root:
    model: mellum2
    description: A helpful AI assistant running on a local llama.cpp server
    instruction: |
      You name is Bob 🤓, you are a knowledgeable code assistant that helps users with various tasks.
      Be helpful, accurate, and concise in your responses.
      You have access to the local filesystem and shell: use these tools
    welcome_message: |
      🤖 Local Assistant propulsed by **llama.cpp** 🦙
      
    toolsets:
      - type: filesystem
      - type: shell

So we have defined:

• An OpenAI API compatible "LLM provider": llamacpp
• A model (LLM): mellum2
• Then a main agent: root with its system instructions, its model and a set of tools (toolsets) to interact with the host system (and this is where I have to tell you that it's better to run a code agent in a sandbox, a VM or a container, and to go have a look at Docker SBX, which offers a container inside a micro VM).

All that's left is to launch our new agent.

Starting Docker Agent

docker-agent run agent.yaml

or docker agent run agent.yaml depending your installation

You'll land on this TUI:

And you can start interacting with your new code agent:

That's all for today (feel free to comment or ask questions). In an upcoming blog post, we'll see how to use Docker Agent in ACP (Agent Client Protocol) mode with Zed Editor.