A mini code agent with Docker Agent - Part 4: the `rag` toolset

We have seen how to make an agent that can explore (01), work with files (02), think and stay on track (03). Now we need it to be able to know things it never learned.

A model only "knows" what it saw during training, and it saw it all "blended": every version of a language, every writing fashion, every contradictory tutorial, with no distinction. And a small model will answer in an approximate and probably incomplete way. So we are going to hand it "framed" information: a corpus of documents we already own, and that we want it to consult before answering.

RAG (Retrieval-Augmented Generation) is the answer to that: before replying, the agent searches your documents and only answers from what it found.

Here the corpus is a Rust cookbook: 16 recipes in knowledge/. Rust is almost certainly already in the model's training data (though it all depends on the model), so the point is not that it could not know, it is that its answer be anchored to that particular reference (our Rust cookbook corpus) and citable, instead of coming out of its "fuzzy" memory.

The principle

The principle is simple: we index the documents, then we query the index to find the relevant passages before answering. The model only sees those passages, not the rest of the corpus.

• Embedding: turning a text into a vector of numbers (here 768 dimensions) such that two texts with a close meaning give two close vectors.
• Chunk: a piece of a document. We do not index a whole file: we cut it up, so that only the useful passage goes back to the model.
• Cosine similarity: measuring how close two vectors are (1 = identical). It is distance computation between vectors.

Declaring a knowledge base

Unlike the other toolsets, a RAG base is declared once in a top-level rag: block, then referenced by its name from any agent:

agents:
  root:
    toolsets:
      - type: rag
        ref: rag_knowledge      # ← reference

rag:
  rag_knowledge:                # ← declaration
    tool:
      description: >
        Search the team's Rust cookbook: syntax, ownership and borrowing,
        Option and Result, iterators, collections, testing, concurrency...
    docs:
      - ./knowledge
    strategies:
      - type: chunked-embeddings
        embedding_model: embedder
        database: ./knowledge-embeddings.db
        vector_dimensions: 768

In this example the Markdown files of the Rust corpus live in the ./knowledge folder. They will be cut into chunks, then turned into vectors by the embedder embedding model. The vectors are stored in a SQLite index knowledge-embeddings.db so they can be reused on every query.

An embedding model is a model whose only job is to turn a text into a vector of numbers. Two texts with a close meaning give two close vectors. RAG uses that property to find the relevant passages in the corpus.

Two agents can therefore share the same index without building it twice.

tool.description: what the model reads to decide

This is the only thing the model sees before choosing whether to call the tool or not. If your description is too "vague" (a plain "search documents"), a small model may well never call it. To help it, list the subjects covered, like here: "syntax, ownership, Option and Result, iterators..." — that list is what triggers the search when the question falls into one of them.

Often the model believes it already knows everything, and often it answers from memory without ever calling the RAG tool, so you will most likely have to give it more directive instructions: "search it for every question, even one you are sure you know".

Now let's define the embedding model, which is different from the chat model. Our RAG setup needs two models, with very different roles.

A second model: the embedder

A RAG needs two models, with very different roles:

• the chat model (here mellum): it understands the question, decides what to do, writes the answer.
• the embedding model (here embedder): it turns a text into a vector of numbers, to measure how close in meaning two texts are.

models:
  mellum:                       # the one that reasons and answers
    provider: dmr
    model: huggingface.co/jetbrains/mellum2-12b-a2.5b-instruct-gguf-q4_k_m:Q4_K_M
    base_url: http://host.docker.internal:12434/engines/v1

  embedder:                     # the one that turns text into vectors
    provider: dmr
    model: huggingface.co/unsloth/embeddinggemma-300m-gguf:Q8_0
    base_url: http://host.docker.internal:12434/engines/v1
    max_tokens: 2048

Remember to pull the embedding model before starting the agent, otherwise it will not be able to index the documents:

docker model pull huggingface.co/unsloth/embeddinggemma-300m-gguf:Q8_0

✋ vector_dimensions must be exact. EmbeddingGemma produces 768-dimension vectors (you have to go and read its model card to know that: embeddinggemma-300m-GGUF). A value that does not match the model does not "degrade" the search: it breaks it. And max_tokens: 2048 is that model's context window — the chunks must stay under it.

Two important settings: threshold and limit

The threshold is the minimum similarity below which a chunk is thrown away. Its default is 0.5. What matters is not the score of the best chunk but bringing back passages that are close to what you are searching for. Too high a threshold and you risk getting no result at all, too low a one and you will get a lot of noise. On a small model, noise is very costly: it fills the context window and the model loses track of the question. That is also why there is a limit setting, so as not to drown the model in too many results.

"Option default without evaluating it"      "avoid cloning a String"
0.6029  rust-option.md      <- the answer    0.5043  rust-strings.md  <- the answer
0.5664  rust-option.md                       0.4897  ────────── cut by the 0.5 default
0.5301  rust-option.md                       0.4852  ──────────
0.5236  rust-error-handling.md               0.4840  ──────────

Settings:

threshold: 0.3
limit: 8

If your RAG "never finds anything", or does find something but answers in a strangely incomplete way, this is the first setting to look at, before suspecting the chunking or the model.

Chunking the documents: chunking

Everything the RAG returns goes into the model's context. On a small model that is the scarcest resource, so you have to try and find the right trade-off between chunks that are too small (a lot of noise) and chunks that are too big (few results). The chunking is done in the background at startup, and replayed when a document changes.

chunking:
  size: 1200                      # big enough to keep a whole section
  overlap: 100                    # avoids cutting a sentence in half
  respect_word_boundaries: true

📝 Docker Agent offers other chunking and search strategies, which you can look up in the documentation. Here we picked chunked-embeddings for its simplicity and robustness.

The index

• A local SQLite file, at the location given by database:. Here knowledge-embeddings.db.
• Indexing is done in the background at startup, and replayed when a document changes.
• To force a full re-index: delete the .db.

The complete agent configuration file for this lesson

agents:
  root:
    model: mellum
    description: A Rust assistant that answers from the team's Rust cookbook, not from memory.
    instruction: |
      You are a Rust assistant. Answer ONLY from the team's Rust cookbook: search
      it for every question, even one you are sure you know.
      If the passages you retrieved do not answer the question, reply exactly
      "The cookbook does not cover this topic." Never answer from your own knowledge
      of Rust, and never cite a file that does not contain the answer.
      List the source files you used.
    toolsets:
      # A knowledge base is declared once under `rag:` and referenced by name.
      - type: rag
        ref: rag_knowledge

rag:
  rag_knowledge:
    tool:
      # This description is what the model reads to decide whether to search.
      # Name the subject explicitly: a small model needs the hint.
      description: >
        Search the team's Rust cookbook: syntax, ownership and borrowing, Option
        and Result, error handling, iterators, collections, strings, structs and
        enums, traits and generics, testing, concurrency, file I/O, JSON, CLI
        parsing, and the most common compiler errors with their fixes.
    docs:
      - ./knowledge

    strategies:
      - type: chunked-embeddings
        embedding_model: embedder
        # The index is a plain local SQLite file. Delete it to force a re-index.
        database: ./knowledge-embeddings.db
        # EmbeddingGemma produces 768-dimension vectors: this MUST match.
        vector_dimensions: 768
        similarity_metric: cosine_similarity
        # A small embedding model scores lower than a big one — 0.5 is too strict.
        threshold: 0.3
        limit: 8
        chunking:
          size: 1200
          overlap: 100
          respect_word_boundaries: true

models:
  mellum:
    provider: dmr
    model: huggingface.co/jetbrains/mellum2-12b-a2.5b-instruct-gguf-q4_k_m:Q4_K_M
    # If you run the agent from inside a container, use this URL
    base_url: http://host.docker.internal:12434/engines/v1
    # If you run the agent from the host, use this URL
    # base_url: http://localhost:12434/engines/v1

  # A second, much smaller model — used ONLY to turn text into vectors.
  embedder:
    provider: dmr
    model: huggingface.co/unsloth/embeddinggemma-300m-gguf:Q8_0
    # If you run the agent from inside a container, use this URL
    base_url: http://host.docker.internal:12434/engines/v1
    # If you run the agent from the host, use this URL
    # base_url: http://localhost:12434/engines/v1

    # EmbeddingGemma's context is 2048 tokens: chunks must stay under it.
    max_tokens: 2048

📝 You'll find all the configuration files over here: 04-rag-knowledge

Running the agent

cd 04-rag-knowledge

# TUI
docker-agent run localhost.agent.yaml

# Or TUI with sbx (sandbox):
sbx run docker-agent --kit . -- run agent.yaml

In this example, we ask the agent to explain the Option concept in Rust to us, drawing on its documentation corpus. The agent will search the cookbook files, then write an answer from that corpus:

Key takeaways

• The rag toolset anchors the answers to a corpus you choose.
• You need two models: the one that answers, and a tiny embedder (300 MB), everything stays local.
• vector_dimensions must match the embedder exactly (768 for EmbeddingGemma).
• A badly tuned threshold will make a RAG "fail" "silently".
• A tool.description that lists the subjects helps the small model search, and on a corpus it thinks it already knows, you also have to force it to use the rag toolset with explicit instructions.

In the next blog post, we'll see how to use an MCP server with Docker Agent to give it information coming from the outside.