Practice · RAG

Building RAG by hand:
a small start without a framework.

August 27, 2026 · approx. 7 min read

This example answers questions about your own Markdown files. It consists of one Python file, one JSON file, and a locally running Ollama model.

What a RAG is

A language model doesn't automatically know your documents. Asked a question, it can therefore produce a plausible but wrong answer. A RAG searches your own document collection for matching passages before answering, and hands them to the model together with the question.

The number sequences are called embeddings. They roughly capture the meaning of a text. That lets the search also find paragraphs that express the same thing in different words. The model still only gets the paragraphs found, not the entire document collection.

Setting up the folder

Create an empty folder with this layout. The Markdown files are your knowledge base. In this small example, every paragraph becomes one chunk.

mein-rag/
├── rag.py
└── wissen/
    ├── versand.md
    └── produkte.md

For testing, a file like wissen/versand.md is enough:

# Versand

Bestellungen nach Deutschland kommen in zwei bis vier Werktagen an.

Der Versand nach Österreich kostet 8 Euro.

The full program

The complete code is still available to download as rag.py. Here it's broken down in the same order: code first, then its explanation.

ollama pull embeddinggemma
ollama pull gemma3

python rag.py index
python rag.py ask "Was kostet der Versand nach Österreich?"

The first command builds the index. Every paragraph is sent to the embedding model once and stored with its number sequence in index.json. After changes in wissen/, run python rag.py index again.

Code and explanation

1. Imports and settings

import json
import math
import sys
import urllib.request
from pathlib import Path

DOCS = Path("wissen")
INDEX = Path("index.json")
OLLAMA = "http://localhost:11434"
EMBED_MODEL = "embeddinggemma"
CHAT_MODEL = "gemma3"

Path handles folders and file names. json writes the index to a file, and math is needed for the similarity calculation. The five constants below define the document folder, the index file, the local Ollama address, and the two model names. If you have other models loaded, change only EMBED_MODEL and CHAT_MODEL.

2. Sending JSON to Ollama

def post(path, data):
    request = urllib.request.Request(
        OLLAMA + path,
        data=json.dumps(data).encode(),
        headers={"Content-Type": "application/json"},
    )
    with urllib.request.urlopen(request) as response:
        return json.load(response)

post() takes an API path and Python data. json.dumps(...).encode() turns it into JSON bytes for the HTTP request. urlopen() sends it to the local server. The last line turns its JSON response back into a Python dictionary. This small function replaces an SDK here.

3. Translating text into numbers and comparing them

def embed(text):
    return post("/api/embed", {"model": EMBED_MODEL, "input": text})["embeddings"][0]

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    length_a = math.sqrt(sum(x * x for x in a))
    length_b = math.sqrt(sum(y * y for y in b))
    return dot / (length_a * length_b) if length_a and length_b else 0

embed() sends text to /api/embed and takes the first number sequence from the response. cosine() computes a similarity value from it: dot multiplies the values pairwise and sums them, while length_a and length_b normalize the value. The check at the end guards against division by zero.

4. Splitting Markdown files into chunks

def chunks():
    for file in DOCS.glob("*.md"):
        for paragraph in file.read_text(encoding="utf-8").split("\n\n"):
            text = paragraph.strip()
            if text:
                yield {"source": file.name, "text": text}

DOCS.glob("*.md") walks through every Markdown file in the wissen folder. split("\n\n") splits on two consecutive line breaks, i.e. a blank line. A single line break would be \n. yield returns one chunk at a time instead of building a large list up front. This is deliberately simple. For structured manuals, headings should later be included in the chunk as well.

5. Building the index

def build_index():
    entries = []
    for chunk in chunks():
        chunk["embedding"] = embed(chunk["text"])
        entries.append(chunk)
        print("indexiert:", chunk["source"])
    INDEX.write_text(json.dumps(entries, ensure_ascii=False), encoding="utf-8")
    print(f"Fertig: {len(entries)} Chunks in {INDEX}")

build_index() iterates over the chunks, adds an embedding key to each, and appends the result to entries. json.dumps(..., ensure_ascii=False) preserves special characters in the JSON file. With a handful of files, this is easy to follow and good enough. A vector database only becomes interesting once linear search gets too slow or you need filters and frequent updates.

6. Finding matching chunks

def search(question, limit=3):
    question_embedding = embed(question)
    entries = json.loads(INDEX.read_text(encoding="utf-8"))
    return sorted(
        entries,
        key=lambda entry: cosine(question_embedding, entry["embedding"]),
        reverse=True,
    )[:limit]

The question gets embedded too. sorted(..., key=..., reverse=True) sorts all entries by their similarity value, which the lambda expression computes for each entry. [:limit] keeps the first three hits. The value itself isn't a quality judgment; what matters is whether the right sources land among the first hits.

7. Sending context and question to the model

def answer(question):
    found = search(question)
    context = "\n\n".join(f"[{entry['source']}]\n{entry['text']}" for entry in found)
    prompt = f"""Beantworte die Frage nur mit dem folgenden Kontext.
Wenn die Antwort nicht im Kontext steht, schreibe das deutlich.
Nenne die Quelldatei in Klammern.

KONTEXT:
{context}

FRAGE: {question}"""
    result = post("/api/chat", {
        "model": CHAT_MODEL,
        "messages": [{"role": "user", "content": prompt}],
        "stream": False,
    })
    print(result["message"]["content"])

The chunks found are assembled into context together with their source file. The f-string inserts that context and the question into the prompt. stream=False has Ollama return a complete JSON response, and the last line prints its text. The prompt instruction reduces hallucinations, but doesn't prevent them. Check the stated source for anything that matters.

8. Running commands

if __name__ == "__main__":
    if sys.argv[1:] == ["index"]:
        build_index()
    elif len(sys.argv) > 2 and sys.argv[1] == "ask":
        answer(" ".join(sys.argv[2:]))
    else:
        print('Nutze: python rag.py index | python rag.py ask "Frage"')

This block only runs when you launch rag.py directly. sys.argv contains the words after python rag.py. With index, the index gets built. With ask, every following word is reassembled into a question.

What a framework takes over later

This small setup is a good starting point because every step stays visible. First put together ten to twenty real questions with their expected source. Only once the results fall short with those does the next building block pay off.