Key takeaways

Training changes the model; RAG does not. Training updates weights. RAG finds external text and adds it to the context of a single request. Mixing the two is expensive.

“Embedding” means two different things. Inside a Transformer it is a token vector that attention computes with. In RAG it is a document-chunk vector used for similarity search. Related idea, different jobs.

A Transformer does not turn text into search vectors. The tokenizer emits IDs, the embedding layer emits vectors, the Transformer processes context, and the output head scores the next token.

A vector database is not RAG. RAG means retrieve → place in context → generate. You can start with Markdown, an index file, and keyword search. The vector index is a derived layer you can replace.

Document structure is an investment. Headings, meaning-sized blocks, and metadata help humans, help the model in context, and help future retrieval. Embeddings and agents stack on the same source of truth.

An LLM is not a knowledge base

The model does not store python.md or a team policy as files. It stores parameters: billions of numbers fitted to next-token prediction. Those weights compress statistical regularities of language and the world as of training time. That is not a versioned catalog and not an access-control system.

External documentation lives elsewhere: Git, a wiki, tickets, PDFs, tables. RAG does not weld it into the weights. At answer time it finds relevant pieces and places them next to the question. The model sees them as ordinary text in context — the same way it sees a system prompt and chat history.

Keep four mechanisms distinct:

Text, tokens, and two kinds of embeddings

The tokenizer: a string is not yet meaning

The model does not receive Unicode the way a person reads it. A tokenizer first splits the string into pieces from its vocabulary and replaces each piece with an ID.

"Hello, how are you?"
        ↓
Tokenizer
        ↓
["Hello", ",", " how", " are", " you", "?"]
        ↓
[18432, 11, 927, 4812, 30]

The numbers are illustrative: every model has its own vocabulary. The ID is an index, not “the meaning of the sentence.” One word may become several tokens; a leading space often glues to a word; code and rare terms fragment more than people expect.

Three practical consequences follow. Context windows are measured in tokens, not words. Many APIs bill by tokens. Chunking “every 500 words” lines up poorly with what the model will actually see.

The embedding layer inside the model

A token ID is useless to a neural net by itself. It is looked up in an embedding table: each vocabulary slot has a vector of fixed length.

token ID
   ↓
embedding
   ↓
[0.12, -0.73, 0.44, ...]

This is the first meaning of “embedding”: the internal token representation the Transformer computes with. Tokens that appear in similar contexts often end up nearby after training, but the job of this layer is to give the net numbers, not to index your wiki.

Document embeddings for search

The second meaning shows up in RAG. A separate model (often smaller and different) collapses a text chunk into one vector. Passages with similar meaning should lie close even when they do not share the same words.

chunk
  ↓
embedding model
  ↓
[0.12, -0.03, 0.88, ...]

Confusion starts when both senses share one name. Token embeddings live inside the LLM and change from layer to layer through attention. Document embeddings live in a search index and are compared to a query vector. The enterprise view of semantic search, hybrid exact match, and access control is in why embeddings changed search.

Transformers and answer generation

Self-attention with a simple example

Take: “Masha put the book on the table because she was tired.” A person almost certainly binds “she” to Masha, not the book. The model does not “understand characters.” At each layer every token rebuilds its representation by looking at other tokens in the sequence — more strongly at those that help predict what comes next.

each token
   ↓
attends to other tokens in context
   ↓
gets a more informed representation

That is self-attention: not a synonym dictionary, but weighted information exchange inside the window. You do not need the matrices. Remember that a token’s representation depends on its neighbors, and that anything outside the window is invisible for this call.

A small Transformer trained on a narrow corpus is walked through in a 6.4M-parameter model from scratch. For an application engineer the takeaway is: answer quality depends on what entered the sequence, not on magical “file understanding.”

A Transformer block is not “text-to-digits”

One block, simplified:

Input
 ↓
Self-Attention
 ↓
Normalization
 ↓
Feed Forward Network
 ↓
Normalization
 ↓
Output

The model is a stack of these blocks. A common myth: “the Transformer turns text into numbers.” The chain is different:

1. the tokenizer turns text into IDs;
2. the embedding layer turns IDs into vectors;
3. the Transformer processes those representations in context;
4. the output head produces a probability distribution over the next token.

Numbers appear at step 2. The Transformer already works with representations.

Autoregression: one token at a time

Generation is not one “whole answer.” It is a loop. The model looks at the current context, scores the next token, picks one (greedy or sampled), appends it, and repeats.

I want to drink
        ↓
water 0.61
coffee 0.18
tea   0.11
...
        ↓
water

I want to drink water
        ↓
now   0.31
and   0.27
...

Two properties later explain both RAG and hallucinations. First, every new token depends on what was already generated: an early error drags a fluent continuation with it. Second, the model does not separate “facts from documents” from “statistically common text” until you put the document in context and require it to ground the answer.

Training changes weights; context does not

Where “knowledge” in the parameters comes from

The training task, simplified: given a prefix, predict the continuation.

"Paris is the capital of ___"
             ↓
          "France"

The model emits a distribution, compares it to the true token, computes a loss, and nudges weights. After a huge number of such steps the parameters hold compressed regularities: language, facts that appeared often in the corpus, reasoning templates. This is not a filesystem. You cannot open “the Paris article” inside the weights and patch a date.

Fine-tuning further adapts weights for format, style, or a narrow task. RAG does not. The map of pretraining, SFT, LoRA, and tools is in how LLMs are trained. Here one row is enough:

The context window

The window is finite. Teams usually try to fit a system prompt, conversation history, the user question, retrieved passages, and tool results.

System instructions
+
Conversation
+
User question
+
Retrieved documents
+
Tool results
        ↓
       LLM

Whatever did not fit does not exist for this call. That is why retrieval exists: you cannot ship ten million documentation tokens every time. You must pick a few passages that are likely to help.

The model as an interface to knowledge, not as enterprise memory, is argued in why LLMs appeared.

Why RAG exists

Imagine internal docs totaling millions of tokens. You cannot dump them into every request: cost, latency, and the model would still lose the right page in the middle. You need selection.

10,000,000 tokens of documentation
              ↓
           retrieval
              ↓
       5–20 relevant chunks
              ↓
             LLM

RAG (retrieval-augmented generation) means: find external information, add it to context, answer with it. This is not “a smart model that remembers the wiki.” It is a pipeline of retrieval plus generation.

On a demo the pipeline looks short. In a product it is usually retrieval that breaks, not “insufficient creativity”: bad chunking, a stale index, semantics where you needed an exact error code, no reranking. The production loop — chunking, hybrid search, reranking, evaluation — is in production RAG engineering. The platform view is in enterprise RAG architecture.

Markdown as the source of truth

Physically, a knowledge base for a prototype — and for many teams — is ordinary files.

knowledge/
├── README.md
├── python/
│   ├── functions.md
│   ├── objects.md
│   └── classes.md
├── ai/
│   ├── llm.md
│   ├── embeddings.md
│   └── rag.md
└── programming/
    └── algorithms.md

RAG does not require a secret format. A textbook, internal runbooks, this blog, agent skills in SKILL.md can all be the primary source. Markdown is readable by humans, versioned in Git, built into a site, and the same text can be chunked for search.

File structure matters more than it looks. A document with real headings beats a wall of text: easier for people, easier for the model in context, easier to cut on meaning boundaries, easier to attach metadata, later easier to embed.

# Python Functions

    ## Definition
    ...
    ## Arguments
    ...
    ## Mutable arguments
    ...
    ## Common mistakes
    ...
    ## Related concepts
    - references
    - mutability
    - objects

One source of truth can feed both a website and an index:

                Markdown
                    │
          source of truth
                    │
          ┌─────────┴─────────┐
          ↓                   ↓
        Website            RAG index
                              │
                     ┌────────┴────────┐
                     ↓                 ↓
                keyword search    embeddings
                                       ↓
                                  vector DB

Embeddings here are derived. You can rebuild the index. You can swap the model. You can improve retrieval without rewriting the meaning of the documents. The reverse is also true: a pretty vector stack will not save confused sources.

Chunking and metadata

A whole document is rarely a good unit for both search and context. You split it.

document
   ↓
chunk 1
chunk 2
chunk 3
...

Bad chunking cuts a thought in half:

...can be modified after
creation. This means that when a list is...

Better to cut on a meaning boundary and carry the heading along:

    ## Mutable objects

Lists can be modified after creation...

In practice you care about chunk size, overlap, keeping the heading, a pointer back to the source file, and descriptive fields. Too short and you lose why the sentence exists. Too long and it searches badly and burns the window. Overlap saves a thought on a boundary and also duplicates. A heading in every chunk gives the model an anchor: this is not “a paragraph from the middle of python.md,” it is the mutable-objects section.

Metadata lets you mix meaning with filters:

---
title: Mutable and immutable objects
topic: python
level: intermediate
concepts:
  - mutability
  - references
  - functions
---

Or in the index:

{
  "source": "chapter-03.md",
  "section": "Mutable arguments",
  "topic": "python",
  "level": "intermediate"
}

Semantic search finds nearby meaning. Filters drop the wrong section, level, or source. Exact API names need more than meaning. Structure-preserving splits are explored in RAG chunking experiments.

Search: vectors, hybrid, and reranking

Similarity search

The question is embedded with the same embedding model and compared to chunk vectors. Cosine similarity is common: closer directions score higher.

Question:
"Why can a function change a list?"
             ↓
       embedding model
             ↓
        query vector
             ↓
       similarity search
             ↓
chunk A → 0.94
chunk B → 0.91
chunk C → 0.87
chunk D → 0.31

The top chunks go into the prompt. That is not a proof of truth; it is closeness in that embedding model’s space. Short queries, mixed languages, code, and rare identifiers break the picture more often than slide decks admit.

A vector database is not RAG

A vector store typically holds an ID, text, embedding, and metadata.

{
  "id": 1837,
  "text": "Lists are mutable...",
  "embedding": [0.12, -0.73, 0.44],
  "metadata": {
    "topic": "python",
    "section": "mutable objects",
    "source": "python.md"
  }
}

The catalog is long: PostgreSQL with pgvector, Qdrant, Weaviate, Milvus, Chroma, Pinecone, Elasticsearch / OpenSearch. In an experiment you can keep vectors in JSON. You want a database when volume, filters, updates, and latency show up. It replaces neither documents, nor chunking, nor the prompt. The index layer is surveyed in vector databases.

Hybrid search and reranking

Semantics is weak on exact function names, error codes, version numbers, and rare terms. Those belong in lexical search (classic full text, BM25), fused with vector hits, plus metadata filters.

keyword search / BM25
        +
semantic search
        +
metadata filtering
        ↓
better retrieval

Production systems often do not hand the model “top 5 of 10,000” from the first stage. A cheap search gathers tens of candidates; a reranker then scores the pair “question + passage” and keeps the best.

10000 documents
       ↓
vector / hybrid search
       ↓
50 candidates
       ↓
reranker
       ↓
5 best chunks
       ↓
LLM

How to fuse BM25 and vectors is in hybrid search with reciprocal-rank fusion. How to keep reranking inside latency and cost budgets is in reranking in production RAG.

What you retrieved is still just text in a prompt:

SYSTEM:
You answer questions about Python.

CONTEXT:
Lists are mutable objects...
A function receives a reference to the list...

QUESTION:
Why can a function change a list?

RAG does not write knowledge into weights. It temporarily occupies part of the context window. If the passages are wrong, the model will be confidently wrong “with a citation.”

From a Markdown prototype to an agent

Retrieval without a vector database

The first working loop can be boring, which is the point:

Markdown
 ↓
INDEX.md
 ↓
metadata
 ↓
keyword / structural search
 ↓
relevant chunks
 ↓
LLM

A table of contents, keyword search, a filter on topic in YAML, a few whole files in context if the corpus is tiny. That tests source quality before you pick Qdrant. RAG ≠ vector database.

Skills as operational knowledge

For coding agents a “knowledge base” often looks like a skill: instructions, procedures, examples, sometimes tools.

Skill
├── instructions
├── knowledge
├── procedures
├── tools
└── examples

On disk that is still Markdown:

skills/
├── SKILL.md
├── concepts/
│   ├── embeddings.md
│   ├── chunking.md
│   └── retrieval.md
├── procedures/
│   └── build-rag.md
└── examples/
    └── example.md

The model uses a skill as operational knowledge: not “a fact about Paris,” but “how this repository publishes an article.” The same text is usable by a human. Later you can add embeddings if skills no longer fit the window. Tools in production, when an agent calls external APIs, are covered in MCP in production.

Tools and the agent loop

RAG supplies knowledge. Tools supply actions: search, filesystem, database, calculator, HTTP API.

LLM
 ↓
decides what to do next
 ↓
Tool
 ├── search
 ├── database
 ├── filesystem
 ├── calculator
 └── API
 ↓
result
 ↓
LLM

An agent ties reasoning, retrieval, and tools into a loop. That is an orchestration layer, not “another index.” Turn it on when one retrieval pass is not enough: you must join release notes, an error code, and a runbook. For “what port does the service listen on,” an extra loop only adds latency.

Grow in stages

Add layers; do not start with “the platform.”

Stage 1 — Markdown. A knowledge/ tree with INDEX.md, concepts, guides, and examples.

Stage 2 — meaning-sized chunks. Headings, blocks, YAML, links between concepts.

Stage 3 — simple retrieval. Table of contents, keywords, field filters.

Stage 4 — embeddings. Chunks → embedding model → e.g. embeddings.json.

Stage 5 — vector search. A database or a database extension.

Stage 6 — hybrid and reranking. Words + meaning + metadata + a reranker.

Stage 7 — agent. Tools and the right for the model to choose the next action.

Each stage leaves the previous source of truth in place. You change how you find, not how you store meaning.

Common mistakes

Mixing training and RAG: “we’ll fine-tune on the PDFs and the model will remember the policy.” The policy changes on Friday; the weights do not.

Treating the Transformer as “a converter from text into search vectors.” Search embeddings are a separate model and a separate index.

Chunking by character count without headings or links, then wondering why the model cites a stump.

Equating RAG with Qdrant. Without a corpus, chunking, a prompt, and evaluation, the index is useless.

Relying on semantics alone where you need a function name and a version number.

Stuffing too many “similar” chunks into context: the model drowns, cost rises, the right paragraph vanishes.

Confusing an agent with search. Extra tool calls will not create a missing page in the knowledge base.

Four ideas worth pinning above the monitor:

FAQ

Further reading

This piece is a mechanics map. Neighbors go deeper on single layers.

The architectural choice “weights vs outside knowledge” is how LLMs are trained. Why the model should be an interface, not enterprise memory: why LLMs appeared and why enterprises need RAG. Semantic search in a corporate loop: why embeddings matter. Taking the pipeline to production: production RAG engineering. Chunking, hybrid, and reranking: chunking experiments, hybrid search, reranking.

Conclusion

A modern AI system is not “a smart model.” It is independent layers: tokenization, representations, Transformer, context window, knowledge base, retrieval, tools, and orchestration. A good Markdown corpus can be a full source of truth. Embeddings, a vector database, and reranking do not replace it; they help you find the right piece as the system grows.

A concrete step this week: take one folder of docs, align headings with meaning boundaries, write INDEX.md, and check whether the answer is findable with keyword search before you choose a vector database. In the lab that looks like assembling a circuit with named joints, not buying a “knowledge platform” on day one.