Tools · Fundamentals
How LLMs work:
five stages to rebuild.
DE Auf Deutsch lesen
The how-llms-work project by w3cj breaks the language-model pipeline into five stages, all of which run in the browser: from a rule-based chatbot to a transformer trained without an ML library. I went through the build and worked out which stages actually help in everyday work with LLMs.
What it's about
Many explanations of language models rely on word clouds, attention arrows, or a picture of vector space. how-llms-work adds runnable code to these pictures. Each of the five sections has its own server route, client hook, and visualization. While the computation runs, the server streams progress via Server-Sent Events.
The stack stays narrow: Hono for server, JSX, and client components, Vite for development and bundling. No ML library. What gets computed is written out as TypeScript in the repository, including the backward pass and the optimizer. The project on GitHub.
pnpm install
pnpm dev
Requires pnpm and Node 20 or newer. If pnpm blocks the build scripts, approve the
esbuild package with pnpm approve-builds and reinstall.
Stage 1: A chatbot without intelligence
Simple Chat
ELIZA logic with modern streaming
Basis: Weizenbaum (1966), ELIZA
The first stage is a chatbot built from conditions and text patterns, the kind Joseph Weizenbaum built in 1966. It knows no probabilities and learns nothing. It checks whether a pattern matches and replies with the stored sentence. The reply arrives word by word via Server-Sent Events, which makes the output resemble the streaming of modern chat interfaces.
The trick in this stage lies in the contrast. The interface feels like a language model because streaming, the typing animation, and the chat history are identical. Behind it sits an if-then chain. Anyone who has seen a prototype that impressed mainly because the interface felt right will recognize the pattern.
For practice, this means the interface itself shapes the impression an AI application makes. That includes time to first token, streaming speed, and how the history is displayed. A smooth interface can mask a model's weaknesses at first.
Stage 2: XOR and the limit of a single layer
Neural Net
Why a perceptron fails on four data points
Basis: Rumelhart, Hinton & Williams (1986), Learning Representations by Back-Propagating Errors
Here a network trains live in the browser on the XOR function, i.e. on four input pairs with four known answers. A single-layer perceptron can't solve this because the solution isn't linearly separable. Minsky and Papert described such limits of single-layer perceptrons in 1969. The later decline in research funding and public interest, however, had several technical and political causes.
With a hidden layer and backpropagation, the error does drop. On a problem small enough to fit on a sheet of paper, you can watch the weights shift and the loss curve fall.
On four data points, something a diagram about gradient descent easily hides becomes visible: the backward pass computes how much each parameter contributed to the error. An optimizer then uses these gradients to update the weights. When training a GPT model, these same two steps form the foundation, supplemented by many additional techniques and considerably more compute.
Stage 3: The tokenizer that writes the bill
BPE
Byte pair encoding, trained on your own text
Basis: Sennrich, Haddow & Birch (2016), Neural Machine Translation of Rare Words with Subword Units
The user enters text, and the algorithm starts at individual characters, merging the most frequent pair into a new unit on each pass. The visualization shows every merge step, until letters have become syllables and syllables have become whole words.
Of all five stages, this one has the biggest direct payoff for my everyday work, because the tokenizer determines what a call costs and how much fits into the context window. Running your own text through the demo shows that the tokenizer's language and training material shape how text gets split up. German compound words, umlauts, and inflectional endings can end up split into more units than equivalent English phrasing. In my RAG projects, the overhead for German technical text usually landed between a third and a half compared to an English translation of the same content. That's an experience value from those projects, not a general rule.
Tokenization is also part of why models are bad at counting letters within a word, process long numbers imprecisely, or burn through a lot of context on deeply nested JSON. The model works with token IDs and their embeddings. Individual characters are only indirectly present in that, depending on the tokenization.
Stage 4: Embeddings, and why the model matters for RAG
Word2Vec
Skip-gram with negative sampling
Basis: Mikolov et al. (2013), Efficient Estimation of Word Representations and Distributed Representations of Words and Phrases
The model learns a vector for each word by predicting which words appear in its neighborhood. Negative sampling draws a handful of random non-neighbors instead of normalizing over the entire vocabulary, which is what makes training affordable in the first place. The demo shows words with similar usage drifting together over the course of training.
Word2Vec dates from 2013; today's search systems mostly use context-dependent transformer embeddings. Both approaches learn from linguistic context but differ substantially in architecture and training objective. Proximity in vector space therefore doesn't reliably map to sameness of meaning. Semantic search tends to fail on opposite pairs, for instance. "Contract terminated" and "contract renewed" sit in almost identical contexts and end up close together accordingly.
For a knowledge base, the embedding model directly affects which passages get preselected. The downstream language model can't reliably retrieve matches that never made the shortlist. For German-language content, multilingual models outperformed English-heavy default models in my comparisons. The gap was larger for short chunks than for long ones. This, too, is an observation from my projects, not a general ranking.
Stage 5: A transformer without a library
GPT from scratch
Decoder-only, all done by hand
Basis: Vaswani et al. (2017), Attention Is All You Need · Radford et al. (2018), GPT-1
The final stage builds a decoder-only transformer and trains it in the browser. Multi-head
causal self-attention, layer normalization, feed-forward blocks, the backward pass, and the
Adam optimizer are all spelled out. For speed, training runs across multiple threads using
SharedArrayBuffer.
The components come from a series of papers the project links individually, and which reappear in the code as named functions:
| Component | Source | Role |
|---|---|---|
| Xavier initialization | Glorot & Bengio, 2010 | Scales starting weights so signals in deep networks neither vanish nor explode |
| Layer normalization | Ba, Kiros & Hinton, 2016 | Normalizes activations before attention and feed-forward so training stays stable |
| Adam | Kingma & Ba, 2014 | Adapts the learning rate per parameter instead of globally |
| Causal self-attention | Vaswani et al., 2017 | Each token weighs the preceding tokens; future ones stay masked |
| Nucleus sampling | Holtzman et al., 2019 | When generating output, draw only from the probability mass covering share p |
For applications, this stage gets concrete at sampling.
Holtzman et al.
examined in 2019 why always picking the most likely continuation often leads to repetitive text,
and why unrestricted sampling from the full distribution can become unstable. Top-p cuts off the
tail of the distribution and draws from what remains. Anyone tuning temperature and
top_p in an application is shifting exactly this boundary, and in the demo the
effect can be observed directly on a tiny model.
How the project is structured
All five sections follow the same three-step pattern, which makes reading easier once you've understood one of them:
- Server route under
src/routes/: a Hono POST endpoint that accepts the input and streams computation progress back as SSE events. - Client hook under
src/client/hooks/: holds the UI state and connects the event stream to the display viauseSSEChat. - Result component under
src/client/components/: renders the incoming data as a visualization.
Shared code includes the BPE tokenizer in src/server/lib/bpe.ts, used jointly by
stages 3 through 5, and the SSE helper functions on both sides. Anyone looking to check a single
detail gets there quickly through these files.
Which stages help in day-to-day project work
For teams running LLM applications, stages 3 and 4 matter most. Tokenizer and embeddings are the two places where your own decisions are made and where mistakes hit budget and match quality directly. Stages 2 and 5 explain why models behave the way they do, but change little about your own architecture.
A transformer trained in the browser on a small corpus demonstrates individual mechanisms. The behavior of large models can't be derived from it. As parameter count changes, so do compute cost, data mix, training procedure, and evaluation.
The effort involved is modest. Clone the repository, pnpm install,
pnpm dev, and then all five sections can be fed your own text.
Building an LLM application?
I build RAG and LLM applications for production, from choosing the embedding model through chunking and evaluation to monitoring in live operation. 30-minute intro call, free of charge.