One model, many tasks — powerful because general, unreliable for the same reason
Transformers and foundation models: tokens, embeddings, attention, and scaling
The question: How can one model perform many tasks without being trained separately for each one?
You need enough technical fluency to question vendor claims, design evaluations, and understand why foundation models are both powerful and unreliable. This is the vocabulary lesson that makes every later lesson concrete.
What the lesson covers
Language models do not read like humans. Text is split into tokens — word fragments — and each token becomes an embedding: a vector where meaning is geometry (similar meanings sit close together). Everything a model "knows" during a request must fit in its context window, and every token in and out costs money. Tokens explain cost, context limits, and a surprising share of odd model behaviour.
The transformer (Vaswani et al., 2017, "Attention Is All You Need") relates every token to every other token through attention — letting the model use context flexibly, and letting training parallelise across massive compute. Training objective: predict the next token, over internet-scale text. From that single objective, capabilities emerge: translation, summarisation, coding, reasoning-like behaviour.
Foundation models are pretrained broadly, then adapted: prompting (cheapest), retrieval (grounding in your data), fine-tuning (behaviour shaping), and tools (letting the model call functions). The deployment stack in practice: interface → prompt → context → model → tools/RAG → evaluation → monitoring. Vendors sell the middle; value and risk live at the ends.
Scaling improved capability dramatically — but not equally across reliability, truthfulness, security, or business fit. Hallucination is structural: the model generates plausible continuations, not verified facts. Benchmarks are gamed and saturate; the 2026 AI Index theme of "artificial jagged intelligence" captures it — gold-medal Olympiad maths next to a 50.1% success rate reading an analog clock. The only benchmark that matters is your task, evaluated on your data.
Managerial takeaway: treat model capability claims like audited accounts treat management claims — trust after verification. Build a small evaluation set (real questions, expected answers, a rubric) before you commit to any model or vendor.
Key points
- Tokens → embeddings → attention: meaning becomes geometry; context is finite and priced.
- One pretraining objective (next token) yields many capabilities — adaptation happens via prompting, retrieval, fine-tuning, and tools.
- Hallucination is structural, not a bug to patch; grounding and verification are design requirements.
- Capability is jagged: benchmark scores do not predict your task. Build your own eval set.
- You own the ends of the stack: context design in, evaluation and monitoring out.
Framework — Foundation Model Stack
interface → prompt → context → model → tools/RAG → evaluation → monitoring. Vendor demos show the model; your value and your risk concentrate in prompt/context design and in evaluation/monitoring — the layers you own.
The lab
Touch tokens, similarity, and hallucination — then build your first eval questions.
Open this lesson, its lab and its quiz
Sources and further reading
- Attention Is All You Need — Vaswani et al. (2017)
- On the Opportunities and Risks of Foundation Models — Bommasani et al. (2021)
- Morgan Stanley case — OpenAI