Basecamp 1: Prompting & RAGs

Three circles overlap to make an AI generalist. AI literacy — knowing how these systems actually work and how to operate them well. Domain expertise — knowing which problems in your field are worth solving and how they really behave. Operational execution — knowing which tool to reach for, while holding tools loosely, because today's tool is replaced next quarter and the fundamentals aren't.

The trainer is explicit about what this role is not: not an AI/ML specialist who builds transformers, not a tool expert, not someone who outsources their thinking to a model, and not "a prompt engineer." It's someone who takes a problem end to end and ships something. And the pressure is real — coding benchmarks have climbed to the point where narrow specialists are the ones at risk, while people with one deep domain plus an open mind convert fastest.

Step one, tokenization: your sentence is chopped into the model's smallest units and swapped for fixed number IDs. "Explain" is always token 176289. English has a finite vocabulary — roughly 450k words — so every one of them gets a number, and anything the model has never seen gets a catch-all UNK token instead of an error. Rough conversion: 75 words ≈ 100 tokens. This is also the billing unit, which is why cost is measured here.

Step two, embeddings: those meaningless numbers become long vectors that carry meaning learned from what words sit near each other across the whole internet. That's how the model knows "calories in apple" is the fruit and "market cap of apple" is the company — nothing was hand-labelled, the co-occurrence did it. Step three, self-attention: the model decides which words in your input matter most. Step four, prediction: it emits the next token, probabilistically. Steps one, two and four are fixed machinery. Step three is the one you can steer, and prompting is nothing more than steering it.

Step four is prediction from a probability distribution, and a distribution always has a most-likely next word. There is no branch in that machinery that outputs "I have no idea." So when you ask about something that doesn't exist, the model does the only thing it can: it assembles the most statistically comfortable answer from the pieces it knows.

The tell is that hallucinations are *reasonable*. Asked about a model that never existed, ChatGPT correctly noted there's no official model by that name — and then helpfully explained it was probably a nickname meaning a powerful, dominating model. Every step of that reasoning is sensible; the subject is fiction. That's what makes hallucination dangerous: it doesn't look like an error, it looks like an answer.

The context window is the maximum number of tokens the model can hold in one conversation. Not what it remembers — what it can physically accommodate. The journal analogy is exact: a hundred-page notebook takes a hundred pages, a two-hundred-page notebook takes two hundred, and when it's full it's full.

The part people miss is what counts against it. It isn't just your typed messages and the model's replies — every uploaded document is tokenized into the same budget. Uploading one prompting guide PDF consumed 38,000 tokens before a single question was asked. Context sizes are also the fastest-moving spec in the industry, which is why the trainer doesn't memorize them: he opens a comparison leaderboard and sorts by intelligence index, output tokens per second, price, or window size, depending on what the task needs.

Karpathy's framing, quoted in the session: context engineering is "the delicate art and science of filling the context window with just the right information." The rename matters because it moves the job from wording tricks to information supply. Prompt engineering sounds like phrasing; context engineering is about what the model is allowed to know — documents, memory, tools, examples, all of it.

The four-part checklist is the practical form. World context: who am I, what do I do, who am I talking to. Task context: what specifically needs to happen. Examples context: what a good answer looks like. Constraint context: what you must not do. Every model silently assumes all four whether or not you supply them — and it will assume wrong, because it has no way to know you go to the gym for 45 minutes, five days a week, and are lactose intolerant.

Picture a metal chain: linear interlocking links, and you can ask for ten metres or fifty. Chain-of-thought is that shape applied to reasoning — the problem broken into linked steps rather than answered in one jump. Google Brain published the technique in 2023, testing it on reasoning puzzles the models were getting wrong; the only change was appending that one instruction, and accuracy jumped sharply (the trainer cites ~70% in the paper).

Two things follow. First, it isn't just for puzzles — a go-to-market strategy, a financial plan or a competitor analysis are all problems that decompose into steps. Second, and this is the practical unlock: the paper predates every current model's training cutoff, so the model already knows the technique. You don't need to hand-write the steps. Describe your problem and ask it to "make a chain-of-thought prompt going step by step" — it writes the framework for you, typically opening with a role, then an objective, then numbered steps starting with "ask the user for the missing information."

The model was trained on maths, history, geography, marketing, medicine — everything. Ask a bare question and it searches all of it. Name a role and self-attention narrows the search to the relevant region: "you are an elite market researcher who thinks like a McKinsey partner" pulls the model toward consulting frameworks, structured analysis, and the vocabulary of that world.

The trainer's proof is a time-travel argument. Before AI, how did you learn to do market research? You searched, and you landed on HBS articles and Salesforce's guide — the same authoritative sources the model trained on. So writing "you are an expert in market research who has worked for brands like Salesforce" isn't flattery or theatre; it's a pointer at that body of material. Notice too that when you ask a model to write you a prompt, it opens with a role unprompted — because the guides it learned from all say to.

Google, OpenAI and Anthropic have each published a guide to prompting their models. Nobody has time to read three documents and synthesize them. So don't — upload them and ask the model to assimilate the best practices into a reusable "prompt designer." The trainer calls this the father of all techniques, and it's the one thing from this session that outlives every tool in it.

It's built in layers, and each layer contributes something visible. Google's guide gives you "start with a strong role" and chain-of-thought. OpenAI's GPT-5 cookbook adds delimiters and explicit reasoning effort. Anthropic's adds XML tagging, because Claude works best with open/close markup. The result is a block of text that already knows every technique and picks the right one for the problem you hand it — including asking you clarifying questions before it writes anything, which is precisely the failure the bad diet-plan prompt demonstrated an hour earlier.

RAG means the answer comes from documents you supplied rather than from the model's memory. That's it. Attach your resume and ask what projects you worked on at Microsoft, and the model has no choice but to read — it cannot predict facts it was never trained on. Hallucination, for anything the documents cover, disappears.

The library analogy carries the mechanism. A hall of unsorted books with no librarian is the bare model: asked for one fact, you wander and guess. Add a librarian and shelf tags and you get direction — that's chunking and indexing, where a 100-page document becomes ten 10-page chunks stored in a vector database. Add four friends searching different books at once and you get parallel retrieval: your question is matched against every chunk simultaneously, and only the chunks that match — say chunk 5 and chunk 6 for photosynthesis — get pulled into the context window. The rest stays in the database, which is exactly why this scales to documents far larger than the window.

The trainer's framing: everything happening in AI today was technically possible ten years ago. What changed is that AI started generating the three channels humans actually use to communicate. Text — every chat model. Images — every generator. Audio — every voice tool. There is no fourth channel: video is images at some frames per second, a phone call is audio, radio is audio.

The payoff is a bird's-eye filter for the overwhelming flood of new tools. Any AI application you'll ever see is a permutation of those three generative capabilities wired to a problem. Underneath, it's all numbers anyway — an image is a 3×1024×1024 matrix of RGB pixel values, text is token IDs, audio is samples. Knowing that, a new product launch stops being noise and becomes a question: which of the three does this generate, and what problem is it pointed at?