0:01 The newest AI models can read about a million tokens at once. Google says that's eight novels. But the more you give them, the worse they read it. Anthropic's own docs now warn about it. That hits agents hardest. Every step adds more to read: every file it opens, every result that comes back. Soon the few lines it needs are buried under thousands it doesn't.
0:31 And the page it needs most may not be there at all. It's somewhere in your company's documents, which the model has never seen. So what should a model see? And how does it find the right page in 10,000?
0:53 What the model sees
Here's the plan. What the model sees. How an agent fills it, and why more isn't better. Then how to keep it clean, how it finds the right documents, and what happens when it finds the wrong ones. So what can a model actually see? Only what's in this one request. Remember the form the app fills in. Instructions at the top, then your question, then an empty part for the answer.
1:22 The model reads that form, and nothing else. No files, no inbox, no internet, unless something writes them in. In an agent, the form holds more. Every tool call, and every result that comes back, is added below, lap after lap. Everything on that form is the model's context. And the most the form can hold, counted in tokens, is the context window. How big is it? The newest models from Anthropic, OpenAI and Google take about a million tokens.
1:59 And everything has to fit inside it. The instructions, the tool menu, the conversation, every result, every document, and the thinking and the answer it hasn't written yet.
2:17 How fast does an agent fill it?
So how fast does an agent fill it? Take the sales agent's question. Why did sales drop last month? Before lap one, the form holds the instructions, three tools and the question. About 300 tokens. Lap one lists the data folder. 70 more. Lap two reads the sales file. About 700 more. Lap three asks for EMEA's orders, both months at once.
2:46 Say the tool is the simple kind, that hands back every order. Back come over 3,000 rows. That's 125,000 tokens. Lap four answers. EMEA orders dropped, from about 2,000 to 1,200. To write that, it needed two numbers from all those rows. How many in each month. But the rows stay. They're now 99% of the form, and it's all sent again on every lap.
3:19 Keep asking why, and every lap piles on more. So what happens at the limit? The request is refused. Too long. Your code has to decide what to cut. Some chat apps just drop the oldest messages, so the model quietly forgets how the conversation began. Others sum them up instead.
3:44 Why cut anything?
But why cut anything? A million tokens is a lot of room. Because a model doesn't read all of its form equally well. In 2023, researchers at Stanford gave a model 20 documents and a question. Only one document held the answer. When that one came first, GPT-3.5 got about three in four right. When it sat in the middle, at worst, barely half. Worse than when it was given no documents at all.
4:16 They called it lost in the middle. Newer models are far better at finding one fact. But in 2025, Chroma, a company that makes search tools for AI, tested 18 leading models. As the input grew, they got less reliable, even on simple tasks. Google's 2026 tests still show it. This is called context rot. Agents show it too. In one long run of Gemini playing Pokémon, back in 2025, Google saw that past about 100,000 tokens, it tended to repeat old moves instead of making new plans.
4:56 So the context window is how much a model can hold. Not how much it can use well.
5:06 How do you keep the form clean?
So how do you keep the form clean? Anthropic's engineers put the goal in one line. The smallest possible set of high-signal tokens. The few that matter most for this step. Choosing them is called context engineering. Here it is on the same run. First, the better tool from the episode on tools. One that answers, instead of handing back every order: just the total for each month. 125,000 tokens becomes about 100.
5:38 Same answer. Second, clear out old results once they've been used. Leave a short note in their place, so the agent can fetch them again if it needs to. This is called tool result clearing. In one long web search test, Anthropic found it cut the tokens used by 84%. Third, when a run gets long, have the model sum it up. The goal, what it's found, what's still open. Then start a fresh form with just that summary, and carry on.
6:11 This is called compaction. Here, 126,000 tokens become about 80. It isn't free. A summary can drop the one detail you need later. So a good one keeps decisions and open problems, and throws away old tool results.
6:34 What about a question the data can't answer?
So what about a question the data can't answer? Why did EMEA orders drop? The answer's in a memo, somewhere in 10,000 pages of company documents. That's about 4 million tokens. Four times what the form can hold. And the model was never trained on any of it. So you find the few pieces that matter, and put just those in. Finding them is called retrieval. And the whole idea is called retrieval-augmented generation, or RAG.
7:09 Ahead of time, every document is cut into chunks, a paragraph or so each, and filed in a search index. Then each search runs two ways. By keywords, which finds the exact words, like a product code. And by meaning, which finds a chunk that says orders slowed, when you searched for sales dropped. To search by meaning, each chunk becomes a list of numbers. Picture it as a spot on a map of meaning, where chunks about similar things land close together.
7:43 That list is called an embedding. Your question gets one too, and the search picks the chunks that land closest to it. The best few go into the form, each with where it came from. The model answers from them, and cites them, so you can check. Here, the top chunk says, from October 1, EMEA shipping costs €15 an order. So the agent answers: orders fell because shipping went up.
8:12 Source: the shipping memo.
8:18 Is that right?
But is that right? Look at the memo that chunk came from. Its heading says, proposal for 2027, not approved. When the memo was cut up, the heading landed in a different chunk. So the chunk the search found looked like a fact about this year. And the agent gave a confident, wrong answer, with a real source attached. So RAG can still be wrong.
8:46 Not by making things up from nothing, but by trusting the wrong piece. The citation is how a person catches it. One fix: before anything is filed, write a line of context onto every chunk. Which document, which region, which year. Anthropic tested this in 2024. Along with keyword search, it cut failed searches by about half. Now the right memo comes first. On October 1, EMEA moved to a new ordering site, and the old links broke.
9:20 There's another way to find things. Let the agent search for itself. Give it tools to search files for a word, open one, and read a few lines. It loops, searching and reading, until it has what it needs, or your code stops it. This is called agentic search. Early versions of Anthropic's coding agent, Claude Code, used RAG, with an index. The team switched to searching with tools, and said it just worked better.
9:52 By feel, they admit, more than by formal tests. It costs more steps and more tokens. But there's no index to go stale. If a file changed this morning, the next search sees the change. Cursor found agents do best when they mix the two.
10:13 What should a model see?
So what should a model see? Only what this step needs. Not everything that fits. The goal, the right tools, what it's found so far, and the few pages that matter. It finds those pages by searching, by words, by meaning, or with tools of its own. And it keeps the form clean. Trim what's been used. Sum up what's long. That's a context that doesn't fill up with junk.
10:44 And when the conversation ends, the next one starts with an empty form. Next: how agents remember.
