What an agent actually sees [1]

Image: Pale Blue Dot (revised, 2024) — NASA/JPL-Caltech

In the previous post we established the difference between an LLM, an agent, and a multi-agent system. One of the defining characteristics of an agent — compared to a bare LLM — is its context.

Let’s define context precisely, then talk about why managing it is most of the job.

What context means for an agent

Context is everything an agent sees at the moment it has to respond. Not everything that happened, not everything that exists in the system — just what gets passed into the model for that specific invocation.

This distinction matters more than it seems.

Three tiers of context

Not all information is treated equally. There are three categories:

1. What the agent always sees

Every time the agent is invoked, it receives:

System prompt — its instructions, role, and behavioral rules
Conversation history — previous turns in the session
Current user message — the actual question or task
Tool and sub-agent descriptions — so it knows what actions are available

This is the core context. It’s always there.

2. What the agent never sees

Some information is intentionally hidden. The clearest example: intermediate steps in a multi-agent pipeline.

When an agent is used as a tool by another agent (the “agent as tool” pattern), the orchestrator doesn’t expose the sub-agent’s internal reasoning to the rest of the system. The caller sees only the final output. This isolation is deliberate — inner workings of a sub-agent are rarely useful to other parts of the pipeline, and passing them would bloat the context for no gain.

Same logic applies to intermediate tool logs: if agent A calls a search API, processes 40 results, and extracts 3 relevant chunks, the other agents in the system don’t need to see all 40 results — only what was extracted.

3. What the agent sees only when needed

Some context is loaded conditionally:

Artifacts — binary files like PDFs, spreadsheets, or images. These are only injected into the context when they’re actually needed for the task at hand, not by default.
Skills — a pattern where a tool carries not just its function but also its own instructions. The agent pulls in the skill’s context only when it decides to use it.

Why context management is the real challenge

You might think this is a technical implementation detail. It’s not. Here’s why it matters.

Token limits

Some frontier models now support up to one million tokens in context. That sounds like a lot — until you’re running a long session with a multi-agent system processing large documents. Limits get hit faster than expected, especially in production scenarios.

Cost

Every token in the context costs money. If you’re passing information the agent doesn’t need for the current task, you’re paying for noise. At scale, this adds up quickly.

Quality degradation

This is the most counterintuitive one. Longer context doesn’t mean better answers — it often means worse ones.

Research has formalized this as context rot: measurable degradation in output quality as context length increases, even when the model’s limit hasn’t been reached. One of the underlying causes is the lost-in-the-middle effect — models tend to pay stronger attention to information at the beginning and end of the context, and lose track of what’s in the middle. Think about how you read a paper: abstract first, then conclusions, and only if something’s unclear do you go through the body. LLMs behave similarly. Studies have recorded accuracy drops of 30% or more on retrieval tasks caused by this effect alone — documented in Liu et al., 2023.

Effective context management isn’t just about staying under the token limit. It’s about keeping the context clean and relevant.

Handling long sessions: context compaction

What happens when a session runs long? The naive approach — pass everything — runs into all three problems above. The standard solution is context compaction.

The idea is simple: once the context reaches a threshold (measured in tokens or number of turns), older messages get summarized rather than passed verbatim. The model receives a compressed representation of what happened earlier, not the full transcript.

This is the same thing that happens to us in practice. If you’ve used Claude or Gemini for a long session, you’ve noticed that the model’s recall of early exchanges becomes imprecise after a while. It’s not a bug — it’s compression at work.

Compaction keeps sessions functional over time without blowing up the context. The tradeoff is precision: summaries lose detail. The design question is always which details matter enough to preserve.

Context engineering is the job

Here’s the reframe: when building agentic systems, most of the work isn’t training a model or picking the right framework. It’s deciding what goes in the context and what doesn’t.

Before adding anything to the context, three questions worth asking:

Does this agent actually need this information to complete the current task? If the answer isn’t clearly yes, leave it out.
Does this information belong here, or should it be fetched on demand? Permanent context is expensive; conditional context is cheap.
If this context grows over time, what’s the compaction strategy? A system with no answer to this question will degrade silently.

These are architecture decisions, not implementation details. Getting them wrong leads to systems that are slow, expensive, and inconsistent — and the degradation is rarely obvious until it’s already affecting quality in production.

What’s next

We’ve been talking about what agents see at runtime. A related problem is how they access knowledge that doesn’t fit in the context window at all — large document bases, structured data, real-time information.

That’s what RAG solves. Next post.

If you have questions or want to suggest a topic, reach out.