For two years, "prompt engineering" was the discipline of getting models to behave. It worked, mostly. Then context windows got bigger, models got smarter, and the bottleneck moved. What we call it now is context engineering — and it's a meaningfully different discipline.
This is the shift the best AI engineers have already made. If you're still optimizing one prompt at a time, you're optimizing the wrong thing.
The shift
Prompt engineering asked: what's the best way to phrase this request?
Context engineering asks: what's the best way to populate this 200K-token window?
The first treats the LLM as a black box you coax. The second treats your entire input pipeline — retrieval, history, tools, examples, system prompt — as a system you design.
The mindset is different. You're no longer a copywriter. You're a librarian.
The three layers of context
Static context
The system prompt, tool definitions, persona, response format rules. Stable across requests. Should be heavily cached. Lives at the top of your prompt structure.
Session context
Conversation history, user preferences, recent interactions. Lives across multiple turns. Needs compaction strategies for long sessions.
Query-specific context
Retrieved documents, current user input, freshly-fetched data. Built per-request. Where most of your engineering effort should go.
Designing each layer's lifecycle and budget is where the new craft lies. Each layer has different update cadences, cache implications, and quality requirements.
What changes in practice
- Retrieval is part of the prompt. The best chunks beat the cleverest prompt. Invest there.
- Order matters. Information at the start and end of context gets more attention than information in the middle ("lost in the middle"). Place the user's actual question carefully.
- Format the context for the model. Markdown headers, XML tags, structured JSON — pick a format and use it consistently. Models trained recently were exposed to all of these; pick what reads cleanly.
- Manage context budget. A 200K window doesn't mean fill it. Long contexts can hurt accuracy and slow inference. Treat context like memory — finite, valuable.
- Cache aggressively. Stable prefixes are dramatically cheaper. Design context structure for cache hits.
Concrete techniques
Context compression
Summarize older conversation turns into a short brief. Trade fidelity for room. A weekly conversation summarized into 200 tokens beats stuffing 30K tokens of raw history.
Selective retrieval
Retrieve more candidates than you'll use, then rerank. Quality > quantity. Five well-ranked chunks beat fifty mediocre ones.
Context attribution
Tag each piece of context with its source. Makes auditing failures dramatically easier ("the model cited chunk 3, which came from doc X"). Bonus: the model can pass source attribution to the user.
Multi-step context building
A first LLM call decides what context the second call needs. Often cheaper and more accurate than stuffing everything upfront. The "plan, then fetch, then answer" pattern.
Hierarchical retrieval
Retrieve high-level summaries first; fetch detailed sections only if the summary is relevant. Saves tokens, often improves accuracy because the summary acts as a filter.
The economics of context engineering
Better context engineering is both higher quality and lower cost. The instinct to "just stuff more in the prompt" is expensive and usually doesn't help past a certain point. Lean, well-structured context wins on both axes.
Concrete: a system that retrieves 5 carefully-reranked chunks plus a 500-token system prompt will typically outperform one that retrieves 50 raw chunks plus a 2,000-token system prompt — and cost 5x less.
Mistakes to avoid
- Treating context as a memory dump. Every token should earn its place.
- Ignoring order. Position matters more than people think.
- Mixing static and dynamic content in the prompt prefix. Breaks caching. Always put dynamic last.
- Not separating instructions from data. Use clear delimiters so the model knows what's "instructions to follow" vs "data to reason over."
- Skipping retrieval evaluation. If retrieval is wrong, the model is set up to fail. Evaluate retrieval quality separately from generation quality.
The mindset
Think like a librarian, not like a copywriter. Your job is no longer to craft the perfect sentence — it's to put the right information in front of the model at the right time, in the right format, within the right budget. The teams winning at LLM apps in 2026 are the ones who've made this shift.
If you want to learn context engineering as part of a complete production AI workflow, the JoinAI MasterClass includes a full week dedicated to it.
