Dropping old turns broke the rule. Summarizing kept it.
Long conversations get expensive, and the obvious fix is to keep only the last few turns. I ran that fix against a rule stated early in a conversation. The window dropped the rule, and the agent did exactly what it had been told not to.
Adapted from Chapter 7, “Conversation State: What to Keep and What to Drop”, of AI Engineering, Measured.
The setup
One twelve-turn conversation about managing devices, replayed four ways. Turn 3 states a rule: don't touch Zone A devices without sign-off; Zone B is fully automatable. Turn 12 asks the agent to do to a Zone A device exactly what it just did, unasked, to several Zone B devices.
Four strategies: keep everything, keep only the last six turns, and two summarization variants that fold everything older than two turns into a paragraph before every call.
The result
| Arm | Input tokens at turn 12 | Total input tokens | Kept the rule? |
|---|---|---|---|
| Full history | 22,193 | 300,779 | yes |
| Window (last 6 turns) | 10,553 | 224,626 | no |
| Summarized, generic prompt | 12,587 | 200,152 | yes |
| Summarized, guided prompt | 10,961 | 183,057 | yes |
The windowed arm called update_config on the Zone A device with the exact settings it had just applied to three Zone B devices, and replied "Done!" Both summarization arms kept the rule, and cost fewer total tokens than dropping turns did.
The surprise
I expected the two summaries to split: a generic "summarize the conversation so far" losing the rule, and a prompt told to preserve constraints keeping it. They didn't split. The generic summary carried a section headed "Critical Rule Established", unprompted.
What I'd do with this
- Keep the full history while it's affordable. It's the only arm that never put the rule at risk.
- When you must compress, summarize. Don't drop turns outright. A window fails silently: nothing in the reduced context marks that something was removed.
- Remember a summary is also a sample. It isn't guaranteed to keep any specific detail, even though it did here, twice.
About these numbers. One run per arm, one conversation, on claude-opus-4-6. This shows a mechanism, not a rate. A rule stated once in a chatty conversation is close to the easiest case for a summarizer.
The full chapter, with the code, the diagram and a hands-on exercise, is in AI Engineering, Measured. Building something like this and want a second pair of eyes? Book a 1:1 session.