Short-Term Memory

Context windows define the hard limit on what an agent can "see" at once

Attention mechanisms determine what the agent focuses on within that window

Short-term memory is temporary—anything outside the window is forgotten

Larger windows ≠ better: they cost more, run slower, and can dilute attention

Typical ranges: 4K-8K (chat), 32K-64K (documents), 128K+ (full books)

Cost scales linearly with window size—2x tokens = 2x price

Self-attention: tokens attend to other tokens in the same sequence

Cross-attention: tokens in one sequence attend to another (RAG, translation)

Multi-head attention: multiple parallel attention operations capture diverse patterns

Sliding window: simple FIFO, drops oldest first (good for short sessions)

Summarization: compress old messages to preserve key info (long conversations)

Retrieval: store externally, fetch relevant chunks (unlimited history)

Always preserve the system prompt—it defines agent behavior

Keep the last 3-5 messages in full for conversational coherence

Monitor token usage proactively to trigger compression before hitting limits