Cache-Friendly Mimari: Statik Baş, Dinamik Kuyruk Prensibi

Temel Prensip#

Yanlış sıralama#

messages = [
    {"role": "user", "content": user_query},  # ← DİNAMİK, en başta
    {"role": "system", "content": SYSTEM},   # ← statik, ama prompt'ta kullanıcıdan sonra
]

Doğru sıralama#

messages = [
    {"role": "system", "content": SYSTEM + STATIC_KB},  # ← STATİK, başta
    {"role": "user", "content": user_query},             # ← dinamik, sonda
]

Conversation History — En Karmaşık Senaryo#

messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": turn1_user},
    {"role": "assistant", "content": turn1_assistant},
    {"role": "user", "content": turn2_user},
    {"role": "assistant", "content": turn2_assistant},
    # ... 10 turn
    {"role": "user", "content": current_query},
]

Cache pattern#

Anthropic'te explicit cache_control#

messages = [
    {
        "role": "system",
        "content": [
            {"type": "text", "text": SYSTEM,
             "cache_control": {"type": "ephemeral"}},
        ],
    },
    # History — her assistant turn'ünde cache_control ekle
    {"role": "user", "content": turn1_user},
    {
        "role": "assistant",
        "content": [
            {"type": "text", "text": turn1_assistant,
             "cache_control": {"type": "ephemeral"}},  # ← cache after this
        ],
    },
    {"role": "user", "content": current_query},  # dinamik
]

RAG Pattern — Statik Kbase + Dinamik Retrieval#

Layer 1 — Statik knowledge base (FAQ, manuallar)#

Layer 2 — Retrieved chunks#

messages = [
    {
        "role": "system",
        "content": [
            # Cache layer 1 — STATİK
            {"type": "text", "text": SYSTEM_PROMPT,
             "cache_control": {"type": "ephemeral"}},
            {"type": "text", "text": COMPANY_FAQ + GENERAL_KB,
             "cache_control": {"type": "ephemeral"}},
        ],
    },
    {
        "role": "user",
        "content": (
            # Layer 2 — DİNAMİK (cache yok)
            f"RETRIEVED CONTEXT:\n{retrieved_chunks}\n\n"
            f"QUESTION: {user_query}"
        ),
    },
]

Optimum split#

Tool Definitions — Cache'in Kara Atı#

Anti-pattern — Conditional tools (cache breaker)#

# YANLIŞ — her isteğin tool listesi farklı, cache miss
tools = get_user_specific_tools(user_id)  # 5-10 tool, user'a göre değişir
response = anthropic.messages.create(
    tools=tools,
    messages=[...],
)

Doğru pattern — Static tool universe#

# Tüm tool'ları her zaman gönder, model permission'a göre kullanmıyor
ALL_TOOLS = sorted([...], key=lambda t: t["name"])  # alfabetik

response = anthropic.messages.create(
    tools=ALL_TOOLS,
    system=[
        {"type": "text", "text": "Permission: " + user_permission,
         "cache_control": {"type": "ephemeral"}},
        {"type": "text", "text": SYSTEM_PROMPT,
         "cache_control": {"type": "ephemeral"}},
    ],
    messages=[...],
)

Trade-off#

• Static tool universe: cache hit ✅, ama 5-10 ekstra tool token (model conditional yapsa daha az olurdu)
• Conditional tools: cache miss ❌, ama tool token tasarrufu

Few-Shot Examples — Statik mi Dinamik mi?#

A — Statik few-shot (cache-friendly)#

SYSTEM = """...

Örnekler:
Soru: A → Cevap: A'
Soru: B → Cevap: B'
Soru: C → Cevap: C'
"""

B — Dynamic few-shot (cache-hostile)#

examples = retrieve_similar_examples(user_query, k=3)
SYSTEM_FOR_THIS_REQUEST = template.format(examples=examples)

Kompromis — Statik few-shot for cache + dynamic supplement in user message#

messages = [
    {
        "role": "system",
        "content": [
            {"type": "text", "text": SYSTEM_WITH_STATIC_EXAMPLES,
             "cache_control": {"type": "ephemeral"}},
        ],
    },
    {
        "role": "user",
        "content": (
            f"İlgili örnek 1: {dynamic_example_1}\n"
            f"İlgili örnek 2: {dynamic_example_2}\n\n"
            f"Sorum: {user_query}"
        ),
    },
]

Cache Anti-Pattern'leri — Sakın Yapma#

1. Tarih/saat sistem prompt'unda#

SYSTEM = f"Şu an: {datetime.now()}\n\n..."  # her saniye değişir!

2. Random ID prompt'ta#

SYSTEM = f"Request: {uuid.uuid4()}\n\n..."  # her uuid farklı

3. User-specific bilgi system'da#

SYSTEM = f"User name: {user.name}\n\n..."  # her user farklı

4. Whitespace değişikliği#

SYSTEM = "Sen asistansın.\n\nKurallar..."  # \n\n
# vs
SYSTEM = "Sen asistansın.\n\n\nKurallar..."  # \n\n\n — cache miss!

5. List sıralaması#

tools = [...]  # sıralı değil — her dict orderı bozar cache
tools = sorted(tools, key=lambda t: t["name"])  # ← her zaman sırala