Enterprise APM'lerle LLM Cost Entegrasyonu: Sentry, Datadog, New Relic Pattern'leri

1️⃣ Datadog LLM Observability#

Feature seti#

• ✅ Built-in tracing for OpenAI, Anthropic, Cohere, Mistral
• ✅ Cost tracking (per provider)
• ✅ Cost attribution (Datadog's tag system)
• ✅ Eval framework (basic LLM-as-judge)
• ✅ Prompt drift detection
• ⚠️ Self-host Datadog yok (cloud-only)

Fiyatlandırma#

Entegrasyon#

from ddtrace.llmobs import LLMObs

LLMObs.enable(
    ml_app="my-llm-product",
    api_key=os.environ["DD_API_KEY"],
    site=os.environ["DD_SITE"],
)

# OpenAI auto-instrumented
response = openai.chat.completions.create(...)

Avantaj/Dezavantaj#

2️⃣ Sentry AI Monitoring#

Feature seti#

• ✅ Tracing (OpenAI, Anthropic, etc.)
• ✅ Cost tracking
• ✅ AI-specific error capture (token limit, rate limit, content filter)
• ✅ Performance overlay (latency p95)
• ✅ Self-host (Sentry'nin enterprise self-host'u)
• ❌ Eval framework yok
• ❌ Prompt management yok

Fiyatlandırma#

Entegrasyon#

import sentry_sdk
sentry_sdk.init(
    dsn="...",
    integrations=[
        sentry_sdk.integrations.openai.OpenAIIntegration(),
        sentry_sdk.integrations.anthropic.AnthropicIntegration(),
    ],
    enable_tracing=True,
)

Avantaj/Dezavantaj#

3️⃣ New Relic AI Monitoring#

Feature seti#

• ✅ Tracing
• ✅ Cost tracking
• ✅ Model performance benchmarks
• ✅ Token usage analytics
• ✅ Built-in alerts
• ⚠️ Self-host enterprise-only

Fiyatlandırma#

Entegrasyon#

import newrelic.agent
newrelic.agent.initialize("newrelic.ini")

# OpenAI auto-instrumented (newrelic.ai modülü)

Hızlı Karşılaştırma#

Karar#

Hybrid Pattern — En Yaygın Production Setup#

[App] → LiteLLM proxy
           ↓ callback 1: Datadog (cross-service trace, error tracking)
           ↓ callback 2: Langfuse (LLM-specific eval, prompt mgmt, cost dashboard)

import litellm

litellm.success_callback = [
    "datadog",
    "langfuse",
]
litellm.failure_callback = [
    "datadog",
    "langfuse",
    "sentry",
]

Custom Metric Pattern'leri#

Datadog custom metric örneği#

from datadog import statsd

# Her LLM çağrısının sonunda
statsd.gauge(
    "llm.cost_per_request",
    cost_usd,
    tags=[f"model:{model}", f"feature:{feature}", f"user:{user_id}"],
)

statsd.histogram(
    "llm.tokens.input",
    input_tokens,
    tags=[f"model:{model}", f"cached:{is_cached}"],
)

statsd.histogram(
    "llm.tokens.output",
    output_tokens,
    tags=[f"model:{model}"],
)

statsd.increment(
    "llm.cache_hit" if cache_hit else "llm.cache_miss",
    tags=[f"model:{model}"],
)

Sentry custom metric#

sentry_sdk.metrics.distribution(
    "llm.cost_per_request",
    cost_usd,
    unit="usd",
    tags={"model": model, "feature": feature},
)

Modül 3 Özet#

1. Telemetry felsefe — ölçemediğin neyi optimize edemezsin
2. Usage objesi anatomi — 3 sağlayıcı, 3 farklı format
3. Streaming nuance'leri — production'da 7 yaygın hata
4. Araç karşılaştırma — Langfuse, Helicone, LangSmith, Phoenix, OTel
5. Self-hosted stack — ClickHouse + Grafana ile maksimum kontrol
6. Enterprise APM entegrasyonu — Datadog, Sentry, New Relic pattern'leri