Back to full roadmap
topiccore
Production Observability Stack
Langfuse / LangSmith / Helicone / Phoenix — real-time monitor + trace + alert for agents.
3 hours3 resources1 prereqs
You must monitor production agents. Stack layers:
- Tracing: OpenTelemetry (vendor-agnostic) or OpenLLMetry (LLM-specific)
- Visualization: Langfuse / LangSmith / Phoenix dashboards
- Metrics: Prometheus → Grafana (token/cost/latency time series)
- Alerting: Datadog / Sentry / PagerDuty (P95 spike, error rate, cost anomaly)
- User feedback: thumbs up/down, comments → auto-added to eval dataset
Pro: every agent run = traceId; for support you can say "review this trace".