Feature-Flag → Cost-Flag: Engineering-Level Measurement of A/B Test's Real $/User Impact

Senaryo#

• Control (50%): Normal Elasticsearch search
• Variant A (50%): GPT-5-mini ile reranked Smart Search

Klasik ölçüm#

Maliyet katmanı#

Control:  +$1.26
Variant:  +$1.728 - $0.036 = +$1.692

Cost-Flag Pattern'ı#

Feature flag kütüphanesi (GrowthBook, LaunchDarkly, custom)#

from growthbook import GrowthBook

def get_search_variant(user_id: str) -> str:
    gb = GrowthBook(api_host="...", client_key="...")
    gb.set_attributes({"user_id": user_id})
    if gb.is_on("smart_search_experiment"):
        return "variant_a"
    return "control"

# LLM çağrısı
def perform_search(query: str, user_id: str):
    variant = get_search_variant(user_id)

    if variant == "control":
        return elasticsearch_search(query)

    # Variant A — LLM-powered
    response = completion(
        model="gpt-5-mini",
        messages=[
            {"role": "system", "content": "Rerank search results semantically..."},
            {"role": "user", "content": query},
        ],
        metadata={
            "tenant_id": ...,
            "user_id": user_id,
            "feature": "smart_search",
            "variant": variant,  # ← KEY
            "experiment_id": "smart_search_experiment",
        },
    )
    return llm_rerank(response, base_results)

Variant Dashboard#

SELECT
    JSONExtractString(raw_metadata, 'variant') AS variant,
    count() AS total_requests,
    sum(cost_total_usd) AS total_cost,
    avg(cost_total_usd) AS avg_cost_per_request,
    count(DISTINCT user_id) AS users
FROM llm_telemetry.requests
WHERE
    ts >= '2026-04-01'
    AND JSONExtractString(raw_metadata, 'experiment_id') = 'smart_search_experiment'
GROUP BY variant

Bir hafta sonraki rapor#

• Control conversions: 12.500 × %2.8 = 350 → $0/conversion
• Variant conversions: 12.300 × %3.6 = 443 → $1.008/443 = **$2.28/extra-conversion**

Statistical Significance — Cost Boyutu#

Per-user cost dağılımı#

Variant A 12.300 kullanıcının her birinin LLM maliyeti:
  90 percentile: $0.20
  50 percentile: $0.04
  10 percentile: $0.01

Mean: $0.082
Std dev: $0.18  ← yüksek (power user'lar var)

Pratik yaklaşım#

1. Median ve P90'ı karşılaştır (mean'in yanı sıra)
2. Power user'ları ayrı analiz et (top %10)
3. Multiple metrics aynı anda — conversion ↑ ve cost/user ↓ aynı anda?
4. Confidence interval'lar ile karar ver, p-value'ya saplanma

Kullanışlı statsmodels örneği#

import scipy.stats as st

control_revenue_per_user = ...   # array
variant_revenue_per_user = ...
control_cost_per_user = ...
variant_cost_per_user = ...

# Net (revenue - cost) farkı için Welch test
control_net = control_revenue_per_user - control_cost_per_user
variant_net = variant_revenue_per_user - variant_cost_per_user

t, p = st.ttest_ind(variant_net, control_net, equal_var=False)
print(f"Net mean lift: {variant_net.mean() - control_net.mean():.2f}")
print(f"p-value: {p:.4f}")

LTV-Aware Karar#

Variant A:
  Extra revenue per user (lifetime): $48 × 1.2× LTV multiplier = $57.6
  Extra cost per user (lifetime): $0.082 × 12 ay = $0.984
  Net LTV impact: +$56.6 per user

A/B Testte Abuse İzleme#

SELECT
    user_id,
    sum(cost_total_usd) AS user_cost,
    count() AS requests
FROM llm_telemetry.requests
WHERE
    JSONExtractString(raw_metadata, 'variant') = 'variant_a'
    AND ts >= '2026-04-01'
GROUP BY user_id
HAVING user_cost > 50  -- anormal harcayanları yakala
ORDER BY user_cost DESC

• Gerçek heavy user mı?
• Otomasyon (bot) mı?
• Prompt injection abuse mu?