FP8 Inference: vLLM SmoothQuant + TensorRT-LLM — Production-Ready on RTX 4090

1. vLLM Native FP8 Support#

# 1. Llama 3.1 8B'yi FP8'e çevir (one-shot)
from llmcompressor.modifiers.quantization import GPTQModifier, SmoothQuantModifier
from llmcompressor.transformers import oneshot

# SmoothQuant + FP8
oneshot(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    recipe=[
        SmoothQuantModifier(smoothing_strength=0.8),
        GPTQModifier(targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"]),
    ],
    output_dir="llama-3.1-8b-fp8",
    num_calibration_samples=512,
)

# 2. vLLM ile serve
vllm serve llama-3.1-8b-fp8 --quantization fp8

2. RTX 4090 Inference Throughput Karşılaştırma#

• batch=1 (single user): AWQ int4 hızlı (kernel optimized)
• batch=16+: FP8 en hızlı (memory bandwidth artar, kernel daha verimli)
• FP8 kalite kaybı INT4'ten çok az (PPL +%0.3 vs +%1.0-1.9)
• FP8 size INT4'ten 2x büyük (8GB vs 4.5GB)

• Single-user / low concurrency → AWQ int4
• High concurrency / batch serving → FP8