Whisper Architecture: Log-Mel Spectrogram + Encoder-Decoder + Language Tokens

1. Whisper Pipeline#

audio (16kHz mono) → 30-second window
  → STFT → power spectrum → mel-filterbank (80 bins) → log → log-mel spectrogram
  → conv1d (downsample 2x) → conv1d → encoder transformer (12-32 layer)
  → cross-attention with decoder
  → decoder transformer (12-32 layer)
  → text tokens (BPE 50K)

Special tokens:
  <|startoftranscript|>
  <|tr|>                              # language code (Turkish)
  <|transcribe|> / <|translate|>      # task
  <|notimestamps|> / <|0.00|>...      # timestamp tokens
  <|endoftext|>

Model Variants#

Model	Params	Multilingual	RTL ses	RTX 4090 inference
whisper-tiny	39M	✅	OK (TR sınırlı)	35× realtime
whisper-base	74M	✅	OK	30×
whisper-small	244M	✅	iyi	22×
whisper-medium	769M	✅	iyi	16×
whisper-large-v3	1.55B	✅	çok iyi	12×
whisper-large-v3-turbo	809M	✅	çok iyi	14× (large-v3'ten 4× hızlı)