GloVe + FastText: Global Co-Occurrence Matrix + Subword N-Gram Extension

Ders Haritası (11 Bölüm)#

1. Word2Vec'in eksiği — local vs global statistics tartışması
2. Co-occurrence matrisi — X_ij interpretasyonu
3. GloVe objective — weighted least squares
4. GloVe vs Word2Vec — empirical karşılaştırma
5. FastText motivasyonu — OOV ve morfoloji
6. Subword n-gram modeli — character n-gram embedding
7. FastText training — extension of Skip-Gram
8. Türkçe için FastText — morfolojik dil testi
9. Pratik kullanım — gensim FastText demo
10. Modern era — niye eski tek-statik embedding'ler hâlâ önemli
11. Seçim kılavuzu — Word2Vec/GloVe/FastText hangisi ne zaman

1. Word2Vec'in Eksiği — Local vs Global#

1.1 Word2vec local window#

For each sentence in corpus:
    For each token w_t:
        Look at window [w_{t-k}, w_{t+k}]
        Update embedding based on local context

1.2 Local'in dezavantajı#

• Distant relationships kaybolur ('Türkiye'nin başkenti Ankara'dır' — Türkiye, Ankara aynı window'da değil)
• Statistical efficiency düşük: aynı co-occurrence bilgisi her cümlede tekrarlanır
• Long-tail patterns nadir görüldüğü için kaçırılır

1.3 GloVe iddiası#

1.4 X_ij — co-occurrence count#

X["İstanbul", "Boğazı"] = 12,543      # çok beraber geçer
X["İstanbul", "Ankara"]  = 3,210       # bazen
X["İstanbul", "kuantum"] = 4            # nadiren

1.5 Sparse matrix#

3. GloVe Objective — Weighted Least Squares#

3.1 GloVe insight#

u_i^T v_j + b_i + b_j ≈ log X_{ij}

3.2 Loss function#

J = Σ_{i,j} f(X_{ij}) (u_i^T v_j + b_i + b_j - log X_{ij})^2

• Squared error: weighted least squares regression
• f(X_{ij}): weighting function — frequent pair'leri daha çok cezalandır
• bias terms (b_i, b_j): scalar bias her vector için

3.3 Weighting function f#

f(x) = (x / x_max)^α      if x < x_max
        1                  if x >= x_max

• x = 0: f = 0 (sıfır co-occurrence loss'a katkı yok)
• 0 < x < 100: gradient artar
• x >= 100: plateau (the/a gibi çok frequent pair'ler dominate etmesin)

3.4 Training procedure#

def train_glove(X, V, d, epochs=50):
    u = np.random.uniform(-0.5/d, 0.5/d, (V, d))
    v = np.random.uniform(-0.5/d, 0.5/d, (V, d))
    b_u = np.zeros(V)
    b_v = np.zeros(V)
    
    # AdaGrad optimizer
    for epoch in range(epochs):
        for i, j, x_ij in X.nnz_iter():
            weight = min(1, (x_ij / x_max) ** 0.75)
            log_x = np.log(x_ij)
            diff = u[i] @ v[j] + b_u[i] + b_v[j] - log_x
            loss_grad = weight * diff
            # AdaGrad updates
            u[i] -= lr * loss_grad * v[j]
            v[j] -= lr * loss_grad * u[i]
            b_u[i] -= lr * loss_grad
            b_v[j] -= lr * loss_grad
    return u + v  # final embedding: input + context vectors sum

3.5 Final embedding#

3.6 Avantajlar#

• Statistical efficiency: corpus tek pass
• Global context yakalanır
• AdaGrad optimizer'la deterministic convergence

3.7 Dezavantajlar#

• Co-occurrence matrix RAM-heavy (sparse olsa bile)
• Word2Vec'le mukayese empirical olarak benzer kalite (clear winner yok)
• Implementation karmaşıklığı yüksek

5. FastText Motivasyonu — OOV ve Morfoloji#

5.1 Word2Vec problem#

• OOV (out-of-vocab): training'de görülmemiş kelime → vector yok
• Morphological variants: 'anlaşmak', 'anlaştık', 'anlaşamadık' — her biri ayrı vector. İlişkili olduklarını model bilmez.

5.2 Türkçe ekstrem örnek#

anlaş + ama + dık + larımız + dan

5.3 FastText fikri#

'merhaba' = '<me', 'mer', 'erh', 'rha', 'hab', 'aba', 'ba>'

v('merhaba') = v('<me') + v('mer') + v('erh') + v('rha') + v('hab') + v('aba') + v('ba>')

5.4 OOV avantajı#

v('anlaşamayanların') = v('<an') + v('anl') + v('nla') + ... + v('rın') + v('ın>')

5.5 Türkçe için ideal#

6. Subword N-Gram Modeli#

6.1 N-gram extraction#

def get_ngrams(word, n_min=3, n_max=6):
    word = '<' + word + '>'   # boundary markers
    ngrams = set()
    for n in range(n_min, n_max + 1):
        for i in range(len(word) - n + 1):
            ngrams.add(word[i:i+n])
    ngrams.add(word)   # whole word
    return list(ngrams)

print(get_ngrams('merhaba'))
# ['<me', 'mer', 'erh', 'rha', 'hab', 'aba', 'ba>',
#  '<mer', 'merh', 'erha', 'rhab', 'haba', 'aba>', 
#  '<merh', ..., '<merhaba>']

6.2 N-gram count for typical word#

• 3-grams: 5 - 3 + 1 + 2 (boundary) = 5
• 4-grams: 4 + 2 = 6
• 5-grams: 3 + 2 = 5
• 6-grams: 2 + 2 = 4
• Word itself: 1
• Total: ~21 n-grams

6.3 N-gram vocab size#

• V_ngram = 2M
• d = 300
• Params: 600M

6.4 FastText training (Skip-Gram extension)#

def skipgram_fasttext_loss(center_word, context_word, W_subword, W_out):
    # Center word's vector = sum of its n-grams
    center_ngrams = get_ngrams(center_word)
    v_center = sum(W_subword[ng] for ng in center_ngrams)
    
    # Context: standard word vector (not subword-decomposed)
    u_context = W_out[context_word]
    
    # Same as Word2Vec from here
    score = sigmoid(u_context @ v_center)
    # ...

6.5 Inference (final word vector)#

def word_vector(word, W_subword):
    ngrams = get_ngrams(word)
    return sum(W_subword[ng] for ng in ngrams) / len(ngrams)

6.6 Pretrained FastText for Turkish#

https://fasttext.cc/docs/en/crawl-vectors.html

wget https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/cc.tr.300.bin.gz
gunzip cc.tr.300.bin.gz

6.7 Gensim FastText#

from gensim.models import FastText
model = FastText(
    sentences=turkish_corpus,
    vector_size=300,
    window=5,
    min_count=5,
    workers=8,
    sg=1,           # Skip-gram
    min_n=3,        # min n-gram
    max_n=6,        # max n-gram
    epochs=10,
)
model.save('fasttext-tr.model')

8. Türkçe için FastText — Morfolojik Dil Testi#

8.1 Test 1: OOV handling#

# Word2Vec
from gensim.models import Word2Vec
w2v = Word2Vec.load('word2vec-tr-100d.model')
try:
    vec = w2v.wv['anlaşamayanların']   # OOV
except KeyError:
    print('OOV in word2vec')

# FastText
from gensim.models import FastText
ft = FastText.load('fasttext-tr.model')
vec = ft.wv['anlaşamayanların']        # works!
print(f'Vector shape: {vec.shape}')    # [300]

8.2 Test 2: Morphological similarity#

sim_w2v = w2v.wv.similarity('anlaşmak', 'anlaştık')
sim_ft = ft.wv.similarity('anlaşmak', 'anlaştık')
print(f'Word2Vec: {sim_w2v:.3f}')   # 0.55 (decent)
print(f'FastText: {sim_ft:.3f}')    # 0.87 (excellent)

8.3 Test 3: Rare word quality#

# 'kahvecioğlu' surname — corpus'ta 3 kez geçer
sim_w2v = w2v.wv.similarity('kahveci', 'kahvecioğlu')
sim_ft = ft.wv.similarity('kahveci', 'kahvecioğlu')
print(f'Word2Vec: {sim_w2v:.3f}')   # 0.42 (low, rare word)
print(f'FastText: {sim_ft:.3f}')    # 0.79 (high, subword shared)

8.4 Empirical: Türkçe analogies#

• Word2Vec (10M token): 0.61 Spearman correlation
• GloVe: 0.60
• FastText: 0.71 (+10 puan)

10. Modern Era — Niye Eski Static Embedding Hâlâ Önemli#

10.1 Modern alternatif: contextual embedding#

10.2 Static embedding'in halen avantajları#

1. Lightweight: 300d FastText 6 GB, BERT-base 440 MB ama 110M params (inference slower)
2. Deterministic: aynı kelime hep aynı vector — caching kolay
3. Lookup hızı: O(1) constant time vs BERT O(seq_len^2) attention
4. Mobile/edge: küçük cihazlarda runnable
5. Pre-training-free: kendi corpus'unda hızlı eğit

10.3 Pratik kullanım scenarios (2026)#

• Semantic search (small scale): FastText + cosine sim
• Text classification (TF-IDF alternative): word2vec avg pool + LinearSVC
• OOV handling in toolkits: FastText fallback
• Educational: linguistics + NLP intro courses
• Specialized domains: legal/medical with custom corpus

10.4 Hybrid approach#

11. Seçim Kılavuzu#

Senaryo 1: Genel-amaç NLP (modern)#

Senaryo 2: Lightweight semantic search, mobile#

Senaryo 3: Educational/research#

Senaryo 4: Custom domain (legal/medical Türkçe)#

Senaryo 5: Multilingual transfer#

Senaryo 6: LLM pre-training input#

Karar matrisi#

Sıradaki Ders: Modern LLM Embedding Katmanı#