TR Agglutination Pitfalls: Suffix Tokenization + İ/I/ı/i Casefold Bug

1. Agglutinative Pitfalls#

"ev" + "ler" + "im" + "iz" + "de" + "ki" + "ler"
= "evlerimizdekiler"

2. İ/I/ı/i Casefold — Klasik Bug#

"İSTANBUL".lower()    # → "i̇stanbul" (i + combining dot above!)
"istanbul".upper()    # → "ISTANBUL" (yanlış!)
"İSTANBUL".casefold() # → "i̇stanbul"

def tr_lower(s):
    return s.replace("İ", "i").replace("I", "ı").lower()

def tr_upper(s):
    return s.replace("i", "İ").replace("ı", "I").upper()

3. Unicode NFC vs NFD#

import unicodedata

s_nfc = "İstanbul"                                # NFC normalize
s_nfd = unicodedata.normalize("NFD", "İstanbul")  # NFD: I + combining dot

len(s_nfc), len(s_nfd)  # 8, 9
s_nfc == s_nfd          # False — looks same, different bytes

text = unicodedata.normalize("NFC", text)