Qwen 2.5 32B / 72B Math + Code Mastery: GSM8K + MATH-500 + HumanEval FT Reçetesi

Qwen 2.5 32B/72B — math + code'da Llama 70B'yi geçen baseline. Math-heavy dataset mix (GSM8K + MATH + AIME + MetaMathQA), step-by-step CoT format, code execution loop, hyperparameter farkları (lr daha düşük, ep daha çok). 4×H100 80GB QLoRA 32B reçetesi (~3 saat).

Şükrü Yusuf KAYA

32 dakikalık okuma

14.05.2026

İleri

Qwen 2.5 32B / 72B Math + Code Mastery: GSM8K + MATH-500 + HumanEval FT Reçetesi

1. Memory Budget#

Setup	Hardware	W (NF4)	Total peak	Sığar?
Qwen 2.5 32B QLoRA	1×H100 80GB	16 GB	38 GB	✅
Qwen 2.5 32B QLoRA	4×H100 80GB FSDP	4 GB/GPU	12 GB/GPU	✅ rahat
Qwen 2.5 32B QLoRA	1× RTX 4090 24GB	16 GB	OOM	❌
Qwen 2.5 32B QLoRA	2× RTX 4090 FSDP	8 GB/GPU	18 GB/GPU	⚠️ marjinal
Qwen 2.5 72B QLoRA	8×H100 80GB FSDP	4.5 GB/GPU	14 GB/GPU	✅
Qwen 2.5 72B QLoRA	1×H100 80GB CPU offload	36 GB	70 GB	⚠️ yavaş

python

# === Math-Heavy Dataset Mix ===
from datasets import load_dataset, concatenate_datasets
 
# Cookbook'un math mix'i
gsm8k = load_dataset("openai/gsm8k", "main", split="train")           # 7.4K
math = load_dataset("hendrycks/competition_math", split="train")       # 7.5K
metamath = load_dataset("meta-math/MetaMathQA", split="train")         # 395K
orca_math = load_dataset("microsoft/orca-math-word-problems-200k", split="train")  # 200K
 
# Step-by-step CoT format
def to_chat_cot(ex, source):
    if source == "gsm8k":
        problem = ex["question"]
        solution = ex["answer"]   # already step-by-step
    elif source == "math":
        problem = ex["problem"]
        solution = ex["solution"]
    elif source == "metamath":
        problem = ex["query"]
        solution = ex["response"]
    return {
        "text": tokenizer.apply_chat_template([
            {"role": "user", "content": problem},
            {"role": "assistant", "content": solution},
        ], tokenize=False)
    }
 
# Mix with τ=0.4 sampling
import numpy as np
sizes = np.array([len(gsm8k), len(math), len(metamath), len(orca_math)])
weights = (sizes ** 0.4) / (sizes ** 0.4).sum()
 
# Train — Qwen 2.5 32B + 4×H100 + ~3 saat