Skip to content

Code Eval: HumanEval + MBPP + BigCodeBench + LiveCodeBench + SWE-Bench-Lite

Code LLM standard benchmark suite: HumanEval (164 Python), MBPP (974 Python), BigCodeBench (1140 calls, 139 libs), LiveCodeBench (data-leak resistant), SWE-Bench-Lite (300 real GitHub issues). Pass@1 vs pass@10, code execution sandbox. Running bench on RTX 4090.

Şükrü Yusuf KAYA
26 min read
Advanced
Code Eval: HumanEval + MBPP + BigCodeBench + LiveCodeBench + SWE-Bench-Lite

1. Code Benchmark Tablosu#

BenchmarkSizeTypeNotlar
HumanEval164function-level Pythonklasik, data-leak riski
HumanEval-X (multilingual)164 × 66 dilEN + ZH + ...
MBPP (Mostly Basic Python Problems)974basic algorithmiciyi baseline
BigCodeBench1140real-world library calls (139 lib)en realistic
LiveCodeBench400+LeetCode-style, monthly refresheddata-leak resistant
SWE-Bench-Lite300real GitHub issue → PRen zor, agentic
RepoBench27Kcode completionrepo-level
Cookbook standart eval suite: HumanEval + MBPP (sanity) + BigCodeBench + LiveCodeBench (real-world) + SWE-Bench-Lite (agentic).
Pass@k: k completion'dan en az birinin doğru olma olasılığı.
pass@1
(greedy) ve
pass@10
(temperature 0.8) cookbook default.
✅ Teslim
  1. lm-eval-harness ile HumanEval koş. 2) Pre/post FT karşılaştır. 3) Sonraki ders: 8.8 — Code-LLM Safety.

Yorumlar & Soru-Cevap

(0)
Yorum yazmak için giriş yap.
Yorumlar yükleniyor...

Related Content

Part 0 — Engineering Foundations

Welcome to the Fine-Tuning Cookbook: System, Stage Taxonomy, and the Reproducibility Contract

Start Learning
Part 0 — Engineering Foundations

Reproducibility Stack: Seeds, cuDNN Flags, and Deterministic CUDA — End the 'Works on My Machine' Problem

Start Learning
Part 0 — Engineering Foundations

Environment Pinning: uv + pyproject.toml, CUDA Version Matrix, and Container Recipes

Start Learning