NumPy ile Tensor Autograd Sıfırdan: Broadcasting-Aware Mini-Tinygrad İnşası

Ders Haritası (Detaylı)#

1. Skaler → Tensor: ne değişiyor?
2. Tensor
   class iskeleti — data, grad, _backward, parents, requires_grad
3. Operasyonlar #1: element-wise (add, mul, exp, log, relu)
4. Broadcasting-aware backward: sum-along-broadcast-dims trick
5. Operasyonlar #2: matmul ve gradient (iki transpose)
6. Operasyonlar #3: reshape, transpose, view — shape ops
7. Operasyonlar #4: reduction (sum, mean, max) ve gradient
8. Operasyonlar #5: softmax, log_softmax, cross-entropy
9. backward()
   — topological sort + reverse
10. Optimizer
    ve
    Module
    class'ları — PyTorch-tarzı API
11. Demo: MNIST'i mini-tinygrad ile eğit
12. PyTorch ile karşılaştırma — bit-exact gradient check

1. Skaler → Tensor: Ne Değişiyor?#

Değişiklikler#

1. Data: tek sayı değil, NumPy ndarray
2. Gradient: tek sayı değil, aynı shape'te ndarray
3. Operatörler: artık element-wise + broadcasting + reduction + matmul
4. Backward: shape-aware → broadcasting'i tersine çevirmek gerekiyor
5. Memory: çok daha büyük tensor'lar → bellek bilinçli olmalı

Kritik gözlem: gradient shape = parameter shape#

Niye broadcasting backward zor?#

2. Tensor Class İskeleti#

import numpy as np

class Tensor:
    def __init__(self, data, _children=(), _op='', requires_grad=False):
        self.data = np.asarray(data, dtype=np.float32)
        self.grad = np.zeros_like(self.data) if requires_grad else None
        self._backward = lambda: None
        self._prev = set(_children)
        self._op = _op
        self.requires_grad = requires_grad

    def __repr__(self):
        return f"Tensor(shape={self.data.shape}, op='{self._op}', requires_grad={self.requires_grad})"

    @property
    def shape(self):
        return self.data.shape

Tasarım kararları#

• data
  :
  np.float32
  zorla — float64 default'tan kaçın
• grad
  : lazy init (only if
  requires_grad
  )
• _backward
  : closure
• _prev
  : parent tensor set
• requires_grad
  : leaf node kontrolü (parametre vs sabit)

Broadcasting backward helper#

def unbroadcast(grad, target_shape):
    """Gradient'i broadcast eksenleri boyunca topla."""
    # 1. Eğer grad rank > target rank, baştan eksenleri reduce et
    while grad.ndim > len(target_shape):
        grad = grad.sum(axis=0)
    # 2. Boyut 1 olan target dim'lerinde, grad'i topla
    for i, dim in enumerate(target_shape):
        if dim == 1:
            grad = grad.sum(axis=i, keepdims=True)
    return grad

3. Element-wise Operasyonlar#

Toplama:

y = a + b

(broadcasting destekli)#

def __add__(self, other):
    other = other if isinstance(other, Tensor) else Tensor(other)
    out = Tensor(self.data + other.data,
                 _children=(self, other), _op='+',
                 requires_grad=self.requires_grad or other.requires_grad)

    def _backward():
        if self.requires_grad:
            self._ensure_grad()
            # ∂out/∂self = 1 → gradient passes through
            self.grad += unbroadcast(out.grad, self.shape)
        if other.requires_grad:
            other._ensure_grad()
            other.grad += unbroadcast(out.grad, other.shape)
    out._backward = _backward
    return out

Çarpma:

y = a * b

(element-wise)#

def __mul__(self, other):
    other = other if isinstance(other, Tensor) else Tensor(other)
    out = Tensor(self.data * other.data,
                 _children=(self, other), _op='*',
                 requires_grad=self.requires_grad or other.requires_grad)

    def _backward():
        if self.requires_grad:
            self._ensure_grad()
            self.grad += unbroadcast(out.grad * other.data, self.shape)
        if other.requires_grad:
            other._ensure_grad()
            other.grad += unbroadcast(out.grad * self.data, other.shape)
    out._backward = _backward
    return out

ReLU#

def relu(self):
    out = Tensor(np.maximum(0, self.data),
                 _children=(self,), _op='relu',
                 requires_grad=self.requires_grad)

    def _backward():
        if self.requires_grad:
            self._ensure_grad()
            # d/dx max(0, x) = 1 if x > 0 else 0
            self.grad += out.grad * (self.data > 0)
    out._backward = _backward
    return out

5. Matmul ve Gradient'i — İki Transpose#

Niye iki transpose?#

Batched matmul#

6. Shape Operasyonları — reshape, transpose, sum#

reshape#

def reshape(self, *shape):
    out = Tensor(self.data.reshape(*shape),
                 _children=(self,), _op='reshape',
                 requires_grad=self.requires_grad)

    def _backward():
        if self.requires_grad:
            self._ensure_grad()
            # Gradient eski shape'e geri
            self.grad += out.grad.reshape(self.shape)
    out._backward = _backward
    return out

Transpose#

def transpose(self, *axes):
    if not axes:
        axes = tuple(reversed(range(self.data.ndim)))
    out = Tensor(np.transpose(self.data, axes),
                 _children=(self,), _op='transpose',
                 requires_grad=self.requires_grad)

    def _backward():
        if self.requires_grad:
            self._ensure_grad()
            # Inverse transpose
            inv_axes = np.argsort(axes)
            self.grad += np.transpose(out.grad, tuple(inv_axes))
    out._backward = _backward
    return out

Sum (reduction)#

def sum(self, axis=None, keepdims=False):
    out_data = self.data.sum(axis=axis, keepdims=keepdims)
    out = Tensor(out_data, _children=(self,), _op='sum',
                 requires_grad=self.requires_grad)

    def _backward():
        if self.requires_grad:
            self._ensure_grad()
            # Gradient'i tüm reduced axes'e broadcast et
            grad = out.grad
            if not keepdims and axis is not None:
                axes = (axis,) if isinstance(axis, int) else axis
                for ax in sorted(axes):
                    grad = np.expand_dims(grad, ax)
            self.grad += np.broadcast_to(grad, self.shape).copy()
    out._backward = _backward
    return out

7. Softmax + Cross-Entropy — Birleştirilmiş ve Stabil#

Numerically stable log_softmax#

def log_softmax(self, axis=-1):
    # log-sum-exp trick
    m = self.data.max(axis=axis, keepdims=True)
    shifted = self.data - m
    lse = np.log(np.exp(shifted).sum(axis=axis, keepdims=True)) + m
    out_data = self.data - lse  # log_softmax

    out = Tensor(out_data, _children=(self,), _op='log_softmax',
                 requires_grad=self.requires_grad)

    def _backward():
        if self.requires_grad:
            self._ensure_grad()
            # d/dx log_softmax(x)_i = δ_ij - softmax(x)_j
            # ∂L/∂x_i = ∂L/∂y_i - softmax_i · Σ_j ∂L/∂y_j
            softmax = np.exp(out_data)
            grad_sum = out.grad.sum(axis=axis, keepdims=True)
            self.grad += out.grad - softmax * grad_sum
    out._backward = _backward
    return out

Cross-entropy (with integer targets)#

def cross_entropy(logits, targets):
    """
    logits: Tensor (N, C)
    targets: array of class indices, shape (N,)
    """
    log_probs = logits.log_softmax(axis=-1)
    N = logits.shape[0]
    # Pick log_prob of target class
    picked = log_probs.data[np.arange(N), targets]
    loss_data = -picked.mean()
    loss = Tensor(loss_data, _children=(logits,), _op='ce',
                  requires_grad=logits.requires_grad)

    def _backward():
        if logits.requires_grad:
            logits._ensure_grad()
            # ∂L/∂logits = (softmax - one_hot(targets)) / N
            softmax = np.exp(log_probs.data)
            one_hot = np.zeros_like(softmax)
            one_hot[np.arange(N), targets] = 1.0
            grad_logits = (softmax - one_hot) / N
            logits.grad += grad_logits * loss.grad
    loss._backward = _backward
    return loss

8.

backward()

Metodu — Topological Sort + Reverse#

def backward(self):
    topo = []
    visited = set()

    def build(v):
        if id(v) in visited:
            return
        visited.add(id(v))
        for child in v._prev:
            build(child)
        topo.append(v)

    build(self)

    # Initial gradient
    self._ensure_grad()
    self.grad = np.ones_like(self.data)

    for node in reversed(topo):
        node._backward()

Gradient zeroing#

def zero_grad(self):
    if self.grad is not None:
        self.grad.fill(0)

9.

Module

ve

Optimizer

— PyTorch-tarzı API#

Module

class#

class Module:
    def __init__(self):
        self._params = []

    def __call__(self, *args, **kwargs):
        return self.forward(*args, **kwargs)

    def forward(self, *args, **kwargs):
        raise NotImplementedError

    def parameters(self):
        return self._params


class Linear(Module):
    def __init__(self, in_features, out_features):
        super().__init__()
        # Kaiming init
        bound = np.sqrt(2.0 / in_features)
        self.W = Tensor(np.random.randn(in_features, out_features) * bound,
                        requires_grad=True)
        self.b = Tensor(np.zeros(out_features), requires_grad=True)
        self._params.extend([self.W, self.b])

    def forward(self, x):
        return x @ self.W + self.b


class Sequential(Module):
    def __init__(self, *layers):
        super().__init__()
        self.layers = layers
        for layer in layers:
            if isinstance(layer, Module):
                self._params.extend(layer.parameters())

    def forward(self, x):
        for layer in self.layers:
            x = layer(x) if isinstance(layer, Module) else layer(x)
        return x

Optimizer

#

class SGD:
    def __init__(self, params, lr=0.01, momentum=0.9):
        self.params = list(params)
        self.lr = lr
        self.momentum = momentum
        self.velocities = [np.zeros_like(p.data) for p in self.params]

    def step(self):
        for p, v in zip(self.params, self.velocities):
            v *= self.momentum
            v += p.grad
            p.data -= self.lr * v

    def zero_grad(self):
        for p in self.params:
            if p.grad is not None:
                p.grad.fill(0)

11. PyTorch ile Bit-Exact Karşılaştırma#

import torch
import torch.nn as nn

# Sync seeds
np.random.seed(42)
torch.manual_seed(42)

# Sabit input ve target
x_np = np.random.randn(4, 8).astype(np.float32)
y_np = np.array([0, 1, 2, 1])

# Mini-tinygrad
W_np = np.random.randn(8, 3).astype(np.float32)
b_np = np.zeros(3).astype(np.float32)

W_mini = Tensor(W_np.copy(), requires_grad=True)
b_mini = Tensor(b_np.copy(), requires_grad=True)

logits_mini = Tensor(x_np) @ W_mini + b_mini
loss_mini = cross_entropy(logits_mini, y_np)
loss_mini.backward()

# PyTorch
W_pt = torch.tensor(W_np.copy(), requires_grad=True)
b_pt = torch.tensor(b_np.copy(), requires_grad=True)
x_pt = torch.tensor(x_np)

logits_pt = x_pt @ W_pt + b_pt
loss_pt = torch.nn.functional.cross_entropy(logits_pt, torch.tensor(y_np))
loss_pt.backward()

# Karşılaştır
print(f"Loss mini: {loss_mini.data:.6f}")
print(f"Loss pyt:  {loss_pt.item():.6f}")
print(f"W grad max diff: {np.abs(W_mini.grad - W_pt.grad.numpy()).max():.2e}")
print(f"b grad max diff: {np.abs(b_mini.grad - b_pt.grad.numpy()).max():.2e}")
# Diff < 1e-5 → bit-exact (FP32 hassasiyetinde)

12. Ne Eksik Bıraktık? (Tinygrad ile Karşılaştırma)#

Eksiklerimiz#

• GPU/Metal/CUDA backend: bizimki saf CPU NumPy
• Lazy evaluation: tinygrad operations'ı plan ediyor, sonra çalıştırıyor
• Kernel fusion: ardışık ops birleştirilip tek kernel'a çevriliyor
• JIT: bir kez compiled, tekrar tekrar koş
• Broader ops: conv, attention, pooling, distributed
• Mixed precision: FP16, BF16
• Quantization: INT8 inference

Bizim seçimimiz#

• Eager eval: hemen hesapla
• NumPy CPU: portabilite + öğretici
• Sade op set: pedagoji odaklı

Sonuç#

13. Mini Egzersizler#

1. exp

   operatörü: Tensor class'a

   exp

   ekle. Backward:

   out.grad * out.data

   .
2. mean

   operatörü:

   sum / count

   olarak yaz. Backward: gradient'i count ile böl.
3. Conv1d (1D convolution): kernel × input convolution. Backward'ı çıkar.
4. Dropout: forward'da random mask, backward'da aynı maskı uygula. Eğitim/eval mod ayrımı.
5. Memory leak avı: birikmiş computation graph'leri tutmuyoruz mu? Detach helper'ı yaz.

Bu Derste Neler Öğrendik?#

Sıradaki Ders#