Glossary Library

Technical GlossaryDeep Learning

Leaky ReLU

An activation function that leaves a small nonzero slope in the negative region to alleviate the dying ReLU problem.

Leaky ReLU was developed to reduce the learning loss caused by standard ReLU zeroing out everything in the negative region. By preserving a small slope for negative inputs, it can prevent some neurons from becoming permanently inactive. It may provide more stable training behavior, especially under fragile optimization conditions or in smaller-data scenarios.

You Might Also Like

Explore these concepts to continue your artificial intelligence journey.

Glossary Cover

yapay-zeka-temelleri

Optimization

The process of systematically improving model parameters according to an objective function in order to increase performance.

Glossary Cover

matematik-istatistik-optimizasyon

Gradient Descent

A fundamental optimization method that updates parameters in the opposite direction of the gradient to minimize a loss function.

Glossary Cover

matematik-istatistik-optimizasyon

Stochastic Gradient Descent

An optimization approach that updates parameters using single examples or small subsets instead of the full dataset at each step.