What Is a Decision Tree? A Guide to Classification and Regression in Machine Learning

What is a decision tree? A decision tree is a rule-based, interpretable machine learning (an approach that learns patterns from data) model that splits data from a root node through a series of yes/no questions and produces a prediction (a class or a number) at each leaf. At every split, the question that best separates the data is chosen with a criterion.

The appeal of the decision tree is here: you can read the reason for a prediction like a human. Questions such as "Is income above a certain threshold? If yes, are past payments regular?" branch the flow in a tree structure. This guide answers, from an expert perspective, what a decision tree is, how it works, what criteria like information gain and gini do, why pruning is necessary, and how it relates to random forest.

Definition

Decision Tree (Karar Ağacı)

A rule-based, interpretable machine learning model that splits data from a root node through a series of yes/no questions (feature-based splits) and produces a prediction (a class or a number) at each leaf. At every split the most discriminative question is chosen using a criterion such as information gain or gini; used for both classification and regression.

Also known as: Decision Tree, classification tree, regression tree, CART

Much of modern AI consists of black-box models whose behavior is hard to explain. The decision tree offers the opposite: it shows step by step how a prediction was reached. This transparency makes it both one of the first algorithms learned in education and a preferred tool in regulated sectors.

Consider a bank's credit decision. "The model rejected it" is not enough; the regulator wants a justification given to the customer. The decision tree is a natural solution here, because the reason for the decision is written directly in the branches of the tree. That is why decision trees sit at the center of the explainable AI debate. Decision trees are also the building block of far more powerful ensemble methods (random forest, gradient boosting); modest on their own, together they form some of the most accurate models available.

A decision tree consists of three types of nodes: the root node (the start where all data enters), internal nodes (branching points that split data with a question), and leaf nodes (the ends where the final prediction is given). During training, the algorithm searches top-down for questions that split the data into subgroups that are as "pure" as possible.

The critical question is: which feature and which threshold to split on? The algorithm tries every possible split and selects the one that best separates the data. "Best" is defined by an impurity criterion: the closer the subgroups after the split are to a single class, the better the split. This process is repeated recursively until a stopping condition (maximum depth, minimum sample count, or full purity) is reached.

How to

Steps to train a decision tree

The core steps from raw data to a predicting decision tree.

1. 1

   Start at the root

   All training data is placed at the root node.

2. 2

   Find the best split

   For each feature and threshold, information gain or gini is computed; the question that best separates the data is chosen.

3. 3

   Split the data

   The data is divided into two (or more) subgroups by the chosen question.

4. 4

   Repeat recursively

   The same process is repeated for each subgroup until a stopping condition is met.

5. 5

   Label the leaves

   Each leaf is labeled with the majority class (or the average) of the samples it contains.

The prediction phase is very simple: a new sample enters at the root, descends the relevant branch based on its answer to each node's question, and reaches a leaf. That leaf's label is the model's prediction.

At the heart of a decision tree is the decision "which question best separates the data?", and a number answers this. The two most common criteria are information gain and gini impurity.

Information gain is based on the concept of entropy. Entropy measures how mixed a group is: a group containing samples all from the same class has zero entropy, while a half-mixed group has high entropy. The information gain of a split is the difference between the entropy before the split and the weighted entropy after it — that is, how much disorder was reduced thanks to that question. The algorithm selects the split with the highest information gain.

Gini impurity approaches the same goal from a different angle: it measures the probability of misclassifying a randomly chosen sample from a group when it is randomly labeled according to the group's class distribution. A pure group has a gini value of zero. In practice, information gain and gini produce very similar trees; because gini contains no logarithm, it is slightly faster to compute and is therefore the default in many libraries.

Comparison of information gain and gini impurity

Criterion	Information Gain	Gini Impurity
Underlying concept	Entropy (disorder)	Misclassification probability
Pure group value	Entropy 0 → gain maximal	Gini 0
Compute cost	Contains logarithm, slightly slower	No logarithm, slightly faster
Typical use	ID3 / C4.5 algorithms	CART, default in most libraries
Result quality	Very similar in practice	Very similar in practice

Decision trees are not a single pattern; they vary by both output type and training algorithm. By output type they split in two: classification trees predict a category (spam or not, loan approved or rejected), while regression trees produce a continuous number (the price of a home, the quantity of a demand). The difference lies mostly in the split criterion — regression uses variance reduction instead of purity.

By algorithm family there are historical variants. ID3 and its improved form C4.5 are classic methods that use information gain. CART (Classification and Regression Trees) makes gini-based binary splits and is the basis of most implementations today. More than the detail of these variants, it is important to keep in mind their shared idea: purifying the data through successive questions. The decision tree belongs to the supervised learning branch of the broader machine learning family and is usually positioned as a non-linear, rule-based alternative to linear models such as logistic regression.

The biggest weakness of decision trees is overfitting. A tree allowed to grow without limits can deepen enough to memorize even every bit of noise in the training data; it then looks flawless in training but fails on data it has not seen before. When only a single sample remains at a leaf, the tree has stopped learning and started memorizing.

The main way to manage this risk is pruning. Pre-pruning limits the tree as it grows: it stops early by setting a maximum depth, a minimum sample count at a node, or a minimum information gain threshold. Post-pruning grows the tree fully and then simplifies it by cutting branches that add nothing to generalization. Pruning both improves accuracy and preserves the interpretability advantage by keeping the tree more readable. This approach is a simple but effective example of the more general deep learning idea of regularization.

A single decision tree is intuitive but fragile: a small change in the data can completely change the tree's structure. To overcome this instability, ensemble methods were developed, and the best known is random forest.

Random forest trains many decision trees with randomization: each tree is grown on a random subset of the data and a random subset of the features. At prediction time all trees vote and the majority (or the average) wins. This "wisdom of the crowd" balances the errors of individual trees against each other, producing a much more accurate and stable model. Gradient boosting is a different ensemble strategy: it adds trees sequentially, each new tree focusing on correcting the errors of the previous ones.

Comparison of a single decision tree and random forest

Dimension	Single Decision Tree	Random Forest
Accuracy	Medium, fragile	High, stable
Overfitting	High risk	Markedly lower
Interpretability	Very high, single path readable	Low, hundreds of trees
Training cost	Low	Higher
Typical use	Simple models needing explanation	Tabular data where accuracy is priority

The trade-off is clear: a single tree gains interpretability, random forest gains accuracy. In scenarios where justification is mandatory, such as credit scoring, a single tree or shallow trees may be preferred, while for tabular data problems where raw prediction accuracy is the priority, random forest and gradient boosting are the standard choice.

You do not have to code a decision tree from scratch today; mature libraries have standardized the job. In the Python ecosystem, scikit-learn provides ready CART-based decision tree and random forest implementations for both classification and regression; the default split criterion is usually gini, but you can switch to information gain (entropy) with a single line. When higher accuracy is needed, gradient boosting libraries such as XGBoost, LightGBM, and CatBoost combine many decision trees sequentially and frequently top tabular-data competitions.

A practical setup order is as follows: first split the data into train and test, build a baseline with a single decision tree, apply pruning by tuning the tree's depth and the minimum samples per leaf, then compare this baseline against random forest or gradient boosting. This staged approach lets you both understand how the model decides and consciously set the balance between accuracy and interpretability. Taking a model to production then brings in MLOps practices — versioning, monitoring, and retraining. To learn these steps end to end and hands-on, see the training programs.

Decision trees and derived ensemble models are common especially in sectors dominated by tabular (row-column) data. In banking and finance, credit scoring, fraud detection (see anomaly detection), and customer churn prediction are typical applications; in these areas both accuracy and the ability to justify the decision are critical.

In healthcare, patient risk classification; in retail, demand forecasting and customer segmentation; and in manufacturing, quality control and failure prediction are addressed with decision-tree-based models. In the Türkiye context, compliance with KVKK is decisive for such decisions involving personal data: being able to present the justification for an automated decision is not only a technical but a legal requirement. The natural interpretability of the decision tree turns into an advantage exactly at this point.

You see the answer to what a decision tree is more clearly when you compare it with a popular alternative — the neural network. Both do classification and regression in supervised learning, but their philosophies are opposite. A decision tree produces explicit, human-readable rules; a neural network learns a representation spread across millions of weights that cannot be read directly.

This distinction drives a practical choice. For problems with tabular (row-column) data, a small number of meaningful features, and a mandatory justification for the decision, decision trees and derived ensemble models are usually both more accurate and more defensible. In contrast, for raw, high-dimensional data such as images, audio, and text, neural networks and deep learning are far ahead; because for these data types the model itself must learn meaningful features rather than having them defined by hand. So the answer to "which is better" depends on the data type and the need for explainability; the decision tree is the natural choice for tabular and regulated scenarios.

Although the decision tree is powerful and intuitive, it has limits to be aware of. The best known is the overfitting we discussed earlier; but that is not the only issue.

• Instability: A small change in the training data can produce a completely different tree structure. A single tree may therefore not be robust enough to build critical decisions upon.
• Axis-parallel splits: Decision trees split data with vertical/horizontal thresholds; capturing diagonal (non-linear) boundaries requires many splits, which grows the tree unnecessarily.
• Sensitivity to imbalanced data: If one class dominates, the tree tends to ignore the minority class; class weighting may be needed.
• Coarseness on continuous targets: Regression trees produce stepwise (staircase) predictions; they are not as elegant as linear methods at modeling very smooth relationships.

Most of these limits are mitigated by using an ensemble (random forest, gradient boosting) instead of a single tree, or through pruning. The right tool choice depends on whether the problem prioritizes accuracy or explainability.

What is the difference between a decision tree and random forest?

A decision tree is a single tree and is prone to overfitting on its own. Random forest is an ensemble method that trains many decision trees on random subsets of data and features and combines their votes. Random forest is usually more accurate and more stable, but cannot be interpreted as easily as a single tree.

What is the difference between information gain and gini?

Both measure how well a split separates the data. Information gain is based on entropy and computes how much disorder is reduced after the split; gini impurity measures the probability of misclassifying a randomly chosen sample. In practice their results are very similar; gini is slightly faster to compute.

Why is pruning necessary?

A decision tree grown without limits memorizes the training data and fails on new data; this is called overfitting. Pruning simplifies the model by cutting branches that rest on very few samples or add nothing to generalization. This improves both accuracy and interpretability.

Does a decision tree do classification or regression?

It does both. If the leaf node produces a category (for example 'loan approved/rejected') it is a classification tree; if it produces a continuous number (for example the price of a home) it is a regression tree. The split criterion changes accordingly: gini/information gain for classification, variance reduction for regression.

Why are decision trees considered interpretable?

Because the reason for a prediction can be read directly: the path from root to leaf turns into explicit rules like 'if income > X and age < Y then approve'. Unlike neural networks it is not a black box. This transparency makes decision trees valuable for explainable AI and regulated sectors.

In short, the answer to what is a decision tree is: a rule-based, interpretable machine learning model that splits data with successive yes/no questions and produces a prediction at each leaf. Split decisions are made with information gain or gini; overfitting is managed with pruning; and when accuracy is needed, it is strengthened with ensemble methods such as random forest and gradient boosting. For the basics see the what is machine learning and what is an algorithm guides, start with AI consulting for an enterprise AI roadmap, and visit the learning center to strengthen the fundamentals.