One dataset, two ensembles

Gradient Boosting and Random Forest both combine many small decision trees — but the trees play completely different roles. Step through each side to see how a single tree fits into its ensemble, using the same six points and only depth-1 trees (stumps).

The shared toy dataset

	P1	P2	P3	P4	P5	P6
x	1	2	3	4	5	6
y	2	2	6	6	10	10

A three-level staircase. One stump can only make one cut — so no single tree can fit it. Mean of y = 6.

Notation — what every symbol means

x: The input feature. Just a number here (1…6) — in real problems it's a feature vector.
y: The ground truth — the observed target value that came with the dataset. It is never a model output; it's what both models are trying to reproduce. Training = making predictions close to y.
ŷ: A prediction (inference output) — "y-hat". Any value a tree or ensemble produces when you ask it about some x. The whole game is ŷ ≈ y.
x*: A test point — an x we query at inference time (here x* = 4). Marked with a star to stress it's a question we ask the trained model, not a training row.

F(x): The combined ensemble prediction — boosting's ŷ. F₀ is the flat baseline (mean of y = 6); each round produces the next version: F₁, F₂, … Fₖ = Fₖ₋₁ + η·Tₖ(x).
T₁, T₂…: The individual stumps, in training order. Tₖ is the tree fit in round k — and its target is not y but the residuals left over by Fₖ₋₁. So Tₖ(x) outputs a correction (e.g. −4 or +2), not a standalone guess at y. Alone, T₂(x) = +3 is meaningless; it only makes sense added onto F₁.
r: The residuals: r = y − F(x), the signed error of the current ensemble at each training point (red stems in the chart). This is the "shrinking target" each new tree is trained on.
η: The shrinkage / learning rate (0.5 here). Each tree's correction is deliberately damped before being added, so no single tree overcommits.

Tree A, B, C: Named with letters, not numbers, on purpose: there is no order. Each is a complete standalone model trained directly on (x, y) pairs from its own bootstrap sample — its output is a ŷ (e.g. Tree B: ŷ(4) = 5), unlike the boosting Tₖ whose output is only a correction. The forest's final ŷ is the plain average of the trees' ŷ values.

Inside a single tree — how a stump is actually computed

Both panels treat "fit one tree" as an atomic step. This section opens that box. A depth-1 tree (stump) is fully described by three numbers — a threshold t, a left-leaf value, a right-leaf value — and fitting one is the same brute-force search everywhere, whether the tree lives in a boosting ensemble, a forest, or stands alone:

Enumerate candidate thresholds. Only midpoints between consecutive distinct x values can change the grouping. With x = 1…6 there are exactly five candidates: 1.5, 2.5, 3.5, 4.5, 5.5.
Assign each leaf its best constant. Split the points into left (x < t) and right (x > t), and give each leaf the single value that best summarizes its side's targets.
Score the split with an impurity measure — how mixed each side still is, weighted by size — and keep the threshold with the best score, i.e. the biggest drop: impurity(parent) − weighted impurity(children).

The recipe is a template. Steps 2 and 3 have two standard instantiations, depending on what the target is:

Regression — targets are numbers (what both panels on this page do)

Leaf value: the mean of the side's targets — the constant minimizing squared error. Impurity: SSE = Σ(target − leaf mean)², so the search maximizes variance reduction.

Worked through on boosting round 1, where the targets are the residuals (−4, −4, 0, 0, 4, 4):

threshold t	left targets	left mean	right targets	right mean	SSE after split
1.5	−4	−4	−4, 0, 0, 4, 4	0.8	44.8
2.5	−4, −4	−4	0, 0, 4, 4	+2	16.0 ✓
3.5	−4, −4, 0	−2.67	0, 4, 4	+2.67	21.3
4.5	−4, −4, 0, 0	−2	4, 4	+4	16.0 (tie)
5.5	−4, −4, 0, 0, 4	−0.8	4, 4	+4	44.8

t = 2.5 and t = 4.5 genuinely tie at SSE 16; ties break leftmost, so T₁ = (t = 2.5, leaves −4 | +2) — exactly the stump in the boosting panel. Sequentiality makes the tie harmless: with 2.5 taken, round 2's residuals make 4.5 the clear winner. The forest trees run the identical search on raw y from their own bootstrap samples — which is why Tree A (never drew P3/P4) lands on t = 3.5 while B and C land on 4.5. Real forests add one more twist: each split considers only a random subset of features; with a single feature here there's nothing to randomize.

Classification — targets are classes (the version taught with entropy)

Leaf value: the side's majority class (or its class proportions, if you want probabilities). Impurity: entropy H = −Σ pᶜ·log₂ pᶜ over class proportions, so the search maximizes information gain — this is ID3/C4.5; CART's Gini index is a cheaper stand-in with the same intent.

Our toy y takes only three values (2, 6, 10), so the same six points double as a 3-class problem: parent entropy for 2+2+2 points of three classes is H = log₂3 ≈ 1.585 bits. Scoring the same five thresholds:

threshold t	H(left)	H(right)	weighted H after	information gain
1.5	0	1.522	1.268	0.317
2.5	0 (pure: all "2")	1.0	0.667	0.918 ✓
3.5	0.918	0.918	0.918	0.667
4.5	1.0	0 (pure: all "10")	0.667	0.918 (tie)
5.5	1.522	0	1.268	0.317

Identical ranking, identical 2.5 / 4.5 tie as the regression table — both criteria reward carving off a pure, homogeneous chunk. That's no coincidence: variance is to numbers what entropy is to classes (a Gaussian's entropy, ½·log(2πeσ²), grows monotonically with variance, so minimizing SSE is entropy minimization under a Gaussian view of each leaf). The panels above use the regression form because their targets are continuous — boosting's round-4 residuals like −0.875 have no class proportions to take logs of. Everything downstream of this section is unchanged by the choice: boosting still sums leaf outputs, the forest still averages them (majority-votes, in classification).

Watch a tree grow — recursive splitting, live

The tables above score one split. A real tree repeats that search inside every impure leaf until nothing is mixed anymore. Step through the full construction on the six points — and flip the criterion to see SSE and entropy steer the very same tree.

data space — where the cuts land

the tree being built

Gradient Boosting sequential · trees fix errors

F(x) = 6 + 0.5·T₁(x) + 0.5·T₂(x) + 0.5·T₃(x) + 0.5·T₄(x)

F₀
mean = 6

→residuals

T₁ · x < 2.5
−4 | +2

→residuals

T₂ · x < 4.5
−1.5 | +3

→residuals

T₃ · x < 4.5
−0.75 | +1.5

→residuals

T₄ · x < 2.5
−0.875 | +0.4375

Residual SSE:

7.75

2.69

0.96

Random Forest parallel · trees vote

Bootstrap A → P1 P2 P3 P4 P5×2 P6×2

Bootstrap B → P1 P2 P3 P4×2 P5×2 P6

Bootstrap C → P1 P2 P3 P4×2 P5 P6×2

Tree A

split x < 3.5

ŷ(4) = 10

Tree B

split x < 4.5

ŷ(4) = 5

Tree C

split x < 4.5

ŷ(4) = 6

No arrows between trees — none needs another to exist. They could all be trained at the same instant.

10 + 5 + 6 ── ÷ 3 ──▶

forest ŷ(4) = 7

true y at x = 4 is 6 — the average tames Tree A's wild 10

What just happened, side by side

Every row below refers to the numbers you stepped through above — not generic textbook claims.

	Gradient Boosting	Random Forest
Each tree is trained on…	The residuals of the current ensemble — a different target every round. T₁ saw (−4,−4,0,0,4,4); by round 4 the target had shrunk to (−0.875,−0.875,0.125,0.125,0.75,0.75).	The original y values (2,2,6,6,10,10), every time — but each tree sees a different bootstrap sample of the rows (Tree A never saw P3 or P4 at all).
Dependency between trees	Strictly sequential. T₂'s training target (−2,−2,−1,−1,3,3) literally does not exist until F₀ + 0.5·T₁ has been computed. Delete T₁ and every later tree is wrong.	None. Trees A, B, C were fit independently against the same y. Delete Tree B and the other two are unchanged — you'd just average 10 and 6 instead.
How trees combine	Shrinkage-weighted sum: F(x) = 6 + 0.5·T₁ + 0.5·T₂ + 0.5·T₃ + 0.5·T₄. Each term is a small correction added on top of the rest.	Plain average: (10 + 5 + 6) / 3 = 7. Each tree is a full standalone prediction; the forest is their mean.
Growing the ensemble mainly improves…	Bias. One stump can't fit a 3-step staircase, but stacked corrections drove the SSE 64 → 28 → 7.75 → 2.69 → 0.96. More rounds keep bending the model toward the data (and eventually overfit).	Variance. Individual trees swung wildly at x = 4 (predicting 10, 5, 6) depending on which rows their bootstrap happened to contain. Averaging smoothed that to a stable 7. More trees stabilize — they don't make the average more expressive.