What gradient descent is
Gradient descent updates the model’s parameters (weights and bias) with a short loop. Wrongness is height on a hill: you measure how high you are, feel which way is uphill, step the other way, and repeat until the hill flattens.
Same sold-house story as Your First Learning Model. Same five-house model as Linear Regression. Walk the four steps below, then watch them on the bowl chart.
- 1. Calculate the error
- The model predicts on training data. A cost function (here MSE) measures how wrong those predictions are — the height on the hill.
- 2. Compute the gradient
- Find the slope of that error. The gradient always points uphill — toward more error.
- 3. Step downhill
- Adjust parameters in the opposite direction of the gradient (the negative gradient), scaled by the learning rate, so error falls.
- 4. Repeat
- Iterate until the slope is near zero — the model has converged near the lowest error it can reach with this shape.
Intuition
One weight, one downhill step
Start with one house and one weight — the chart you can actually see. Height is loss; left-to-right is the weight. The curve is the bowl. The dot is you. Each click walks error → gradient → downhill step → repeat.
Weight is 0. Predict $0 for a house that sold for $295,000. The cost (error) is huge — you are high on the rim of the bowl.
What happens in this step
sqft = 1500 actual = 295000 weight = 0 predict = 1500 × 0 = 0 error = 295000 − 0 = 295000
Bridge
Many knobs, same downhill rule
Vectors and Weights packed three features into a list. Linear Regression trained three weights plus a bias on all five houses. That was not a new idea — it was this same downhill step, on every knob, every house, every epoch, until MSE fell.
You cannot draw a 4D bowl. You still nudge each weight downhill on MSE. Same walk; more dimensions.
w += (actual − pred) × x × lr. Frameworks often write w -= (pred − actual) × x × lr. Flip the error sign and the updates match. You already ran gradient descent.Solution in TypeScript
step is one downhill nudge on one house. train repeats it over epochs. With lr = 1e-8 you recover the honest Linear Regression finish — ~[189, 0.93, 27] and bias ~0.69.
type Vector = number[];
const HOUSES: [Vector, number][] = [
[[1200, 2, 15], 245_000],
[[1800, 3, 8], 310_000],
[[2200, 4, 3], 420_000],
[[ 900, 2, 40], 180_000],
[[1500, 3, 10], 295_000],
];
class LinearModelGD {
weights: Vector;
bias = 0;
constructor(features: number) {
this.weights = Array(features).fill(0);
}
predict(x: Vector): number {
return x.reduce((s, xi, i) => s + xi * this.weights[i], 0) + this.bias;
}
/** One downhill step on one house. */
step(x: Vector, y: number, lr: number): void {
const pred = this.predict(x);
const error = pred - y; // uphill direction for MSE
for (let i = 0; i < this.weights.length; i++) {
this.weights[i] -= lr * error * x[i];
}
this.bias -= lr * error;
}
}
function mse(model: LinearModelGD, data: [Vector, number][]): number {
return (
data.reduce((sum, [x, y]) => sum + (model.predict(x) - y) ** 2, 0) /
data.length
);
}
function train(
model: LinearModelGD,
data: [Vector, number][],
epochs: number,
lr: number,
): void {
for (let epoch = 0; epoch < epochs; epoch++) {
for (const [x, y] of data) {
model.step(x, y, lr);
}
}
}
const model = new LinearModelGD(3);
train(model, HOUSES, 5000, 1e-8);
console.log(model.weights.map((w) => w.toFixed(2))); // ~[189, 0.93, 27]
console.log(model.bias.toFixed(2)); // ~0.69Dial
Step size matters
Same hill. Different shoe size. The chart below follows the just-right run (1e-8) — MSE falling across epochs. The cells compare three dials at once.
Now the model has three weights plus bias on all five houses. Gradient descent is unchanged — only the learning rate (step size) changes.
What happens in this step
All three runs begin at weights = [0, 0, 0], bias = 0 MSE ≈ 90.4 billion
| Learning rate | On HOUSES |
|---|---|
1e-10 | Crawls — still high MSE after many epochs |
1e-8 | Smooth — same finish as Linear Regression |
1e-6 | Explodes — MSE jumps up on epoch 1 |
Detail
How many houses before a nudge?
An epoch is one full pass over the five houses. A batch is how many you look at before one weight update. Same downhill step — different rhythm. Step through the three strategies.
- in the batch / updating
All five sold houses are the training set. An epoch means you will visit every house once. The question is: how many before you nudge the weights?
| Strategy | Houses before a nudge | Updates / epoch on HOUSES |
|---|---|---|
| SGD | 1 | 5 |
| Mini-batch | e.g. 2 | 3 (2 + 2 + 1) |
| Full batch | all 5 | 1 |
Next
Model stays linear; search is named
Build → train → infer is unchanged from the
Learning overview. Training is now explicitly “walk downhill on loss.” Inference still freezes the weights and only calls predict.
Trail so far: First Learning Model → Vectors and Weights → Linear Regression → Gradient Descent → Single Neuron.