What a neural network is
A neural network stacks neurons into layers. Information flows forward for a prediction; error flows backward so every weight can take a gradient-descent step.
One neuron could not learn XOR. With a hidden layer of ReLU neurons plus a linear output, the same four GD beats — error → gradient → downhill → repeat — unlock patterns with no single straight boundary.
- Layer
- A group of neurons that all see the same inputs and produce a vector of outputs.
- Hidden layer
- Middle layer(s) with activations (ReLU here). This is where the bend lives.
- Forward pass
- Input → hidden → output. One prediction.
- Backpropagation
- Send output error backward through each layer so every weight knows its share of the blame.
Build model
Architecture for this lesson
Smallest proof that hidden layers matter: XOR — 2 inputs → 4 hidden (ReLU) → 1 output (linear). Later you point the same code at HOUSES with 3 inputs and a bigger hidden layer.
- in play
XOR uses two bits. For houses you would plug in [sqft, beds, age] the same way — only the input size changes.
Forward
Forward pass
Data only moves input → hidden → output:
- current hop
Want target 1. Fresh network (seed 1) has not learned yet.
Backward
Backpropagation — blame flows back
After the forward pass, compare prediction to target. Then walk the network in reverse so every weight gets a downhill nudge.
- 1. Calculate the error
- At the output:
target − prediction(or the opposite sign — same as Gradient Descent). - 2. Update the output layer
- Output is linear — same update as a one-layer model, using hidden activations as its “features.”
- 3. Chain into the hidden layer
- Each hidden neuron’s error = (how much it fed the output) × ReLU derivative (0 if it was off).
- 4. Update hidden weights
- Same GD step on W₁ and b₁. Repeat for every example, every epoch.
- updating
Prediction was too low or too high. Nudge output weights and bias — they touch the answer directly.
Solution in TypeScript
Layer can be ReLU or linear. NeuralNetwork wires hidden → output. The training loop is forward, then the backward updates above. Seed 1 makes the printed XOR result reproducible on this page.
type Vector = number[];
type Matrix = number[][]; // rows = neurons, cols = inputs
function relu(x: number): number {
return Math.max(0, x);
}
function reluDeriv(x: number): number {
return x > 0 ? 1 : 0;
}
/** Mulberry32 — fixed seed so this lesson’s numbers are reproducible. */
function mulberry32(seed: number): () => number {
return () => {
seed |= 0;
seed = (seed + 0x6d2b79f5) | 0;
let t = Math.imul(seed ^ (seed >>> 15), 1 | seed);
t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
};
}
class Layer {
weights: Matrix;
biases: Vector;
activation: "relu" | "linear";
constructor(
inputSize: number,
outputSize: number,
activation: "relu" | "linear",
rnd: () => number,
) {
this.activation = activation;
this.weights = Array.from({ length: outputSize }, () =>
Array.from({ length: inputSize }, () => rnd() * 0.5 - 0.25),
);
this.biases = Array(outputSize).fill(0);
}
forward(inputs: Vector): { outputs: Vector; preActivations: Vector } {
const preActivations: Vector = [];
const outputs: Vector = [];
for (let j = 0; j < this.weights.length; j++) {
let sum = this.biases[j];
for (let i = 0; i < inputs.length; i++) {
sum += inputs[i] * this.weights[j][i];
}
preActivations.push(sum);
outputs.push(this.activation === "relu" ? relu(sum) : sum);
}
return { outputs, preActivations };
}
}
class NeuralNetwork {
hidden: Layer;
output: Layer;
constructor(inputSize: number, hiddenSize: number, seed = 1) {
const rnd = mulberry32(seed);
this.hidden = new Layer(inputSize, hiddenSize, "relu", rnd);
this.output = new Layer(hiddenSize, 1, "linear", rnd);
}
forward(inputs: Vector): number {
const h = this.hidden.forward(inputs);
const o = this.output.forward(h.outputs);
return o.outputs[0];
}
}
const xor: [Vector, number][] = [
[[0, 0], 0],
[[0, 1], 1],
[[1, 0], 1],
[[1, 1], 0],
];
const net = new NeuralNetwork(2, 4);
const lr = 0.1;
for (let epoch = 0; epoch < 5000; epoch++) {
let loss = 0;
for (const [x, target] of xor) {
const h = net.hidden.forward(x);
const o = net.output.forward(h.outputs);
const pred = o.outputs[0];
const error = target - pred;
loss += error * error;
// output layer (linear) — same GD step as before
for (let j = 0; j < net.output.weights.length; j++) {
for (let i = 0; i < h.outputs.length; i++) {
net.output.weights[j][i] += error * h.outputs[i] * lr;
}
net.output.biases[j] += error * lr;
}
// hidden layer — chain error through output weights × ReLU derivative
const hiddenErrors: Vector = Array(h.outputs.length).fill(0);
for (let i = 0; i < h.outputs.length; i++) {
let sum = 0;
for (let j = 0; j < net.output.weights.length; j++) {
sum += net.output.weights[j][i] * error;
}
hiddenErrors[i] = sum * reluDeriv(h.preActivations[i]);
}
for (let j = 0; j < net.hidden.weights.length; j++) {
for (let i = 0; i < x.length; i++) {
net.hidden.weights[j][i] += hiddenErrors[j] * x[i] * lr;
}
net.hidden.biases[j] += hiddenErrors[j] * lr;
}
}
if (epoch % 1000 === 0) {
console.log(`epoch ${epoch} loss ${(loss / xor.length).toFixed(4)}`);
}
}
for (const [x, target] of xor) {
console.log(x, "→", net.forward(x).toFixed(3), `(want ${target})`);
}
// With seed 1: near-perfect XOR after a few thousand epochsTraining
Train XOR until it snaps
Four examples per epoch. Learning rate 0.1 is fine on this tiny problem (HOUSE prices still need tiny rates like 1e-8). Watch predictions lock onto 0/1.
Seed 1, fresh weights. Predictions are wrong — the network has not walked downhill yet.
What happens in this step
[0,0] → 0.000 (want 0) [0,1] → 0.059 (want 1) [1,0] → −0.005 (want 1) [1,1] → 0.059 (want 0) loss ≈ 0.45
Next
Point this at houses
Swap NeuralNetwork(2, 4) for NeuralNetwork(3, 8) (or 16), feed [sqft, beds, age], keep a linear output for price — or put a softmax head on three tiers in
Price Tier Classifier.