Logistic Regression
The classification problem
Regression predicts a number. Classification predicts a category. Is this email spam or not spam? Will this customer churn? Is this tumour malignant or benign?
You might think you could just use linear regression and threshold the output — if the prediction is above 0.5, call it positive. This actually works sometimes, but it breaks down badly when outliers are present (a single extreme data point can shift the decision boundary), and the raw outputs don't have a meaningful probabilistic interpretation.
Logistic regression solves both problems. It outputs a genuine probability between 0 and 1, uses a loss function designed for probabilities, and is robust to outliers in the input features.
The sigmoid function
Logistic regression starts exactly like linear regression — compute a weighted sum of inputs:z = w₀ + w₁x₁ + w₂x₂ + ... + wₙxₙ
Then instead of outputting z directly, it passes z through the sigmoid function:σ(z) = 1 / (1 + e⁻ᶻ)
The sigmoid squashes any real number into the range (0, 1). Highly positive values of z map to probabilities near 1. Highly negative values map near 0. At z = 0, the output is exactly 0.5.
The model's prediction is the probability that the example belongs to the positive class: P(y=1 | x) = σ(w · x)
Cross-entropy loss
Logistic regression does not use Mean Squared Error. When outputs are probabilities, MSE has a poor gradient landscape — it trains slowly. Instead, logistic regression uses cross-entropy loss (also called log loss):Loss = -[y × log(ŷ) + (1-y) × log(1-ŷ)]
where y is the true label (0 or 1) and ŷ is the predicted probability.
If the true label is 1 and the model predicts 0.99 → loss is tiny. If it predicts 0.01 → loss is very large (log(0.01) ≈ −4.6). The logarithm makes confident wrong answers extremely costly, pushing the model strongly towards correct confidence.
Note
gradient = (predicted - actual) × input. This is not a coincidence — it's a consequence of the sigmoid being the inverse of the log-odds. The mathematical elegance is why logistic regression has been the dominant linear classifier for decades.The decision boundary
The threshold at which you call something positive is a design choice. The default is 0.5 — if the model gives 50%+ probability of positive, predict positive.
But this is not always right. In medical screening, you might prefer 0.2 — better to flag 80% of healthy patients (false positives) than to miss a single cancer case (false negative). In fraud detection, you might use 0.9 — the cost of false positives (blocking a legitimate transaction) is high.
This tradeoff is visualised with the ROC curve (True Positive Rate vs. False Positive Rate at every possible threshold). The AUC-ROC score — the area under this curve — tells you how good the model is regardless of threshold choice.
Multiclass logistic regression
The binary version (spam or not spam) extends naturally to multiple classes via softmax. Instead of one sigmoid output, you compute one score per class and pass them all through softmax:softmax(zᵢ) = e^zᵢ / Σe^zⱼ
This produces a probability distribution over all classes — guaranteed to sum to 1. The predicted class is whichever has the highest probability. The loss is still cross-entropy, now summed over all classes.
Note
What's next
Logistic regression draws straight-line boundaries. Decision trees instead ask a series of yes/no questions about features, creating complex, non-linear boundaries. They're also naturally interpretable — you can read the tree and understand exactly why a prediction was made.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.