Logistic Regression

What This Is

Logistic regression is the classification sibling of linear regression. It uses the same linear score z = Xw + b but passes it through a sigmoid (or softmax, for multi-class) to produce a probability: p(y=1 | x) = σ(z).

The practical lesson is that logistic regression is an honest baseline whose coefficients, thresholds, and calibration are all things you can read directly. If a complicated classifier cannot beat it meaningfully, the complicated classifier is adding risk without value.

When You Use It

• a classification baseline before you try trees or deep models
• any task where you actually need a probability, not just a label
• any task where the coefficients need to be read and defended by a human
• tasks with lots of features but limited training examples — a well-regularized linear classifier is hard to beat in that regime
• text-bag-of-words and other very sparse problems — linear classifiers dominate this regime

Do Not Use It When

• the classes are not linearly separable in the feature space you have and you are unwilling to add feature interactions
• you need nonlinear shape and tree ensembles are available — trees will usually handle it
• the probability calibration story is more important than the decision and you are not willing to calibrate — combine with Calibration and Thresholds

Tooling

• LogisticRegression
• SGDClassifier(loss="log_loss") for very large problems
• Pipeline + StandardScaler
• predict_proba
• decision_function
• class_weight
• C — inverse regularization strength
• penalty="l1", penalty="l2", penalty="elasticnet"
• solver="liblinear", "saga", "lbfgs"
• multi_class="multinomial" for softmax

The Loss And The Gradient

Logistic regression minimizes log-loss (a.k.a. binary cross-entropy):

L(w) = -(1/n) Σ [ y_i log σ(z_i) + (1 - y_i) log (1 - σ(z_i)) ]

The gradient is beautifully clean:

∂L/∂w = (1/n) X^T (σ(z) - y)

That is the same structure as the linear-regression gradient — the residual, projected through the features — which is why gradient descent "just works" here as long as the features are scaled.

Regularization Cheat Sheet

Penalty	Effect	Use when
l2	shrinks all coefficients smoothly	default for well-behaved features
l1	drives some coefficients to zero	automatic feature selection, sparse inputs
elasticnet	mix of L1 and L2	correlated features where sparsity is still wanted

In scikit-learn C = 1 / λ, so smaller C = more regularization. This inversion trips people up regularly.

Minimal Example

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

model = make_pipeline(StandardScaler(), LogisticRegression())
model.fit(X_train, y_train)
probs = model.predict_proba(X_valid)[:, 1]

Scaling is not optional here. Without it the penalty hits features with large magnitudes and under-penalizes small-magnitude features — the coefficients you read are then artifacts of units.

Worked Pattern

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV

clf = LogisticRegression(max_iter=1000, solver="liblinear")
grid = GridSearchCV(
    clf,
    {"C": [0.01, 0.1, 1.0, 10.0, 100.0], "penalty": ["l1", "l2"]},
    cv=5,
    scoring="roc_auc",
)
grid.fit(X_train, y_train)

The logarithmic C grid is the right shape because the penalty is exponential in effect. Scoring by ROC-AUC separates "can the model rank positives above negatives" from "did we pick the right threshold" — pair with Calibration and Thresholds when the cutoff matters.

Multi-Class

For multi-class, logistic regression becomes softmax regression:

p(y=k | x) = exp(z_k) / Σ_j exp(z_j)

In scikit-learn, pass multi_class="multinomial" (or leave it on the default, which picks it automatically for most solvers) rather than the one-versus-rest fallback — the joint model is better calibrated when classes compete.

What To Inspect

• the ROC curve and AUC — the model's ranking ability independent of any threshold
• the calibration curve — is a score of 0.8 really about 80% positive?
• per-class precision/recall, not just overall accuracy
• coefficients on standardized features — which columns are pulling the decision
• class_weight="balanced" on imbalanced problems — does it move the minority recall
• the confusion matrix under two different operating points

Failure Pattern

Confusing C and λ. C = 1/λ, so lowering C is increasing regularization. Running a grid from [10, 100, 1000] when you meant "heavy regularization" produces the opposite effect.

A second failure is treating predict_proba as calibrated by default. For log-loss-trained linear models the calibration is usually close, but if you used class_weight="balanced" or heavy regularization the probabilities may be off — check the calibration curve and see Calibration and Thresholds.

A third failure is reading the coefficients before scaling. Without scaling, the coefficient on a dollar feature versus a percent feature is a unit artifact, not a signal.

Quick Checks

1. Is the pipeline scaling the features before the classifier?
2. Is max_iter high enough that the solver converges cleanly?
3. Is C being searched on a log grid?
4. Is the threshold you are using the one predict chose, or the one the cost structure demands?
5. Did you look at the calibration curve, not only the accuracy?

Practice

1. Fit logistic regression on a toy 2D binary problem and plot the linear decision boundary.
2. Compare C = 0.01, C = 1, and C = 100 on the same split. Note what the coefficients do.
3. Switch penalty="l2" to penalty="l1" and count how many coefficients go to zero.
4. Add a second class and compare multinomial logistic regression against one-versus-rest.
5. Measure the calibration curve with CalibrationDisplay and explain what you see.
6. Move the decision threshold from 0.5 to 0.3 and explain which cost structure that implies.
7. Explain why scaling matters for regularized logistic regression.
8. Describe one signal that class_weight="balanced" is helping and one signal that it is distorting the probabilities.
9. Explain why log-loss is the natural loss function for a probabilistic classifier.
10. State one problem for which logistic regression will not be competitive no matter how well you regularize.

Longer Connection

Logistic regression sits next to:

• Calibration and Thresholds — how to turn scores into decisions
• Evaluation Metrics Deep Dive — the metrics to trust for classifiers
• Feature Selection — L1 regularization as a selection lens
• SVM Margins and Kernels — the margin-based alternative with different trade-offs
• Linear Regression — same gradient structure, continuous target

Logistic regression is the honest default. When it wins, you ship it. When it loses to a more complex model, you now know exactly what the complex model is buying you.