scikit-learn: the full lifecycle of a model

scikit-learn is “classic” machine learning: regressions, trees, random forests, SVMs, k-means. No GPU, no deep networks, and yet it solves most problems on tabular data. Its strength is a single interface: every model is used the same way.

The interface: fit, predict, score

from sklearn.linear_model import LogisticRegression

model = LogisticRegression()
model.fit(X_train, y_train)           # learn
y_pred = model.predict(X_test)        # predict
model.score(X_test, y_test)           # evaluate (accuracy by default)

Switching algorithms means changing one line. That’s what makes it quick to compare several approaches.

The golden rule: never evaluate on training data

A model that has “learned by heart” gets a perfect score on what it has already seen, and is useless on new data: that’s overfitting. So we set a test set aside, and only touch it at the very end.

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

stratify=y keeps the same class proportions in both sets, which is essential when one class is rare.

Pipelines: preprocessing is part of the model

Normalising the data before splitting it leaks information from the test set into training: the computed mean has “seen” the test data. A Pipeline solves this by chaining preprocessing and model, fitted on the training data only.

from sklearn.pipeline import make_pipeline
from sklearn.compose import make_column_transformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.ensemble import RandomForestClassifier

numeric = ["age", "fare", "family_size"]
categorical = ["sex", "pclass", "embarked"]

preprocessing = make_column_transformer(
    (make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), numeric),
    (make_pipeline(SimpleImputer(strategy="most_frequent"), OneHotEncoder(handle_unknown="ignore")), categorical),
)

model = make_pipeline(preprocessing, RandomForestClassifier(random_state=42))
model.fit(X_train, y_train)

All the preprocessing lives inside the model object: you save it, deploy it, and a new row of raw data goes through exactly the same steps.

Cross-validation and hyperparameter tuning

A single train/test split gives a score that depends on chance. Cross-validation trains and evaluates $k$ times on different splits:

from sklearn.model_selection import cross_val_score, GridSearchCV

scores = cross_val_score(model, X_train, y_train, cv=5)
print(scores.mean(), scores.std())

grid = {
    "randomforestclassifier__n_estimators": [100, 300],
    "randomforestclassifier__max_depth": [None, 5, 10],
}
search = GridSearchCV(model, grid, cv=5, n_jobs=-1)
search.fit(X_train, y_train)
search.best_params_

The test set stays untouched until the end: it’s used once, for the final score.

Choosing the right metric

Accuracy is misleading as soon as classes are imbalanced: a fraud detector that always answers “no fraud” reaches 99% on a dataset where 1% of transactions are fraudulent.

from sklearn.metrics import classification_report, confusion_matrix

print(classification_report(y_test, y_pred))   # precision, recall, F1 per class
confusion_matrix(y_test, y_pred)

• Precision: of everything the model flagged, how much was correct?
• Recall: of all the real cases, how many did the model find?

Choosing between the two is a business decision: missing a fraud and bothering an honest customer don’t cost the same.

Which algorithm to try first?

1. A simple model as a baseline: linear or logistic regression. It’s the direct continuation of my article on linear regression.
2. A random forest or gradient boosting (HistGradientBoostingClassifier): often the best choice on tabular data.
3. Only move to neural networks if the data calls for it: images, text, sound, very large volumes.

Project to validate the step

Take the Titanic notebook from the pandas step and:

1. build a complete Pipeline (preprocessing + model);
2. compare logistic regression, random forest and gradient boosting with cross-validation;
3. tune the best one’s hyperparameters with GridSearchCV;
4. report the final score on the test set, once, with the confusion matrix;
5. submit the predictions on Kaggle.

Resources

• The scikit-learn tutorials and user guide, among the best documentation in the ecosystem.
• The “Intermediate Machine Learning” course on Kaggle Learn, focused on pipelines and data leakage.