§ 1.5Module I

Steps in Developing a Machine Learning Application

1.5 — Steps in Developing a Machine Learning Application

Recall first. Put these in a plausible order: deploy, choose a metric, collect data, test once, fit a model, define the problem, monitor. Which step is most dangerous to skip?

The complete loop

A machine-learning application is an iterative socio-technical process, not just the line model.fit(X, y). A robust high-level workflow is:

define problem and metric
          ↓
collect, label, and understand data
          ↓
split data without leakage
          ↓
create a baseline and preprocessing pipeline
          ↓
fit candidate models on training data
          ↓
validate, diagnose, and tune
          ↓
run the untouched test evaluation once
          ↓
deploy → monitor → collect feedback → revisit

The arrows are not a one-way waterfall. Monitoring can reveal drift or bad labels and send the project back to data collection or problem definition.¹²

Step 1 — Define the problem and success

Write the task in operational terms:

What prediction or decision is needed?
Who will use it, and at what time?
What inputs are available then?
What counts as a mistake?
What metric and threshold reflect the real cost?
What constraints exist: latency, privacy, interpretability, safety, or budget?

A vague goal such as “use ML to improve student performance” is not testable. “Predict which students need a tutoring invitation before the first assessment, while keeping the false-invitation rate below a chosen level” is a defined starting point.

Step 2 — Collect and understand data

Identify sources, units, time range, permissions, missing values, label definitions, duplicates, and sampling process. Explore distributions and relationships before selecting a model. Confirm that the target is a meaningful proxy for the decision rather than merely convenient to record.

Labels can be expensive, delayed, subjective, or biased. A larger dataset does not automatically repair a poor label or a missing population.

Step 3 — Split before fitting transformations

Keep evaluation data separate from the information used to fit the model. A common arrangement is:

Training set: fit parameters.
Validation set or cross-validation: choose models and hyperparameters.
Test set: final estimate after choices are finished.

The exact split depends on sample size, groups, time, and task. For time-dependent data, a random split may allow future information into the past. For repeated observations from one entity, split by entity when the intended deployment is on unseen entities.

Preprocessing belongs inside the split-aware workflow. If a scaler or imputer computes statistics from the entire dataset before splitting, the test set has influenced the training process. A pipeline keeps transformations and estimators together and helps prevent this leakage.¹

Step 4 — Build a baseline

A baseline is a simple reference that must be beaten for complexity to be justified:

predict the training mean for regression;
predict the majority class for classification;
use an existing rule or current human process;
use a simple model with interpretable features.

A model with 95% accuracy is not impressive if a majority-class baseline achieves 96%.

Step 5 — Choose features and a model

Select representations that are available at inference time. Start with a model whose capacity and assumptions fit the data and whose behavior can be diagnosed. Feature engineering, transformations, and model choice are hypotheses about what structure matters; they are not free improvements.

Step 6 — Fit and validate

Fit parameters using training data. Use validation or cross-validation to compare candidates, tune hyperparameters, inspect errors, and check whether performance is stable across meaningful subgroups or time periods. Do not repeatedly tune against the final test set: then the test set becomes part of model selection.

Step 7 — Final test and deployment

After the design is frozen, evaluate once on the untouched test set and report uncertainty and relevant slices where possible. Before deployment, decide what happens when the model is uncertain, wrong, unavailable, or outside its training distribution.

Deployment needs monitoring for:

input and label distribution drift;
performance decay;
data pipeline failures;
fairness or subgroup gaps;
latency, cost, and user behavior;
feedback loops caused by the model’s own decisions.

Worked example — predicting exam scores

Goal: estimate a student’s final score early enough to offer support.

Define: regression; metric MAE; prediction made after the first assessment.
Data: attendance and early-assessment features; exclude final-score information.
Split: use historical cohorts so a later cohort can act as a realistic test.
Baseline: predict the training-cohort mean score.
Pipeline: impute missing attendance using training data only, encode categories if needed, then fit a regression model.
Validation: compare a linear model with a more flexible model; inspect errors by cohort.
Test: evaluate once on the held-out cohort.
Deploy: monitor whether course format or student population changes; define a human review for uncertain predictions.

The workflow makes the model answerable: what it predicts, when it predicts it, how it is judged, and what happens after the number is produced.

Exercise

A team normalizes every row of a dataset, randomly splits it, tries 50 models, selects the one with the best test score, and reports that score as final performance. Identify at least three process errors.

Answer

Normalization before splitting can leak test-set statistics; the random split may be inappropriate for grouped or temporal data; selecting among 50 models using the test score overfits the test set, so it is no longer an untouched final estimate. The team should split appropriately, fit preprocessing on training folds only, use validation/cross-validation for selection, and reserve the test set for the final report.

Exam lens

Ten-step answer: define problem → collect/understand data → label → split → preprocess → baseline → choose model/features → train → validate/tune → final test → deploy/monitor. Combine steps when writing a short answer, but do not omit the separation between training, validation, and testing.

Common traps:

test data is not a second training set;
preprocessing can leak information;
a higher score is not automatically a better application;
deployment is not the end of the ML lifecycle.

Rapid revision

Can I explain the purpose of train, validation, and test data?
Can I identify when a random split is unsafe?
Can I explain why a baseline is necessary?
Can I name at least three things to monitor after deployment?

Sources

scikit-learn, “Getting Started.” https://scikit-learn.org/stable/getting_started.html — pipelines, train/test split, cross-validation, and leakage prevention. ↩ ↩²
scikit-learn, “Cross-validation: evaluating estimator performance.” https://scikit-learn.org/stable/modules/cross_validation.html — validation strategies and estimator evaluation. ↩