Machine Learning Basics

1.1 — Machine Learning Basics

Recall first. Imagine you must write a program that detects spam email. What rules would you write, and what happens when spam changes its wording? Write two sentences before reading.

What problem does machine learning solve?

Traditional programming gives a computer explicit rules:

rules + input data → output

Machine learning instead gives a learning procedure examples and lets it infer a model:

examples + correct answers (when available) → learned model
learned model + new input → prediction

The model is not a human-like understanding of the problem. It is a parameterized pattern that is useful for a defined task and performance measure. The central question is not “did the model memorize the training records?” but “does it make useful predictions on new cases?”¹²

A precise definition

Tom Mitchell’s standard formulation is:

A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P if its performance at tasks in T, as measured by P, improves with experience E.¹

This definition prevents a vague answer such as “the computer learns automatically.” Every ML problem should name all three parts:

If the system improves only on the records it already saw, that is not enough. The experience must support performance on the intended task.

The vocabulary of one dataset

Suppose a dataset contains one row per student:

• A sample / instance / example is one row.
• A feature / attribute / predictor is an input used to make a prediction: attendance, study-hours, or previous-score.
• A target / label / response is what the model is asked to predict: final-score.
• A model is the learned mapping from features to a prediction.
• A parameter is learned from data, such as a regression coefficient.
• A hyperparameter is chosen by the practitioner or a search procedure, such as a tree’s maximum depth.
• A loss function measures how bad a prediction is during training.
• Training / fitting is the process of using data to choose model parameters.
• Inference / prediction is using the fitted model on new input.

In matrix notation, X usually stores features and y stores targets. A supervised dataset is often written as:

D = {(x₁, y₁), (x₂, y₂), ..., (xₙ, yₙ)}

where xᵢ is the feature vector for example i and yᵢ is its target.³

The ML workflow in miniature

data → representation/features → learning algorithm → model
                                                ↓
new features ───────────────────────────────→ prediction

The algorithm is the general procedure (for example, linear regression or a decision-tree learner). The model is the result after that procedure has been fitted to a particular dataset. Two datasets passed through the same algorithm can produce different models.

A practical library such as scikit-learn separates fitting and prediction: an estimator is fitted on X and y, then used to predict targets for new X. For unsupervised estimators, a target y may not be supplied.³

What machine learning is—and is not

ML is useful when:

• the task contains recurring patterns;
• examples or measurements are available;
• a useful performance measure can be defined; and
• writing complete rules by hand is difficult or too costly.

ML does not automatically provide:

• causal explanations — prediction and causation are different questions;
• correct labels — the model can learn systematic labelling mistakes;
• fairness — historical data can encode unequal treatment;
• robustness to future conditions — deployment data can differ from training data; or
• a reason to use a complicated model when a clear rule or ordinary database is sufficient.

These are issues to diagnose throughout Module I, not promises made by the word “learning.”

Worked example — specify the learning problem

A college wants to predict whether a student will pass a course.

1. Task: binary classification — predict pass or fail.
2. Experience: past student records containing permitted features and the final outcome.
3. Performance: for example, recall for the fail class if missing an at-risk student is costly.
4. Input: attendance and early-assessment features available before the prediction is made.
5. Output: a predicted class or a probability of passing.

Notice the last point: a feature such as the final examination score would make the task look accurate but would be unavailable at prediction time. It would also leak the answer into the inputs.

Exercise

For each scenario, name the likely task, one feature, the target (if any), and one sensible performance measure.

1. Predict tomorrow’s electricity demand from weather and calendar data.
2. Group news articles by topic when no topic labels are provided.
3. Decide whether a transaction is fraudulent.

Exam lens

Definition: Machine learning is the study of algorithms that improve performance on a task through experience, where the task, experience, and performance measure are made explicit.

Do not confuse:

• algorithm vs. fitted model;
• feature vs. target;
• parameter vs. hyperparameter;
• training/fitting vs. inference/prediction;
• prediction vs. causal explanation.

Long-answer skeleton: define ML using T, E, and P; describe X and y; show training and prediction; then state why unseen-data performance matters.

Rapid revision

• Can I give Mitchell’s T–E–P definition in my own words?
• Can I label features, targets, parameters, and hyperparameters in a table?
• Can I explain the difference between an algorithm and a model?
• Can I state one limitation of ML that is not solved by more model complexity?

Sources