AI-Supervised Learning

Supervised Learning

Objectives

• Be able to correctly identify supervised learning problems.
• Learn how to formulate supervised learning problems.
• Understand what inputs and outputs are passed to learning algorithms.
• Identify how the performance of such algorithms can be measured and select suitable metrics.
• Understand the theoretical limitations of supervised learning.

Introduction

Computer programs are written to precise rules, but we live in a world where the rules are unclear, changing, or noisy. In many cases, seeing examples can make learning easier or more successful.

Supervised learning (SL) is learning by example. SL is provided with input-output pairs (examples) and tries to map inputs to outputs. For example:

• **Cat or Dog:** Inputs are photos, outputs are labels (e.g., "Cat" or "Dog").
• **Fraudulent Bank Transactions:** Inputs are transaction details, outputs indicate whether the transaction is fraudulent.
• **Speech Recognition:** Inputs are audio files, outputs are transcribed text.

Mathematical Formulation

The inputs to the algorithm are denoted by \( x \), and the corresponding outputs are \( y \). These are typically high-dimensional vectors or matrices. The set of all possible inputs and outputs are denoted as \( X \) and \( Y \), respectively. Provided examples (\( D \)) can be expressed as:

\[ D = \{(x_1, y_1), (x_2, y_2), \dots, (x_n, y_n)\} \]

Classification and Regression

• **Classification:** When \( Y \) is finite and small, we classify the problem as classification. Example: Predicting the animal in a picture.
• **Regression:** When predicting a number or vector, it is a regression problem. Example: Predicting stock prices based on the time of day.

Relationship Between Classification and Regression

• **Classification as Regression:** Data points can be separated using a line or plane, allowing regression solutions to separate classes.
• **Regression as Classification:** Continuous values (e.g., rent prices) can be discretized into buckets (e.g., £500–£1000, £1000–£1500).

Applications

1. 1. 1. Self-Driving Cars

1. **Data Collection:** Collect data from human-driven cars equipped with sensors (e.g., cameras, LIDAR). 2. **Labeling the Data:** Pair sensor data with actions taken by human drivers (e.g., steering angle, braking). 3. **Feature Extraction:** Extract features like road lane markings and traffic signs. 4. **Model Training:** Train a supervised learning model to map features to actions. 5. **Model Evaluation:** Test the model with unseen scenarios to ensure generalization. 6. **Deployment:** Deploy the model for real-world driving and fine-tune based on collected data.

1. 1. 1. Predicting Commute Time

Inputs:

• Start Time
• Day of the Week
• Weather Conditions
• Traffic Data
• Distance
• Route Taken
• Special Events
• Road Incidents

Outputs:

• Predicted commute time (continuous or discrete categories).
• Recommended departure time or route.

Metrics in Supervised Learning

The objective is to minimize a loss function \( L \) which quantifies the difference between predicted values \( \hat{y} \) and actual values \( y \):

• **Regression Metrics:**

 * **Mean Squared Error (MSE):** \( \text{MSE} = \frac{1}{n} \sum (y_i - \hat{y}_i)^2 \)
 * **Mean Absolute Error (MAE):** \( \text{MAE} = \frac{1}{n} \sum |y_i - \hat{y}_i| \)

• **Classification Metrics:**

 * **Accuracy:** \( \text{Accuracy} = \frac{\text{True Positives} + \text{True Negatives}}{\text{Total Predictions}} \)
 * **Precision and Recall:** Useful for imbalanced datasets.
   * \( \text{Precision} = \frac{\text{True Positives}}{\text{True Positives} + \text{False Positives}} \)
   * \( \text{Recall} = \frac{\text{True Positives}}{\text{True Positives} + \text{False Negatives}} \)
 * **F1 Score:** Balances precision and recall.

Overfitting and Generalization

• **Overfitting:** When a model performs well on training data but poorly on unseen data.
• **Generalization:** The ability of a model to perform well on unseen data. Achieved by splitting the dataset into training and testing subsets.

Theoretical Foundations

1. 1. 1. Approximation Theory

• **Weierstrass Approximation Theorem:** A continuous function can be approximated arbitrarily well with polynomials.
• **Neural Networks:** A neural network with one hidden layer can approximate any continuous function.

Summary

Supervised learning aims to learn mappings from inputs to outputs using labeled examples. Careful attention is needed to prevent overfitting and ensure generalization. This technique is widely used in fields like self-driving cars, predictive modeling, and classification problems.

References

• [Weierstrass Approximation Theorem](https://en.wikipedia.org/wiki/Weierstrass_approximation_theorem)
• [Neural Networks and Deep Learning](http://neuralnetworksanddeeplearning.com/chap4.html)