Lesson 18-Machine Learning Toolbox and Regression Analysis

Introduction

Machine learning has revolutionized the way we analyze data, make predictions, and automate decision-making. From recommending movies to detecting fraud, from diagnosing diseases to driving cars, machine learning algorithms are everywhere. And at the heart of many of these applications is regression analysis—the statistical technique for predicting continuous outcomes.

MATLAB provides a comprehensive ecosystem for machine learning through the Statistics and Machine Learning Toolbox. This toolbox, which we’ve touched on in previous tutorials, is MATLAB’s flagship for statistical and machine learning algorithms. It provides a unified framework for:

  • Supervised learning: Learning from labeled data (regression and classification)
  • Unsupervised learning: Discovering patterns in unlabeled data (clustering, dimensionality reduction)
  • Model selection and evaluation: Cross-validation, hyperparameter tuning, performance assessment

In this extensive tutorial, we will dive deep into MATLAB’s machine learning ecosystem, with a special focus on regression analysis. Regression is one of the most widely used techniques in machine learning and statistics. It’s used to predict continuous outcomes (like house prices, temperature, stock prices) based on input features. We’ll explore the theory behind regression, the various regression algorithms available in MATLAB, and—crucially—how to select, train, and evaluate regression models effectively.

We’ll cover simple linear regression, regularized regression (Lasso, Ridge, Elastic Net), tree-based methods (regression trees, random forests), Gaussian process regression, and even neural network regression. Along the way, we’ll explore the bias-variance tradeoff—one of the most fundamental concepts in machine learning—and understand why ensemble methods (like random forests) often outperform single models.

By the end of this article, you’ll have both the theoretical understanding and the practical MATLAB skills to build, evaluate, and deploy regression models for real-world problems. Whether you’re predicting house prices, modeling engine thermal behavior, or forecasting energy consumption, the techniques you’ll learn here will serve you well.

MATLAB’s Machine Learning Ecosystem: An Overview

MATLAB’s machine learning ecosystem is centered around the Statistics and Machine Learning Toolbox. This toolbox provides a comprehensive set of functions and apps for machine learning, covering the entire workflow from data preprocessing to model deployment.

The Statistics and Machine Learning Toolbox

The Statistics and Machine Learning Toolbox (which we’ll refer to as “Stats Toolbox” for brevity) is organized around several key functional areas:

Data Preprocessing (covered in Tutorial 17):

  • Handling missing data: ismissing(), fillmissing(), rmmissing()
  • Outlier detection and handling: isoutlier(), filloutliers()
  • Feature normalization: zscore(), mapstd(), mapminmax()
  • Feature selection: pca(), fscchi2(), fsrnif()

Supervised Learning:

  • Regression: fitlm(), fitrlinear(), fitrsvm(), fitrtree(), fitrensemble(), fitrgp(), fitrnet()
  • Classification: fitcsvm(), fitctree(), fitcensemble(), fitcnet(), fitcdiscr(), fitcnaivebayes()

Unsupervised Learning:

  • Clustering: kmeans(), hierarchical(), dbscan()
  • Dimensionality Reduction: pca(), tsne(), ppca()

Model Evaluation and Selection:

  • Cross-validation: cvpartition(), crossval(), kfoldLoss()
  • Performance metrics: loss(), predict(), resubLoss()
  • Hyperparameter tuning: bayesopt(), gridsearch()

Apps (Interactive Environments):

  • Regression Learner: Interactively train, compare, and export regression models
  • Classification Learner: Interactively train, compare, and export classification models
  • Cluster Analysis: Interactively perform clustering

The toolbox is designed around a consistent “workflow-centric” philosophy. Most functions follow a similar pattern:

  1. Prepare your data: Clean, normalize, and split into training and test sets
  2. Choose a model type: Select the appropriate algorithm for your problem
  3. Train the model: Use the fit* functions (e.g., fitlm() for linear regression)
  4. Evaluate the model: Use predict() to make predictions, loss() to compute error metrics
  5. Tune hyperparameters: Use cross-validation or Bayesian optimization to improve performance
  6. Deploy the model: Export to C/C++, generate MATLAB code, or use in production

This consistent workflow makes it easy to experiment with different algorithms and compare their performance.

Supervised vs. Unsupervised Learning in MATLAB

Machine learning is broadly categorized into supervised and unsupervised learning. Understanding the difference is fundamental to choosing the right algorithm for your problem.

Supervised Learning

In supervised learning, you have a labeled dataset: each training example consists of an input (features) and a desired output (label or target). The goal is to learn a function that maps inputs to outputs, which can then be used to predict outputs for new inputs.

Supervised learning is further divided into:

  • Regression: The output is continuous (e.g., predicting house prices, temperature, stock prices)
  • Classification: The output is discrete (e.g., spam/not spam, cat/dog/bird, malignant/benign)
% ============================================
% Supervised Learning Example: Regression
% ============================================

% Load sample data
load carsmall;
% Features: Horsepower, Weight
% Target: MPG (miles per gallon, continuous)

% Prepare data
X = [Horsepower, Weight];
y = MPG;

% Split into training and test sets (70% train, 30% test)
rng(42);
n = length(y);
train_idx = randsample(n, round(0.7 * n));
test_idx = setdiff(1:n, train_idx);

X_train = X(train_idx, :);
y_train = y(train_idx);
X_test = X(test_idx, :);
y_test = y(test_idx);

% Train a linear regression model
mdl = fitlm(X_train, y_train);

% Make predictions on test set
y_pred = predict(mdl, X_test);

% Evaluate the model
rmse = sqrt(mean((y_test - y_pred).^2));
r2 = 1 - sum((y_test - y_pred).^2) / sum((y_test - mean(y_test)).^2);

fprintf('RMSE: %.4f\n', rmse);
fprintf('R²: %.4f\n', r2);

% Visualize predictions vs. actual
figure('Name', 'Supervised Learning: Regression', 'Position', [100, 100, 800, 600]);
scatter(y_test, y_pred, 50, 'b', 'filled', 'MarkerFaceAlpha', 0.5);
hold on;
plot([min(y_test), max(y_test)], [min(y_test), max(y_test)], 'r-', 'LineWidth', 2);
xlabel('Actual MPG');
ylabel('Predicted MPG');
title('Actual vs. Predicted (Test Set)');
legend('Predictions', 'Perfect Prediction', 'Location', 'best');
grid on;

Unsupervised Learning

In unsupervised learning, you have an unlabeled dataset: you only have inputs (features), and no desired outputs. The goal is to discover patterns, structure, or relationships in the data.

Common unsupervised learning tasks include:

  • Clustering: Group similar data points together (e.g., customer segmentation)
  • Dimensionality reduction: Reduce the number of features while preserving information (e.g., PCA, t-SNE)
  • Anomaly detection: Identify unusual data points (e.g., fraud detection)
% ============================================
% Unsupervised Learning Example: Clustering
% ============================================

% Generate synthetic data (2 clusters)
rng(42);
cluster1 = randn(100, 2) + [0, 0];  % Cluster 1: centered at [0, 0]
cluster2 = randn(100, 2) + [5, 5];  % Cluster 2: centered at [5, 5]
X = [cluster1; cluster2];

% Perform k-means clustering (k=2)
[idx, centers] = kmeans(X, 2);

% Visualize the clusters
figure('Name', 'Unsupervised Learning: Clustering', 'Position', [100, 100, 800, 600]);
gscatter(X(:, 1), X(:, 2), idx, 'rb', 'o*');
hold on;
plot(centers(:, 1), centers(:, 2), 'kx', 'MarkerSize', 15, 'LineWidth', 3);
xlabel('Feature 1');
ylabel('Feature 2');
title('K-Means Clustering (k=2)');
legend('Cluster 1', 'Cluster 2', 'Centroids', 'Location', 'best');
grid on;

Key differences:

Supervised LearningUnsupervised Learning
Needs labeled data (inputs + outputs)Only needs inputs (no outputs)
Goal: Predict outputs for new inputsGoal: Discover patterns/structure
Examples: Regression, classificationExamples: Clustering, PCA
Evaluation: Use test set performanceEvaluation: Use intrinsic metrics (e.g., silhouette score)

In this tutorial, we’ll focus on supervised learning for regression. Let’s start with the simplest and most widely used regression algorithm: linear regression.

Regression Models in MATLAB: fitlm, fitrlinear, and fitrsvm

Regression models are used to predict continuous outcomes. MATLAB provides a wide variety of regression algorithms, each with its own strengths and weaknesses. Let’s explore the most commonly used ones.

Linear Regression: fitlm()

Linear regression is the simplest and most widely used regression algorithm. It models the relationship between the input features and the output as a linear function:

y=β0+β1x1+β2x2+⋯+βpxp+ϵy = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \dots + \beta_p x_p + \epsilon

Where:

  • yy is the output (response variable)
  • x1,x2,…,xpx_1, x_2, \dots, x_p are the input features
  • β0,β1,…,βp\beta_0, \beta_1, \dots, \beta_p are the model coefficients (parameters to learn)
  • ϵ\epsilon is the error term (assumed to be normally distributed with mean 0 and variance σ2\sigma^2)

In MATLAB, you train a linear regression model using fitlm().

% ============================================
% Linear Regression with fitlm()
% ============================================

% Load sample data
load carsmall;
% Features: Horsepower, Weight
% Target: MPG

% Create a table (easier for fitlm)
tbl = table(Horsepower, Weight, MPG);

% Train a linear regression model
% Formula: 'MPG ~ Horsepower + Weight' means:
%   MPG = beta0 + beta1*Horsepower + beta2*Weight
mdl = fitlm(tbl, 'MPG ~ Horsepower + Weight');

% Display the model summary
disp(mdl);

% The output shows:
% - Coefficients (beta0, beta1, beta2) and their p-values
% - R-squared (proportion of variance explained by the model)
% - Adjusted R-squared (adjusted for number of predictors)
% - F-statistic and p-value (test of overall significance)

Interpreting the Linear Regression Output

The output of fitlm() is a LinearModel object that contains a wealth of information. Let’s interpret the key parts:

% Extract coefficient information
coeff_table = mdl.Coefficients;
disp('========== COEFFICIENT TABLE ==========');
disp(coeff_table);

% The table has columns:
% - Estimate: The fitted coefficient (beta)
% - SE: Standard error of the coefficient
% - tStat: t-statistic for testing whether the coefficient = 0
% - pValue: p-value for the t-test

% Interpretation:
% 1. If pValue < 0.05, the coefficient is statistically significant
%    (i.e., the predictor is associated with the response)
% 2. The sign of the coefficient tells you the direction of the relationship:
%    - Positive: As predictor increases, response increases
%    - Negative: As predictor increases, response decreases
% 3. The magnitude of the coefficient tells you the strength of the relationship.

% Example interpretation for our model:
% Suppose the coefficient for Weight is -0.006
% Interpretation: For each additional pound of weight, MPG decreases by 0.006
% (i.e., heavier cars have lower MPG)

% Predict using the model
new_data = table(100, 3000, 'VariableNames', {'Horsepower', 'Weight'});
prediction = predict(mdl, new_data);
fprintf('Predicted MPG for Horsepower=100, Weight=3000: %.2f\n', prediction);

% Compute R-squared and Adjusted R-squared
r2 = mdl.Rsquared.Ordinary;
adj_r2 = mdl.Rsquared.Adjusted;
fprintf('R²: %.4f\n', r2);
fprintf('Adjusted R²: %.4f\n', adj_r2);

% The difference between R² and Adjusted R²:
% - R² always increases when you add more predictors (even useless ones)
% - Adjusted R² penalizes for adding predictors that don't improve the model
% - Use Adjusted R² for model comparison (higher is better)

Assessing Model Fit

How do you know if your linear regression model is any good? Several diagnostic tools can help:

% Diagnostic plots
figure('Name', 'Regression Diagnostics', 'Position', [100, 100, 1400, 800]);

% Subplot 1: Actual vs. Predicted
subplot(2, 3, 1);
predicted = predict(mdl, tbl);
scatter(tbl.MPG, predicted, 50, 'b', 'filled', 'MarkerFaceAlpha', 0.5);
hold on;
plot([min(tbl.MPG), max(tbl.MPG)], [min(tbl.MPG), max(tbl.MPG)], 'r-', 'LineWidth', 2);
xlabel('Actual MPG');
ylabel('Predicted MPG');
title('Actual vs. Predicted');
grid on;

% Subplot 2: Residuals vs. Predicted
subplot(2, 3, 2);
residuals = mdl.Residuals.Raw;
scatter(predicted, residuals, 50, 'b', 'filled', 'MarkerFaceAlpha', 0.5);
hold on;
plot([min(predicted), max(predicted)], [0, 0], 'r-', 'LineWidth', 2);
xlabel('Predicted MPG');
ylabel('Residuals');
title('Residuals vs. Predicted');
grid on;
% Ideally, residuals should be randomly scattered around 0 (no pattern)

% Subplot 3: Histogram of Residuals
subplot(2, 3, 3);
histogram(residuals, 15, 'Normalization', 'pdf', 'FaceColor', 'b', 'FaceAlpha', 0.5);
hold on;
x = linspace(min(residuals), max(residuals), 100)';
y = normpdf(x, 0, std(residuals));
plot(x, y, 'r-', 'LineWidth', 2);
xlabel('Residuals');
ylabel('Probability Density');
title('Histogram of Residuals (should be normal)');
grid on;

% Subplot 4: Q-Q Plot of Residuals
subplot(2, 3, 4);
qqplot(residuals);
title('Q-Q Plot of Residuals (should be straight line)');
grid on;

% Subplot 5: Cook's Distance (influence diagnostic)
subplot(2, 3, 5);
cooks_distance = mdl.Diagnostics.CooksDistance;
plot(cooks_distance, 'bo-', 'MarkerSize', 8, 'LineWidth', 2);
xlabel('Observation Index');
ylabel('Cook''s Distance');
title('Cook''s Distance (identifies influential points)');
grid on;

% Subplot 6: Coefficient Confidence Intervals
subplot(2, 3, 6);
coefCI = coefCI(mdl);  % 95% confidence intervals for coefficients
coefficients = mdl.Coefficients.Estimate;
errorbar(1:length(coefficients), coefficients, ...
    coefficients - coefCI(:, 1), coefCI(:, 2) - coefficients, ...
    'bo', 'MarkerSize', 8, 'LineWidth', 2);
xlabel('Coefficient Index');
ylabel('Coefficient Value');
title('Coefficients with 95% Confidence Intervals');
set(gca, 'XTick', 1:length(coefficients), 'XTickLabel', mdl.CoefficientNames);
xtickangle(45);
grid on;

Key diagnostics:

  1. Residuals vs. Predicted: Look for patterns. If there’s a pattern (e.g., residuals increase as predictions increase), the model might be misspecified.
  2. Histogram of Residuals / Q-Q Plot: Residuals should be approximately normally distributed.
  3. Cook’s Distance: Identifies influential observations (outliers that heavily influence the fitted model).
  4. R-squared: Proportion of variance explained. Higher is better, but don’t obsess over it.
  5. Adjusted R-squared: Like R-squared, but penalizes for including unnecessary predictors.

Linear Regression with High-Dimensional Data: fitrlinear()

When you have a large number of features (high-dimensional data), ordinary linear regression can overfit. The model might fit the training data perfectly but generalize poorly to new data. This is where regularized linear regression comes in.

MATLAB’s fitrlinear() function trains a linear regression model with regularization. Regularization adds a penalty term to the loss function to prevent overfitting.

We’ll discuss regularization in detail in the next section. For now, let’s see how to use fitrlinear():

% ============================================
% Linear Regression with Regularization: fitrlinear()
% ============================================

% Generate high-dimensional data (p > n scenario)
rng(42);
n = 100;   % Number of samples
p = 500;  % Number of features (much larger than n!)
X = randn(n, p);
true_coefficients = [2, -1, 1.5, zeros(1, p-3)]';  % Only first 3 features are relevant
y = X * true_coefficients + randn(n, 1) * 0.5;  % Add noise

% Split into training and test sets
train_idx = 1:round(0.7 * n);
test_idx = (round(0.7 * n) + 1):n;

X_train = X(train_idx, :);
y_train = y(train_idx);
X_test = X(test_idx, :);
y_test = y(test_idx);

% Method 1: Ordinary linear regression (will overfit!)
mdl_ols = fitlm(X_train, y_train);  % This will warn about rank-deficient matrix

% Method 2: Regularized linear regression (using fitrlinear)
% 'Lambda' controls the strength of regularization
% 'Learner' can be 'leastsquares', 'svm', etc.
mdl_regularized = fitrlinear(X_train, y_train, 'Lambda', 0.1);

% Compare models
y_pred_ols = predict(mdl_ols, X_test);
y_pred_reg = predict(mdl_regularized, X_test);

rmse_ols = sqrt(mean((y_test - y_pred_ols).^2));
rmse_reg = sqrt(mean((y_test - y_pred_reg).^2));

fprintf('========== MODEL COMPARISON ==========\n');
fprintf('OLS (no regularization): RMSE = %.4f\n', rmse_ols);
fprintf('Regularized: RMSE = %.4f\n', rmse_reg);
fprintf('Regularization improved RMSE by %.4f\n', rmse_ols - rmse_reg);

When to use fitrlinear():

  • When you have high-dimensional data (p > n)
  • When you suspect multicollinearity (features are highly correlated)
  • When you want to perform feature selection (Lasso regularization automatically sets some coefficients to 0)

Support Vector Regression: fitrsvm()

Support Vector Regression (SVR) is a regression algorithm based on Support Vector Machines (SVMs). While SVMs are more commonly used for classification, SVR is a powerful regression technique that’s particularly effective in high-dimensional spaces and when the data has a nonlinear relationship.

The key idea of SVR is to find a function that deviates from the actual values by at most epsilon (ϵ\epsilon), while being as flat as possible.

% ============================================
% Support Vector Regression with fitrsvm()
% ============================================

% Generate nonlinear data
rng(42);
n = 200;
X = linspace(-3, 3, n)';
y = X.^2 + randn(n, 1) * 0.5;  % y = x^2 + noise

% Split into training and test sets
train_idx = 1:round(0.7 * n);
test_idx = (round(0.7 * n) + 1):n;

X_train = X(train_idx);
y_train = y(train_idx);
X_test = X(test_idx);
y_test = y(test_idx);

% Train SVR model with RBF kernel (handles nonlinear relationships)
mdl_svr = fitrsvm(X_train, y_train, 'KernelFunction', 'rbf', ...
    'BoxConstraint', 1, 'Epsilon', 0.1);

% Predict on test set
y_pred_svr = predict(mdl_svr, X_test);

% Evaluate
rmse_svr = sqrt(mean((y_test - y_pred_svr).^2));
fprintf('SVR with RBF kernel: RMSE = %.4f\n', rmse_svr);

% Compare with linear regression (which will struggle with nonlinear data)
mdl_linear = fitlm(X_train, y_train);
y_pred_linear = predict(mdl_linear, X_test);
rmse_linear = sqrt(mean((y_test - y_pred_linear).^2));
fprintf('Linear regression: RMSE = %.4f\n', rmse_linear);

% Visualize
figure('Name', 'SVR vs. Linear Regression', 'Position', [100, 100, 1000, 400]);

subplot(1, 2, 1);
scatter(X_test, y_test, 50, 'b', 'filled', 'MarkerFaceAlpha', 0.5);
hold on;
plot(X_test, y_pred_svr, 'r-', 'LineWidth', 2);
xlabel('X');
ylabel('y');
title('SVR with RBF Kernel');
legend('Actual', 'Predicted', 'Location', 'best');
grid on;

subplot(1, 2, 2);
scatter(X_test, y_test, 50, 'b', 'filled', 'MarkerFaceAlpha', 0.5);
hold on;
plot(X_test, y_pred_linear, 'r-', 'LineWidth', 2);
xlabel('X');
ylabel('y');
title('Linear Regression');
legend('Actual', 'Predicted', 'Location', 'best');
grid on;

Key parameters for SVR:

  • KernelFunction: The kernel function to use. Options: 'linear', 'rbf' (Radial Basis Function, default), 'polynomial', 'sigmoid'.
  • BoxConstraint: The box constraint (C) controls the tradeoff between flatness and tolerating deviations larger than epsilon.
  • Epsilon: The epsilon parameter specifies the margin of tolerance where no penalty is given to errors.

When to use SVR:

  • When the data has a nonlinear relationship
  • When you have high-dimensional data
  • When you want a model that’s robust to outliers (SVR uses epsilon-insensitive loss)

In the next section, we’ll dive deeper into regularization—a fundamental technique for preventing overfitting in regression models.

Membership Required

You must be a member to access this content.

View Membership Levels

Already a member? Log in here
Share your love