Lesson 17-Statistical Analysis and Data Preprocessing Techniques

Introduction

Statistical analysis and data preprocessing are the unsung heroes of data science and machine learning. While much attention is focused on fancy algorithms—deep neural networks, support vector machines, ensemble methods—the reality is that the quality of your analysis depends far more on the quality of your data and how well you prepare it. As the old adage goes: “Garbage in, garbage out.” No matter how sophisticated your statistical method or machine learning algorithm, if your data is poorly preprocessed, your results will be unreliable, misleading, or outright wrong.

MATLAB provides a comprehensive ecosystem for statistical analysis and data preprocessing through the Statistics and Machine Learning Toolbox. This toolbox, which we’ll refer to as the “Stats Toolbox” for brevity, contains hundreds of functions for descriptive statistics, probability distributions, hypothesis testing, regression, multivariate analysis, and—crucially—data preprocessing. These preprocessing capabilities are what transform raw, messy, real-world data into the clean, structured datasets that machine learning algorithms need to perform well.

In this extensive tutorial, we will explore the full pipeline of statistical data analysis in MATLAB: from computing basic descriptive statistics, through probability distributions and hypothesis testing, to regression analysis and—most importantly—the often-overlooked but absolutely critical steps of data preprocessing. We’ll explore why data preprocessing is not just a “nice to have” but an absolute necessity, delve into the mathematical reasons why normalization dramatically improves the performance of gradient descent algorithms, and work through practical examples of preprocessing sensor data for machine learning. By the end of this article, you’ll have both the theoretical understanding and the practical MATLAB skills to prepare real-world data for statistical analysis and machine learning.

The Statistics and Machine Learning Toolbox: An Overview

The Statistics and Machine Learning Toolbox (often abbreviated as “Stats Toolbox” or simply “stats”) is one of MathWorks’ most comprehensive toolboxes. It provides functions for:

Descriptive Statistics: Functions to summarize and describe data: mean(), median(), std(), var(), prctile(), iqr(), moment(), and many more.

Probability Distributions: A complete framework for working with probability distributions: probability density functions (pdf), cumulative distribution functions (cdf), random number generation (random), and distribution fitting (fitdist).

Hypothesis Testing: Functions for statistical hypothesis tests: ttest(), ttest2(), chi2gof(), kstest(), anova1(), anova2(), friedman(), and more.

Regression Analysis: Functions for fitting regression models: fitlm() (linear regression), fitglm() (generalized linear models), stepwiselm() (stepwise regression), fitrsvm() (support vector regression), and many more.

Multivariate Analysis: Techniques for analyzing data with multiple variables: pca() (principal component analysis), factoran() (factor analysis), manova1() (multivariate ANOVA), classify() (discriminant analysis), and more.

Machine Learning: A comprehensive set of machine learning algorithms: fitcsvm() (support vector machines), fitctree() (classification trees), fitcensemble() (ensemble methods), fitcnet() (neural networks), and more.

Data Preprocessing: Functions for cleaning and preparing data: ismissing(), fillmissing(), rmmissing(), isoutlier(), filloutliers(), zscore(), mapstd(), mapminmax(), and more.

The toolbox is designed around a consistent set of conventions that make it easy to use. Most functions follow a “function-based” interface where you pass in your data and parameters, and the function returns results. Many functions also support a “object-oriented” interface where you create a “model object” that encapsulates the fitted model and provides methods for prediction, visualization, and further analysis.

For example, to fit a linear regression model, you can use the function-based interface:

% Load data
load carsmall;
% Fit linear regression: MPG = beta0 + beta1*Horsepower + beta2*Weight
mdl = fitlm([Horsepower, Weight], MPG);

% Display the results
disp(mdl);

Or you can use the object-oriented interface (which is the same in this case, since fitlm() returns a model object):

% Fit the model
mdl = fitlm([Horsepower, Weight], MPG);

% Use the model object for prediction
new_data = [150, 3000; 200, 4000];  % Two new cars
predictions = predict(mdl, new_data);

% Visualize the model
plot(mdl);

The Stats Toolbox is extensively documented, with detailed help for each function, examples, and “how-to” guides for common tasks. If you’re serious about data analysis in MATLAB, spending time with the Stats Toolbox documentation is an excellent investment.

In the following sections, we’ll explore each of these functional areas in depth, with a special focus on data preprocessing—the critical but often neglected step that determines the success or failure of your analysis.

Descriptive Statistics: Summarizing Data with mean, median, std, var, and prctile

Descriptive statistics are the foundation of data analysis. Before you can build models, test hypotheses, or draw conclusions, you need to understand your data. What are the central tendencies? How spread out is the data? Are there outliers? Descriptive statistics provide the answers to these questions.

Measures of Central Tendency: mean() and median()

The mean (arithmetic average) is perhaps the most familiar measure of central tendency. In MATLAB, you compute the mean using the mean() function.

% Generate some sample data
rng(42);  % Set random seed for reproducibility
data = randn(100, 1);  % 100 random numbers from standard normal distribution

% Compute the mean
mean_val = mean(data);
fprintf('Mean: %.4f\n', mean_val);

% For matrices, mean() operates column-wise by default
data_matrix = randn(100, 3);  % 100x3 matrix
mean_cols = mean(data_matrix);
fprintf('Mean of each column: %.4f, %.4f, %.4f\n', mean_cols(1), mean_cols(2), mean_cols(3));

% To compute the mean of the entire matrix, use 'all'
mean_all = mean(data_matrix, 'all');
fprintf('Mean of all elements: %.4f\n', mean_all);

The median is the middle value when the data is sorted. It’s more robust to outliers than the mean.

% Compute the median
median_val = median(data);
fprintf('Median: %.4f\n', median_val);

% Demonstrate robustness to outliers
data_with_outlier = [data; 1000];  % Add an extreme outlier
mean_with_outlier = mean(data_with_outlier);
median_with_outlier = median(data_with_outlier);

fprintf('With outlier - Mean: %.2f, Median: %.4f\n', mean_with_outlier, median_with_outlier);
% The mean is strongly affected by the outlier, but the median is not!

When to use mean vs. median:

  • Use the mean when your data is symmetric and doesn’t have outliers
  • Use the median when your data is skewed or has outliers
  • The median is more “robust” (less sensitive to extreme values)

Measures of Dispersion: std() and var()

The standard deviation (std()) measures how spread out the data is. A small standard deviation means the data points are close to the mean; a large standard deviation means they’re spread out.

The variance (var()) is the square of the standard deviation. It’s useful for mathematical derivations, but standard deviation is more interpretable because it’s in the same units as the data.

% Compute standard deviation and variance
std_val = std(data);
var_val = var(data);

fprintf('Standard Deviation: %.4f\n', std_val);
fprintf('Variance: %.4f\n', var_val);
fprintf('Verification: std^2 = %.4f (should equal variance)\n', std_val^2);

% For a sample (as opposed to a population), MATLAB uses n-1 in the denominator
% To use n (population formula), specify the second argument as 1
std_sample = std(data);        % Sample standard deviation (default)
std_population = std(data, 1);  % Population standard deviation
fprintf('Sample std: %.4f, Population std: %.4f\n', std_sample, std_population);

Understanding the difference between sample and population:

  • If your data is the entire population of interest, use std(data, 1) (divides by n)
  • If your data is a sample from a larger population, use std(data) (divides by n-1, which gives an unbiased estimator)
  • In practice, if you’re working with real-world data (which is almost always a sample), use the default std(data)

Percentiles and Quantiles: prctile() and quantile()

Percentiles divide the data into 100 equal parts. The 50th percentile is the median. The 25th and 75th percentiles are the first and third quartiles, respectively.

% Compute percentiles
p25 = prctile(data, 25);   % 25th percentile (first quartile)
p50 = prctile(data, 50);   % 50th percentile (median)
p75 = prctile(data, 75);   % 75th percentile (third quartile)
p95 = prctile(data, 95);   % 95th percentile

fprintf('25th percentile: %.4f\n', p25);
fprintf('50th percentile (median): %.4f\n', p50);
fprintf('75th percentile: %.4f\n', p75);
fprintf('95th percentile: %.4f\n', p95);

% Compute multiple percentiles at once
percentiles = [25, 50, 75, 90, 95, 99];
values = prctile(data, percentiles);
for i = 1:length(percentiles)
    fprintf('%dth percentile: %.4f\n', percentiles(i), values(i));
end

Quantiles are like percentiles but divide the data into any number of equal parts. The 0.25 quantile is the same as the 25th percentile.

% Compute quantiles
q1 = quantile(data, 0.25);  % First quartile
q2 = quantile(data, 0.50);  % Median (second quartile)
q3 = quantile(data, 0.75);  % Third quartile

fprintf('First quartile: %.4f\n', q1);
fprintf('Median: %.4f\n', q2);
fprintf('Third quartile: %.4f\n', q3);

The Interquartile Range: iqr()

The interquartile range (IQR) is the difference between the 75th and 25th percentiles. It’s a robust measure of dispersion that’s not affected by outliers.

% Compute the interquartile range
iqr_val = iqr(data);
fprintf('Interquartile Range: %.4f\n', iqr_val);

% The IQR is used in the "1.5 * IQR rule" for detecting outliers
% Any value below Q1 - 1.5*IQR or above Q3 + 1.5*IQR is considered an outlier
lower_bound = p25 - 1.5 * iqr_val;
upper_bound = p75 + 1.5 * iqr_val;

outliers = data(data < lower_bound | data > upper_bound);
fprintf('Number of outliers (1.5*IQR rule): %d\n', length(outliers));

Higher Moments: skewness() and kurtosis()

The skewness measures the asymmetry of the data distribution. A symmetric distribution has skewness 0. A distribution with a long right tail has positive skewness; a distribution with a long left tail has negative skewness.

The kurtosis measures the “tailedness” of the distribution. A normal distribution has kurtosis 3. A distribution with heavier tails than normal has kurtosis > 3; a distribution with lighter tails has kurtosis < 3.

% Compute skewness and kurtosis
skew_val = skewness(data);
kurt_val = kurtosis(data);

fprintf('Skewness: %.4f (0 = symmetric)\n', skew_val);
fprintf('Kurtosis: %.4f (3 = normal)\n', kurt_val);

% Generate data with different skewness and kurtosis
data_right_skewed = exprnd(1, 1000, 1);  % Exponential distribution (right-skewed)
data_left_skewed = -exprnd(1, 1000, 1);  % Negative exponential (left-skewed)
data_uniform = rand(1000, 1);  % Uniform distribution (lower kurtosis)

fprintf('\nExponential (right-skewed): Skewness = %.4f\n', skewness(data_right_skewed));
fprintf('Negative Exponential (left-skewed): Skewness = %.4f\n', skewness(data_left_skewed));
fprintf('Uniform (light tails): Kurtosis = %.4f\n', kurtosis(data_uniform));

Visualizing Descriptive Statistics

Descriptive statistics are much easier to interpret when visualized. MATLAB provides excellent plotting functions for data exploration.

% Generate sample data
rng(42);
data = [randn(100, 1); randn(100, 1) + 3];  % Bimodal distribution

% Create a figure with multiple plots
figure('Name', 'Descriptive Statistics Visualization', 'Position', [100, 100, 1400, 800]);

% Subplot 1: Histogram
subplot(2, 3, 1);
histogram(data, 20);
hold on;
% Overlay mean and median
x_mean = mean(data) * ones(1, 2);
y_mean = ylim();
plot(x_mean, y_mean, 'r-', 'LineWidth', 2);
x_median = median(data) * ones(1, 2);
plot(x_median, y_mean, 'g-', 'LineWidth', 2);
legend('Data', 'Mean', 'Median');
title('Histogram with Mean (red) and Median (green)');

% Subplot 2: Box Plot
subplot(2, 3, 2);
boxplot(data);
title('Box Plot');
% The box shows Q1, median, Q3. The whiskers show Q1 - 1.5*IQR and Q3 + 1.5*IQR.
% Points beyond the whiskers are plotted individually (outliers).

% Subplot 3: CDF Plot
subplot(2, 3, 3);
cdfplot(data);
title('Empirical CDF');

% Subplot 4: QQ Plot (Quantile-Quantile Plot)
subplot(2, 3, 4);
qqplot(data);
title('Q-Q Plot (against normal distribution)');
% If the data is normally distributed, the points will lie approximately on the red line.

% Subplot 5: Probability Density Plot
subplot(2, 3, 5);
pd = fitdist(data, 'Normal');  % Fit a normal distribution
x = linspace(min(data), max(data), 100)';
y = pdf(pd, x);
hold on;
histogram(data, 20, 'Normalization', 'pdf');
plot(x, y, 'r-', 'LineWidth', 2);
title('PDF: Histogram and Fitted Normal Distribution');
legend('Data (normalized)', 'Fitted Normal PDF');

% Subplot 6: Scatter Plot with Mean and Std
subplot(2, 3, 6);
data_sorted = sort(data);
cumulative = (1:length(data)) / length(data);
plot(data_sorted, cumulative, 'b-', 'LineWidth', 2);
hold on;
% Overlay mean +/- std
x_std1 = [mean(data) - std(data), mean(data) + std(data)];
y_std1 = [0.5, 0.5];
plot(x_std1, y_std1, 'r-', 'LineWidth', 3);
title('Cumulative Distribution with Mean +/- Std');

This comprehensive visualization gives you a complete picture of your data’s distribution. The histogram shows the shape, the box plot shows the quartiles and outliers, the CDF and QQ plots help assess normality, and the PDF plot shows the probability density.

Practical Example: Analyzing Sensor Data

Let’s put it all together with a practical example: analyzing data from a temperature sensor.

% ============================================
% Practical Example: Analyzing Sensor Data
% ============================================

% Generate synthetic sensor data (temperature readings in Celsius)
rng(42);
n_samples = 1000;
true_temperature = 25;  % True temperature: 25°C
noise = randn(n_samples, 1) * 0.5;  % Measurement noise (std = 0.5°C)
sensor_data = true_temperature + noise;

% Add some outliers (sensor glitches)
outlier_indices = randi(n_samples, [20, 1]);
sensor_data(outlier_indices) = sensor_data(outlier_indices) + 10;  % +10°C glitch

% Step 1: Compute descriptive statistics
fprintf('\n========== SENSOR DATA ANALYSIS ==========\n');
fprintf('Number of samples: %d\n', n_samples);
fprintf('Mean: %.4f°C\n', mean(sensor_data));
fprintf('Median: %.4f°C\n', median(sensor_data));
fprintf('Standard Deviation: %.4f°C\n', std(sensor_data));
fprintf('Variance: %.4f°C²\n', var(sensor_data));
fprintf('IQR: %.4f°C\n', iqr(sensor_data));
fprintf('Skewness: %.4f\n', skewness(sensor_data));
fprintf('Kurtosis: %.4f\n\n', kurtosis(sensor_data));

% Step 2: Detect outliers using the 1.5*IQR rule
Q1 = prctile(sensor_data, 25);
Q3 = prctile(sensor_data, 75);
IQR = Q3 - Q1;
lower_bound = Q1 - 1.5 * IQR;
upper_bound = Q3 + 1.5 * IQR;

outlier_mask = sensor_data < lower_bound | sensor_data > upper_bound;
n_outliers = sum(outlier_mask);
fprintf('Outliers detected (1.5*IQR rule): %d (%.2f%%)\n', ...
    n_outliers, 100 * n_outliers / n_samples);

% Step 3: Visualize the data
figure('Name', 'Sensor Data Analysis', 'Position', [100, 100, 1200, 800]);

subplot(2, 2, 1);
plot(sensor_data, 'b.', 'MarkerSize', 8);
hold on;
plot([1, n_samples], [mean(sensor_data), mean(sensor_data)], 'r-', 'LineWidth', 2);
plot([1, n_samples], [median(sensor_data), median(sensor_data)], 'g-', 'LineWidth', 2);
xlabel('Sample Index');
ylabel('Temperature (°C)');
title('Sensor Readings with Mean (red) and Median (green)');
legend('Data', 'Mean', 'Median');

subplot(2, 2, 2);
boxplot(sensor_data);
title('Box Plot of Sensor Data');
ylabel('Temperature (°C)');

subplot(2, 2, 3);
histogram(sensor_data, 30, 'Normalization', 'pdf');
hold on;
% Overlay normal distribution with same mean and std
x = linspace(min(sensor_data), max(sensor_data), 100)';
y = normpdf(x, mean(sensor_data), std(sensor_data));
plot(x, y, 'r-', 'LineWidth', 2);
title('Histogram and Normal Distribution Fit');
xlabel('Temperature (°C)');
ylabel('Probability Density');

subplot(2, 2, 4);
qqplot(sensor_data);
title('Q-Q Plot (assessing normality)');

% Step 4: Remove outliers and recompute statistics
sensor_data_clean = sensor_data(~outlier_mask);
fprintf('\n========== ANALYSIS AFTER OUTLIER REMOVAL ==========\n');
fprintf('Number of clean samples: %d\n', length(sensor_data_clean));
fprintf('Mean: %.4f°C\n', mean(sensor_data_clean));
fprintf('Median: %.4f°C\n', median(sensor_data_clean));
fprintf('Standard Deviation: %.4f°C\n', std(sensor_data_clean));
fprintf('True temperature: %.1f°C\n', true_temperature);
fprintf('Error after cleaning: %.4f°C\n', mean(sensor_data_clean) - true_temperature);

This example demonstrates the complete workflow of descriptive statistics: compute statistics, detect outliers, visualize the data, remove outliers, and recompute. This is a pattern you’ll use again and again in data analysis.

In the next section, we’ll build on these foundations to explore probability distributions—the mathematical models that describe how data is generated.

Membership Required

You must be a member to access this content.

View Membership Levels

Already a member? Log in here
Share your love