Lesson 20-Neural Network Fundamentals and BP Network Design

Introduction to Neural Networks in MATLAB: From Neural Network Toolbox to Deep Learning Toolbox

Let me take you on a journey through one of the most transformative technologies in modern engineering and data science. If you’re reading this, you probably already know that neural networks have revolutionized everything from image recognition to autonomous driving. But here’s the thing—MATLAB has been in the neural network game longer than most people realize, and understanding this evolution will give you a much deeper appreciation for the tools at your fingertips.

Back in the day—and when I say “back in the day,” I’m talking about the late 1990s and early 2000s—MATLAB had what was called the “Neural Network Toolbox.” It was revolutionary for its time. You could create perceptrons, train multilayer perceptrons with backpropagation, and even do some basic pattern recognition. But it was, shall we say, a bit clunky by modern standards. The syntax was less intuitive, the documentation was sparse, and if you wanted to do anything beyond the basics, you’d find yourself diving deep into the underlying C code.

Then came the deep learning revolution around 2012 with AlexNet, and everything changed. MATLAB responded by evolving the Neural Network Toolbox into what we now know as the “Deep Learning Toolbox.” This wasn’t just a rebranding exercise—it was a complete architectural overhaul. The new toolbox integrated seamlessly with MATLAB’s GPU computing capabilities, added support for convolutional neural networks (CNNs), recurrent neural networks (RNNs), and later, transformer architectures.

But here’s what’s really interesting, and what most tutorials won’t tell you: the old functions didn’t go away. When you call feedforwardnet() or patternnet() today, you’re still using code that traces its lineage back to that original Neural Network Toolbox. The network object, the train() function, the whole ecosystem of training functions like traingd, trainlm, trainscg—these are all part of what we call the “Shallow Neural Network” framework. They coexist alongside the newer trainNetwork() function that’s used for deep learning.

Why does this matter? Well, if you’re working on a project that involves a relatively small dataset (say, a few thousand samples) and you need a quick, interpretable model, the old-school feedforwardnet() approach is often better than trying to wrestle with a deep learning model that might overfit or take forever to train. I’ve seen too many engineers jump straight to deep learning when a simple three-layer MLP would have solved their problem in minutes rather than hours.

In this article, we’re going to dive deep into the fundamentals. We’ll start with the perceptron—the great-grandfather of all neural networks—and work our way through multi-layer perceptrons, the mathematics of backpropagation, and finally to practical implementations in MATLAB. By the time we’re done, you’ll understand not just how to use functions like feedforwardnet() and patternnet(), but why they work the way they do.

And trust me, once you understand the “why,” the “how” becomes so much more powerful.

Let’s get started.

The Perceptron: The Simplest Neural Network and Its Limitations

Imagine you’re an engineer in the late 1950s. The field of artificial intelligence is brand new—in fact, the term hasn’t even been coined yet. You’re trying to create a machine that can “learn” from examples, and you come up with the simplest possible idea: take some inputs, multiply them by weights, add them up, and if the sum exceeds a threshold, output a 1; otherwise, output a 0.

Congratulations—you’ve just invented the perceptron.

The perceptron, introduced by Frank Rosenblatt in 1958, is elegantly simple. Here’s the mathematical formulation:

Given an input vector x = [x₁, x₂, …, xₙ]ᵀ, a weight vector w = [w₁, w₂, …, wₙ]ᵀ, and a bias term b, the perceptron computes:

y = f(w₁x₁ + w₂x₂ + ... + wₙxₙ + b)

where f is a step function:

f(z) = 1 if z ≥ 0
f(z) = 0 if z < 0

That’s it. That’s the entire perceptron. If you’ve ever taken a course in linear algebra or machine learning, you’ll recognize this as a linear classifier. The perceptron is trying to find a hyperplane in n-dimensional space that separates your data into two classes.

Implementing a Perceptron from Scratch in MATLAB

Let’s actually build one from scratch. No toolboxes, no fancy functions—just raw MATLAB code. This will help you understand exactly what’s happening under the hood.

% ============================================
% PERCEPTRON FROM SCRATCH IN MATLAB
% ============================================
% This example demonstrates the fundamental mechanics
% of a perceptron. We'll train it to classify two
% simple 2D clusters.

clear; clc; close all;

% Step 1: Create a simple dataset
% We'll use two clusters: Class 0 and Class 1
% Class 0: points where x1 + x2 < 0 (roughly)
% Class 1: points where x1 + x2 >= 0 (roughly)

rng(42);  % Set random seed for reproducibility

% Generate 100 samples for each class
n_samples = 100;

% Class 0: centered around (-1, -1)
X0 = randn(n_samples, 2) * 0.5 - 1;
T0 = zeros(n_samples, 1);  % Target = 0

% Class 1: centered around (1, 1)
X1 = randn(n_samples, 2) * 0.5 + 1;
T1 = ones(n_samples, 1);   % Target = 1

% Combine into training set
X = [X0; X1]';  % MATLAB expects inputs as columns (2 x 200)
T = [T0; T1]';  % Targets as row vector (1 x 200)

% Step 2: Initialize perceptron parameters
% We add a bias term by appending a 1 to each input
w = randn(3, 1) * 0.1;  % Weights for [bias, x1, x2]
% w(1) is the bias, w(2) is weight for x1, w(3) is weight for x2

learning_rate = 0.1;
n_epochs = 100;

% Step 3: Training loop (Perceptron Learning Rule)
fprintf('Training perceptron for %d epochs...\n', n_epochs);

for epoch = 1:n_epochs
    n_misclassified = 0;
    
    for i = 1:size(X, 2)  % Loop through each sample
        % Add bias term to input
        x_i = [1; X(:, i)];  % 3 x 1 vector [bias; x1; x2]
        
        % Compute perceptron output (step function)
        y_i = (w' * x_i >= 0);  % 1 if w'*x >= 0, else 0
        
        % True target
        t_i = T(i);
        
        % Perceptron update rule
        % If misclassified, update weights
        if y_i ~= t_i
            n_misclassified = n_misclassified + 1;
            % w_new = w_old + eta * (t - y) * x
            w = w + learning_rate * (t_i - y_i) * x_i;
        end
    end
    
    % Display progress
    if mod(epoch, 10) == 0 || epoch == 1
        fprintf('Epoch %d: %d misclassified\n', epoch, n_misclassified);
    end
    
    % Early stopping if perfectly classified
    if n_misclassified == 0
        fprintf('Converged at epoch %d!\n', epoch);
        break;
    end
end

% Step 4: Visualize the decision boundary
fprintf('\nFinal weights: w = [%.4f, %.4f, %.4f]\n', w(1), w(2), w(3));
fprintf('Decision boundary: %.4f + %.4fx₁ + %.4fx₂ = 0\n', w(1), w(2), w(3));

figure('Position', [100, 100, 800, 600]);
hold on;

% Plot the two classes
scatter(X0(:,1), X0(:,2), 50, 'b', 'filled', 'DisplayName', 'Class 0');
scatter(X1(:,1), X1(:,2), 50, 'r', 'filled', 'DisplayName', 'Class 1');

% Plot the decision boundary
% The boundary is where w(1) + w(2)*x1 + w(3)*x2 = 0
% Solve for x2: x2 = -(w(1) + w(2)*x1) / w(3)
x1_range = linspace(-3, 3, 100);
if abs(w(3)) > 1e-10
    x2_boundary = -(w(1) + w(2) * x1_range) / w(3);
    plot(x1_range, x2_boundary, 'k-', 'LineWidth', 2, 'DisplayName', 'Decision Boundary');
else
    % Vertical line case (w(3) = 0)
    x1_boundary = -w(1) / w(2);
    plot([x1_boundary, x1_boundary], [-3, 3], 'k-', 'LineWidth', 2, 'DisplayName', 'Decision Boundary');
end

xlabel('x₁ (Feature 1)');
ylabel('x₂ (Feature 2)');
title('Perceptron Decision Boundary');
legend('Location', 'best');
grid on;
axis([-3, 3, -3, 3]);
hold off;

% Step 5: Test accuracy
correct = 0;
for i = 1:size(X, 2)
    x_i = [1; X(:, i)];
    y_i = (w' * x_i >= 0);
    if y_i == T(i)
        correct = correct + 1;
    end
end
accuracy = correct / size(X, 2) * 100;
fprintf('Training accuracy: %.2f%%\n', accuracy);

Now, here’s where the story takes a dramatic turn. In 1969, Marvin Minsky and Seymour Papert published a book called “Perceptrons” that basically killed neural network research for a decade. They proved, mathematically, that a single-layer perceptron cannot solve the XOR problem.

The XOR Problem: Why the Perceptron Fails

The XOR (exclusive OR) problem is deceptively simple. Here’s the truth table:

x₁x₂XOR Output
000
011
101
110

Go ahead, try to draw a single straight line that separates the 1s from the 0s. I’ll wait.

You can’t do it. And neither can a perceptron.

Here’s why: the XOR problem is not linearly separable. The points (0,0) and (1,1) belong to class 0, while (0,1) and (1,0) belong to class 1. If you plot these points, you’ll see that you need two lines to separate them—or, equivalently, a non-linear decision boundary.

This is the fundamental limitation of the perceptron: it can only learn linear decision boundaries. If your data isn’t linearly separable, the perceptron will never converge. It will keep oscillating forever, or it will converge to a suboptimal solution that misclassifies some points.

Minsky and Papert’s book basically said, “Hey, these perceptrons are cool, but they’re really limited.” And the AI community, being the fickle beast that it is, largely abandoned neural networks for about a decade.

But here’s the plot twist: Minsky and Papert knew that adding one more layer of neurons would solve the XOR problem. They even mentioned it in their book. But they argued that such “multi-layer” perceptrons would be too difficult to train because nobody had figured out how to compute gradients for the hidden layer.

It would take another 17 years—until 1986—for Rumelhart, Hinton, and Williams to publish their famous paper on backpropagation, which finally showed how to train multi-layer networks. And that, my friends, is the story of how the perceptron’s limitations led to the development of modern deep learning.

But before we get to backpropagation, we need to understand what a multi-layer perceptron actually looks like.

Membership Required

You must be a member to access this content.

View Membership Levels

Already a member? Log in here

Share your love