Data Analysis and Statistics
Data Exploration and Visualization
Comprehensive Data Exploration Process:
% Load example dataset
load patients
tbl = table(Age, Height, Weight, Smoker, Systolic, Diastolic);
% 1. Data overview
summary(tbl); % Display statistical summary
% 2. Univariate analysis
figure;
subplot(2,2,1);
histogram(tbl.Age);
title('Age Distribution');
subplot(2,2,2);
boxplot(tbl.Systolic);
title('Systolic Blood Pressure Boxplot');
% 3. Multivariate relationships
figure;
heatmap(tbl, 'Smoker', 'Sex', 'ColorVariable', 'Age');
title('Smoker Age Heatmap (by Gender)');
% 4. High-dimensional visualization
figure;
gscatter(tbl.Height, tbl.Weight, tbl.Smoker);
xlabel('Height'); ylabel('Weight');
title('Body Type Distribution for Smokers and Non-Smokers');
% 5. Correlation analysis
corrMatrix = corr([tbl.Age, tbl.Height, tbl.Weight, tbl.Systolic]);
figure;
heatmap(corrMatrix);
title('Physiological Indicators Correlation Matrix');
Statistical Hypothesis Testing
Inter-Group Difference Analysis Methods:
% 1. Independent samples t-test (compare blood pressure between smokers and non-smokers)
smokerIdx = tbl.Smoker == 1;
nonSmokerIdx = tbl.Smoker == 0;
[h_p,t_p,ci_p,stats_p] = ttest2(tbl.Systolic(smokerIdx), tbl.Systolic(nonSmokerIdx));
fprintf('Blood pressure t-test result: p-value=%.4f\n', h_p);
% 2. One-way ANOVA (compare blood pressure across genders)
[anova_p, anova_table, stats_anova] = anova1(tbl.Systolic, tbl.Sex);
multcompare(stats_anova); % Multiple comparisons
% 3. Non-parametric test (Wilcoxon rank-sum test)
[h_w, p_w] = ranksum(tbl.Systolic(smokerIdx), tbl.Systolic(nonSmokerIdx));
fprintf('Blood pressure Wilcoxon test: p-value=%.4f\n', p_w);
% 4. Correlation test (Pearson/Spearman)
[rho_p, p_rho] = corr(tbl.Age, tbl.Systolic);
[rho_s, p_s] = corr(tbl.Age, tbl.Systolic, 'Type', 'Spearman');
fprintf('Age-Blood Pressure Pearson correlation: rho=%.2f, p=%.4f\n', rho_p, p_rho);
Regression Analysis Techniques
Linear and Nonlinear Modeling:
% 1. Multiple linear regression
X = [ones(height(tbl),1), tbl.Age, tbl.Height];
y = tbl.Systolic;
mdl_linear = fitlm(X, y);
disp(mdl_linear);
% 2. Stepwise regression for variable selection
mdl_stepwise = stepwiselm(tbl, 'linear', ...
'PredictorVars', {'Age', 'Height', 'Weight', 'Smoker'}, ...
'ResponseVar', 'Systolic');
% 3. Nonlinear regression (exponential model)
% Assume model: y = a*exp(b*x) + c
xdata = tbl.Age;
ydata = tbl.Systolic;
fun = @(b,x) b(1)*exp(b(2)*x) + b(3);
beta0 = [1, 0.05, 100]; % Initial guess
mdl_nonlinear = fitnlm(xdata, ydata, fun, beta0);
disp(mdl_nonlinear);
% 4. Generalized linear model (logistic regression)
tbl.SmokerNum = double(tbl.Smoker); % Convert to numeric
mdl_glm = fitglm(tbl, 'SmokerNum ~ Age + Height', 'Distribution', 'binomial');
disp(mdl_glm);
Dimensionality Reduction and Factor Analysis
PCA and Factor Extraction:
% 1. Principal Component Analysis
[coeff, score, latent, tsquared, explained] = pca([tbl.Age, tbl.Height, tbl.Weight]);
% Visualize results
figure;
pareto(explained);
xlabel('Principal Component'); ylabel('Percentage of Variance Explained');
title('PCA Variance Explained Ratio');
% 2. Factor Analysis
[loadings, specificVar, stats_fa] = factoran([tbl.Age, tbl.Height, tbl.Weight], 2);
figure;
biplot(loadings(:,1:2), 'Scores', score(:,1:2), 'VarLabels', {'Age','Height','Weight'});
title('Factor Analysis Biplot');
% 3. Result interpretation
fprintf('Principal component loadings matrix:\n');
disp(coeff);
fprintf('Factor loadings matrix:\n');
disp(loadings);
Clustering Analysis Methods
K-means and Hierarchical Clustering:
% 1. K-means clustering
X = [tbl.Age, tbl.Weight];
[idx_kmeans, C_kmeans] = kmeans(X, 3); % Divide into 3 clusters
% Visualize clustering results
figure;
gscatter(X(:,1), X(:,2), idx_kmeans);
hold on;
plot(C_kmeans(:,1), C_kmeans(:,2), 'kx', 'MarkerSize', 15);
title('K-means Age-Weight Clustering');
% 2. Hierarchical clustering
Y = pdist(X); % Compute distance matrix
Z = linkage(Y, 'ward'); % Ward's minimum variance method
c = cluster(Z, 'maxclust', 3); % Divide into 3 clusters
% Plot dendrogram
figure;
dendrogram(Z);
title('Hierarchical Clustering Dendrogram');
% 3. Clustering evaluation (silhouette coefficient)
silhouette_vals = silhouette(X, idx_kmeans);
figure;
silhouette(X, idx_kmeans);
title('K-means Clustering Silhouette Analysis');
Machine Learning Techniques
Supervised Learning Models
Classification and Regression Tasks:
% 1. Classification tree model
Mdl_tree = fitctree(tbl, 'Smoker');
view(Mdl_tree, 'Mode', 'graph');
% Predict new data
newData = table(45, 170, 70, 0, NaN, NaN, ...
'VariableNames', {'Age','Height','Weight','Smoker','Systolic','Diastolic'});
[pred_smoker, score_smoker] = predict(Mdl_tree, newData);
% 2. Support Vector Machine regression
X_train = [tbl.Age, tbl.Height];
y_train = tbl.Systolic;
Mdl_svr = fitrsvm(X_train, y_train, 'KernelFunction', 'gaussian');
% 3. Random Forest regression
Mdl_rf = TreeBagger(100, X_train, y_train, 'Method', 'regression');
% 4. Model evaluation
cv_Mdl_tree = crossval(Mdl_tree, 'KFold', 5);
loss_tree = kfoldLoss(cv_Mdl_tree);
fprintf('Classification tree cross-validation error rate: %.2f%%\n', loss_tree*100);
Unsupervised Learning Algorithms
Clustering and Dimensionality Reduction Techniques:
% 1. Gaussian Mixture Model clustering
GMModel = fitgmdist([tbl.Age, tbl.Weight], 2);
idx_gmm = cluster(GMModel, [tbl.Age, tbl.Weight]);
% 2. t-SNE dimensionality reduction visualization
X = [tbl.Age, tbl.Height, tbl.Weight];
Y = tsne(X, 'NumDimensions', 2);
figure;
gscatter(Y(:,1), Y(:,2), tbl.Smoker);
title('t-SNE Age-Height-Weight Dimensionality Reduction');
% 3. Principal Component Regression
[coeff, score, ~, ~, ~] = pca([tbl.Age, tbl.Height, tbl.Weight]);
X_pca = score(:,1:2); % Use first two principal components
Mdl_pcr = fitrlinear(X_pca, tbl.Systolic);
Model Evaluation Techniques
Cross-Validation and Performance Metrics:
% 1. k-fold cross-validation
cv_Mdl_rf = crossval(Mdl_rf, 'KFold', 10);
loss_rf = kfoldLoss(cv_Mdl_rf);
fprintf('Random Forest 10-fold CV MSE: %.2f\n', loss_rf);
% 2. Confusion matrix (classification problem)
tbl_class = tbl;
tbl_class.SmokerCat = categorical(tbl.Smoker, [0 1], {'NonSmoker','Smoker'});
Mdl_class = fitctree(tbl_class, 'SmokerCat');
cv_Mdl_class = crossval(Mdl_class, 'KFold', 5);
label_pred = kfoldPredict(cv_Mdl_class);
confMat = confusionmat(categorical(tbl_class.SmokerCat), label_pred);
figure;
confusionchart(confMat);
% 3. ROC curve analysis
[X_roc, Y_roc, T_roc, AUC] = perfcurve(tbl_class.SmokerCat, ...
predict(Mdl_class, tbl_class), 'Smoker');
figure;
plot(X_roc, Y_roc);
xlabel('False Positive Rate'); ylabel('True Positive Rate');
title(sprintf('ROC Curve (AUC=%.2f)', AUC));
Deep Learning Fundamentals
Neural Network Design and Training:
% 1. Create feedforward neural network
layers = [ ...
featureInputLayer(3) % 3 input features (age, height, weight)
fullyConnectedLayer(10)
reluLayer()
fullyConnectedLayer(1)
regressionLayer()]; % Regression output layer
% 2. Training options setup
options = trainingOptions('adam', ...
'MaxEpochs', 50, ...
'MiniBatchSize', 32, ...
'ValidationData', {X_val, y_val}, ...
'Plots', 'training-progress');
% 3. Train network
net = trainNetwork([tbl.Age, tbl.Height, tbl.Weight], tbl.Systolic, layers, options);
% 4. CNN image classification example
imageInputLayer = imageInputLayer([128 128 3]);
convLayer = convolution2dLayer(3, 16, 'Padding', 'same');
poolLayer = maxPooling2dLayer(2, 'Stride', 2);
fullyConnLayer = fullyConnectedLayer(10);
softmaxLayer = softmaxLayer();
classificationLayer = classificationLayer();
cnnLayers = [ ...
imageInputLayer
convLayer
reluLayer()
poolLayer
fullyConnLayer
softmaxLayer
classificationLayer];
% 5. RNN time series prediction
numFeatures = 1;
numResponses = 1;
numHiddenUnits = 200;
layers = [ ...
sequenceInputLayer(numFeatures)
lstmLayer(numHiddenUnits)
fullyConnectedLayer(numResponses)
regressionLayer];
options = trainingOptions('adam', ...
'MaxEpochs', 100, ...
'GradientThreshold', 1, ...
'InitialLearnRate', 0.005, ...
'Plots', 'training-progress');
% Prepare time series data
XTrain = ... % Time steps × features × observations
YTrain = ... % Response variable
net = trainNetwork(XTrain, YTrain, layers, options);
Model Deployment Techniques
Production Environment Deployment Process:
% 1. Create prediction function
function [y_pred, scores] = predictBloodPressure(age, height, weight)
persistent Mdl_rf;
if isempty(Mdl_rf)
Mdl_rf = load('bloodpressure_model.mat');
end
inputTbl = table(age, height, weight);
[y_pred, scores] = predict(Mdl_rf, inputTbl);
end
% 2. Compile into standalone application
compiler.build.productionServerDeployment('predictBloodPressure.m', ...
'ServiceName', 'BloodPressurePrediction', ...
'OutputDir', 'deployment');
% 3. Generate RESTful API
compiler.build.productionServerDeployment('predictBloodPressure.m', ...
'ServiceName', 'BP_API', ...
'RequirementsFile', 'requirements.txt');
% 4. MATLAB Production Server configuration
% In the server management interface:
% - Upload deployment package
% - Configure API routes (/predict_bp)
% - Set resource limits (CPU/memory)
% 5. Client call example
% MATLAB client
url = 'http://server.example.com/predict_bp';
inputData = struct('age', 45, 'height', 170, 'weight', 70);
options = weboptions('ContentType', 'application/json');
response = webwrite(url, inputData, options);
predictedBP = jsondecode(response).prediction;
% Python client
import requests
import json
url = "http://server.example.com/predict_bp"
data = {"age": 45, "height": 170, "weight": 70}
headers = {'Content-Type': 'application/json'}
response = requests.post(url, data=json.dumps(data), headers=headers)
predicted_bp = response.json()['prediction']
Deep Learning and AI
Network Design and Training
Custom Network Architecture:
% 1. Create layer graph
layers = [ ...
imageInputLayer([224 224 3], 'Name', 'input')
convolution2dLayer(7, 64, 'Stride', 2, 'Padding', 'same', 'Name', 'conv1')
reluLayer('Name', 'relu1')
maxPooling2dLayer(3, 'Stride', 2, 'Name', 'pool1')
groupNormalizationLayer('Name', 'gn1')
residualBlock('Name', 'res1') % Custom residual block
fullyConnectedLayer(1000, 'Name', 'fc')
softmaxLayer('Name', 'softmax')
classificationLayer('Name', 'output')];
% 2. Define training options
options = trainingOptions('sgdm', ...
'InitialLearnRate', 0.01, ...
'MaxEpochs', 30, ...
'MiniBatchSize', 64, ...
'Shuffle', 'every-epoch', ...
'ValidationData', {valImages, valLabels}, ...
'ValidationFrequency', 30, ...
'Verbose', false, ...
'Plots', 'training-progress');
% 3. Train network
net = trainNetwork(trainImages, trainLabels, layers, options);
% 4. Custom residual block implementation
function layers = residualBlock(name)
layers = [ ...
convolution2dLayer(3, 64, 'Padding', 'same', 'Name', [name '_conv1'])
groupNormalizationLayer('Name', [name '_gn1'])
reluLayer('Name', [name '_relu1'])
convolution2dLayer(3, 64, 'Padding', 'same', 'Name', [name '_conv2'])
groupNormalizationLayer('Name', [name '_gn2'])];
% Add skip connection
layers = [layers ...
additionLayer(2, 'Name', [name '_add']) ...
reluLayer('Name', [name '_relu2'])];
end
Computer Vision Applications
Image Classification and Object Detection:
% 1. Fine-tuning pre-trained model
net = resnet50; % Load pre-trained ResNet-50
lgraph = layerGraph(net);
% Replace classification layer
numClasses = 10; % Number of classes in new dataset
newLayers = [ ...
fullyConnectedLayer(numClasses, 'Name', 'fc_new')
softmaxLayer('Name', 'softmax_new')
classificationLayer('Name', 'output_new')];
lgraph = replaceLayer(lgraph, 'fc1000', newLayers(1));
lgraph = replaceLayer(lgraph, 'ClassificationLayer_fc1000', newLayers(3));
% Freeze base layers
for i = 1:172
layer = lgraph.Layers(i);
if isa(layer, 'nnet.cnn.layer.Convolution2DLayer') || ...
isa(layer, 'nnet.cnn.layer.GroupNormalizationLayer')
layer.WeightLearnRateFactor = 0;
layer.BiasLearnRateFactor = 0;
end
end
% 2. Object detection (using Faster R-CNN)
load('fasterRCNNResNet50VehicleExample.mat'); % Pre-trained vehicle detector
I = imread('testImage.jpg');
[bboxes, scores, labels] = detect(detector, I);
% Visualize results
annotatedImage = insertObjectAnnotation(I, 'rectangle', bboxes, labels);
figure;
imshow(annotatedImage);
title('Vehicle Detection Results');
% 3. Semantic segmentation
load('deeplabv3plusResnet18Cityscapes.mat'); % Pre-trained segmentation network
C = semanticseg(I, net);
B = labeloverlay(I, C);
figure;
imshow(B);
title('Semantic Segmentation Results');
Natural Language Processing
Text Analysis and Classification:
% 1. Text preprocessing
documents = tokenizedDocument([
'MATLAB is a high-level language'
'Deep learning requires GPU acceleration'
'Statistical analysis is important'
]);
% Remove stop words
documents = removeStopWords(documents);
% Stemming
documents = normalizeWords(documents, 'Style', 'lemma');
% 2. Word embedding (Word2Vec)
emb = fastTextWordEmbedding; % Load pre-trained word vectors
avgEmbedding = mean(wordEmbedding(emb, tokens(documents)), 1);
% 3. Text classification LSTM
inputSize = emb.NumWords;
numHiddenUnits = 200;
numClasses = 3; % Number of classification classes
layers = [ ...
sequenceInputLayer(inputSize)
bilstmLayer(numHiddenUnits, 'OutputMode', 'last')
fullyConnectedLayer(numClasses)
softmaxLayer
classificationLayer];
% Prepare training data
XTrain = doc2sequence(emb, documents);
YTrain = categorical([1 2 3]); % Example labels
options = trainingOptions('adam', ...
'MaxEpochs', 10, ...
'MiniBatchSize', 1, ...
'Plots', 'training-progress');
net = trainNetwork(XTrain, YTrain, layers, options);
% 4. Sentiment analysis example
sentimentNet = alexnet; % Transfer learning example (should use text-specific network in practice)
% Need to convert text to image features (illustrative only)
Reinforcement Learning Framework
Q-learning and Deep Q-Network:
% 1. Environment definition (custom grid world)
env = GridWorldEnvironment;
% 2. Q-learning algorithm
numStates = env.getNumStates;
numActions = env.getNumActions;
QTable = zeros(numStates, numActions);
% Learning parameters
gamma = 0.9; % Discount factor
alpha = 0.1; % Learning rate
epsilon = 0.1; % Exploration rate
% Training loop
for episode = 1:1000
state = env.reset;
done = false;
while ~done
% ε-greedy policy to select action
if rand < epsilon
action = randi(numActions);
else
action = find(QTable(state,:) == max(QTable(state,:)), 1);
end
% Execute action
[nextState, reward, done] = env.step(action);
% Q-value update
QTable(state, action) = QTable(state, action) + ...
alpha * (reward + gamma * max(QTable(nextState,:)) - QTable(state, action));
state = nextState;
end
end
% 3. Deep Q-Network (DQN) implementation
layers = [ ...
featureInputLayer(env.getNumObservations)
fullyConnectedLayer(128)
reluLayer()
fullyConnectedLayer(128)
reluLayer()
fullyConnectedLayer(numActions)];
% Training options
options = trainingOptions('adam', ...
'MaxEpochs', 100, ...
'MiniBatchSize', 64, ...
'GradientThreshold', 1);
% Create DQN agent
agent = dqnAgent(env, layers, options);
% Train agent
trainStats = train(agent, env, 'MaxSteps', 10000);
% 4. Policy evaluation
initialState = env.reset;
done = false;
totalReward = 0;
while ~done
action = getAction(agent, initialState);
[nextState, reward, done] = env.step(action);
totalReward = totalReward + reward;
initialState = nextState;
end
fprintf('Test episode total reward: %.2f\n', totalReward);
AI Model Optimization Techniques
Performance Acceleration and Compression:
% 1. GPU-accelerated computation
if canUseGPU
% Transfer network to GPU
net = assembleNetwork(layers); % Assemble network first
net = move(net, 'gpu');
% GPU data conversion
XTrain = gpuArray(XTrain);
YTrain = gpuArray(YTrain);
% Train on GPU
trainedNet = trainNetwork(XTrain, YTrain, net.Layers, options);
else
warning('GPU unavailable, using CPU computation');
trainedNet = trainNetwork(XTrain, YTrain, layers, options);
end
% 2. Network quantization (reduce model size)
quantizedNet = quantizeNetwork(trainedNet, ...
'ExecutionEnvironment', 'cpu', ...
'QuantizationDataType', 'uint8');
% Evaluate quantized network
YPred = classify(quantizedNet, XTest);
accuracy = mean(YPred == YTest);
fprintf('Quantized network accuracy: %.2f%%\n', accuracy*100);
% 3. Network pruning (reduce parameter count)
prunedNet = pruneNetwork(trainedNet, ...
'PruningThreshold', 0.01, ...
'PruneMode', 'global');
% 4. Model compression package generation
compiler.build.productionCodeForDeployment('trainedNet.mat', ...
'PackageName', 'MyCompactModel', ...
'TargetLang', 'C++');
% 5. Performance benchmarking
fprintf('Original network size: %.2f MB\n', whos('trainedNet').bytes/1e6);
fprintf('Quantized network size: %.2f MB\n', whos('quantizedNet').bytes/1e6);
fprintf('Pruned network size: %.2f MB\n', whos('prunedNet').bytes/1e6);
% 6. Inference acceleration (using code generation)
cfg = coder.config('lib');
cfg.TargetLang = 'C++';
cfg.DeepLearningConfig = coder.DeepLearningConfig('cudnn');
inputArgs = {coder.typeof(single(0), [224 224 3 1])};
codegen -config cfg -args inputArgs predictNetwork -report;
% Generate MEX file for acceleration
codegen -config cfg predictNetwork -args inputArgs -o predictNetwork_mex;



