Lesson 07-Advanced Data Processing

Time Series Analysis Techniques

Time Series Object Creation and Management

Basic Time Series Construction:

% Create regular time series
t = datetime(2023,1,1):caldays(1):datetime(2023,1,31);
data = rand(1,31);
ts = timeseries(data, t);
ts.Name = 'Daily Random Data';

% Add metadata
ts.UserData.Description = 'Simulated 31-day temperature changes';
ts.TimeInfo.StartDate = '2023-01-01';

% Quick creation of regular time series
ts_regular = timeseries(rand(1,24), 0:3600:86399, 'Name', 'Hourly Data');
ts_regular.TimeInfo.Units = 'seconds';

Irregular Time Series Processing:

% Create irregular time points
irregular_times = [datetime(2023,1,1,8,0,0); 
                  datetime(2023,1,1,10,30,0);
                  datetime(2023,1,1,15,15,0)];
irregular_data = [23.5; 24.1; 22.8];
ts_irregular = timeseries(irregular_data, irregular_times);

% Interpolation processing
new_times = datetime(2023,1,1,8,0,0):calhours(1):datetime(2023,1,1,18,0,0);
ts_interp = resample(ts_irregular, new_times);

Time Series Collections and Operations

tscollection Creation and Management:

% Create time series collection
ts1 = timeseries(rand(1,10), datetime(2023,1,1)+caldays(0:9));
ts2 = timeseries(rand(1,10), datetime(2023,1,1)+caldays(0:9));
tsc = tscollection({ts1, ts2});
tsc.Name = 'Multivariate Time Series Collection';

% Add new time series
ts3 = timeseries(rand(1,10), datetime(2023,1,1)+caldays(0:9));
tsc = addts(tsc, ts3, 'Temperature');

% Collection operations
sum_ts = addsample(tsc, 'Temperature', datetime(2023,1,5), 10);
removeSample(tsc, 'Humidity', datetime(2023,1,3));

Time Series Analysis Functions:

% Resampling and aggregation
daily_avg = resample(ts, 'daily', 'mean');

% Moving window statistics
window_size = hours(6);
moving_avg = movmean(ts.Data, [window_size/minutes(1) 0]);

% Time series decomposition
[trend, seasonal, residual] = decompose(ts, 'additive');

Advanced Operations on Multidimensional Arrays

Array Dimension Management

Dimension Query and Reconstruction:

% Create 4D array
data_4d = rand(10,20,3,4);

% Query array dimensions
dims = ndims(data_4d);  % 4
size_info = size(data_4d);  % [10 20 3 4]

% Dimension reconstruction
reshaped_data = reshape(data_4d, [200 3 4]);  % 10 * 20=200

% Dimension permutation
permuted_data = permute(data_4d, [2 1 3 4]);  % Swap first two dimensions

High-Dimensional Array Slicing Operations:

% Extract specific dimension slices
slice_2d = data_4d(:,:,2,3);  % Extract layer 2 of the 3rd 3D matrix

% Batch indexing
idx = repmat({':'},1,ndims(data_4d));
idx{2} = 5:15;  % Dimensions 2: 5-15
batch_slice = data_4d(idx{:});

% Logical indexing
mask = data_4d > 0.8;
logical_slice = data_4d(mask);

Efficient Array Computations

Vectorized High-Dimensional Computations:

% Multidimensional array statistics
mean_3d = mean(data_4d, [1 2]);  % Mean along first two dimensions
std_4d = std(data_4d, 0, 3);     % Standard deviation along 3rd dimension

% Broadcasting mechanism application
weights = rand(1,1,3,4);
weighted_sum = sum(data_4d .* weights, [1 2]);

% High-dimensional convolution
kernel = rand(3,3,3);
conv_result = convn(data_4d, kernel, 'same');

Advanced Table Operations

Table Creation and Conversion

Table Construction Methods:

% Create table from matrix
data_matrix = rand(100,3);
col_names = {'Var1','Var2','Var3'};
tbl1 = array2table(data_matrix, 'VariableNames', col_names);

% Create table from structure
data_struct = struct('Name',{'Alice','Bob'}, 'Age',{25,30}, 'Score',{85.5,92.0});
tbl2 = struct2table(data_struct);

% Create table from cell array
cell_data = {'Name', {'Alice','Bob'}; 'Age', {25,30}; 'Score', {85.5,92.0}};
tbl3 = cell2table(cell_data(2:end,:), 'VariableNames', cell_data(1,:));

Table Merging and Joining:

% Horizontal merge (same number of rows)
tbl_a = table(1:5, rand(5,1), 'VariableNames', {'ID','ValueA'});
tbl_b = table(1:5, rand(5,1), 'VariableNames', {'ID','ValueB'});
tbl_combined = [tbl_a, tbl_b];

% Vertical concatenation (same variables)
tbl_c = table(6:10, rand(5,1), 'VariableNames', {'ID','ValueA'});
tbl_vertical = [tbl_a; tbl_c];

% Key-based join
tbl_left = table({'A';'B';'C'}, [1;2;3], 'VariableNames', {'Key','Value1'});
tbl_right = table({'B';'C';'D'}, [4;5;6], 'VariableNames', {'Key','Value2'});
tbl_joined = innerjoin(tbl_left, tbl_right, 'Keys', 'Key');

Advanced Table Operations

Grouped Aggregation Computations:

% Create example table
sales_data = table(...
    repmat({'East';'West';'North';'South'},3,1), ...
    repmat(2020:2022,4,1), ...
    rand(12,1)*1000, ...
    'VariableNames', {'Region','Year','Sales'});

% Group statistics
region_stats = groupsummary(sales_data, 'Region', ...
    {'mean','sum','max'}, 'Sales');

% Multi-variable grouping
year_region_stats = groupsummary(sales_data, {'Year','Region'}, ...
    'includeemptygroups', true, 'Sales', 'mean');

% Custom aggregation functions
custom_stats = groupsummary(sales_data, 'Region', ...
    @(x) [mean(x) std(x)], 'Sales');

Advanced Data Import Techniques

Excel Data Interaction

Advanced Excel Operations:

% Read formatted Excel data
opts = detectImportOptions('financial_data.xlsx');
opts.DataRange = 'A1:D100';
opts.VariableNamesLine = 1;
opts.VariableTypes = {'datetime','double','string','categorical'};
financial_data = readtable('financial_data.xlsx', opts);

% Write to Excel file
writecell({'Header1','Header2'; 1:3, rand(3,1)}, 'output.xlsx', ...
    'Sheet', 2, 'Range', 'A1');

% Dynamically update Excel file
excel = actxserver('Excel.Application');
workbook = excel.Workbooks.Open('data.xlsx');
sheet = workbook.Sheets.Item(1);
sheet.Range('A1').Value = rand(10,1);
workbook.Save();
workbook.Close();
excel.Quit();

HDF5 Big Data Processing

HDF5 File Operations:

% Create HDF5 file
h5create('big_data.h5', '/dataset1', [1e6 10], 'Datatype', 'double');
data = rand(1e6,10);
h5write('big_data.h5', '/dataset1', data);

% Chunked reading of big data
chunk_size = 1e5;
for i = 1:chunk_size:1e6
    idx = i:min(i+chunk_size-1,1e6);
    data_chunk = h5read('big_data.h5', '/dataset1', [idx 1], [length(idx) 10]);
    process_chunk(data_chunk);
end

% Attribute operations
h5create('big_data.h5', '/dataset1/attributes/unit', [1 10], 'Datatype', 'char');
h5writeatt('big_data.h5', '/dataset1', 'unit', 'm/s');

JSON Data Parsing

JSON Interaction Techniques:

% Read JSON file
json_data = jsondecode(fileread('config.json'));

% Process nested structures
if isfield(json_data, 'parameters')
    params = json_data.parameters;
    for i = 1:length(params)
        assignin('base', params(i).name, params(i).value);
    end
end

% Create complex JSON structure
output_data = struct(...
    'metadata', struct('version', '1.0', 'author', 'MATLAB'), ...
    'data', rand(10,1), ...
    'timestamp', datetime('now') );

% Write JSON file
fid = fopen('output.json', 'w');
fprintf(fid, '%s', jsonencode(output_data, 'PrettyPrint', true));
fclose(fid);

Parallel Data Processing Techniques

parfor Parallel Loops

Parallel Computing Implementation:

% Basic parfor example
n = 1e6;
results = zeros(n,1);
parfor i = 1:n
    results(i) = sqrt(i) * log(i+1);
end

% Data partitioning strategy
chunk_size = 1e4;
parfor (i = 1:chunk_size:n, ceil(n/chunk_size))
    end_idx = min(i+chunk_size-1, n);
    results(i:end_idx) = arrayfun(@(x) complex_calc(x), i:end_idx);
end

% Parallel pool configuration
parpool('local',4);  % Start 4 worker processes
parfor i = 1:100
    monte_carlo_simulation(i);
end
delete(gcp);  % Close parallel pool

SPMD Distributed Computing

Single Program Multiple Data Mode:

% Basic SPMD block
spmd
    % Each worker executes independently
    local_data = rand(1000, labindex);  % Data of different sizes
    local_result = mean(local_data);
    
    % Collect all results
    gathered_results = gcat(local_result);
end

% Distributed array operations
spmd
    % Create distributed array
    darray = distributed(rand(10000));
    
    % Distributed computation
    dmean = mean(darray);
    dstd = std(darray);
end

% Retrieve results
final_mean = gather(dmean);

Hybrid Parallel Mode:

% Combine parfor and SPMD
spmd
    % Each worker processes different dataset
    dataset = load(['data_part_' num2str(labindex) '.mat']);
    
    % Use parfor internally
    parfor i = 1:size(dataset,1)
        processed(i) = complex_analysis(dataset(i,:));
    end
    
    % Collect results
    all_results{labindex} = processed;
end

% Merge final results
final_results = vertcat(all_results{:});

Performance Optimization Techniques

Memory Optimization Strategies

Efficient Memory Management:

% Memory preallocation best practices
n = 1e7;
data = zeros(n,1);  % Preallocate double-precision array

% Alternative methods comparison
% Method 1: Dynamic growth (not recommended)
data_bad = [];
for i = 1:n
    data_bad(i) = rand();
end

% Method 2: Preallocation + filling (recommended)
data_good = zeros(n,1);
for i = 1:n
    data_good(i) = rand();
end

% Memory usage comparison
whos data_bad data_good

Data Type Optimization:

% Data type selection guide
large_array = rand(1e6,1);

% Default double precision (8 bytes)
size_double = whos('large_array');

% Single precision (4 bytes)
large_single = single(large_array);
size_single = whos('large_single');

% Unsigned integer (1-4 bytes)
counts = uint32(randi([0,100],1e6,1));
size_uint = whos('counts');

% Logical type (1 byte)
flags = logical(rand(1e6,1)>0.5);
size_logical = whos('flags');

Computation Acceleration Techniques

Vectorization and Preallocation:

% Matrix operations replacing loops
% Inefficient method
result = zeros(1000);
for i = 1:1000
    for j = 1:1000
        result(i,j) = sin(i/100) * cos(j/100);
    end
end

% Efficient vectorization
i = (1:1000)/100;
j = (1:1000)/100;
[i_grid,j_grid] = meshgrid(i,j);
result = sin(i_grid) .* cos(j_grid);

% Preallocate large arrays
big_data = zeros(1e6,100);
for k = 1:100
    big_data(:,k) = rand(1e6,1);
end

Parallel Computing Configuration:

% Parallel pool optimization settings
parpool('local',4, ...
    'SpmdEnabled', true, ...
    'AttachedFiles', {'data_helper.m'}, ...
    'IdleTimeout', 3600);

% GPU computing configuration
if gpuDeviceCount > 0
    gpu = gpuDevice(1);
    reset(gpu);  % Clear GPU memory
    data_gpu = gpuArray(rand(1e6,1));
    result_gpu = sqrt(data_gpu);
    result = gather(result_gpu);
else
    warning('GPU unavailable, using CPU computation');
end
Share your love