Lesson 03-Stack Operations

Detailed Explanation of Stack Registers

Evolution of Stack Registers

The design of stack registers is closely tied to CPU architecture, and its evolution reflects changing demands in memory management and function calling within computer systems.

Early Architectures: 16-bit Stack Registers (8086/8088)

The 8086/8088 CPUs operated in real mode, with 16-bit stack registers:

  • SP (Stack Pointer): 16-bit stack pointer pointing to the top of the stack (offset address).
  • BP (Base Pointer): 16-bit base pointer used to access local variables on the stack (optional).

Characteristics:

  • Limited stack space (maximum 64KB, determined by the SS segment register).
  • Function calls use the CALL instruction to push IP (instruction pointer); RET pops IP to return.

32-bit Architecture: The ESP/EBP Era (80386+)

The 80386 introduced protected mode, extending stack registers to 32 bits:

  • ESP (Extended Stack Pointer): 32-bit stack pointer, replacing the 16-bit SP.
  • EBP (Extended Base Pointer): 32-bit base pointer, replacing the 16-bit BP.

Key Improvements:

  • Support for much larger stack space (4GB address space).
  • Introduction of the frame pointer (Frame Pointer) mechanism (EBP), simplifying stack frame management.

64-bit Architecture: RSP/RBP and Register Extension (x86-64)

The x86-64 architecture further extended registers:

  • RSP (64-bit Stack Pointer): 64-bit stack pointer, backward compatible with 32-bit mode (low 32 bits accessible via ESP).
  • RBP (64-bit Base Pointer): 64-bit base pointer, functionally identical to EBP.
  • New Registers: General-purpose registers RAX–RDI, reducing reliance on the stack (e.g., parameters passed via registers first).

Optimizations:

  • Faster register-based parameter passing (e.g., fastcall convention), reducing stack operation overhead.
RegisterArchitecturePurposeSize
SP16-bit x86Stack pointer register16-bit
BP16-bit x86Base pointer register16-bit
ESP32-bit x8632-bit stack pointer register32-bit
RSP64-bit x8664-bit stack pointer register64-bit

Key Characteristics:

  • SP/ESP/RSP: Always points to the top of the stack; automatically adjusted by PUSH/POP.
  • BP/EBP/RBP: Used to locate local variables and parameters within a stack frame (base addressing).

Push and Pop Operations

PUSH Instruction

Function: Pushes data onto the top of the stack while adjusting the stack pointer (ESP/RSP).

Operation Flow:

  1. Decrement stack pointer (ESP/RSP -= operand size)
  2. Store data at the new stack top

Examples:

mov eax, 0x12345678
push eax        ; 32-bit push
; After execution:
; ESP = ESP - 4
; [ESP] = 0x12345678

push 0xABC      ; Push 8-bit immediate (automatically zero-extended to 32 bits)
; After execution:
; ESP = ESP - 4
; [ESP] = 0x00000ABC

Notes:

  • In 16-bit mode, PUSH defaults to 16-bit operands.
  • In 32/64-bit mode, PUSH defaults to 32/64-bit operands.
  • Push operations implicitly modify the stack pointer.

POP Instruction

Operation Flow:

  1. Read data from the top of the stack
  2. Increment stack pointer (ESP/RSP += operand size)

Examples:

pop ebx         ; Pop into EBX
; After execution:
; EBX = [ESP]
; ESP = ESP + 4

pop word [di]   ; 16-bit pop into memory
; After execution:
; [DI] = [ESP]
; ESP = ESP + 2

Key Rules:

  • The target of POP can be a register or memory location.
  • Cannot pop directly into an immediate value.
  • Pop order must strictly reverse the push order.

Stack Frame Management

A stack frame is temporary space allocated on the stack during a function call, used to store the return address, parameters, local variables, saved registers, etc. Stack frames are managed using the ENTER and LEAVE instructions (or compiler-generated equivalent code).

Typical 32-bit Stack Frame Layout (from top to bottom):

|-------------------|
| Local Variable 3  | ESP+0x0C
| Local Variable 2  | ESP+0x08
| Local Variable 1  | ESP+0x04
| Saved EBX         | ESP+0x00 (caller-saved register)
|-------------------|
| Return Address    | EBP+0x04 (pushed by CALL)
| Saved EBP         | EBP+0x00 (previous EBP value)
|-------------------|
| Parameter 3       | EBP-0x04 (pushed by caller)
| Parameter 2       | EBP-0x08
| Parameter 1       | EBP-0x0C
|-------------------|
| Caller's Stack Frame | (lower addresses)

Key Regions Explained:

  • Return Address: Address of the next instruction pushed by CALL; restored by the callee before returning.
  • Saved Registers: Callee must preserve caller’s registers (e.g., EBX, ESI, EDI) to avoid corruption.
  • Local Variables: Variables defined inside the function (e.g., int a, b;).
  • Parameters: Arguments pushed by the caller (right-to-left order; e.g., func(a,b,c) pushes c → b → a).

ENTER/LEAVE Instructions

ENTER instruction: Automatically sets up a stack frame. Equivalent to (32-bit):

enter imm16, imm8
; Equivalent to:
push ebp            ; Save old base pointer
mov ebp, esp        ; Set new base pointer
sub esp, imm16      ; Allocate space for local variables

LEAVE instruction: Automatically tears down a stack frame. Equivalent to (32-bit):

leave
; Equivalent to:
mov esp, ebp        ; Deallocate local variables
pop ebp             ; Restore old base pointer

Modern Compiler Optimizations:

  • Most compilers replace ENTER with push ebp; mov ebp, esp for clarity and performance.
  • LEAVE is still widely used for readability.

Function Call Mechanism

CALL/RET Instructions

CALL Instruction Execution Flow:

  1. Push return address (address of next instruction) onto stack
  2. Jump to target function address

RET Instruction Execution Flow:

  1. Pop return address from stack top
  2. Jump to that address

Example:

; Caller code
push arg2           ; Push arguments (right to left)
push arg1
call my_func        ; 1. Push return address 2. Jump
add esp, 8          ; Clean up stack arguments (caller responsibility)

; Called function
my_func:
    push ebp        ; Save old EBP
    mov ebp, esp    ; Establish new stack frame
    sub esp, 4      ; Allocate space for local variables
    ; Function body...
    mov esp, ebp    ; Restore ESP
    pop ebp         ; Restore old EBP
    ret             ; Pop return address and jump

Calling Convention Comparison

ConventionParameter PassingStack CleanupPreserved Registers
cdeclRight-to-leftCallerEAX, ECX, EDX
stdcallRight-to-leftCalleeEAX, ECX, EDX
fastcallFirst two args in ECX/EDXCalleeEAX, ECX, EDX
System V ABIRegister-firstCalleeDefined register list

Example: Difference between cdecl and stdcall

  • cdecl: Caller cleans the stack (e.g., after printf("%d", 123), use add esp, 8).
  • stdcall: Callee cleans the stack (e.g., ret 8 pops 8 bytes of arguments).

Stack Safety and Optimization

Stack Alignment Requirements

x86-64 System V ABI requires:

  • Stack pointer must be 16-byte aligned at function entry.
  • Caller must adjust alignment before call:
; Ensure 16-byte alignment before call
and rsp, -16        ; Force alignment to 16-byte boundary
call func

Why Alignment Matters:

  • SSE instructions require 16-byte aligned memory access.
  • Improves cache performance.
  • Some system calls enforce alignment.

Alignment Rules:

  • Natural Alignment: Data type alignment equals its size (e.g., int (4 bytes) aligned to 4-byte multiples; double (8 bytes) to 8-byte multiples).
  • Forced Alignment: Use align pseudo-instruction (e.g., align 16 forces 16-byte alignment).

Example: Compiler Stack Alignment Optimization

GCC automatically aligns the stack (default 16 bytes):

void func() {
    char buf[10]; // 10 bytes, but stack aligned to 16 bytes (allocates 16 bytes)
}

Disassembled:

sub esp, 16     ; Allocates 16 bytes (not 10) to ensure alignment

Buffer Overflow Protection

Common Vulnerability Scenario:

; Unsafe string copy
mov edi, [ebp+8]    ; Destination buffer address
mov esi, [ebp+12]   ; Source string address
mov ecx, 256        ; Fixed length
rep movsb           ; May overflow buffer

Overflow Principle:

void vulnerable() {
    char buf[8];        // 8-byte buffer
    gets(buf);          // Reads from stdin with no length check
}
// Stack layout: buf[0–7] → saved EBP → return address

If input exceeds 8 bytes, it overwrites the saved EBP and return address. The RET instruction then jumps to an attacker-controlled address (e.g., shellcode).

Protection Measures:

  • Stack Canary: Compiler inserts a “canary” value (e.g., 0xdeadbeef) between EBP and local variables; checks it before function return (GCC -fstack-protector).
  • Address Space Layout Randomization (ASLR): OS randomizes base addresses of stack, heap, and libraries, making return address prediction difficult (sysctl -w kernel.randomize_va_space=2 on Linux).
  • Non-Executable Stack (DEP/NX bit): Marks stack memory as non-executable, preventing shellcode execution (Windows PAGE_GUARD, Linux NX bit).
  • Safe Function Alternatives: Use fgets instead of gets, limiting input length (e.g., fgets(buf, sizeof(buf), stdin)).

Protection Techniques Summary:

  • Stack Canaries: Insert random value between EBP and locals; verify before return.
  • ASLR: Randomize stack/heap/library load addresses to increase attack difficulty.
  • Non-Executable Stack (NX): Mark stack as non-executable to block shellcode.

Secure Coding Recommendations:

  • Always check string lengths.
  • Use safe functions (strncpy instead of strcpy).
  • Avoid large buffers on the stack.
  • Enable compiler security options (-fstack-protector).

Stack Optimization: Frame Pointer Omission (FPO)

Modern compilers (e.g., GCC, Clang) enable Frame Pointer Omission (-fomit-frame-pointer) by default, managing the stack directly via general-purpose registers (like ESP) to reduce register pressure and improve performance.

Before Optimization (with frame pointer):

func:
    push ebp            ; Save EBP
    mov ebp, esp        ; EBP points to stack top
    sub esp, 8          ; Allocate local variable space
    ; ... function logic ...
    leave               ; Restore EBP and ESP
    ret

After Optimization (frame pointer omitted):

func:
    sub esp, 8          ; Directly allocate local variable space (ESP = stack top)
    ; ... function logic (access locals directly via ESP) ...
    ret                 ; No need to restore EBP

Impact of Optimization:

  • Advantages: Fewer PUSH/POP instructions → faster execution; frees EBP for other uses.
  • Disadvantages: Harder to traverse stack frames during debugging (requires ESP and symbol table).

Comprehensive Example Analysis

Complete Function Call Example:

section .text
global main
main:
    push rbp
    mov rbp, rsp
    sub rsp, 16                 ; Allocate space for local variables

    ; Prepare arguments (right to left)
    mov rdi, 3                  ; Third argument
    mov rsi, 2                  ; Second argument
    mov rdx, 1                  ; First argument
    call add_three_numbers      ; Parameters passed via registers

    ; No stack cleanup needed (registers used)

    mov rsp, rbp                ; Restore stack pointer
    pop rbp
    ret

add_three_numbers:
    push rbp
    mov rbp, rsp
    sub rsp, 8                  ; Local variable space

    mov [rbp-4], edi            ; Save first parameter
    mov [rbp-8], esi            ; Save second parameter

    ; Compute: rdi + rsi + rdx
    mov eax, edi
    add eax, esi
    add eax, edx

    mov rsp, rbp                ; Restore stack pointer
    pop rbp
    ret

By deeply understanding stack operation mechanisms, you can:

  • Manage function calls more efficiently
  • Write safer assembly code
  • Effectively debug stack-related issues
  • Understand modern compiler optimization strategies
Course: Assembly Language Programming From Low-Level Logic to System Kernel
Share your love

Trending now

Exploring Web 3D Core Technologies Development
Assembly Language Programming From Low-Level Logic to System Kernel
WebRTC Advanced Guide：Media, Transport & Architecture
WebSocket Deep Guide：Building High-Performance Realtime Systems