Lesson 11-Assembly Language and Operating Systems – TianyaSchool – Science, Technology & Engineering Learning Platform

Process and Thread Management

Process Lifecycle Management

Process Creation (fork system call): The kernel duplicates the parent process’s page tables and register state (e.g., the Process Control Block, PCB) via the copy_process function. At the assembly level, the syscall instruction triggers sys_fork, and the kernel copies key registers such as CR3 (page table base register) and EIP (instruction pointer) into the child process’s PCB.

Linux x86_64 fork Example:

mov rax, 57     ; sys_fork system call number
syscall
; Return values:
;   In child process: 0
;   In parent process: PID of child

mov rax, 57     ; sys_fork system call number
syscall
; Return values:
;   In child process: 0
;   In parent process: PID of child

Process Context Switching:

Save context: register states, stack pointer, program counter
Switch page tables: update the CR3 register
Update scheduler queues

Process Termination (exit system call):
A process invokes the exit system call; the kernel releases its memory, closes file descriptors, and finally calls the schedule function to switch to another process.
At the assembly level, the sys_exit handler (e.g., Linux’s do_exit) uses instructions like mov esp, ebp to clean up the stack, releases resources, and ultimately invokes schedule to trigger a context switch.

mov rax, 60     ; sys_exit
mov rdi, 0      ; exit code
syscall

mov rax, 60     ; sys_exit
mov rdi, 0      ; exit code
syscall

Example: Simplified Linux Process Termination in Assembly

; sys_exit system call handler (x86-64)
do_exit:
    mov rdi, rax                ; Save exit status (rax = sys_exit number)
    call do_exit_mm             ; Release memory mappings
    call do_exit_mmap           ; Clean up mm_struct
    call exit_mmu               ; Destroy page tables
    ; Switch to kernel scheduler
    mov rax, TASK_RUNNING
    jmp schedule                ; Invoke scheduler to select next process

; sys_exit system call handler (x86-64)
do_exit:
    mov rdi, rax                ; Save exit status (rax = sys_exit number)
    call do_exit_mm             ; Release memory mappings
    call do_exit_mmap           ; Clean up mm_struct
    call exit_mmu               ; Destroy page tables
    ; Switch to kernel scheduler
    mov rax, TASK_RUNNING
    jmp schedule                ; Invoke scheduler to select next process

Thread Synchronization Mechanisms

Thread synchronization coordinates multi-threaded access to shared resources. Common mechanisms include Mutexes, Semaphores, and Spinlocks. Assembly language directly implements the atomic primitives underlying these mechanisms.

User-Level Thread Context Switching:

; Save current thread context
push rbp
push rbx
push r12
push r13
push r14
push r15

; Restore target thread context
pop r15
pop r14
pop r13
pop r12
pop rbx
pop rbp
ret

; Save current thread context
push rbp
push rbx
push r12
push r13
push r14
push r15

; Restore target thread context
pop r15
pop r14
pop r13
pop r12
pop rbx
pop rbp
ret

Kernel-Level Thread Synchronization – Spinlock Implementation:

; x86_64 spinlock acquire
spin_lock:
    mov al, 1
1:  xchg al, [rdi]      ; Atomic exchange
    test al, al
    jnz 1b              ; Retry if lock was held
    ret

; Spinlock release
spin_unlock:
    mov byte [rdi], 0   ; Clear lock
    ret

; x86_64 spinlock acquire
spin_lock:
    mov al, 1
1:  xchg al, [rdi]      ; Atomic exchange
    test al, al
    jnz 1b              ; Retry if lock was held
    ret

; Spinlock release
spin_unlock:
    mov byte [rdi], 0   ; Clear lock
    ret

x86 Assembly Implementation of Spinlock

// Spinlock structure (simplified)
struct spinlock {
    volatile int lock;  // 0 = unlocked, 1 = locked
};

// Spinlock structure (simplified)
struct spinlock {
    volatile int lock;  // 0 = unlocked, 1 = locked
};

; Acquire lock
spin_lock:
    mov eax, 1          ; Lock value = 1
.spin_loop:
    lock cmpxchg dword [rdi], eax   ; Atomic compare-and-swap: set to 1 if [rdi]==0
    jnz .spin_loop      ; Retry on failure (already locked)
    ret

; Release lock
spin_unlock:
    mov dword [rdi], 0  ; Directly release (memory barrier optional on x86 strong model)
    mfence              ; Memory fence (optional but safe)
    ret

; Acquire lock
spin_lock:
    mov eax, 1          ; Lock value = 1
.spin_loop:
    lock cmpxchg dword [rdi], eax   ; Atomic compare-and-swap: set to 1 if [rdi]==0
    jnz .spin_loop      ; Retry on failure (already locked)
    ret

; Release lock
spin_unlock:
    mov dword [rdi], 0  ; Directly release (memory barrier optional on x86 strong model)
    mfence              ; Memory fence (optional but safe)
    ret

Inter-Process Communication (IPC)

IPC is essential for process collaboration. Common methods include Pipes, Message Queues, and Shared Memory. Assembly language participates in low-level register and memory operations for IPC.

Memory Mapping:
Via shmget/shmat system calls, the same physical memory is mapped into multiple processes’ virtual address spaces.
At the assembly level, the kernel modifies each process’s page tables (via CR3) to establish the mapping. Processes then use mov instructions to read/write shared memory directly.

Shared Memory Access in Assembly

// C code: create shared memory
int shmid = shmget(IPC_PRIVATE, 4096, IPC_CREAT | 0666);
char *shmaddr = shmat(shmid, NULL, 0);

// C code: create shared memory
int shmid = shmget(IPC_PRIVATE, 4096, IPC_CREAT | 0666);
char *shmaddr = shmat(shmid, NULL, 0);

; Process A writes data
mov edi, shmaddr        ; EDI points to shared memory
mov eax, 'Hello'        ; EAX holds data
mov [edi], eax          ; Write to shared memory

; Process B reads data
mov edi, shmaddr
mov eax, [edi]          ; Read from shared memory

; Process A writes data
mov edi, shmaddr        ; EDI points to shared memory
mov eax, 'Hello'        ; EAX holds data
mov [edi], eax          ; Write to shared memory

; Process B reads data
mov edi, shmaddr
mov eax, [edi]          ; Read from shared memory

Pipe Communication Implementation:

; Create pipe
mov rax, 22             ; sys_pipe
mov rdi, pipe_fd        ; Address of fd array
syscall

; Write to pipe
mov rax, 1              ; sys_write
mov rdi, [pipe_fd]      ; Write-end fd
mov rsi, msg
mov rdx, len
syscall

; Read from pipe
mov rax, 0              ; sys_read
mov rdi, [pipe_fd+4]    ; Read-end fd
mov rsi, buf
mov rdx, bufsize
syscall

; Create pipe
mov rax, 22             ; sys_pipe
mov rdi, pipe_fd        ; Address of fd array
syscall

; Write to pipe
mov rax, 1              ; sys_write
mov rdi, [pipe_fd]      ; Write-end fd
mov rsi, msg
mov rdx, len
syscall

; Read from pipe
mov rax, 0              ; sys_read
mov rdi, [pipe_fd+4]    ; Read-end fd
mov rsi, buf
mov rdx, bufsize
syscall

Shared Memory Operations:

; Create shared memory segment
mov rax, 31                     ; sys_shmget
mov rdi, key
mov rsi, size
mov rdx, IPC_CREAT|0666
syscall

; Attach shared memory
mov rax, 30                     ; sys_shmat
mov rdi, shmid
mov rsi, 0                      ; Default address
mov rdx, 0
syscall

; Create shared memory segment
mov rax, 31                     ; sys_shmget
mov rdi, key
mov rsi, size
mov rdx, IPC_CREAT|0666
syscall

; Attach shared memory
mov rax, 30                     ; sys_shmat
mov rdi, shmid
mov rsi, 0                      ; Default address
mov rdx, 0
syscall

Thread Synchronization Primitives

The kernel uses synchronization primitives (e.g., spinlock, mutex) to protect critical sections (e.g., PCB or page table manipulation). Assembly directly implements their low-level logic.

Key Primitive Example: Mutex with Ticket Mechanism
Uses atomic_t-typed owner and ticket fields for fairness.

Assembly Optimization: Uses the LOCK prefix for atomicity and reduces cache coherence traffic.

Simplified Kernel Mutex Implementation in Assembly:

// Mutex structure
struct mutex {
    atomic_t owner;   // CPU ID of current holder
    atomic_t ticket;  // Next ticket number
};

// Mutex structure
struct mutex {
    atomic_t owner;   // CPU ID of current holder
    atomic_t ticket;  // Next ticket number
};

; Acquire mutex
mutex_lock:
    mov eax, smp_processor_id()         ; Get current CPU ID
    lock xadd dword [rdi + ticket], eax ; Atomically increment ticket (eax = old value)
.wait_loop:
    cmp dword [rdi + owner], eax        ; Check if it's our turn
    je .lock_acquired
    pause                               ; Reduce CPU usage (spinlock optimization)
    jmp .wait_loop
.lock_acquired:
    ret

; Release mutex
mutex_unlock:
    mov eax, smp_processor_id()
    lock cmpxchg dword [rdi + owner], eax   ; Atomically release
    jne .error                              ; Error if not owner
    ret
.error:
    ; Handle error (e.g., panic)
    ret

; Acquire mutex
mutex_lock:
    mov eax, smp_processor_id()         ; Get current CPU ID
    lock xadd dword [rdi + ticket], eax ; Atomically increment ticket (eax = old value)
.wait_loop:
    cmp dword [rdi + owner], eax        ; Check if it's our turn
    je .lock_acquired
    pause                               ; Reduce CPU usage (spinlock optimization)
    jmp .wait_loop
.lock_acquired:
    ret

; Release mutex
mutex_unlock:
    mov eax, smp_processor_id()
    lock cmpxchg dword [rdi + owner], eax   ; Atomically release
    jne .error                              ; Error if not owner
    ret
.error:
    ; Handle error (e.g., panic)
    ret

Semaphore Implementation:

; Semaphore P operation (wait)
sem_wait:
    mov eax, [rdi]          ; Load semaphore value
    test eax, eax
    jg .success             ; If > 0, proceed
    ; Otherwise, block current thread (call scheduler)
    call block_thread
.success:
    dec dword [rdi]         ; Decrement count
    ret

; Semaphore V operation (signal)
sem_post:
    inc dword [rdi]         ; Increment count
    ; Wake up waiting thread(s)
    call wakeup_thread
    ret

; Semaphore P operation (wait)
sem_wait:
    mov eax, [rdi]          ; Load semaphore value
    test eax, eax
    jg .success             ; If > 0, proceed
    ; Otherwise, block current thread (call scheduler)
    call block_thread
.success:
    dec dword [rdi]         ; Decrement count
    ret

; Semaphore V operation (signal)
sem_post:
    inc dword [rdi]         ; Increment count
    ; Wake up waiting thread(s)
    call wakeup_thread
    ret

Kernel Assembly Code Analysis

Kernel context switching is a core application of assembly, involving register saving, page table switching, and stack switching.

Key Steps in Linux Context Switching (Simplified):

Save current process context: Push EAX, EBX, ECX, EDX, ESI, EDI, EBP, ESP, EIP onto the current kernel stack.
Switch page tables: Update CR3 to point to the new process’s page table base.
Restore target process context: Pop registers from the target’s kernel stack.

Example: Core Context Switch Snippet (schedule function):

; schedule function (simplified)
schedule:
    ; Save current process context
    pusha                           ; Push all general-purpose registers (EAX–EDI)
    mov eax, esp                    ; EAX = current stack top (saved registers)
    mov [current->thread.sp], eax   ; Save current kernel stack pointer

    ; Select next process (via scheduling algorithm)
    call __schedule                 ; Internally selects 'next' process
    mov current, next               ; Update current process pointer

    ; Switch page tables (CR3)
    mov eax, [next->thread.cr3]
    mov cr3, eax                    ; Switch to target process’s page table

    ; Restore target process context
    mov eax, [next->thread.sp]
    mov esp, eax                    ; Restore target kernel stack pointer
    popa                            ; Restore all general-purpose registers
    ret                             ; Return to target process’s execution point

; schedule function (simplified)
schedule:
    ; Save current process context
    pusha                           ; Push all general-purpose registers (EAX–EDI)
    mov eax, esp                    ; EAX = current stack top (saved registers)
    mov [current->thread.sp], eax   ; Save current kernel stack pointer

    ; Select next process (via scheduling algorithm)
    call __schedule                 ; Internally selects 'next' process
    mov current, next               ; Update current process pointer

    ; Switch page tables (CR3)
    mov eax, [next->thread.cr3]
    mov cr3, eax                    ; Switch to target process’s page table

    ; Restore target process context
    mov eax, [next->thread.sp]
    mov esp, eax                    ; Restore target kernel stack pointer
    popa                            ; Restore all general-purpose registers
    ret                             ; Return to target process’s execution point

x86_64 Process Switching Core Logic:

; switch_to macro/function
switch_to:
    ; Save current process state
    mov [prev_thread + RSP_OFFSET], rsp
    mov [prev_thread + RBP_OFFSET], rbp

    ; Load new process state
    mov rsp, [next_thread + RSP_OFFSET]
    mov rbp, [next_thread + RBP_OFFSET]

    ; Switch kernel stacks
    mov rsp, [next_thread + KERNEL_RSP_OFFSET]

    ; Return to new process
    ret

; switch_to macro/function
switch_to:
    ; Save current process state
    mov [prev_thread + RSP_OFFSET], rsp
    mov [prev_thread + RBP_OFFSET], rbp

    ; Load new process state
    mov rsp, [next_thread + RSP_OFFSET]
    mov rbp, [next_thread + RBP_OFFSET]

    ; Switch kernel stacks
    mov rsp, [next_thread + KERNEL_RSP_OFFSET]

    ; Return to new process
    ret

Memory Management Mechanisms

Memory management enables isolation and efficient utilization through virtual memory, paging, and dynamic allocation. Assembly directly manipulates page tables and the CR3 register.

Virtual Memory Architecture

Virtual memory uses page tables to map virtual addresses (VA) to physical addresses (PA), enabling isolation and overcommitment.

Address Translation Flow:

CPU generates VA
MMU uses CR3 to locate page table base
Multi-level page table walk (Page Directory → Page Table → Page Frame)
TLB caches translations for speed
Final PA accesses physical memory

Page Table Entry (PTE) Flags:

Flag	Meaning
P	Present
R/W	Read/Write
U/S	User/Supervisor
A	Accessed
D	Dirty
PAT	Memory Type

Paging Mechanism Details

Paging divides virtual memory into fixed-size pages (e.g., 4KB). x86-64 uses a 4-level page table.

Translation Flow (x86-64, 4-Level Paging):

VA Decomposition: 64-bit VA split into four indices (PML4, PDPT, PD, PT) + 12-bit offset.
Multi-Level Lookup:
- PML4 index (VA[47:39]) → PML4 entry → PDPT
- PDPT index (VA[38:30]) → PDPT entry → PD
- PD index (VA[29:21]) → PD entry → PT
- PT index (VA[20:12]) → PT entry → Physical Page Frame
PA Calculation: Physical Page Base + Offset (VA[11:0]) = PA

Example: VA = 0x00007fffffffffff → PA
Assume PT entry points to physical frame 0x0000000100000000, offset = 0xfff → PA = 0x0000000100000fff.

4-Level Page Table Structure (x86_64):
CR3 → PML4 Table → PDPT Table → PD Table → PT Table → Physical Page Frame

TLB Management Instruction:

invlpg [mem]    ; Invalidate TLB entry for specified address

invlpg [mem]    ; Invalidate TLB entry for specified address

Large Page Support (2MB/1GB):

; Set large page bit in PDE/PDPTE
mov eax, pde_value
or eax, 0x80        ; PS=1 (Page Size flag)
mov [pde_addr], eax

; Set large page bit in PDE/PDPTE
mov eax, pde_value
or eax, 0x80        ; PS=1 (Page Size flag)
mov [pde_addr], eax

Dynamic Memory Allocation

Dynamic allocation (e.g., malloc) relies on kernel allocators like the Buddy System or Slab Allocator.

Assembly-Level Key Operations:

System Call Trigger: malloc invokes brk or mmap (int 0x80 or syscall).
Page Table Update: Kernel maps new physical pages and sets permissions.

brk System Call Implementation:

; Adjust program break (heap top)
mov rax, 12         ; sys_brk
mov rdi, new_brk
syscall
cmp rax, old_brk
je .no_change       ; No change if failed
; Update heap metadata...
.no_change:

; Adjust program break (heap top)
mov rax, 12         ; sys_brk
mov rdi, new_brk
syscall
cmp rax, old_brk
je .no_change       ; No change if failed
; Update heap metadata...
.no_change:

mmap System Call:

; Map file or anonymous memory
mov rax, 9          ; sys_mmap
mov rdi, addr       ; Suggested address (usually NULL)
mov rsi, length
mov rdx, prot       ; Protection flags (e.g., PROT_READ|PROT_WRITE)
mov r10, flags      ; e.g., MAP_SHARED
mov r8, fd          ; File descriptor
mov r9, offset
syscall

; Map file or anonymous memory
mov rax, 9          ; sys_mmap
mov rdi, addr       ; Suggested address (usually NULL)
mov rsi, length
mov rdx, prot       ; Protection flags (e.g., PROT_READ|PROT_WRITE)
mov r10, flags      ; e.g., MAP_SHARED
mov r8, fd          ; File descriptor
mov r9, offset
syscall

Stack Management Mechanism

The stack stores return addresses, local variables, and register contexts during function calls. Assembly directly manages stack growth (sub rsp, 8) and shrinkage (add rsp, 8).

Key Rules:

Growth Direction: x86/x86-64 stacks grow downward (RSP decreases).
Alignment: Stack pointer must be aligned (e.g., 16-byte for SSE).
Overflow Protection: Stack canaries detect overflow (e.g., GCC -fstack-protector).

Example: Function Call Stack Operations

; Call func(a, b)
push dword [b]      ; Push 2nd arg (stack grows)
push dword [a]      ; Push 1st arg
call func           ; Push return address (RIP), RSP -= 8
add rsp, 8          ; Clean up args (caller cleanup, cdecl)

; Inside func:
func:
    push rbp                ; Save caller's RBP
    mov rbp, rsp            ; Set up new frame
    sub rsp, 8              ; Allocate 8 bytes for locals
    ; ... function body ...
    mov rsp, rbp            ; Deallocate locals
    pop rbp                 ; Restore caller's RBP
    ret                     ; Pop return address, jump back

; Call func(a, b)
push dword [b]      ; Push 2nd arg (stack grows)
push dword [a]      ; Push 1st arg
call func           ; Push return address (RIP), RSP -= 8
add rsp, 8          ; Clean up args (caller cleanup, cdecl)

; Inside func:
func:
    push rbp                ; Save caller's RBP
    mov rbp, rsp            ; Set up new frame
    sub rsp, 8              ; Allocate 8 bytes for locals
    ; ... function body ...
    mov rsp, rbp            ; Deallocate locals
    pop rbp                 ; Restore caller's RBP
    ret                     ; Pop return address, jump back

User Stack ↔ Kernel Stack Switching:

; On interrupt: hardware auto-switches stack
; TSS pre-stores kernel stack pointer
mov [tss + RSP0], kernel_rsp
; CPU automatically switches on privilege change

; On interrupt: hardware auto-switches stack
; TSS pre-stores kernel stack pointer
mov [tss + RSP0], kernel_rsp
; CPU automatically switches on privilege change

Stack Growth Protection:

; Detect overflow via guard page
mov rax, [rsp]
sub rax, 0x1000         ; Probe next page
mov [rax], 0            ; Trigger page fault if unmapped

; Detect overflow via guard page
mov rax, [rsp]
sub rax, 0x1000         ; Probe next page
mov [rax], 0            ; Trigger page fault if unmapped

Memory Protection Techniques

Memory protection uses page table attributes (R/W/X, U/S) and memory region flags (e.g., .text read-only) to enforce security.

Key Mechanisms:

R/W/X Bits: R/W controls write permission; NX (No-eXecute) bit prevents code injection.
U/S Bit: Restricts kernel memory to supervisor mode.

Example: Set Page Table Entry to Read-Only

; Assume pt_entry points to a PTE
mov eax, [pt_entry]     ; Load PTE
and eax, ~0x2           ; Clear R/W bit (0x2 = write-enable mask)
or eax, 0x1             ; Set P (present) and U/S (user-accessible)
mov [pt_entry], eax     ; Write back (now read-only)

; Assume pt_entry points to a PTE
mov eax, [pt_entry]     ; Load PTE
and eax, ~0x2           ; Clear R/W bit (0x2 = write-enable mask)
or eax, 0x1             ; Set P (present) and U/S (user-accessible)
mov [pt_entry], eax     ; Write back (now read-only)

mprotect System Call:

; Change memory region permissions
mov rax, 10             ; sys_mprotect
mov rdi, addr           ; Start address
mov rsi, len            ; Length
mov rdx, prot           ; New protection (e.g., PROT_READ|PROT_EXEC)
syscall

; Change memory region permissions
mov rax, 10             ; sys_mprotect
mov rdi, addr           ; Start address
mov rsi, len            ; Length
mov rdx, prot           ; New protection (e.g., PROT_READ|PROT_EXEC)
syscall

W^X (Write XOR Execute) Policy:
Memory pages cannot be both writable and executable—defends against code injection.
Implementation:

; Make memory read-only + executable
mov rdx, PROT_READ | PROT_EXEC
call mprotect

; Make memory read-only + executable
mov rdx, PROT_READ | PROT_EXEC
call mprotect

Interrupt and Exception Handling

Interrupts include hardware interrupts (keyboard, disk), software interrupts (int 0x80), and exceptions (divide-by-zero). They are either asynchronous (interrupts) or synchronous (exceptions).

Interrupt Classification:

Type	Source	Sync/Async	Example
Hardware Interrupt	External device	Async	Keyboard, timer
Software Interrupt	CPU instruction	Sync	`INT 0x80`
Exception	CPU execution	Sync	Page fault, divide-by-zero

Interrupt Priority (Example):
Machine Check (highest) → NMI → Hardware Interrupts → Software Interrupts (lowest)

Full Interrupt Response Process:

Save context: RFLAGS, CS, RIP
Look up IDT: offset = interrupt number × entry size
Permission check: CPL vs DPL
Stack switch: user → kernel (if needed)
Execute ISR (Interrupt Service Routine)
Return via IRET

Example: x86 Hardware Interrupt Handler (Timer)

timer_interrupt:
    pusha                       ; Save all general-purpose registers
    call update_system_time     ; Handle timer logic
    mov al, 0x20                ; Send EOI to APIC
    out 0x20, al                ; To master PIC
    popa                        ; Restore registers
    iret                        ; Return from interrupt

timer_interrupt:
    pusha                       ; Save all general-purpose registers
    call update_system_time     ; Handle timer logic
    mov al, 0x20                ; Send EOI to APIC
    out 0x20, al                ; To master PIC
    popa                        ; Restore registers
    iret                        ; Return from interrupt

IDT Initialization Example:

; Set IDT entry for IRQ0 (timer)
mov eax, interrupt_handler
mov [idt + irq0*8], eax
mov [idt + irq0*8 + 4], 0x08E00   ; 32-bit interrupt gate, DPL=0

; Set IDT entry for IRQ0 (timer)
mov eax, interrupt_handler
mov [idt + irq0*8], eax
mov [idt + irq0*8 + 4], 0x08E00   ; 32-bit interrupt gate, DPL=0

Exception Handling Mechanism

Exceptions are triggered by the CPU; the kernel handles them via registered handlers.

Page Fault Example:

Trigger: Access unmapped VA → CR2 holds fault address
IDT Lookup: Interrupt 0x0E → page_fault_handler
Handler Logic: Load missing page or terminate process

Page Fault Handler Snippet:

page_fault_handler:
    pusha
    mov rax, cr2                ; RAX = fault address
    ; Check if due to non-present page (P=0)
    test dword [rax + page_table_entry], 0x1
    jz load_page                ; Not present → load it
    ; Else: permission error → send SIGSEGV
    mov rdi, SIGSEGV
    call send_signal_to_process
    jmp exit_process

load_page:
    call load_physical_page     ; Allocate and load page
    or dword [rax + page_table_entry], 0x1  ; Mark as present

exit_process:
    popa
    add rsp, 8                  ; Adjust for error code (if any)
    iret

page_fault_handler:
    pusha
    mov rax, cr2                ; RAX = fault address
    ; Check if due to non-present page (P=0)
    test dword [rax + page_table_entry], 0x1
    jz load_page                ; Not present → load it
    ; Else: permission error → send SIGSEGV
    mov rdi, SIGSEGV
    call send_signal_to_process
    jmp exit_process

load_page:
    call load_physical_page     ; Allocate and load page
    or dword [rax + page_table_entry], 0x1  ; Mark as present

exit_process:
    popa
    add rsp, 8                  ; Adjust for error code (if any)
    iret

Common Exception Handlers:

; Divide-by-zero (#DE)
_divide_error:
    push error_code
    call handle_div_zero
    add rsp, 8
    iret

; Page fault (#PF)
_page_fault:
    mov rax, cr2                ; Get fault address
    call handle_page_fault
    iret

; Divide-by-zero (#DE)
_divide_error:
    push error_code
    call handle_div_zero
    add rsp, 8
    iret

; Page fault (#PF)
_page_fault:
    mov rax, cr2                ; Get fault address
    call handle_page_fault
    iret

Exception Stack Frame Layout:

+------------------+
| Error Code (if any) |
+------------------+
| Return Address (RIP)|
+------------------+
| CS               |
+------------------+
| RFLAGS           |
+------------------+
| User RSP         |
+------------------+
| User SS          |
+------------------+

+------------------+
| Error Code (if any) |
+------------------+
| Return Address (RIP)|
+------------------+
| CS               |
+------------------+
| RFLAGS           |
+------------------+
| User RSP         |
+------------------+
| User SS          |
+------------------+

Interrupt Descriptor Table (IDT)

The IDT stores entries for interrupt/exception handlers. Each entry is a gate descriptor specifying handler address and privilege.

32-bit Gate Descriptor Format:

Field	Size	Description
Segment Selector	16-bit	Points to GDT/LDT code segment (sets DPL)
Offset (Low)	16-bit	Low 16 bits of handler address
P (Present)	1-bit	1 = valid, 0 = invalid (#NP exception)
DPL	2-bit	Descriptor Privilege Level (0–3)
Type	4-bit	0xE = interrupt gate (disables IF), 0xF = trap gate
Offset (High)	16-bit	High 16 bits of handler address

Example: Timer Interrupt (0x20) IDT Entry

; Handler at 0x00100000, kernel code segment 0x08 (DPL=0), interrupt gate
timer_idt_entry:
    dw 0x0000           ; Offset low
    dw 0x08             ; Segment selector
    db 0x00             ; Reserved
    db 0x8E             ; Type=0xE (interrupt gate), DPL=0, P=1
    dw 0x0000           ; Offset high
    dd 0x00100000       ; Full 32-bit offset (for clarity)

; Handler at 0x00100000, kernel code segment 0x08 (DPL=0), interrupt gate
timer_idt_entry:
    dw 0x0000           ; Offset low
    dw 0x08             ; Segment selector
    db 0x00             ; Reserved
    db 0x8E             ; Type=0xE (interrupt gate), DPL=0, P=1
    dw 0x0000           ; Offset high
    dd 0x00100000       ; Full 32-bit offset (for clarity)

64-bit IDT Entry Format:

63..56 55..48 47..40 39..32 31..16        15..0
+------+------+------+------+----------------+-----------+
| SegSel|  IST | Type | DPL  | P  | Offset High 32 | Offset Low 32 |
+------+------+------+------+----------------+-----------+

63..56 55..48 47..40 39..32 31..16        15..0
+------+------+------+------+----------------+-----------+
| SegSel|  IST | Type | DPL  | P  | Offset High 32 | Offset Low 32 |
+------+------+------+------+----------------+-----------+

Load IDTR Register:

lidt [idtr_struct]      ; IDTR contains base address and limit

lidt [idtr_struct]      ; IDTR contains base address and limit

Interrupt Control Techniques

Interrupt control prevents storms and ensures critical code executes uninterrupted.

Key Techniques:

Interrupt Masking: CLI clears IF (disable interrupts); STI sets IF (enable).
Priority Scheduling: APIC assigns priorities (0–15); higher priority interrupts preempt.

Example: Critical Section with Interrupt Masking

cli                     ; Disable interrupts
; Modify global data structure
mov eax, [global_var]
inc eax
mov [global_var], eax
sti                     ; Re-enable interrupts

cli                     ; Disable interrupts
; Modify global data structure
mov eax, [global_var]
inc eax
mov [global_var], eax
sti                     ; Re-enable interrupts

PIC/APIC Control:

; Mask IRQ2 on master PIC
in al, 0x21
or al, 0x04
out 0x21, al

; Enable APIC
mov ecx, 0x808          ; IA32_APIC_BASE MSR
rdmsr
or eax, 0x100           ; Set enable bit
wrmsr

; Set 8259A PIC priority
mov al, 0x11            ; ICW1
out 0x20, al
mov al, 0x20            ; Map IRQ0 to INT 0x20
out 0x21, al

; Mask IRQ2 on master PIC
in al, 0x21
or al, 0x04
out 0x21, al

; Enable APIC
mov ecx, 0x808          ; IA32_APIC_BASE MSR
rdmsr
or eax, 0x100           ; Set enable bit
wrmsr

; Set 8259A PIC priority
mov al, 0x11            ; ICW1
out 0x20, al
mov al, 0x20            ; Map IRQ0 to INT 0x20
out 0x21, al

System Programming Interfaces (SPI)

SPIs enable user programs to interact with the OS via system calls, kernel APIs, and drivers.

System Call Implementation Comparison:

Feature	Linux (x86-64)	Windows (x86-64)
Trigger Instruction	`syscall`	`syscall` (via `ntdll.dll`)
Parameter Passing	First 6 args in `rdi`–`r9`	First 4 args in `rcx`–`r9`; rest on stack
System Call Number	In `rax` (e.g., `SYS_write=1`)	In `rax` (e.g., `NtWriteFile=0x102`)
Return Value	In `rax` (0 = success)	In `rax` (NTSTATUS code)

Linux syscall Example:

mov rax, 1      ; sys_write
mov rdi, 1      ; stdout
mov rsi, msg
mov rdx, len
syscall

mov rax, 1      ; sys_write
mov rdi, 1      ; stdout
mov rsi, msg
mov rdx, len
syscall

Windows SYSENTER Example:

mov eax, 0x7FFE0300     ; SYSENTER_EIP_MSR
mov edx, param_block
sysenter

mov eax, 0x7FFE0300     ; SYSENTER_EIP_MSR
mov edx, param_block
sysenter

Cross-Platform Wrapper:

#ifdef __linux__
#define SYS_CALL(num, ...) syscall(num, ##__VA_ARGS__)
#elif _WIN32
#define SYS_CALL(num, ...) __syscall(num, ##__VA_ARGS__)
#endif

#ifdef __linux__
#define SYS_CALL(num, ...) syscall(num, ##__VA_ARGS__)
#elif _WIN32
#define SYS_CALL(num, ...) __syscall(num, ##__VA_ARGS__)
#endif

Kernel Interface Programming

Kernel modules (e.g., drivers) directly manipulate kernel structures like task_struct and file_operations.

Key Steps (Linux Kernel Module):

Module Init: Register with module_init, allocate resources.
Implement Ops: Define open, read, write, etc.
Module Exit: Release resources with module_exit.

Example: Linux Character Device Driver

#include <linux/module.h>
#include <linux/fs.h>

static int device_open(struct inode *inode, struct file *filp) {
    printk(KERN_INFO "Device opened\n");
    return 0;
}

static ssize_t device_read(struct file *filp, char __user *buf, size_t count, loff_t *f_pos) {
    printk(KERN_INFO "Device read\n");
    return 0;
}

static struct file_operations fops = {
    .owner = THIS_MODULE,
    .open = device_open,
    .read = device_read,
};

static int __init my_init(void) {
    register_chrdev(0, "my_device", &fops); // Register char device
    return 0;
}

static void __exit my_exit(void) {
    unregister_chrdev(0, "my_device"); // Unregister
}

module_init(my_init);
module_exit(my_exit);
MODULE_LICENSE("GPL");

#include <linux/module.h>
#include <linux/fs.h>

static int device_open(struct inode *inode, struct file *filp) {
    printk(KERN_INFO "Device opened\n");
    return 0;
}

static ssize_t device_read(struct file *filp, char __user *buf, size_t count, loff_t *f_pos) {
    printk(KERN_INFO "Device read\n");
    return 0;
}

static struct file_operations fops = {
    .owner = THIS_MODULE,
    .open = device_open,
    .read = device_read,
};

static int __init my_init(void) {
    register_chrdev(0, "my_device", &fops); // Register char device
    return 0;
}

static void __exit my_exit(void) {
    unregister_chrdev(0, "my_device"); // Unregister
}

module_init(my_init);
module_exit(my_exit);
MODULE_LICENSE("GPL");

ioctl System Call:

mov rax, 16         ; sys_ioctl
mov rdi, fd
mov rsi, request
mov rdx, args
syscall

mov rax, 16         ; sys_ioctl
mov rdi, fd
mov rsi, request
mov rdx, args
syscall

ptrace Debugging Interface:

mov rax, 101        ; sys_ptrace
mov rdi, PTRACE_GETREGS
mov rsi, pid
mov rdx, &regs
syscall

mov rax, 101        ; sys_ptrace
mov rdi, PTRACE_GETREGS
mov rsi, pid
mov rdx, ®s
syscall

Driver Development Essentials

Drivers mediate between hardware and OS using Port I/O (x86) or MMIO (Memory-Mapped I/O).

Key Techniques:

Port I/O: in/out instructions (e.g., keyboard port 0x60).
MMIO: mov to memory-mapped registers (e.g., PCIe BAR space).
Interrupt Handling: Register handler via request_irq.

Example: x86 Keyboard Driver (Port I/O)

#include <linux/interrupt.h>
#include <linux/io.h>

#define KEYBOARD_PORT 0x60

static irqreturn_t keyboard_irq_handler(int irq, void *dev_id) {
    unsigned char key_code = inb(KEYBOARD_PORT); // Read from port
    printk(KERN_INFO "Key pressed: 0x%02x\n", key_code);
    return IRQ_HANDLED;
}

static int __init keyboard_init(void) {
    request_irq(1, keyboard_irq_handler, IRQF_SHARED, "keyboard", NULL);
    return 0;
}

static void __exit keyboard_exit(void) {
    free_irq(1, NULL);
}

module_init(keyboard_init);
module_exit(keyboard_exit);
MODULE_LICENSE("GPL");

#include <linux/interrupt.h>
#include <linux/io.h>

#define KEYBOARD_PORT 0x60

static irqreturn_t keyboard_irq_handler(int irq, void *dev_id) {
    unsigned char key_code = inb(KEYBOARD_PORT); // Read from port
    printk(KERN_INFO "Key pressed: 0x%02x\n", key_code);
    return IRQ_HANDLED;
}

static int __init keyboard_init(void) {
    request_irq(1, keyboard_irq_handler, IRQF_SHARED, "keyboard", NULL);
    return 0;
}

static void __exit keyboard_exit(void) {
    free_irq(1, NULL);
}

module_init(keyboard_init);
module_exit(keyboard_exit);
MODULE_LICENSE("GPL");

Kernel Module Assembly Example:

section .text
global _start
_start:
    push rbp
    mov rbp, rsp
    mov rax, ksys_some_function
    call rax
    leave
    ret

section .text
global _start
_start:
    push rbp
    mov rbp, rsp
    mov rax, ksys_some_function
    call rax
    leave
    ret

IOCTL Command Definition:

#define MY_IOCTL_CMD _IOW('M', 1, struct my_args)

#define MY_IOCTL_CMD _IOW('M', 1, struct my_args)

Dynamic Linking Mechanism

Dynamic linking uses linkers (e.g., ld-linux.so) to load shared libraries (e.g., libc.so) into process address space.

Assembly-Level Operations:

DT_NEEDED Entries: ELF .dynamic section lists required libraries.
Relocation: Linker resolves addresses (e.g., R_X86_64_PC32).

Example: ELF DT_NEEDED Entry

Dynamic section at offset 0x200 contains 24 entries:
  Tag        Type                         Name/Value
  0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
  0x000000000000000c (INIT)               0x4004d0
  ...

Dynamic section at offset 0x200 contains 24 entries:
  Tag        Type                         Name/Value
  0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
  0x000000000000000c (INIT)               0x4004d0
  ...

PLT/GOT Resolution:

First call: PLT → GOT → dynamic linker
Subsequent: PLT → GOT (direct function address)

ELF Relocation Example:

mov rax, [got_entry]
test rax, rax
jnz .resolved
call _dl_runtime_resolve     ; Resolve symbol
.resolved:

mov rax, [got_entry]
test rax, rax
jnz .resolved
call _dl_runtime_resolve     ; Resolve symbol
.resolved:

Lazy Binding Optimization:
Resolve symbols on first call to reduce startup time.

PLT Entry Template:

plt_entry:
    jmp *got_entry
    push plt_index
    jmp _dl_runtime_resolve

plt_entry:
    jmp *got_entry
    push plt_index
    jmp _dl_runtime_resolve

Summary and Outlook

The integration of assembly language and operating systems is fundamental to understanding low-level computer operation. From process context switching and memory paging to IDT-based interrupt handling and syscall-based system calls, assembly directly underpins every critical OS component.

Looking forward, with the rise of heterogeneous computing (ARM, RISC-V) and containerization (Docker, Kubernetes), the interaction between assembly and operating systems will become even more diverse. Developers must continuously deepen their grasp of底层 principles and combine them with toolchains (perf, GDB) and real-world scenarios to excel in systems programming.

Course: Assembly Language Programming From Low-Level Logic to System Kernel