Lesson 27-Assembly Applications in Operating System Kernel Development

There’s a persistent myth in the software engineering world that operating system kernels are written entirely in C or C++. “Why would you need assembly?” the thinking goes. “Modern compilers generate great code. Just write everything in a high-level language and let the compiler do the work.”

If only things were that simple.

The reality is that every production operating system kernel—Linux, Windows, macOS, FreeBSD, you name it—has significant amounts of assembly language code. Not just a few lines here and there for “performance-critical sections,” but thousands upon thousands of lines of carefully crafted assembly that handles the tasks that C simply can’t do.

In this article, we’re going to dive deep into the world of OS kernel development with assembly language. We’ll look at exactly why kernels still need assembly, which parts of the kernel have to be written in assembly, and how to write those parts correctly. We’ll cover context switching (the crown jewel of OS assembly code), interrupt handling, system call dispatch, SMP initialization, and a whole lot more.

By the time we’re done, you’ll understand not just that kernels use assembly, but exactly where and exactly why. And you’ll have the knowledge you need to write your own OS kernel assembly code—whether for a hobbyist project or just to understand how your computer really works.

Let’s get started.

Why OS Kernels Still Have Assembly: Context Switching, Interrupt Handling, CPU-Specific Initialization

Let me start with the obvious question: Why? Why, in 2024, with compilers that can optimize the hell out of C code, do operating system kernels still need assembly language?

The answer comes down to three fundamental facts about modern CPUs and operating systems:

  1. There are things that C can’t do. C gives you a portable, high-level abstraction over the hardware. But the hardware can do things that C simply can’t represent. Switching the CPU from user mode to kernel mode. Loading a new GDT (Global Descriptor Table). Executing a “HLT” instruction to put the CPU to sleep. These are things that only assembly can do, because they require executing specific CPU instructions that have no C equivalent.
  2. Someone has to set up the execution environment that C expects. C assumes that there’s a stack, that memory is set up, that interrupts are configured, that there’s a main() function to call. But something has to set all of that up before main() can run. That “something” is assembly code—the very first code that runs when the kernel is loaded.
  3. Performance matters in the kernel. The kernel is the most performance-critical code on the system. A slow syscall dispatch means every system call is slow. A slow context switch means every thread switch is slow. A slow interrupt handler means every I/O operation is slow. And while compilers are good, they’re not perfect—there are still places where hand-written assembly can outperform even the best compiler output.

Let’s break this down into the specific places where assembly is absolutely required in an OS kernel.

Context Switching

A context switch is what happens when the operating system switches the CPU from running one thread to running another. The kernel has to:

  1. Save the entire state (all registers) of the currently running thread
  2. Choose the next thread to run (the scheduler)
  3. Restore the entire state of the next thread
  4. Jump to where that thread was last executing

This is fundamentally an assembly-language operation. Why? Because you can’t save the CPU registers in C! Think about it: if you’re writing a C function to save registers, by the time your function starts executing, the C compiler has already clobbered some registers (the function call ABI specifies which registers are callee-saved and which are caller-saved). You can’t save the complete CPU state from C, because C itself uses the CPU state.

Here’s what a context switch function looks like in x86-64 assembly (Linux kernel style):

; Context switch: switch from thread A to thread B
; Input: RDI = pointer to thread A's saved state (thread_struct)
;        RSI = pointer to thread B's saved state
; Output: None (doesn't return in the traditional sense—returns as thread B)
; C prototype: void switch_to(struct thread_struct *prev, struct thread_struct *next)

global switch_to
switch_to:
    ; Save callee-saved registers of thread A
    mov [rdi + 0x00], rbx   ; Save RBX
    mov [rdi + 0x08], rbp   ; Save RBP
    mov [rdi + 0x10], r12   ; Save R12
    mov [rdi + 0x18], r13   ; Save R13
    mov [rdi + 0x20], r14   ; Save R14
    mov [rdi + 0x28], r15   ; Save R15
    mov [rdi + 0x30], rsp   ; Save RSP (stack pointer)
    
    ; Save the return address (RIP) so thread A can resume later
    ; The return address is at [RSP] (pushed by the CALL instruction)
    mov rax, [rsp]
    mov [rdi + 0x38], rax   ; Save RIP
    
    ; Restore callee-saved registers of thread B
    mov rbx, [rsi + 0x00]   ; Restore RBX
    mov rbp, [rsi + 0x08]   ; Restore RBP
    mov r12, [rsi + 0x10]   ; Restore R12
    mov r13, [rsi + 0x18]   ; Restore R13
    mov r14, [rsi + 0x20]   ; Restore R14
    mov r15, [rsi + 0x28]   ; Restore R15
    mov rsp, [rsi + 0x30]   ; Restore RSP (switch to thread B's stack!)
    
    ; "Return" to thread B's saved RIP
    ; Thread B's saved RIP is stored at offset 0x38 in its thread_struct
    push qword [rsi + 0x38] ; Push RIP onto thread B's stack
    ret                        ; RET pops RIP from stack and jumps there

This is the heart of the operating system. Every time the OS switches threads, this code runs. And it has to be written in assembly, because only assembly can save and restore the complete CPU state.

Interrupt Handling

Interrupts are the way that hardware devices (keyboard, disk, network card, timer, etc.) get the CPU’s attention. When a hardware device needs service, it asserts an interrupt request line, and the CPU stops what it’s doing, saves its current state, and jumps to an interrupt service routine (ISR) that the OS has registered for that interrupt.

Interrupt handling is another area where assembly is mandatory. When an interrupt fires, the CPU automatically:

  1. Saves the current RFLAGS, CS, and RIP on the stack (for x86-64)
  2. Jumps to the interrupt handler (the address is looked up in the IDT—Interrupt Descriptor Table)
  3. The handler has to save the rest of the CPU state (all the other registers)
  4. Do the actual work of handling the interrupt
  5. Restore the CPU state
  6. Execute IRETQ to return from the interrupt

The CPU’s interrupt mechanism is fundamentally an assembly-level construct. The IDT entries are data structures that contain segment selectors and offsets (assembly-level concepts). The IRETQ instruction that returns from an interrupt is an assembly instruction. And the prologue/epilogue code that saves and restores registers has to be written in assembly.

Here’s what a basic interrupt handler looks like in x86-64 assembly:

; Interrupt handler template (x86-64)
; This saves the CPU state, calls a C handler, and restores state

global timer_interrupt_handler
extern timer_handler_c       ; C function that does the real work

timer_interrupt_handler:
    ; The CPU has already pushed: SS, RSP, RFLAGS, CS, RIP
    ; (This is the hardware interrupt frame)
    
    ; Save remaining registers
    push rax
    push rbx
    push rcx
    push rdx
    push rsi
    push rdi
    push rbp
    push r8
    push r9
    push r10
    push r11
    push r12
    push r13
    push r14
    push r15
    
    ; Call the C handler
    call timer_handler_c
    
    ; Restore registers (in reverse order)
    pop r15
    pop r14
    pop r13
    pop r12
    pop r11
    pop r10
    pop r9
    pop r8
    pop rbp
    pop rdi
    pop rsi
    pop rdx
    pop rcx
    pop rbx
    pop rax
    
    ; Return from interrupt (pops RIP, CS, RFLAGS, RSP, SS)
    iretq

(Modern kernels use a slightly more sophisticated approach—they save the registers to a struct pt_regs on the stack and pass a pointer to that struct to the C handler. But the basic idea is the same: the interrupt entry/exit code is assembly.)

CPU-Specific Initialization

Every CPU has features that can only be configured via assembly instructions. Setting up the GDT (Global Descriptor Table). Loading the IDT (Interrupt Descriptor Table). Enabling the floating-point unit (FPU). Enabling SSE/AVX instructions. Configuring the control registers (CR0, CR2, CR3, CR4, EFER). Setting up the syscall/MSR registers for fast system call dispatch.

All of these are privileged operations that can only be done from kernel mode (ring 0), and they all require specific assembly instructions (LGDT, LIDT, LTR, MOV to CRx, WRMSR, etc.) that have no C equivalent.

Here’s what the early kernel initialization code looks like (setting up the GDT and IDT):

; Early kernel initialization (x86-64)
; Sets up GDT, IDT, and enters the kernel's proper execution environment

global early_init
extern main               ; The C kernel entry point

early_init:
    ; Set up the stack (we're running in assembly, need a stack)
    mov rsp, early_stack_top
    
    ; Load the GDT (Global Descriptor Table)
    lgdt [gdt_descriptor]
    
    ; Load the IDT (Interrupt Descriptor Table)
    lidt [idt_descriptor]
    
    ; Set up the TSS (Task State Segment) - needed for stack switching on syscalls
    mov ax, 0x18          ; TSS selector (from GDT)
    ltr ax                 ; Load task register
    
    ; Enable SSE/AVX (needed for floating-point in the kernel)
    mov rax, cr0
    and ax, 0xFFFB         ; Clear CR0.EM (emulation bit)
    or ax, 0x0002          ; Set CR0.MP (monitor coprocessor)
    mov cr0, rax
    
    mov rax, cr4
    or ax, 0x200           ; Set CR4.OSFXSR (enable FXSAVE/FXRSTOR)
    or ax, 0x400           ; Set CR4.OSXMMEXCPT (enable SIMD exceptions)
    mov cr4, rax
    
    ; Enable syscall/sysret (fast system call mechanism)
    mov ecx, 0xC0000080    ; EFER MSR
    rdmsr
    or ax, 0x0001          ; Set EFER.SCE (syscall enable)
    wrmsr
    
    ; Set up syscall targets
    mov ecx, 0xC0000081    ; STAR MSR (syscall target)
    rdmsr
    mov edx, 0x00180008    ; Kernel CS=0x08, User CS=0x18 (adjust for your GDT)
    wrmsr
    
    ; Jump to the C kernel entry point
    call main
    
    ; If main() returns, halt the CPU
.hang:
    hlt
    jmp .hang

; =============================================================================
; GDT (Global Descriptor Table)
; =============================================================================

section .data

align 16
gdt_start:
    ; Null descriptor (required)
    dq 0x0000000000000000
    
    ; Kernel code segment (selector = 0x08)
    dq 0x00AF9A000000FFFF
    
    ; Kernel data segment (selector = 0x10)
    dq 0x00AF92000000FFFF
    
    ; User code segment (selector = 0x18)
    dq 0x00AFFA000000FFFF
    
    ; User data segment (selector = 0x20)
    dq 0x00AFF2000000FFFF
    
    ; TSS descriptor (selector = 0x28)
    ; (TSS descriptor is 16 bytes in x86-64, special format)
    dq 0x0000000000000000  ; Placeholder — set up at runtime
    dq 0x0000000000000000

gdt_descriptor:
    dw gdt_end - gdt_start - 1
    dq gdt_start

gdt_end:

; =============================================================================
; Stack for early initialization
; =============================================================================

section .bss
align 16
early_stack_bottom:
    resb 16384           ; 16KB stack
early_stack_top:

; =============================================================================
; IDT (Interrupt Descriptor Table) — set up by C code, loaded here
; =============================================================================

; The actual IDT is typically set up by C code (easier to manage 256 entries)
; We just need a minimal IDT for early init, then the C code replaces it

align 16
idt_descriptor:
    dw 0x0FFF            ; Limit = 256 entries × 8 bytes - 1
    dq idt_start

idt_start:
    times 256 dq 0       ; 256 null IDT entries (filled in by C code)

This is the “C runtime startup” code for an OS kernel. It’s the assembly code that runs before main() (or kmain()), setting up the CPU and the execution environment so that C code can run.

Without this assembly code, you can’t run C code. There’s no C function that can load the GDT, or set up the IDT, or enable the FPU. These are fundamentally assembly-level operations, and they have to be done in assembly.

Summary: Where Assembly is Required in an OS Kernel

To summarize, here are the places in an OS kernel where assembly is absolutely required:

  1. Context switching (saving/restoring thread state)
  2. Interrupt handling (ISR entry/exit code, IDT setup)
  3. System call dispatch (the gateway between user mode and kernel mode)
  4. CPU initialization (GDT, IDT, TSS, control registers, MSRs)
  5. Memory management (setting CR3 for page table switches, TLB invalidation with INVLPG)
  6. SMP initialization (starting application processors with SIPI)
  7. Power management (HLT instruction for idle loop, ACPI methods)
  8. Atomic operations (LOCK prefix, CMPXCHG, XCHG for locks)
  9. User-mode transitions (setting up the initial user stack and jumping to user code)
  10. Signal delivery (modifying the user stack to “inject” a signal handler)

In the sections that follow, we’re going to dive deep into each of these areas. We’ll look at the assembly code that implements them, the design decisions behind that code, and the subtle bugs that can bite you if you get it wrong.

But first, let’s talk about the “crown jewel” of OS assembly code: the context switch.

Membership Required

You must be a member to access this content.

View Membership Levels

Already a member? Log in here

Course: Assembly Language Programming From Low-Level Logic to System Kernel
Share your love

Trending now

Exploring Web 3D Core Technologies Development
Assembly Language Programming From Low-Level Logic to System Kernel
WebRTC Advanced Guide：Media, Transport & Architecture
WebSocket Deep Guide：Building High-Performance Realtime Systems