Lesson 28-Assembly Programming in Multithreaded Environments – Caveats

Introduction: The Multithreading Reality Check

Let me start with a confession: for the longest time, I thought multithreading was someone else’s problem. “I write assembly,” I told myself. “My code is simple. I don’t need to worry about threads.”

I was wrong. Spectacularly wrong.

Here’s the thing: even “single-threaded” C programs use threads. The C runtime library (libc) uses threads for things like FILE* buffering, atexit() handlers, and memory allocation (malloc/free). If your assembly code modifies memory that libc is using, you have a race condition—even if you never explicitly created a thread.

And if you are writing multithreaded code (which you probably are, if you’re reading this article), you need to understand how assembly instructions interact with:

  • The memory subsystem (caches, store buffers, TLB)
  • The memory consistency model (what order do memory operations appear in to other threads?)
  • Atomic operations (how to do “read-modify-write” operations without races)
  • Locking primitives (spinlocks, mutexes, semaphores)
  • Lock-free data structures (using CMPXCHG16B and other atomic operations)

In this article, we’re going to dissect multithreading from the assembly programmer’s perspective. We’ll cover:

  1. Why multithreading matters for assembly programmers
  2. The problem with shared memory: Race conditions
  3. LOCK prefix: Making memory operations atomic
  4. Memory ordering: x86 has a strong memory model (TSO)
  5. Spinlocks in assembly: Using LOCK CMPXCHG or XCHG
  6. Semaphores and mutexes: Why you shouldn’t implement them in assembly
  7. Thread-local storage (TLS): Accessing thread-specific data
  8. False sharing: Why two variables on the same cache line kill SMP performance
  9. Writing a lock-free data structure in assembly
  10. Complete example: A spinlock implementation and critical section

By the end of this article, you’ll understand how to write assembly code that’s thread-safe. Let’s dive in.

Why Multithreading Matters for Assembly Programmers

You might be thinking: “I write simple assembly. Just MOV, ADD, CMP, JMP. Why do I need to care about threads?”

Here’s the reality: even if you don’t use threads, threads are using you.

The Hidden Threads in “Single-Threaded” Programs

Modern operating systems are multithreaded. The C runtime (libc) uses threads for:

  • Memory allocation (malloc/free): The heap is shared between threads. If two threads call malloc at the same time, they might get the same memory block (race condition) unless the heap is properly synchronized.
  • File I/O buffering: FILE* streams (stdin, stdout, stderr) are buffered. If two threads write to stdout at the same time, the output might be interleaved or corrupted.
  • Signal handlers: Signals (like SIGINT, SIGSEGV) are delivered to a thread, not a process. If your assembly code is executing when a signal arrives, the signal handler runs in the same thread, on the same stack. This can corrupt your stack if you’re not careful.

A Concrete Example: The Race Condition You Didn’t Know You Had

Let me show you a race condition that can happen even in “single-threaded” code.

C code:

#include <stdio.h>
#include <stdlib.h>

int global_counter = 0;

void increment_counter() {
    global_counter++;  // This is NOT atomic!
}

int main() {
    increment_counter();
    printf("%d\n", global_counter);  // Might print 0, 1, or garbage!
    return 0;
}

You might think this is safe. After all, main() only calls increment_counter() once, right?

Wrong. The C compiler might optimize global_counter++ into a non-atomic read-modify-write. And if a signal handler (running in the same thread) also modifies global_counter, you have a race condition.

The assembly for global_counter++ (x86-32, unoptimized):

; global_counter++ (non-atomic!)
MOV EAX, [global_counter]   ; Step 1: Read global_counter into EAX
INC EAX                        ; Step 2: Increment EAX
MOV [global_counter], EAX   ; Step 3: Write EAX back to global_counter

If a signal handler runs between Step 1 and Step 3, and the signal handler also modifies global_counter, the write in Step 3 will overwrite the signal handler’s modification. Data corruption!

The Solution: Atomic Operations

To make global_counter++ atomic, you need to use the LOCK prefix:

; global_counter++ (atomic!)
LOCK INC DWORD [global_counter]   ; Atomically increment memory

The LOCK prefix ensures that the INC instruction:

  1. Reads the memory location into a temporary buffer
  2. Increments the temporary
  3. Writes the result back to memory
  4. Does all of this atomically (no other processor/thread can access the memory location during the operation)

We’ll cover LOCK in detail in Section 3. But first, let’s understand why race conditions happen.

The Problem with Shared Memory: Race Conditions

A race condition occurs when two or more threads access shared memory, and the final result depends on the order in which the accesses occur.

A Classic Race Condition: The Increment Problem

Let me show you the classic example of a race condition.

C code:

#include <pthread.h>

int counter = 0;  // Shared variable

void* increment(void* arg) {
    for (int i = 0; i < 1000000; i++) {
        counter++;  // NON-atomic!
    }
    return NULL;
}

int main() {
    pthread_t t1, t2;
    pthread_create(&t1, NULL, increment, NULL);
    pthread_create(&t2, NULL, increment, NULL);
    pthread_join(t1, NULL);
    pthread_join(t2, NULL);
    
    printf("counter = %d\n", counter);  // Expected: 2000000
    return 0;
}

If counter++ were atomic, the final value of counter would be 2,000,000 (1,000,000 increments × 2 threads).

But counter++ is not atomic. The assembly for counter++ is:

; counter++ (compiled from C, no optimization)
MOV EAX, [counter]   ; Read counter into EAX
INC EAX                  ; Increment EAX
MOV [counter], EAX   ; Write back to counter

This is a read-modify-write sequence. And it’s three instructions, not one. If two threads execute this sequence concurrently, they can interleave, leading to lost updates.

How the Race Condition Happens (Step-by-Step)

Let’s trace what happens when two threads execute counter++ concurrently.

Initial state: counter = 0

Thread 1: MOV EAX, [counter] → EAX = 0 Thread 2: MOV EAX, [counter] → EAX = 0 (Thread 2 reads before Thread 1 writes!)

Thread 1: INC EAX → EAX = 1 Thread 2: INC EAX → EAX = 1 (Thread 2 also increments from 0!)

Thread 1: MOV [counter], EAX → counter = 1 Thread 2: MOV [counter], EAX → counter = 1 (Thread 2 overwrites Thread 1’s update!)

Final state: counter = 1 (But we expected counter = 2!)

This is a lost update. Thread 1’s increment was “lost” because Thread 2 read counter before Thread 1 wrote the updated value.

Why Two Assembly Instructions Aren’t Atomic

You might be thinking: “OK, I’ll just use INC counter (a single instruction). That’s atomic, right?”

Wrong. Even a single INC mem instruction is not atomic on x86 (unless you use the LOCK prefix). Here’s why:

The INC mem instruction (without LOCK) does the following:

  1. Reads the memory operand from RAM into a temporary buffer (this is a load).
  2. Increments the temporary.
  3. Writes the result back to memory (this is a store).

Between steps 1 and 3, another processor (or thread) can access the same memory location. The INC instruction is not atomic unless you use LOCK.

Demonstrating the Race Condition in Assembly

Let me write a complete assembly program that demonstrates the race condition.

; ============================================
; race_condition.asm - Demonstrates a race condition
; Compile: nasm -f elf32 race_condition.asm
; Link:   gcc -m32 race_condition.o -o race_condition -lpthread
; ============================================

SECTION .data
counter DD 0                  ; Shared counter
THREAD_COUNT EQU 2
ITERATIONS EQU 1000000

SECTION .text
GLOBAL main
EXTERN pthread_create, pthread_join, printf

; ---------- Thread function ----------
thread_func:
    PUSH EBP
    MOV EBP, ESP
    
    MOV ECX, ITERATIONS    ; Loop counter
    
.loop:
    ; ---------- NON-atomic increment (RACE CONDITION!) ----------
    MOV EAX, [counter]   ; Read counter
    INC EAX                  ; Increment
    MOV [counter], EAX   ; Write back
    
    ; ---------- ATOMIC increment (FIXED) ----------
    ; LOCK INC DWORD [counter]   ; Uncomment this to fix the race!
    
    LOOP .loop
    
    MOV ESP, EBP
    POP EBP
    RET

; ---------- Main function ----------
main:
    PUSH EBP
    MOV EBP, ESP
    
    ; Create threads
    SUB ESP, THREAD_COUNT * 4   ; Space for thread handles
    MOV EBX, ESP                    ; EBX = array of thread handles
    
    MOV EAX, 0                      ; Thread argument (unused)
    PUSH EAX
    PUSH thread_func
    PUSH EAX
    CALL pthread_create
    ADD ESP, 12
    
    MOV [EBX], EAX                ; Save thread handle
    
    ; ... create second thread ...
    
    ; Join threads
    PUSH EAX
    PUSH DWORD [EBX]
    CALL pthread_join
    ADD ESP, 8
    
    ; ... join second thread ...
    
    ; Print result
    PUSH DWORD [counter]
    PUSH format_str
    CALL printf
    ADD ESP, 8
    
    MOV ESP, EBP
    POP EBP
    RET

SECTION .data
format_str DB "counter = %d (expected %d)", 10, 0

When you run this program, counter will be less than 2,000,000 (the expected value). The exact value depends on timing (that’s why it’s called a “race” condition).

Fixing the Race Condition

To fix the race condition, you need to make the increment atomic. There are two ways to do this:

  1. Use the LOCK prefix (hardware-supported atomic operation)
  2. Use a mutex (mutual exclusion lock)

We’ll cover LOCK in the next section. For now, just know that any read-modify-write sequence on shared memory is a potential race condition.

sequenceDiagram
    participant T1 as Thread 1
    participant Mem as Shared Memory<br/>(counter=0)
    participant T2 as Thread 2
    
    T1->>Mem: Read counter (0)
    Note over T1: EAX = 0
    
    T2->>Mem: Read counter (0)
    Note over T2: EAX = 0
    
    T1->>T1: Increment EAX (1)
    T2->>T2: Increment EAX (1)
    
    T1->>Mem: Write counter (1)
    Note over Mem: counter = 1
    
    T2->>Mem: Write counter (1)
    Note over Mem: counter = 1 (LOST UPDATE!)
    
    Note over T1,T2: Both threads incremented.<br/>But counter = 1, not 2!

Membership Required

You must be a member to access this content.

View Membership Levels

Already a member? Log in here
Course: Assembly Language Programming From Low-Level Logic to System Kernel
Share your love

Trending now

Exploring Web 3D Core Technologies Development
Assembly Language Programming From Low-Level Logic to System Kernel
WebRTC Advanced Guide：Media, Transport & Architecture
WebSocket Deep Guide：Building High-Performance Realtime Systems