Lesson 06-Dynamic Analysis Techniques

Dynamic Analysis Introduction

Dynamic analysis involves running a program in a controlled environment and observing its runtime behavior (including instruction execution, memory access, and system interactions) to analyze the code. Compared to static analysis, dynamic analysis can reveal runtime logic, decrypt dynamically generated content, and detect hidden behaviors. This tutorial covers the following core topics:

  • Debugging Fundamentals: Master breakpoints, register/memory monitoring, and single-step execution to control program execution.
  • Dynamic Behavior Analysis: Monitor API calls, network activity, file system, and registry operations to understand program-system interactions.

This tutorial will explain these techniques step by step, providing Python code (combined with debugger APIs and monitoring tools) and practical cases using tools like Radare2, GDB, x64dbg, and Wireshark.

Preparation

Before starting, ensure the following tools are installed:

  • Python 3.x: For writing debugging and monitoring scripts.
  • Capstone: Disassembly engine (pip install capstone).
  • pywin32 (Windows): For Windows debugging APIs (pip install pywin32).
  • GDB: Linux/macOS debugger (included with the system or installed via package manager).
  • x64dbg: Windows debugger (download from the official website).
  • Radare2: Open-source reverse engineering framework (install from the official website).
  • API Monitor: Windows API call monitoring tool.
  • Procmon: Windows system monitoring tool (Sysinternals suite).
  • Wireshark: Network traffic analysis tool.
  • Fiddler: HTTP/HTTPS proxy tool.
  • Sample Binary Files: PE, ELF, or Mach-O files for x86/x64/ARM architectures (obtainable from open-source projects or CTF challenges).
  • Virtual Machine: Such as VMware or VirtualBox, for safely running malicious samples.

Debugging Fundamentals

Debugging is the process of controlling program execution, inspecting state, and modifying behavior to analyze code. Debuggers (such as GDB, x64dbg) provide features like breakpoints, register/memory monitoring, and single-step execution, forming the foundation of dynamic analysis.

Breakpoint Types

Breakpoints pause program execution, allowing the analyst to inspect the state. Common breakpoint types include:

  • Software Breakpoints:
    • Principle: Replace the target instruction with an interrupt instruction (e.g., x86 int 3, 0xCC).
    • Advantages: Easy to set, unlimited quantity.
    • Disadvantages: Easily detected (anti-debugging techniques may check for 0xCC).
    • Example: Set a software breakpoint at address 0x401000.
  • Hardware Breakpoints:
    • Principle: Use processor debug registers (e.g., x86 DR0-DR3) to monitor specific addresses.
    • Advantages: Hard to detect, suitable for monitoring specific instructions or memory.
    • Disadvantages: Limited quantity (usually 4).
    • Example: Monitor execution at function entry 0x401000.
  • Conditional Breakpoints:
    • Principle: Pause when a specified condition (e.g., register value or memory content) is met.
    • Advantages: Precise control, reduces irrelevant interruptions.
    • Disadvantages: High performance overhead.
    • Example: Pause when eax == 0x1234.

Example: Setting Breakpoints in x64dbg

Set breakpoints in x64dbg:

  1. Open the binary, navigate to the target address (e.g., 0x401000).
  2. Press F2 to set a software breakpoint; the icon turns red.
  3. Right-click and select “Hardware Breakpoint” to set a hardware breakpoint.
  4. In the “Breakpoints” window, add a conditional breakpoint (e.g., eax == 0x1234).

Example: Setting Breakpoints in GDB

Set breakpoints in GDB:

gdb sample.elf
(gdb) break *0x401000  # Software breakpoint
(gdb) hbreak *0x401000  # Hardware breakpoint
(gdb) condition 1 eax==0x1234  # Conditional breakpoint (assuming breakpoint number 1)
(gdb) run

Register and Memory Monitoring

Registers and memory store the runtime state of a program; monitoring them tracks data flow and control flow.

  • Register Monitoring:
    • Key Registers:
      • x86/x64: eax/rax (return value), esp/rsp (stack pointer), eip/rip (instruction pointer).
      • ARM: r0-r3 (parameters/return values), sp (stack pointer), lr (return address), pc (instruction pointer).
    • Tools: Debugger register windows (e.g., x64dbg “Registers” panel, GDB info registers).
  • Memory Monitoring:
    • Methods: Set memory breakpoints (read/write/execute) or directly view memory contents.
    • Tools: x64dbg “Memory” panel, GDB x command.

Example: GDB Register and Memory Monitoring

gdb sample.elf
(gdb) break main
(gdb) run
(gdb) info registers  # View registers
(gdb) x/10x $esp     # View top 10 words of the stack
(gdb) watch *$esp    # Set memory write breakpoint
(gdb) continue

Output Example:

rax            0x0                 0
rbx            0x7fffffffde48      140737488346696
rsp            0x7fffffffde38      0x7fffffffde38
rip            0x401000            0x401000 <main>
...
0x7fffffffde38: 0x00401234 0x00000000 0x7fffffffdf08 0x00000001

Example: x64dbg Memory Monitoring

  1. Open x64dbg, load the binary.
  2. In the “Memory” panel, right-click the target address (e.g., 0x401000), select “Set Breakpoint” -> “Read/Write”.
  3. Run the program; check memory contents when the breakpoint triggers.

Single-Step Execution and Tracing

Single-step execution and tracing control instruction-by-instruction or function-by-function execution, ideal for detailed logic analysis.

  • Single-Step Execution:
    • Step Into: Execute the next instruction, including function calls (e.g., GDB stepi, x64dbg F7).
    • Step Over: Skip function calls, execute the next instruction (e.g., GDB nexti, x64dbg F8).
    • Step Out: Execute until the current function returns (e.g., GDB finish, x64dbg Shift+F8).
  • Tracing:
    • Record instruction execution path, register changes, or memory access.
    • Tools: x64dbg “Trace” function, GDB trace command.

Example: GDB Single-Step Execution

gdb sample.elf
(gdb) break main
(gdb) run
(gdb) disassemble main
(gdb) si  # Step into
(gdb) ni  # Step over
(gdb) info registers

Example: x64dbg Tracing

  1. Load the binary in x64dbg, set a breakpoint.
  2. Press F7 to step into, observe register and memory changes.
  3. Enable tracing in the “Trace” panel to record the execution path.

Code Example: Python Debugger (Windows)

The following code uses pywin32 to implement a simple Windows debugger, setting breakpoints and monitoring registers:

import win32api
import win32con
import win32process
import struct

def debug_process(executable):
    # Start the process
    startup_info = win32process.STARTUPINFO()
    process_info = win32process.CreateProcess(None, executable, None, None, False, 
                                             win32con.DEBUG_PROCESS, None, None, startup_info)
    h_process = process_info[0]
    
    # Set software breakpoint (0xCC)
    target_address = 0x401000
    old_byte = win32api.ReadProcessMemory(h_process, target_address, 1)
    win32api.WriteProcessMemory(h_process, target_address, b"\xCC")
    
    # Debug loop
    while True:
        debug_event = win32process.DebugEvent()
        if not win32process.GetDebugEvent(debug_event):
            break
        if debug_event.dwDebugEventCode == win32con.EXCEPTION_DEBUG_EVENT:
            if debug_event.u.Exception.ExceptionRecord.ExceptionCode == win32con.EXCEPTION_BREAKPOINT:
                print(f"Breakpoint hit at 0x{debug_event.u.Exception.ExceptionRecord.ExceptionAddress:08x}")
                # Read registers
                context = win32process.GetThreadContext(process_info[1])
                print(f"EAX: 0x{context.Eax:08x}, EIP: 0x{context.Eip:08x}")
                # Restore original instruction
                win32api.WriteProcessMemory(h_process, target_address, old_byte)
                context.Eip -= 1  # Rewind EIP
                win32process.SetThreadContext(process_info[1], context)
        win32process.ContinueDebugEvent(debug_event.dwProcessId, debug_event.dwThreadId, win32con.DBG_CONTINUE)

debug_process("sample.exe")

Output Example:

Breakpoint hit at 0x00401000
EAX: 0x00000000, EIP: 0x00401000

Debugging Fundamentals Notes

  1. Anti-Debugging Techniques: Malware may detect breakpoints or debuggers; use stealth debugging (e.g., hardware breakpoints).
  2. Performance Overhead: Conditional breakpoints and tracing may slow execution; optimize conditions.
  3. Environment Isolation: Debug in a virtual machine to prevent host infection by malicious code.
  4. Tool Selection: x64dbg is suitable for Windows GUI debugging, GDB for Linux command-line debugging.

Dynamic Behavior Analysis

Dynamic behavior analysis monitors a program’s runtime behavior (such as API calls, network activity, file system, and registry operations) to understand its functionality and intent. This is critical for analyzing malware, detecting vulnerabilities, or verifying program behavior.

API Call Monitoring

API calls reflect program interactions with the operating system, such as file operations, network requests, or process management. Monitoring API calls reveals key logic (e.g., malware C2 communication).

  • Tools:
    • API Monitor: Windows tool, captures API calls and parameters.
    • Procmon: Sysinternals tool, monitors system calls.
    • Frida: Cross-platform dynamic instrumentation framework, ideal for scripted monitoring.

Example: Using API Monitor

  1. Open API Monitor, load the target process (e.g., sample.exe).
  2. Select APIs to monitor (e.g., kernel32.dll’s CreateFileA).
  3. Run the program, view the call log.

Output Example:

CreateFileA("C:\\test.txt", GENERIC_WRITE, 0, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL) -> 0x1234
WriteFile(0x1234, "Hello", 5, 0x12345678, NULL) -> TRUE

Example: Frida API Call Monitoring

The following Frida script monitors kernel32.dll’s CreateFileA:

// create_file_hook.js
const CreateFileA = Module.findExportByName("kernel32.dll", "CreateFileA");

Interceptor.attach(CreateFileA, {
    onEnter(args) {
        console.log("CreateFileA called with file: " + Memory.readUtf8String(args[0]));
    },
    onLeave(retval) {
        console.log("CreateFileA returned: " + retval);
    }
});

Run:

frida -f sample.exe -l create_file_hook.js

Output Example:

CreateFileA called with file: C:\test.txt
CreateFileA returned: 0x1234

Code Example: Python API Call Monitoring

The following code uses pywin32 to intercept CreateFileA:

import win32api
import win32process
import ctypes

def hook_api(process_handle, api_name, dll_name):
    # Get API address
    dll = ctypes.WinDLL(dll_name)
    api_addr = ctypes.cast(getattr(dll, api_name), ctypes.c_void_p).value
    
    # Inject breakpoint
    old_byte = win32api.ReadProcessMemory(process_handle, api_addr, 1)
    win32api.WriteProcessMemory(process_handle, api_addr, b"\xCC")
    return old_byte, api_addr

def monitor_api(executable):
    process_info = win32process.CreateProcess(None, executable, None, None, False, 
                                             win32con.DEBUG_PROCESS, None, None, 
                                             win32process.STARTUPINFO())
    h_process = process_info[0]
    
    # Hook CreateFileA
    old_byte, api_addr = hook_api(h_process, "CreateFileA", "kernel32.dll")
    
    # Debug loop
    while True:
        debug_event = win32process.DebugEvent()
        if not win32process.GetDebugEvent(debug_event):
            break
        if debug_event.dwDebugEventCode == win32con.EXCEPTION_DEBUG_EVENT:
            if debug_event.u.Exception.ExceptionRecord.ExceptionAddress == api_addr:
                print("CreateFileA called")
                context = win32process.GetThreadContext(process_info[1])
                # Read filename (first parameter on stack)
                filename_ptr = win32api.ReadProcessMemory(h_process, context.Esp + 4, 4)
                filename = win32api.ReadProcessMemory(h_process, struct.unpack("I", filename_ptr)[0], 100).decode("ascii")
                print(f"Filename: {filename}")
                # Restore original instruction
                win32api.WriteProcessMemory(h_process, api_addr, old_byte)
                context.Eip -= 1
                win32process.SetThreadContext(process_info[1], context)
        win32process.ContinueDebugEvent(debug_event.dwProcessId, debug_event.dwThreadId, win32con.DBG_CONTINUE)

monitor_api("sample.exe")

Output Example:

CreateFileA called
Filename: C:\test.txt

Network Behavior Analysis

Network behavior analysis monitors a program’s network activity, such as HTTP requests, DNS queries, or C2 communication. Common tools include:

  • Wireshark: Captures and analyzes network packets.
  • Fiddler: Proxy tool, focused on HTTP/HTTPS traffic.
  • tcpdump: Linux command-line tool, captures raw packets.

Example: Wireshark Analysis

  1. Start Wireshark, select a network interface (e.g., Wi-Fi).
  2. Set a filter (e.g., http or tcp.port == 80).
  3. Run the target program, observe traffic.

Output Example:

GET /api/data HTTP/1.1
Host: example.com
User-Agent: SampleApp/1.0

Example: Fiddler HTTPS Analysis

  1. Open Fiddler, enable HTTPS decryption (Tools -> Options -> HTTPS).
  2. Run the program, view requests and responses.

Code Example: Python Network Traffic Capture

Use scapy to capture HTTP traffic:

from scapy.all import *

def packet_callback(packet):
    if packet.haslayer(TCP) and packet.haslayer(Raw):
        if packet[TCP].dport == 80 or packet[TCP].sport == 80:
            payload = packet[Raw].load.decode("ascii", errors="ignore")
            if "GET" in payload or "POST" in payload:
                print("=== HTTP Request ===")
                print(payload)

sniff(filter="tcp port 80", prn=packet_callback, count=10)

Output Example:

=== HTTP Request ===
GET /index.html HTTP/1.1
Host: example.com

File System and Registry Tracking

File system and registry operations reflect a program’s persistence behavior (e.g., creating files, modifying configurations). Monitoring these reveals malicious actions (e.g., ransomware encrypting files).

  • Tools:
    • Procmon: Windows tool, monitors file and registry operations.
    • strace: Linux tool, traces system calls.
    • lsof: Lists open files.

Example: Procmon Monitoring

  1. Open Procmon, set a filter (e.g., Process Name is sample.exe).
  2. Run the program, view file and registry operations.

Output Example:

sample.exe: CreateFile C:\test.txt
sample.exe: RegSetValue HKLM\Software\Sample

Example: strace File Operation Tracking

On Linux, use strace:

strace -e openat,write ./sample.elf

Output Example:

openat(AT_FDCWD, "test.txt", O_WRONLY|O_CREAT|O_TRUNC, 0644) = 3
write(3, "Hello", 5) = 5

Code Example: Python File Operation Monitoring

The following code uses watchdog to monitor file system changes:

from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler

class FileMonitor(FileSystemEventHandler):
    def on_created(self, event):
        print(f"File created: {event.src_path}")
    def on_modified(self, event):
        print(f"File modified: {event.src_path}")

observer = Observer()
observer.schedule(FileMonitor(), path=".", recursive=False)
observer.start()
try:
    while True:
        pass
except KeyboardInterrupt:
    observer.stop()
observer.join()

Output Example:

File created: test.txt
File modified: test.txt

Dynamic Behavior Analysis Notes

  1. Sandbox Evasion: Malware may detect virtual machines or sandboxes; use stealth environments.
  2. Encrypted Traffic: HTTPS traffic requires decryption (e.g., Fiddler HTTPS decryption).
  3. Performance Overhead: Monitoring tools may impact performance; optimize filters.
  4. Tool Combination: Combine Procmon, Wireshark, and Frida for comprehensive behavior capture.

Comprehensive Example: Dynamic Malware Analysis

The following code combines debugging and behavior monitoring to analyze a Windows PE file:

import win32api
import win32process
import pefile
from scapy.all import *

def analyze_malware(file_path):
    # Load PE file, check import table
    pe = pefile.PE(file_path)
    print("=== Imported APIs ===")
    for entry in pe.DIRECTORY_ENTRY_IMPORT:
        for imp in entry.imports:
            print(f"{entry.dll.decode()}: {imp.name.decode()}")
    
    # Debug and set breakpoint
    startup_info = win32process.STARTUPINFO()
    process_info = win32process.CreateProcess(None, file_path, None, None, False, 
                                             win32con.DEBUG_PROCESS, None, None, startup_info)
    h_process = process_info[0]
    
    # Set breakpoint at CreateFileA
    dll = ctypes.WinDLL("kernel32.dll")
    api_addr = ctypes.cast(getattr(dll, "CreateFileA"), ctypes.c_void_p).value
    old_byte = win32api.ReadProcessMemory(h_process, api_addr, 1)
    win32api.WriteProcessMemory(h_process, api_addr, b"\xCC")
    
    # Monitor network
    def packet_callback(packet):
        if packet.haslayer(Raw) and (packet[TCP].dport == 80 or packet[TCP].sport == 80):
            print("HTTP Packet:", packet[Raw].load.decode("ascii", errors="ignore"))
    
    sniff(filter="tcp port 80", prn=packet_callback, count=5, store=False, timeout=10)
    
    # Debug loop
    while True:
        debug_event = win32process.DebugEvent()
        if not win32process.GetDebugEvent(debug_event):
            break
        if debug_event.dwDebugEventCode == win32con.EXCEPTION_DEBUG_EVENT:
            if debug_event.u.Exception.ExceptionRecord.ExceptionAddress == api_addr:
                print("CreateFileA called")
                context = win32process.GetThreadContext(process_info[1])
                filename_ptr = win32api.ReadProcessMemory(h_process, context.Esp + 4, 4)
                filename = win32api.ReadProcessMemory(h_process, struct.unpack("I", filename_ptr)[0], 100).decode("ascii")
                print(f"Filename: {filename}")
                win32api.WriteProcessMemory(h_process, api_addr, old_byte)
                context.Eip -= 1
                win32process.SetThreadContext(process_info[1], context)
        win32process.ContinueDebugEvent(debug_event.dwProcessId, debug_event.dwThreadId, win32con.DBG_CONTINUE)

analyze_malware("sample.exe")

Output Example:

=== Imported APIs ===
kernel32.dll: CreateFileA
kernel32.dll: WriteFile
wininet.dll: InternetOpenA
CreateFileA called
Filename: C:\malware.txt
HTTP Packet: GET /data HTTP/1.1

Using x64dbg and Wireshark for Interactive Analysis

  1. Debugging:
    • Open x64dbg, load sample.exe.
    • Set a breakpoint at CreateFileA (F2).
    • Run the program, check parameters and return values.
  2. Network Monitoring:
    • Start Wireshark, filter http traffic.
    • Observe the program’s network requests.
  3. File Monitoring:
    • Open Procmon, filter sample.exe.
    • View file creation and write operations.

Dynamic Analysis Practice

  1. Debugging Analysis: Set breakpoints at key APIs (e.g., CreateFileA, InternetOpenA), track parameters.
  2. Behavior Monitoring: Use Procmon and Wireshark to capture file, network, and registry activity.
  3. Malware Analysis: Combine debugging and behavior analysis to identify C2 communication and persistence mechanisms.
  4. Automated Scripting: Write Frida or Python scripts to automate monitoring and data extraction.

Advanced Debugging

Advanced debugging techniques address complex protection mechanisms (e.g., anti-debugging) and low-level system analysis (e.g., kernel debugging). This section explains how to identify and bypass anti-debugging techniques and introduces the basics of kernel debugging.

Anti-Debugging Technique Identification

Anti-debugging techniques are mechanisms used by malware or protected software to detect debuggers. Common methods include:

  • IsDebuggerPresent:
    • Windows API, checks the BeingDebugged flag in the process’s PEB (Process Environment Block).
    • Code Example: call IsDebuggerPresent test eax, eax jnz debugger_detected
  • Timing Checks:
    • Measure code execution time to detect delays caused by debuggers (e.g., single-stepping).
    • Common APIs: GetTickCount, QueryPerformanceCounter.
    • Code Example: call GetTickCount mov ebx, eax ; Some operations call GetTickCount sub eax, ebx cmp eax, 1000 ; If time difference is too large, suspect debugging ja debugger_detected
  • Other Techniques:
    • Check for debugger processes (e.g., OllyDbg.exe, x64dbg.exe).
    • Use NtQueryInformationProcess to check debug status.
    • Detect breakpoints (e.g., scan for 0xCC or hardware breakpoint registers).

Example: Identifying IsDebuggerPresent

The following code checks for a debugger:

call IsDebuggerPresent
test eax, eax
jnz debugger_detected

Disassembly:

from capstone import *

CODE = b"\xFF\x15\x00\x20\x40\x00\x85\xC0\x75\x02"

md = Cs(CS_ARCH_X86, CS_MODE_32)
for instr in md.disasm(CODE, 0x401000):
    print(f"0x{instr.address:08x}: {instr.mnemonic} {instr.op_str}")
    if instr.mnemonic == "call" and "IsDebuggerPresent" in instr.op_str:
        print("  -> Anti-Debug: IsDebuggerPresent")
    elif instr.mnemonic == "jnz":
        print("  -> Anti-Debug: Debugger Check Branch")

Output:

0x00401000: call 0x402000
  -> Anti-Debug: IsDebuggerPresent
0x00401006: test eax, eax
0x00401008: jnz 0x40100c
  -> Anti-Debug: Debugger Check Branch

Example: Identifying Timing Checks

call GetTickCount
mov ebx, eax
call GetTickCount
sub eax, ebx
cmp eax, 1000
ja debugger_detected

Disassembly:

CODE = b"\xFF\x15\x04\x20\x40\x00\x8B\xD8\xFF\x15\x04\x20\x40\x00\x2B\xC3\x3D\xE8\x03\x00\x00\x77\x02"

md = Cs(CS_ARCH_X86, CS_MODE_32)
for instr in md.disasm(CODE, 0x401000):
    print(f"0x{instr.address:08x}: {instr.mnemonic} {instr.op_str}")
    if instr.mnemonic == "call" and "GetTickCount" in instr.op_str:
        print("  -> Anti-Debug: Time Check")
    elif instr.mnemonic == "ja":
        print("  -> Anti-Debug: Time Check Branch")

Output:

0x00401000: call 0x402004
  -> Anti-Debug: Time Check
0x00401006: mov ebx, eax
0x00401008: call 0x402004
  -> Anti-Debug: Time Check
0x0040100e: sub eax, ebx
0x00401010: cmp eax, 0x3e8
0x00401015: ja 0x401019
  -> Anti-Debug: Time Check Branch

Bypassing Anti-Debugging

Bypassing anti-debugging techniques requires modifying program behavior, typically through:

  • Modifying Registers:
    • Directly set IsDebuggerPresent return value (e.g., eax = 0).
    • Modify PEB’s BeingDebugged flag to 0.
  • Patching Code:
    • Replace conditional jumps (e.g., jnz) with nop or unconditional jumps (jmp).
    • Modify timing check thresholds.
  • Dynamic Instrumentation:
    • Use Frida or debugger APIs to intercept anti-debugging functions.

Example: Bypassing IsDebuggerPresent

Method 1: Patch Code

Replace jnz debugger_detected with nop:

import win32api

def patch_binary(file_path, offset, new_bytes):
    with open(file_path, "r+b") as f:
        f.seek(offset)
        f.write(new_bytes)

# Replace jnz (0x75 0x02) with nop (0x90 0x90)
patch_binary("sample.exe", 0x401008, b"\x90\x90")

Verification (Radare2):

r2 sample.exe
> s 0x401008
> pd 1
0x00401008 nop

Method 2: Frida Dynamic Instrumentation

// bypass_isdebuggerpresent.js
const IsDebuggerPresent = Module.findExportByName("kernel32.dll", "IsDebuggerPresent");

Interceptor.attach(IsDebuggerPresent, {
    onEnter(args) {
        console.log("IsDebuggerPresent called");
    },
    onLeave(retval) {
        console.log("Forcing IsDebuggerPresent to return 0");
        retval.replace(0);
    }
});

Run:

frida -f sample.exe -l bypass_isdebuggerpresent.js

Output:

IsDebuggerPresent called
Forcing IsDebuggerPresent to return 0

Example: Bypassing Timing Checks

Method 1: Modify Threshold

Change cmp eax, 1000 immediate to 0xFFFF:

# Replace cmp eax, 0x3e8 with cmp eax, 0xffff
patch_binary("sample.exe", 0x401011, b"\x3D\xFF\xFF\x00\x00")

Method 2: Frida Intercept GetTickCount

// bypass_gettickcount.js
const GetTickCount = Module.findExportByName("kernel32.dll", "GetTickCount");

let lastTime = 0;
Interceptor.attach(GetTickCount, {
    onEnter(args) {
        console.log("GetTickCount called");
    },
    onLeave(retval) {
        if (lastTime == 0) {
            lastTime = retval.toInt32();
        } else {
            retval.replace(lastTime + 10); // Simulate small time difference
            lastTime = 0;
        }
    }
});

Run:

frida -f sample.exe -l bypass_gettickcount.js

Kernel Debugging

Kernel debugging is used to analyze the operating system kernel, drivers, or low-level behavior. Common tools include WinDbg and SoftICE (now obsolete, WinDbg is mainstream).

  • WinDbg:
    • Supports user-mode and kernel-mode debugging.
    • Commonly used for driver analysis, blue screen debugging, and rootkit detection.
    • Configuration: Requires target and host systems connected via serial, USB, or network.
  • SoftICE:
    • Early kernel debugger, runs on the target system.
    • Largely replaced by WinDbg.

Example: WinDbg Kernel Debugging Setup

  1. Configure Target System (Windows):
    • Enable debug mode: bcdedit /debug on bcdedit /dbgsettings serial baudrate:115200
    • Restart the system.
  2. Start WinDbg on Host System:
    • Install WinDbg (from Microsoft SDK).
    • Launch: windbg -k com:port=COM1,baud=115200
  3. Basic Commands:
    • Set breakpoint: bp nt!NtCreateFile (at kernel function NtCreateFile).
    • View registers: r.
    • Single-step: t (step into), p (step over).
    • View call stack: k.

Example: WinDbg Driver Debugging

Assume analyzing a malicious driver:

windbg -k com:port=COM1,baud=115200

In WinDbg:

!process 0 0  ; List processes
bp nt!NtCreateFile  ; Set breakpoint at file creation
g  ; Continue execution

When breakpoint triggers:

r  ; View registers
k  ; View call stack
dv  ; View local variables

Code Example: Python Calling WinDbg

Use pykd (WinDbg Python extension) for automated debugging:

import pykd

def kernel_debug():
    pykd.attachKernel()  # Connect to kernel
    print("Attached to kernel")
    
    # Set breakpoint
    bp_id = pykd.setBp("nt!NtCreateFile")
    print(f"Breakpoint set at nt!NtCreateFile: {bp_id}")
    
    # Run
    pykd.go()
    
    # Breakpoint triggered
    print("Breakpoint hit")
    regs = pykd.reg()
    print(f"RAX: 0x{regs['rax']:016x}, RIP: 0x{regs['rip']:016x}")
    
    # Clean up
    pykd.removeBp(bp_id)

kernel_debug()

Output Example:

Attached to kernel
Breakpoint set at nt!NtCreateFile: 0
Breakpoint hit
RAX: 0x0000000000000000, RIP: 0xFFFFF80012345678

Advanced Debugging Notes

  1. Complex Anti-Debugging: Malware may combine multiple anti-debugging techniques; bypass from multiple angles.
  2. Kernel Debugging Risks: Incorrect operations may crash the system; test in a virtual machine.
  3. Tool Selection: WinDbg for kernel debugging, x64dbg for user-mode debugging, Frida for dynamic instrumentation.
  4. Legal Compliance: Bypassing anti-debugging may involve legal risks; operate within authorized scope.

Dynamic Injection and Modification

Dynamic injection and modification techniques modify program behavior at runtime, widely used in reverse engineering (bypassing protection), malware (loading malicious code), and security testing (feature enhancement). This section explains DLL injection, code injection and hooking, and runtime memory patching.

DLL Injection Techniques

DLL injection loads a custom dynamic link library into a target process’s address space, typically to modify behavior or monitor operations. Common methods include:

  • CreateRemoteThread + LoadLibrary:
    • Allocate memory in the target process, write the DLL path, create a remote thread to call LoadLibrary.
    • Advantages: Simple and reliable.
    • Disadvantages: Easily detected.
  • SetWindowsHook:
    • Use Windows hooks (e.g., keyboard, mouse hooks) to load the DLL.
    • Advantages: Suitable for GUI programs.
    • Disadvantages: Depends on specific events.

Example: CreateRemoteThread DLL Injection

The following code injects a DLL into a target process:

import win32api
import win32con
import win32process
import ctypes

def inject_dll(pid, dll_path):
    # Open target process
    h_process = win32api.OpenProcess(win32con.PROCESS_ALL_ACCESS, False, pid)
    
    # Allocate memory
    dll_path_bytes = dll_path.encode("ascii")
    alloc_addr = ctypes.windll.kernel32.VirtualAllocEx(h_process, 0, len(dll_path_bytes), 
                                                       win32con.MEM_COMMIT | win32con.MEM_RESERVE, 
                                                       win32con.PAGE_READWRITE)
    
    # Write DLL path
    win32api.WriteProcessMemory(h_process, alloc_addr, dll_path_bytes)
    
    # Get LoadLibraryA address
    kernel32 = ctypes.WinDLL("kernel32.dll")
    load_library = ctypes.cast(getattr(kernel32, "LoadLibraryA"), ctypes.c_void_p).value
    
    # Create remote thread
    h_thread = ctypes.windll.kernel32.CreateRemoteThread(h_process, None, 0, load_library, alloc_addr, 0, None)
    
    # Wait for thread to finish
    win32api.WaitForSingleObject(h_thread, -1)
    
    # Clean up
    ctypes.windll.kernel32.VirtualFreeEx(h_process, alloc_addr, 0, win32con.MEM_RELEASE)
    win32api.CloseHandle(h_thread)
    win32api.CloseHandle(h_process)

# Example: Inject into notepad.exe
inject_dll(1234, "C:\\mydll.dll")

DLL Code (C language):

#include <windows.h>

BOOL APIENTRY DllMain(HMODULE hModule, DWORD ul_reason_for_call, LPVOID lpReserved) {
    if (ul_reason_for_call == DLL_PROCESS_ATTACH) {
        MessageBoxA(NULL, "DLL Injected!", "Success", MB_OK);
    }
    return TRUE;
}

Compile:

cl /LD mydll.c

Example: SetWindowsHook DLL Injection

import win32api
import win32con
import ctypes

def set_hook_dll(dll_path, pid):
    # Load DLL
    h_module = ctypes.windll.kernel32.LoadLibraryA(dll_path.encode("ascii"))
    
    # Get hook function address (assume DLL exports HookProc)
    hook_proc = ctypes.cast(ctypes.windll.kernel32.GetProcAddress(h_module, b"HookProc"), ctypes.c_void_p).value
    
    # Set keyboard hook
    h_hook = ctypes.windll.user32.SetWindowsHookExA(win32con.WH_KEYBOARD_LL, hook_proc, h_module, 0)
    
    # Wait for events
    ctypes.windll.user32.GetMessageA(0, 0, 0, 0)
    
    # Clean up
    ctypes.windll.user32.UnhookWindowsHookEx(h_hook)
    ctypes.windll.kernel32.FreeLibrary(h_module)

set_hook_dll("C:\\mydll.dll", 1234)

DLL Code:

#include <windows.h>

LRESULT CALLBACK HookProc(int nCode, WPARAM wParam, LPARAM lParam) {
    if (nCode >= 0) {
        MessageBoxA(NULL, "Hook Triggered!", "Success", MB_OK);
    }
    return CallNextHookEx(NULL, nCode, wParam, lParam);
}

BOOL APIENTRY DllMain(HMODULE hModule, DWORD ul_reason_for_call, LPVOID lpReserved) {
    return TRUE;
}

Code Injection and Hooking

Code injection directly injects machine code into a target process; hooking modifies function calls to intercept behavior. Common techniques include:

  • Inline Hook:
    • Modify the target function’s beginning, insert jmp to custom code.
    • Advantages: Precise control.
    • Disadvantages: Complex, requires preserving original instructions.
  • IAT Hook:
    • Modify the Import Address Table (IAT), replace API addresses with custom functions.
    • Advantages: Simple, suitable for API interception.
    • Disadvantages: Limited to imported functions.

Example: Inline Hook

The following code inserts a hook at the beginning of MessageBoxA:

import win32api
import win32process
import ctypes

def inline_hook(pid, target_func, hook_code):
    h_process = win32api.OpenProcess(win32con.PROCESS_ALL_ACCESS, False, pid)
    
    # Get target function address
    user32 = ctypes.WinDLL("user32.dll")
    target_addr = ctypes.cast(getattr(user32, target_func), ctypes.c_void_p).value
    
    # Allocate memory to store hook code
    hook_addr = ctypes.windll.kernel32.VirtualAllocEx(h_process, 0, len(hook_code), 
                                                     win32con.MEM_COMMIT | win32con.MEM_RESERVE, 
                                                     win32con.PAGE_EXECUTE_READWRITE)
    
    # Write hook code
    win32api.WriteProcessMemory(h_process, hook_addr, hook_code)
    
    # Save original instructions
    original_bytes = win32api.ReadProcessMemory(h_process, target_addr, 5)
    
    # Write jmp to hook
    jmp_code = b"\xE9" + struct.pack("<I", hook_addr - target_addr - 5)
    win32api.WriteProcessMemory(h_process, target_addr, jmp_code)
    
    return original_bytes, target_addr, hook_addr

# Hook code: Display custom message
hook_code = (
    b"\x6A\x00"           # push 0
    b"\x68" + struct.pack("<I", 0x1000)  # push "Hooked!"
    b"\x68" + struct.pack("<I", 0x1004)  # push "Success"
    b"\x6A\x00"           # push 0
    b"\xFF\x15\x00\x00\x00\x00"  # call MessageBoxA (needs filling)
    b"\xC3"               # ret
)

original_bytes, target_addr, hook_addr = inline_hook(1234, "MessageBoxA", hook_code)

Example: IAT Hook

Modify the IAT to replace MessageBoxA with a custom function:

import pefile
import win32api
import win32process

def iat_hook(pid, dll_path):
    pe = pefile.PE(dll_path)
    h_process = win32api.OpenProcess(win32con.PROCESS_ALL_ACCESS, False, pid)
    
    # Locate IAT
    for entry in pe.DIRECTORY_ENTRY_IMPORT:
        if entry.dll.lower() == b"user32.dll":
            for imp in entry.imports:
                if imp.name == b"MessageBoxA":
                    # Replace with custom function address
                    custom_addr = 0x10001000  # Assume custom function address
                    win32api.WriteProcessMemory(h_process, imp.address, struct.pack("<I", custom_addr))
    
    win32api.CloseHandle(h_process)

iat_hook(1234, "sample.exe")

Runtime Memory Patching

Runtime memory patching modifies instructions or data in a process’s memory, bypassing protection or altering behavior.

Example: Memory Patch to Bypass Check

Assume the program has a check logic:

cmp eax, 0x12345678
jne error

Patch to unconditional jump:

def memory_patch(pid, address, new_bytes):
    h_process = win32api.OpenProcess(win32con.PROCESS_ALL_ACCESS, False, pid)
    win32api.WriteProcessMemory(h_process, address, new_bytes)
    win32api.CloseHandle(h_process)

# Replace jne (0x75 0x02) with jmp (0xEB 0x02)
memory_patch(1234, 0x401008, b"\xEB\x02")

Example: Frida Memory Patch

// patch_memory.js
const addr = ptr("0x401008");
const new_bytes = [0xEB, 0x02]; // jmp 0x2
Memory.write(addr, new_bytes);
console.log("Patched memory at 0x401008");

Run:

frida -f sample.exe -l patch_memory.js

Dynamic Injection and Modification Notes

  1. Permission Issues: Injection and hooking require write permissions to the target process, may trigger antivirus.
  2. Stability: Inline hooks must preserve original instructions, or crashes may occur.
  3. Detection Risk: Malware may detect injection; hide traces (e.g., use APC injection).
  4. Tool Support: Frida is suitable for cross-platform injection, x64dbg for manual hooking.

Comprehensive Example: Debugging and Injection Analysis

The following code combines anti-debugging bypass, DLL injection, and memory patching to analyze a protected PE file:

import win32api
import win32con
import win32process
import pefile
import ctypes

def analyze_protected_binary(pid, dll_path):
    # Open process
    h_process = win32api.OpenProcess(win32con.PROCESS_ALL_ACCESS, False, pid)
    
    # Bypass IsDebuggerPresent
    peb_addr = ctypes.c_ulong()
    ctypes.windll.kernel32.GetProcessInformation(h_process, 0x1A, ctypes.byref(peb_addr), ctypes.sizeof(peb_addr))
    win32api.WriteProcessMemory(h_process, peb_addr + 2, b"\x00")  # Clear BeingDebugged
    
    # DLL injection
    dll_path_bytes = dll_path.encode("ascii")
    alloc_addr = ctypes.windll.kernel32.VirtualAllocEx(h_process, 0, len(dll_path_bytes), 
                                                       win32con.MEM_COMMIT | win32con.MEM_RESERVE, 
                                                       win32con.PAGE_READWRITE)
    win32api.WriteProcessMemory(h_process, alloc_addr, dll_path_bytes)
    load_library = ctypes.cast(getattr(ctypes.WinDLL("kernel32.dll"), "LoadLibraryA"), ctypes.c_void_p).value
    h_thread = ctypes.windll.kernel32.CreateRemoteThread(h_process, None, 0, load_library, alloc_addr, 0, None)
    win32api.WaitForSingleObject(h_thread, -1)
    
    # Memory patch (bypass check)
    win32api.WriteProcessMemory(h_process, 0x401008, b"\xEB\x02")  # jne -> jmp
    
    # Clean up
    ctypes.windll.kernel32.VirtualFreeEx(h_process, alloc_addr, 0, win32con.MEM_RELEASE)
    win32api.CloseHandle(h_thread)
    win32api.CloseHandle(h_process)

analyze_protected_binary(1234, "C:\\mydll.dll")

Using x64dbg and Frida for Interactive Analysis

  1. Debugging:
    • Open x64dbg, load sample.exe.
    • Set a breakpoint at IsDebuggerPresent, modify return value (eax = 0).
    • Patch check logic (change jne to jmp).
  2. Injection:
    • Use Frida to inject DLL or hook key APIs (e.g., MessageBoxA).
    • Verify injection effect.
  3. Kernel Debugging:
    • Use WinDbg to connect to a virtual machine, set breakpoint at nt!NtCreateFile.
    • Analyze driver behavior.

Dynamic Analysis Practice

  1. Anti-Debugging Bypass: Combine patching and Frida to disable IsDebuggerPresent and timing checks.
  2. DLL Injection: Implement CreateRemoteThread injection to load a custom DLL.
  3. Hook Analysis: Use inline hooks to intercept APIs and capture parameters.
  4. Memory Patching: Dynamically patch check logic to alter program behavior.

Share your love