Lesson 15-Reverse Engineering Practical Cases

This tutorial will detail the application of reverse engineering in real-world security analysis through three typical practical cases: cracking a simple registration machine, analyzing commercial software protection mechanisms, and detecting game cheats. Each case includes a complete analysis process, technical details, and code implementations to help readers master the core skills of reverse engineering.

Case One: Cracking a Simple Registration Machine

Target Analysis

We will analyze a simple registration machine program that requires the user to input a serial number for validation. Our goal is to locate the validation logic and generate a valid registration code.

Program Behavior Observation:

  1. User inputs serial number
  2. Program validates the serial number
  3. Displays validation result (valid/invalid)

Analysis Steps

1. Preliminary Static Analysis

Load the program using IDA Pro or Ghidra:

# Use IDAPython to quickly locate key functions
import idautils
import idc

def find_validation_function():
    # Search for common validation-related strings
    for s in idautils.Strings():
        if "valid" in s.s.lower() or "invalid" in s.s.lower():
            print(f"Found string: {s.s} at {hex(s.ea)}")
            # Find functions referencing this string
            for ref in idautils.XrefsTo(s.ea):
                func_ea = ref.frm
                func_name = idc.get_func_name(func_ea)
                print(f"  Referenced by function: {func_name} at {hex(func_ea)}")
                return func_ea
    return None

validation_func = find_validation_function()
if validation_func:
    print(f"Found potential validation function at {hex(validation_func)}")
else:
    print("No validation function found")

2. Dynamic Debugging Analysis

Run the program using OllyDbg or x64dbg and set breakpoints:

  1. Set breakpoints at GetWindowTextA or similar API calls (to capture user input)
  2. Set breakpoints at validation result output
  3. Run the program and input test serial numbers

Key Breakpoint Settings:

; Set breakpoints in OllyDbg
bp GetWindowTextA  ; Capture user input
bp MessageBoxA     ; Display validation result

3. Reverse the Validation Logic

Observe program behavior through dynamic debugging:

  1. Input known valid and invalid serial numbers
  2. Observe how the program processes these inputs
  3. Locate the key code segment of the validation logic

Common Validation Patterns:

  • Simple string comparison
  • Mathematical operation validation (e.g., CRC check)
  • Timestamp validation
  • Hardware ID binding

4. Generate Valid Registration Code

Write a registration code generator based on reverse engineering results:

# Assume the validation logic is a simple string comparison
def generate_registration_code():
    # Based on reverse engineering, the valid registration code is "VALID-KEY-123"
    return "VALID-KEY-123"

# Test the generated registration code
print(f"Generated registration code: {generate_registration_code()}")

Handling More Complex Validation Logic:
If the validation logic is more complex (e.g., mathematical operations), the algorithm must be fully reversed:

# Assume the validation logic compares the CRC32 of the input serial with a fixed value
import zlib

def validate_serial(serial):
    # Assume the fixed check value is 0x12345678
    fixed_crc = 0x12345678
    # Calculate CRC32 of the input serial
    calculated_crc = zlib.crc32(serial.encode())
    return calculated_crc == fixed_crc

# Generate a valid serial number (requires knowing the fixed check value)
def generate_serial():
    # The fixed check value must be reversed
    fixed_crc = 0x12345678  # Assumed value
    # In practice, finding an input that generates this CRC requires more complex reverse engineering
    return "GENERATED-SERIAL"

print(f"Generated serial: {generate_serial()}")

Complete Cracking Process

  1. Static Analysis: Locate the validation function
  2. Dynamic Debugging: Observe the validation logic
  3. Reverse Algorithm: Fully understand the validation mechanism
  4. Generate Registration Code: Write a generator

Case Two: Analyzing Commercial Software Protection

Target Analysis

Analyze a software protected with commercial protection tools (such as Themida or VMProtect), understand its protection mechanisms, and attempt to bypass them.

Protection Feature Observation:

  1. Performs anti-debugging checks on startup
  2. Critical code is virtualized or encrypted
  3. Memory access is protected

Analysis Steps

1. Anti-Debugging Detection Analysis

Run the program using OllyDbg or x64dbg:

  1. Set breakpoints at common anti-debugging API calls:
    • IsDebuggerPresent
    • CheckRemoteDebuggerPresent
    • NtQueryInformationProcess
    • OutputDebugString
; Set anti-debugging breakpoints in OllyDbg
bp IsDebuggerPresent
bp CheckRemoteDebuggerPresent
bp NtQueryInformationProcess
bp OutputDebugStringA
  1. Run the program and observe anti-debugging behavior

2. Virtualized/Encrypted Code Analysis

Analyze the protected area using IDA Pro or Ghidra:

  1. Look for high-entropy regions (likely encrypted or virtualized code)
  2. Analyze characteristic code of the protection tool

High Entropy Detection Script:

import math

def calculate_entropy(data):
    if not data:
        return 0.0
    entropy = 0.0
    for x in range(256):
        p_x = float(data.count(chr(x))) / len(data)
        if p_x > 0:
            entropy += - p_x * math.log(p_x, 2)
    return entropy

with open("protected_program.exe", "rb") as f:
    data = f.read()

entropy = calculate_entropy(data)
print(f"File entropy: {entropy:.2f}")

3. Bypassing Protection Mechanisms

Attempt to bypass protection based on analysis results:

  1. Anti-Debugging Bypass:
    • Modify the return value of anti-debugging checks
    • Remove anti-debugging code
  2. Virtualization/Encryption Bypass:
    • Use a debugger to trace virtual machine execution
    • Dump decrypted code

Anti-Debugging Bypass Example:

; In OllyDbg, modify the return value of IsDebuggerPresent
; Locate the instruction calling IsDebuggerPresent
; Modify the EAX register to 0 (non-debugging state)

Virtual Machine Code Dumping:

# Use x64dbg Python script to dump decrypted code from memory
import dbgscript

def dump_decrypted_code():
    # Set breakpoint at a key point in virtual machine execution
    dbgscript.breakpoint_set("vm_execution_point", lambda: dump_memory())
    
def dump_memory():
    # Get current memory content
    # Adjust based on actual protection tool
    memory = dbgscript.memory.read(0x00401000, 0x1000)  # Example address and size
    with open("decrypted_code.bin", "wb") as f:
        f.write(memory)
    print("Decrypted code dumped to decrypted_code.bin")

dump_decrypted_code()

Complete Analysis Process

  1. Identify Protection Tool: By characteristic code and behavior
  2. Analyze Protection Mechanisms: Anti-debugging, virtualization, encryption, etc.
  3. Bypass Protection: Modify anti-debugging, dump decrypted code
  4. Verify Bypass: Run the modified program

Case Three: Reverse Engineering Game Cheat Detection

Target Analysis

Develop a tool to detect game cheats by analyzing common cheat features and behavior patterns.

Common Cheat Features:

  1. Memory modification (e.g., modifying player attributes in game memory)
  2. API hooking (e.g., hooking game rendering functions)
  3. Anti-debugging behavior
  4. Abnormal network communication
  5. Unconventional system calls

Analysis Steps

1. Cheat Behavior Monitoring

Use API monitoring tools (such as API Monitor) or custom hooks:

import ctypes
from ctypes import wintypes

# Define API hook function
def hook_function(original_func):
    def hooked(*args, **kwargs):
        print(f"Hooked function {original_func.__name__} called")
        # Call the original function
        return original_func(*args, **kwargs)
    return hooked

# Example: Hook WriteProcessMemory (commonly used for memory modification)
kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)
WriteProcessMemory = kernel32.WriteProcessMemory
WriteProcessMemory.argtypes = (
    wintypes.HANDLE, wintypes.LPCVOID, wintypes.LPVOID,
    wintypes.SIZE_T, ctypes.POINTER(wintypes.SIZE_T))
WriteProcessMemory.restype = wintypes.BOOL

# Hook WriteProcessMemory
original_WriteProcessMemory = WriteProcessMemory
WriteProcessMemory = hook_function(WriteProcessMemory)

2. Memory Scanning Detection

Scan suspicious patterns in the game process memory:

import win32process
import win32api
import win32con

def scan_process_memory(process_id):
    # Open the process
    h_process = win32api.OpenProcess(win32con.PROCESS_ALL_ACCESS, False, process_id)
    
    # Scan memory regions
    memory_regions = []
    try:
        # Get process memory information
        # Requires implementation of memory region enumeration
        # Simplified example; actual implementation is more complex
        pass
    finally:
        win32api.CloseHandle(h_process)
    
    return memory_regions

# Example: Scan game process
game_process_id = 1234  # Replace with actual game process ID
suspicious_regions = scan_process_memory(game_process_id)
print(f"Suspicious memory regions: {suspicious_regions}")

3. Behavioral Analysis Detection

Monitor abnormal behavior of the game process:

import psutil

def monitor_game_process(process_name):
    for proc in psutil.process_iter(['pid', 'name', 'memory_info']):
        if proc.info['name'] == process_name:
            # Check memory usage patterns
            mem_usage = proc.info['memory_info'].rss
            print(f"Process {process_name} (PID: {proc.info['pid']}) memory usage: {mem_usage / 1024 / 1024:.2f} MB")
            
            # Check network connections
            connections = proc.connections()
            for conn in connections:
                print(f"  Connection: {conn.laddr} -> {conn.raddr}")
            
            # Check thread count
            threads = proc.num_threads()
            print(f"  Threads: {threads}")

# Example: Monitor game process
monitor_game_process("game.exe")

4. Cheat Feature Matching

Use YARA rules or custom feature matching:

import yara

# Define YARA rule (example)
rules = """
rule GameHack_MemoryModification {
    meta:
        description = "Detects memory modification patterns"
    strings:
        $write_mem = { 6A ?? 68 ?? ?? ?? ?? 64 A1 ?? ?? ?? ?? 50 64 89 25 ?? ?? ?? ?? 83 EC 10 }
    condition:
        $write_mem
}
"""

# Compile the rule
yara_rules = yara.compile(source=rules)

# Scan file or memory
matches = yara_rules.match(filepath="game.exe")  # Or scan memory
print(f"YARA matches: {matches}")

Complete Cheat Detection System

  1. Behavior Monitoring: API hooks, memory scanning, network monitoring
  2. Feature Matching: YARA rules, custom features
  3. Behavioral Analysis: Detect abnormal behavior (e.g., high-frequency memory modifications)
  4. Report Generation: Generate detection reports and evidence

Comprehensive Practical Techniques and Precautions

General Reverse Engineering Techniques

  1. Static Analysis:
    • Use IDA Pro/Ghidra for function and string analysis
    • Search for key API calls and string hints
  2. Dynamic Debugging:
    • Use OllyDbg/x64dbg to set breakpoints and observe program behavior
    • Trace key function calls and data flow
  3. Feature Extraction:
    • Extract program characteristic code and behavior patterns
    • Analyze protection mechanisms and validation logic
  4. Algorithm Reversal:
    • Fully understand validation and encryption algorithms
    • Write corresponding generators or decryption tools
  1. Legal Authorization:
    • Only analyze software you own or are authorized to analyze
    • Comply with local laws and regulations
  2. Ethical Guidelines:
    • Do not use reverse engineering for malicious purposes
    • Respect software copyright and intellectual property
  3. Security Practices:
    • Conduct analysis in an isolated environment (e.g., virtual machine)
    • Use antivirus software to protect the analysis environment

Learning Path Recommendations

  1. Basic Stage:
    • Learn assembly language and reverse engineering fundamentals
    • Master the use of IDA Pro/Ghidra and debuggers
  2. Intermediate Stage:
    • Deep dive into common protection mechanisms and bypass techniques
    • Study algorithm reversal in reverse engineering
  3. Advanced Stage:
    • Develop automated reverse engineering tools
    • Research advanced protection technologies (e.g., virtual machine protection)
    • Learn malware analysis techniques

Through the study and practice of these three practical cases, you will master the core skills of reverse engineering, enabling you to analyze various protection mechanisms and develop corresponding detection and cracking tools. Remember, reverse engineering is a double-edged sword—always use these skills within legal and ethical boundaries.

Share your love