Packing and virtual machine protection are commonly used protection techniques in malware and commercial software. They increase the difficulty of reverse engineering by compressing, encrypting, or virtualizing code. This tutorial will deeply explore the working principles of common packing tools, manual unpacking processes, and analysis methods for virtual machine protection technologies, helping readers master these key techniques.
Analysis of Common Packing Tools
UPX (Ultimate Packer for eXecutables)
UPX is one of the most popular open-source packing tools. It uses the LZMA compression algorithm to compress executable files and adds custom decompression code.
UPX Packing Features:
- Compression Algorithm: Uses LZMA algorithm for efficient compression
- Decompression Code: Inserts decompression code at the program entry point
- Simple Detection: Can be detected via file header characteristics and entropy values
- Easy Unpacking: Mature automatic unpacking tools are available
UPX Packing Process:
- Compress the original executable file content
- Add UPX header information to the file header
- Insert decompression code at the program entry point
- Modify the import table and relocation information
UPX Unpacking Methods:
- Automatic Unpacking:
upx -d packed_file.exe - Manual Unpacking:
- Use a debugger (such as OllyDbg) to set breakpoints in the decompression code
- Run the program and pause when execution reaches the decompression code
- Dump the decompressed code from memory
- Repair the import table and relocation information
VMProtect
VMProtect is a commercial-grade virtual machine protection tool that converts critical code into virtual machine bytecode, which is interpreted and executed by the virtual machine at runtime.
VMProtect Protection Features:
- Code Virtualization: Converts x86 instructions into custom bytecode
- Virtual Machine Execution: Interprets and executes bytecode at runtime
- Anti-Debugging: Includes various anti-debugging techniques
- Anti-Memory Dumping: Protects code in memory
VMProtect Protection Process:
- Select critical code segments to protect
- Convert x86 instructions into VM bytecode
- Insert virtual machine interpreter code
- Modify program control flow to use the virtual machine
VMProtect Unpacking Methods:
- Static Analysis:
- Identify virtual machine bytecode characteristics
- Analyze virtual machine interpreter code
- Dynamic Analysis:
- Use a debugger to track virtual machine execution
- Dump decoded code from memory
- Hybrid Method:
- Combine static and dynamic analysis
- Use automated tools for assistance
Themida
Themida is an advanced commercial protection tool that combines multiple protection techniques, including virtual machine protection, code obfuscation, and anti-debugging.
Themida Protection Features:
- Multi-Technique Integration: Combines virtual machine, obfuscation, and anti-debugging
- Advanced Anti-Debugging: Includes various anti-debugging techniques
- Memory Protection: Prevents memory dumping and modification
- Code Obfuscation: Increases reverse engineering difficulty
Themida Protection Process:
- Select critical code to protect
- Apply multiple protection techniques (virtual machine, obfuscation, etc.)
- Insert anti-debugging and anti-memory dumping code
- Modify program control flow
Themida Unpacking Methods:
- Static Analysis:
- Identify protection technique characteristics
- Analyze protection code logic
- Dynamic Analysis:
- Use a debugger to track protection code execution
- Bypass anti-debugging techniques
- Hybrid Method:
- Combine static and dynamic analysis
- Use automated tools for assistance
Manual Unpacking Process
Manual unpacking is an important practice for understanding packing techniques. The following is a detailed unpacking process:
Preparation Work
Environment Setup:
- Use a virtual machine (such as VMware or VirtualBox)
- Install a debugger (such as OllyDbg, x64dbg)
- Prepare PE file analysis tools (such as PEiD, CFF Explorer)
Security Measures:
- Ensure operation in an isolated environment
- Back up the original file
- Record all operation steps
Packing Detection
Detection Methods:
- File Header Analysis:
- Use PEiD or CFF Explorer to check the file header
- Identify packer characteristics
- Entropy Calculation:
- Calculate file entropy (high entropy may indicate compression or encryption)
import math
def calculate_entropy(data):
if not data:
return 0.0
entropy = 0.0
for x in range(256):
p_x = float(data.count(chr(x))) / len(data)
if p_x > 0:
entropy += - p_x * math.log(p_x, 2)
return entropy
with open("packed_file.exe", "rb") as f:
data = f.read()
entropy = calculate_entropy(data)
print(f"File entropy: {entropy}")
- Import Table Analysis:
- Check if the import table has been modified
- Identify API calls specific to the packer
Entry Point Analysis
Analysis Steps:
- Load the packed program into a debugger
- Set a breakpoint at the program entry point (such as
mainCRTStartup) - Run the program and observe the entry point code
- Identify the packer’s decompression or decryption code
Decompression/Decryption Process Tracking
Tracking Methods:
- Set breakpoints in the decompression/decryption code
- Run the program and pause when execution reaches the breakpoint
- Analyze the decompression/decryption process in registers and memory
- Identify the location of the decompressed code
Memory Dumping
Dumping Methods:
- Set a memory dump breakpoint after decompression completes
- Dump the decompressed code from memory
- Use OllyDbg’s “Dump memory” function
- Or use tools like Cheat Engine
- Save the dumped memory as a new PE file
Repair Import Table and Relocation
Repair Steps:
- Open the dumped file with a PE file editor (such as CFF Explorer)
- Repair the import table:
- Add necessary DLL and function imports
- Repair relocation information (if needed)
- Save the repaired file
Verify Unpacking Results
Verification Methods:
- Use a disassembler (such as IDA Pro) to analyze the unpacked file
- Check if the program runs normally
- Verify if key functions are available
- Check for any remaining protection code
Virtual Machine Protection Analysis
Virtual machine protection is an advanced protection technique that converts critical code into virtual machine bytecode, which is interpreted and executed by the virtual machine at runtime. The following are analysis methods for two common virtual machine protections:
VMP (Virtual Machine Protector)
VMP is a commercial-grade virtual machine protection tool that converts x86 instructions into custom bytecode.
VMP Protection Features:
- Bytecode Conversion: Converts x86 instructions into VMP bytecode
- Virtual Machine Interpreter: Interprets and executes bytecode at runtime
- Anti-Debugging: Includes various anti-debugging techniques
- Memory Protection: Prevents memory dumping and modification
VMP Analysis Methods:
- Static Analysis:
- Identify VMP bytecode characteristics
- Analyze virtual machine interpreter code
- Identify bytecode conversion patterns
- Dynamic Analysis:
- Use a debugger to track virtual machine execution
- Dump decoded code from memory
- Analyze bytecode execution flow
- Hybrid Method:
- Combine static and dynamic analysis
- Use automated tools for assistance
VMP Unpacking Process:
- Identify VMP protected regions
- Track virtual machine interpreter execution
- Dump decoded code
- Repair code structure and control flow
- Verify unpacking results
Obsidium
Obsidium is an advanced virtual machine protection tool that combines virtual machine and code obfuscation techniques.
Obsidium Protection Features:
- Multi-Layer Protection: Combines virtual machine and obfuscation techniques
- Advanced Anti-Debugging: Includes various anti-debugging techniques
- Memory Protection: Prevents memory dumping and modification
- Code Obfuscation: Increases reverse engineering difficulty
Obsidium Analysis Methods:
- Static Analysis:
- Identify Obsidium protection characteristics
- Analyze protection code logic
- Identify obfuscated and virtualized regions
- Dynamic Analysis:
- Use a debugger to track protection code execution
- Bypass anti-debugging techniques
- Dump decoded code
- Hybrid Method:
- Combine static and dynamic analysis
- Use automated tools for assistance
Obsidium Unpacking Process:
- Identify Obsidium protected regions
- Bypass anti-debugging techniques
- Track virtual machine execution
- Dump decoded code
- Repair code structure and control flow
- Verify unpacking results
Comprehensive Case Study
Let’s practice packing and virtual machine protection analysis techniques through a comprehensive case.
Case: Analyzing an Application Protected with Themida
Goal: Analyze an application protected with Themida, understand its protection mechanism, and attempt unpacking
Steps:
- Sample Acquisition and Preparation:
- Acquire a Themida-protected application from a legitimate source
- Perform analysis in an isolated virtual machine environment
- Packing Detection:
- Use PEiD to detect packer type
- Use Detect It Easy to view file information
- Calculate file entropy
- Entry Point Analysis:
- Load the program into OllyDbg
- Set a breakpoint at the program entry point
- Observe entry point code
- Protection Mechanism Analysis:
- Identify Themida protection characteristics
- Analyze virtual machine interpreter code
- Identify anti-debugging techniques
- Dynamic Analysis:
- Track virtual machine execution
- Dump decoded code from memory
- Bypass anti-debugging techniques
- Unpacking Process:
- Set a memory dump breakpoint after decompression/decryption completes
- Dump decompressed code from memory
- Repair import table and relocation information
- Verify Unpacking Results:
- Use IDA Pro to analyze the unpacked file
- Check if the program runs normally
- Verify if key functions are available
Case Analysis Results:
- Identified the program uses Themida protection
- Analyzed the working principle of the virtual machine interpreter
- Successfully dumped the decoded code
- Repaired the import table and verified the unpacking results
Summary and Advanced Learning
Packing and virtual machine protection are important techniques in software protection. By understanding the working principles and analysis methods of these techniques, we can better perform reverse engineering and security research. This tutorial has covered in detail:
- Working principles of common packing tools (UPX, VMProtect, Themida)
- Detailed manual unpacking process
- Analysis methods for virtual machine protection techniques (VMP, Obsidium)
Advanced Learning Recommendations:
- Deeply study the specific implementation details of different packing tools
- Research the implementation principles of more virtual machine protection techniques
- Learn advanced reverse engineering techniques (such as code obfuscation analysis)
- Explore machine learning-based packing detection techniques
- Participate in open-source packing and unpacking projects
- Learn to write custom packing and unpacking tools
Through continuous learning and practice, you will master advanced packing and virtual machine protection techniques and contribute to software analysis and security research.



