Encryption algorithms are core technologies in information security, widely used in data protection, communication security, and software protection. This tutorial will deeply explore the working principles of common encryption algorithms (AES, RSA, XOR), key extraction techniques, and methods for dynamic and static decryption, helping readers master the core technologies of encryption and decryption.
Analysis of Common Encryption Algorithms
AES (Advanced Encryption Standard)
AES is the most widely used symmetric encryption algorithm today, published by the U.S. National Institute of Standards and Technology (NIST) in 2001, replacing the DES algorithm.
AES Features:
- Symmetric Encryption: Uses the same key for encryption and decryption
- Block Encryption: Divides data into 128-bit (16-byte) blocks for encryption
- Multiple Key Lengths: Supports 128-bit, 192-bit, and 256-bit keys
- Efficiency: Has efficient implementations in both software and hardware
- Security: Extensively analyzed and tested, currently considered secure
AES Operation Modes:
- ECB (Electronic Codebook): The simplest mode, each block is encrypted independently
- CBC (Cipher Block Chaining): Each block is XORed with the previous ciphertext block before encryption
- CFB (Cipher Feedback): Converts block cipher into a self-synchronizing stream cipher
- OFB (Output Feedback): Converts block cipher into a synchronous stream cipher
- CTR (Counter): Uses a counter to generate a keystream
AES Encryption Process:
- Key Expansion: Expands the original key into multiple round keys
- Initial Round: XORs the plaintext with the first round key
- Main Rounds: Performs multiple rounds (10/12/14 rounds) of encryption operations, including:
- SubBytes
- ShiftRows
- MixColumns
- AddRoundKey
- Final Round: The last round omits the MixColumns operation
AES Decryption Process:
Decryption is the reverse of encryption, using the same round keys but in reverse order:
- Initial round: XOR with the first round key
- Main rounds: Perform multiple rounds of decryption operations (InvSubBytes, InvShiftRows, InvMixColumns, AddRoundKey)
- Final round: The last round omits InvMixColumns
RSA (Rivest-Shamir-Adleman)
RSA is an asymmetric encryption algorithm based on the difficulty of large integer factorization, widely used in digital signatures and key exchange.
RSA Features:
- Asymmetric Encryption: Uses a public key for encryption and a private key for decryption
- Based on Number Theory: Relies on the difficulty of large integer factorization
- Key Generation:
- Choose two large prime numbers p and q
- Compute n = p*q
- Compute Euler’s totient function φ(n) = (p-1)*(q-1)
- Choose an integer e coprime to φ(n)
- Compute d such that e*d ≡ 1 mod φ(n)
- Public Key: (n, e)
- Private Key: (n, d)
RSA Encryption Process:
- Convert plaintext to an integer m (0 < m < n)
- Compute ciphertext c = m^e mod n
RSA Decryption Process:
- Compute plaintext m = c^d mod n
- Convert integer m back to plaintext
RSA Security:
- Security depends on the difficulty of large integer factorization
- Currently recommended to use 2048-bit or longer keys
- For the same security level, RSA keys are much longer than ECC keys
XOR (Exclusive OR Encryption)
XOR is a simple symmetric encryption algorithm that implements encryption and decryption through XOR operations.
XOR Features:
- Symmetric Encryption: Uses the same key for encryption and decryption
- Simple and Efficient: Fast computation, low resource consumption
- Reversibility: Encryption and decryption use the same operation
- Low Security: Easily broken, usually combined with other encryption methods
XOR Encryption Process:
- Perform bitwise XOR between plaintext and key
- Obtain ciphertext
XOR Decryption Process:
- Perform bitwise XOR between ciphertext and the same key
- Obtain plaintext
Limitations of XOR Encryption:
- If the key length is shorter than the plaintext, the key is reused, leading to predictable patterns
- Vulnerable to known-plaintext attacks
- Usually needs to be combined with other encryption techniques
Key Extraction Techniques
Key extraction is the technique of recovering encryption keys from encrypted programs or systems and is an important part of reverse engineering and cryptanalysis.
Static Key Extraction
Static key extraction is the technique of extracting keys from program code or data without executing the program.
Common Methods:
- String Extraction:
- Search for plaintext keys in the program
- Use string extraction tools (such as the strings command)
strings encrypted_program.exe | grep -i "key" - Code Pattern Analysis:
- Analyze the implementation code of the encryption algorithm
- Identify hard-coded keys or key generation logic
- Look for encryption function calls and parameters
- Data Segment Analysis:
- Check program data segments (such as .rdata, .data)
- Look for possible key storage locations
- Analyze key setup in initialization code
- Decompilation Analysis:
- Use decompilers (such as IDA Pro, Ghidra) to analyze the program
- Identify encryption-related functions and variables
- Track key generation and usage
Static Key Extraction Example:
# Extract strings from PE file using Python
import pefile
def extract_strings(pe_file):
pe = pefile.PE(pe_file)
strings = []
for section in pe.sections:
data = section.get_data()
# Extract printable strings
for i in range(len(data)):
if data[i] == 0: # String terminator
continue
if 32 <= data[i] <= 126: # Printable ASCII characters
s = ""
j = i
while j < len(data) and 32 <= data[j] <= 126:
s += chr(data[j])
j += 1
if len(s) > 4: # Only extract strings longer than 4
strings.append(s)
i = j - 1
return strings
# Extract strings
strings = extract_strings("encrypted_program.exe")
print("Extracted strings:", strings)
Dynamic Key Extraction
Dynamic key extraction is the technique of capturing encryption keys during program execution, typically used to bypass static protection measures.
Common Methods:
- Debugger Capture:
- Run the program using a debugger (such as OllyDbg, x64dbg)
- Set breakpoints at encryption function calls
- Capture keys during the encryption process
- Memory Dumping:
- Dump program memory during encryption
- Search for keys in memory
- Use memory analysis tools (such as Cheat Engine)
- API Monitoring:
- Monitor encryption-related API calls
- Capture API call parameters and return values
- Identify key passing processes
- Hooking Techniques:
- Use hooks to intercept encryption function calls
- Capture keys during function execution
- Analyze encryption logic
Dynamic Key Extraction Example:
# Use x64dbg Python script to capture encryption key
import dbgscript
def on_breakpoint_hit():
# Get register values
eax = dbgscript.registers['eax']
ebx = dbgscript.registers['ebx']
# Assume key is stored in eax
key = eax
print(f"Captured key: {key}")
# Continue execution
dbgscript.continue_event()
# Set breakpoint at encryption function
dbgscript.breakpoint_set("encrypt_function", on_breakpoint_hit)
Key Recovery Techniques
When keys cannot be directly extracted, attempt to recover them:
- Known-Plaintext Attack:
- If part of the plaintext and corresponding ciphertext are known
- Attempt to recover the key or key generation algorithm
- Side-Channel Attack:
- Analyze physical leakage during encryption (such as power consumption, electromagnetic radiation)
- Recover key or key generation information
- Differential Analysis:
- Analyze differences in encryption algorithm responses to different inputs
- Recover key or key generation algorithm
- Linear Analysis:
- Analyze linear approximations of the encryption algorithm
- Recover key or key generation algorithm
Dynamic Decryption and Static Decryption
Static Decryption
Static decryption is the technique of analyzing and decrypting encrypted data without executing the program.
Common Methods:
- Hard-Coded Key Decryption:
- If the encryption key is known
- Use the same algorithm and key to decrypt the data
- Code Analysis Decryption:
- Analyze the implementation of the encryption algorithm
- Implement the same decryption algorithm
- Use the extracted key to decrypt data
- Pattern Recognition Decryption:
- Identify patterns in encrypted data
- Use known encryption patterns to decrypt
Static Decryption Example:
from Crypto.Cipher import AES
# Known key and IV
key = b'Sixteen byte key'
iv = b'Sixteen byte iv '
# Encrypted data
cipher = AES.new(key, AES.MODE_CBC, iv)
encrypted_data = cipher.encrypt(b'Secret message')
# Static decryption
decipher = AES.new(key, AES.MODE_CBC, iv)
decrypted_data = decipher.decrypt(encrypted_data)
print("Decrypted:", decrypted_data.decode())
Dynamic Decryption
Dynamic decryption is the technique of capturing and decrypting encrypted data during program execution.
Common Methods:
- Debugger Decryption:
- Run the program using a debugger
- Set breakpoints at encryption function calls
- Capture plaintext and ciphertext during encryption
- Implement decryption algorithm to decrypt data
- Memory Dump Decryption:
- Dump program memory during encryption
- Search for encrypted data in memory
- Use extracted key to decrypt data
- Hook Decryption:
- Use hooks to intercept encryption function calls
- Capture plaintext and ciphertext during function execution
- Implement decryption algorithm to decrypt data
- API Monitoring Decryption:
- Monitor encryption-related API calls
- Capture API call parameters and return values
- Identify encrypted data and decryption process
Dynamic Decryption Example:
# Use x64dbg Python script for dynamic decryption
import dbgscript
def on_encrypt_call():
# Get parameters
data_ptr = dbgscript.registers['rcx'] # Assume data pointer in rcx
data_len = dbgscript.registers['rdx'] # Assume data length in rdx
# Read data from memory
data = dbgscript.memory.read(data_ptr, data_len)
# Add decryption logic here
# Assume we know the encryption algorithm and key
# decrypted_data = decrypt_algorithm(data, key)
# Print original data
print(f"Captured data: {data}")
# Continue execution
dbgscript.continue_event()
# Set breakpoint at encryption function
dbgscript.breakpoint_set("encrypt_function", on_encrypt_call)
Comprehensive Case Study
Let’s practice encryption and decryption techniques through a comprehensive case.
Case: Analyzing an Application Using AES Encryption
Goal: Analyze an application using AES encryption, extract the encryption key, and decrypt the data
Steps:
- Sample Acquisition and Preparation:
- Obtain the encrypted application from a legitimate source
- Perform analysis in an isolated virtual machine environment
- Packing Detection:
- Use PEiD to detect packer type
- Use Detect It Easy to view file information
- Calculate file entropy
- Entry Point Analysis:
- Load the program into OllyDbg
- Set breakpoint at program entry point
- Observe entry point code
- Encryption Mechanism Analysis:
- Identify AES encryption characteristics
- Analyze encryption function calls
- Identify key generation and usage
- Key Extraction:
- Use static analysis to find hard-coded keys
- Use dynamic analysis to capture runtime keys
- Analyze key generation algorithm
- Decryption Process:
- Implement AES decryption algorithm
- Use extracted key to decrypt data
- Verify decryption results
- Verify Decryption Results:
- Check if decrypted data is readable
- Verify if key information is complete
- Analyze decrypted content
Case Analysis Results:
- Identified the program uses AES-256-CBC encryption
- Successfully extracted the encryption key
- Implemented decryption algorithm and successfully decrypted data
- Verified the correctness of decryption results
Summary and Advanced Learning
Encryption algorithms and decryption techniques are core areas of information security. By understanding the working principles of common encryption algorithms (AES, RSA, XOR), key extraction techniques, and methods for dynamic and static decryption, we can better perform reverse engineering and security research. This tutorial has covered in detail:
- Working principles and features of common encryption algorithms (AES, RSA, XOR)
- Key extraction techniques (static and dynamic)
- Methods for dynamic and static decryption
Advanced Learning Recommendations:
- Deeply study the specific implementation details of different encryption algorithms
- Research more advanced encryption techniques (such as ECC, homomorphic encryption)
- Learn advanced cryptanalysis techniques (such as differential cryptanalysis, linear cryptanalysis)
- Explore machine learning-based encryption detection techniques
- Participate in open-source encryption and decryption projects
- Learn to write custom encryption and decryption tools
Through continuous learning and practice, you will master advanced encryption and decryption techniques and contribute to information security research and reverse engineering.



