CTF Auto-Solver — Crypto Challenges

CTF Auto-Solver — Crypto Challenges

This writeup documents the development of a Python-based CTF crypto challenge auto-solver tool. The tool automatically detects and solves common cryptographic encodings and ciphers found in Capture the Flag competitions — including Base64, ROT13, Caesar, Vigenere, XOR, and hash identification. It was built to eliminate repetitive manual decoding steps during CTF competitions and speed up the flag discovery process significantly.

Project Info

• Type: Custom CTF Tool — Python
• Language: Python 3
• Dependencies: pycryptodome, requests, hashid, Jinja2
• Features: Auto-detection, multi-layer decoding, hash ID, brute force

Step 1 — Project Structure

The project was structured as a modular CLI tool with a separate solver module for each cipher category:

/ctf-solver
  solver.py           -- main entry point and orchestrator
  detector.py         -- cipher and encoding auto-detection
  solvers/
    classical.py      -- Caesar, ROT13, Vigenere, Atbash
    encoding.py       -- Base64, Base32, Base58, hex, binary
    modern.py         -- XOR, RSA helpers, AES mode checks
    hashing.py        -- hash identification and cracking
  wordlists/
    rockyou-top1k.txt -- top 1000 passwords for quick crack
  requirements.txt

Each solver module is self-contained and callable independently or through the main orchestrator which chains multiple solvers automatically when multi-layer encoding is detected.

Step 2 — Auto-Detection Engine

The detector analyses the ciphertext and scores it against known patterns to identify the most likely encoding or cipher:

# detector.py
import re, base64, binascii

def detect(ciphertext):
    ct = ciphertext.strip()
    scores = {}

    # Base64 detection
    if re.fullmatch(r'[A-Za-z0-9+/]*={0,2}', ct):
        try:
            base64.b64decode(ct)
            scores["base64"] = 90
        except Exception:
            pass

    # Hex detection
    if re.fullmatch(r'[0-9a-fA-F]+', ct) \
       and len(ct) % 2 == 0:
        scores["hex"] = 85

    # Binary detection
    if re.fullmatch(r'[01\s]+', ct):
        scores["binary"] = 80

    # Hash detection (MD5, SHA1, SHA256)
    hash_lengths = {32:"md5", 40:"sha1", 64:"sha256"}
    if re.fullmatch(r'[0-9a-f]+', ct.lower()):
        scores["hash"] = hash_lengths.get(len(ct), None)

    # Caesar / ROT13 heuristic — all alpha characters
    if re.fullmatch(r'[A-Za-z\s]+', ct):
        scores["caesar"] = 70

    best = max(scores, key=scores.get) if scores else "unknown"
    return best, scores

The detector returns a ranked list of candidates — the orchestrator tries the top-scoring solver first and falls back to the next if the result does not contain a recognisable flag pattern.

Step 3 — Encoding Solvers

Common CTF encodings are handled in the encoding module. Each solver returns the decoded string or raises an exception if decoding fails:

# solvers/encoding.py
import base64, binascii

def solve_base64(ct):
    return base64.b64decode(ct).decode("utf-8","ignore")

def solve_base32(ct):
    return base64.b32decode(ct).decode("utf-8","ignore")

def solve_hex(ct):
    return bytes.fromhex(ct).decode("utf-8","ignore")

def solve_binary(ct):
    bits = ct.replace(" ","")
    chars = [bits[i:i+8] for i in range(0,len(bits),8)]
    return "".join(chr(int(c,2)) for c in chars)

def solve_url(ct):
    from urllib.parse import unquote
    return unquote(ct)

# Multi-layer decode — recursively apply until no change
def multilayer_decode(ct, depth=0):
    if depth > 10:
        return ct
    decoded = ct
    for solver in [solve_base64, solve_hex,
                   solve_base32, solve_url]:
        try:
            result = solver(ct)
            if result != ct:
                return multilayer_decode(result, depth+1)
        except Exception:
            continue
    return decoded

Step 4 — Classical Cipher Solvers

ROT13, Caesar brute force, Vigenere, and Atbash are handled in the classical module. Caesar brute forces all 25 shifts and scores each result against English letter frequency:

# solvers/classical.py
import string

def solve_rot13(ct):
    return ct.translate(
        str.maketrans(
            string.ascii_uppercase + string.ascii_lowercase,
            string.ascii_uppercase[13:] +
            string.ascii_uppercase[:13] +
            string.ascii_lowercase[13:] +
            string.ascii_lowercase[:13]
        )
    )

def english_score(text):
    freq = "etaoinshrdlu"
    return sum(text.lower().count(c) for c in freq)

def solve_caesar(ct):
    best_score, best_text, best_shift = 0, ct, 0
    for shift in range(1, 26):
        decoded = ""
        for ch in ct:
            if ch.isalpha():
                base = ord('A') if ch.isupper() else ord('a')
                decoded += chr((ord(ch)-base-shift) % 26 + base)
            else:
                decoded += ch
        score = english_score(decoded)
        if score > best_score:
            best_score, best_text, best_shift = \
              score, decoded, shift
    return best_text, best_shift

def solve_atbash(ct):
    result = ""
    for ch in ct:
        if ch.isalpha():
            base = ord('A') if ch.isupper() else ord('a')
            result += chr(base + 25 - (ord(ch) - base))
        else:
            result += ch
    return result

Step 5 — XOR and Hash Solvers

Single-byte XOR is brute forced across all 256 key values. Hash identification uses pattern matching and length to determine the hash type before cracking:

# solvers/modern.py
def solve_xor_single(ct_bytes):
    results = []
    for key in range(256):
        decoded = bytes(b ^ key for b in ct_bytes)
        try:
            text = decoded.decode("utf-8")
            score = sum(text.lower().count(c)
                        for c in "etaoinshrdlu")
            results.append((score, key, text))
        except Exception:
            continue
    results.sort(reverse=True)
    return results[:5]   # top 5 candidates

# solvers/hashing.py
import hashlib

HASH_TYPES = {
    32:  "md5",
    40:  "sha1",
    56:  "sha224",
    64:  "sha256",
    96:  "sha384",
    128: "sha512"
}

def identify_hash(h):
    return HASH_TYPES.get(len(h.strip()), "unknown")

def crack_hash(h, wordlist="wordlists/rockyou-top1k.txt"):
    algo = identify_hash(h)
    if algo == "unknown":
        return None
    with open(wordlist) as f:
        for word in f:
            word = word.strip()
            digest = hashlib.new(algo,
                      word.encode()).hexdigest()
            if digest == h.lower():
                return word
    return None

Step 6 — Main Orchestrator

The main solver ties all modules together. It detects the cipher, chains solvers automatically, and checks each result for a flag pattern before moving to the next candidate:

# solver.py
import re
from detector import detect
from solvers.encoding import multilayer_decode
from solvers.classical import solve_caesar, solve_rot13
from solvers.modern import solve_xor_single
from solvers.hashing import crack_hash

FLAG_PATTERN = re.compile(
    r'(flag|ctf|thm|htb|picoctf)\{[^}]+\}', re.IGNORECASE
)

def find_flag(text):
    match = FLAG_PATTERN.search(text)
    return match.group(0) if match else None

def auto_solve(ciphertext):
    cipher, scores = detect(ciphertext)
    print(f"[*] Detected: {cipher} | Scores: {scores}")

    # Try encoding solvers first
    result = multilayer_decode(ciphertext)
    flag = find_flag(result)
    if flag:
        print(f"[+] FLAG FOUND: {flag}")
        return flag

    # Try classical solvers
    for solver in [solve_rot13,
                   lambda c: solve_caesar(c)[0]]:
        result = solver(ciphertext)
        flag = find_flag(result)
        if flag:
            print(f"[+] FLAG FOUND: {flag}")
            return flag

    print("[-] Flag not automatically found.")
    print(f"[*] Best guess: {result[:100]}")
    return result

Running the tool against a CTF challenge:

# Solve from command line
python3 solver.py -c "ZmxhZ3tlYXN5X2Jhc2U2NH0="
# [*] Detected: base64
# [+] FLAG FOUND: flag{easy_base64}

# Solve from file
python3 solver.py -f challenge.txt

# Force a specific solver
python3 solver.py -c "Uryyb Jbeyq" --solver rot13
# [+] FLAG FOUND: Hello World

Step 7 — Flag Pattern Library

The tool includes patterns for all major CTF platform flag formats so it can recognise flags across competitions:

# Flag patterns supported
FLAG_PATTERNS = [
    r'flag\{[^}]+\}',         # Generic
    r'CTF\{[^}]+\}',          # Generic CTF
    r'THM\{[^}]+\}',          # TryHackMe
    r'HTB\{[^}]+\}',          # HackTheBox
    r'picoCTF\{[^}]+\}',      # picoCTF
    r'DUCTF\{[^}]+\}',        # DownUnderCTF
    r'BCACTF\{[^}]+\}',       # BCACTF
    r'[A-Z0-9_]+\{[^}]+\}',   # Generic fallback
]

The generic fallback pattern catches custom flag formats used by smaller competitions where the prefix varies between challenges.

Step 8 — Installation and Usage

The tool installs cleanly into any Python 3 environment:

# Clone the repository
git clone https://github.com/user/ctf-solver
cd ctf-solver

# Install dependencies
pip3 install -r requirements.txt

# requirements.txt
pycryptodome==3.19.0
requests==2.31.0
hashid==3.1.4

# Solve a ciphertext directly
python3 solver.py -c "48 65 6c 6c 6f"
# [*] Detected: hex
# [+] FLAG FOUND: Hello

# Brute force XOR against a binary file
python3 solver.py -f cipher.bin --solver xor

# Crack a hash
python3 solver.py --hash \
  5f4dcc3b5aa765d61d8327deb882cf99
# [*] Hash type: md5
# [+] Cracked: password

Key Takeaways

• Auto-detection with scoring eliminates manual guessing — frequency analysis identifies Caesar shifts reliably
• Multi-layer recursive decoding handles nested encodings common in CTF challenges automatically
• English frequency scoring is more reliable than dictionary checks for classical cipher validation
• Modular solver design makes adding new ciphers straightforward — one file per cipher category
• Flag pattern matching across all major platforms makes the tool reusable across competitions
• Single-byte XOR brute force across all 256 keys is fast enough to run on every unknown binary blob

Tools Used

• Python 3 — core language for the solver tool
• pycryptodome — modern cryptographic primitive support
• hashid — hash type identification
• hashlib — hash computation for cracking
• base64 / binascii — encoding and decoding
• argparse — CLI argument parsing
• VS Code — development environment