Cryptography in Capture The Flag (CTF)
By Frigg
1/1/2026
Cryptography is the practice of developing and using coded algorithms to protect and obscure transmitted information so that it may only be read by those with the permission and ability to decrypt it. Put differently, cryptography obscures communications so that unauthorized parties are unable to access them.

Quick orientation (what CTF crypto challenges test)
- Recognize primitives (hash, MAC, symmetric, asymmetric).
- Find incorrect use (nonce reuse, ECB, static IV, unauthenticated encryption).
- Use tooling & small scripts to recover keys / forge data.
- Typical patterns: OTP reuse, RSA low-exponent, common modulus, padding oracle, hash length extension, HMAC misuse, PRNG predictability.
Tools you’ll use a lot
hashcat/john(password/hash cracking)openssl(inspect/convert/generate)python3(+ libraries:pycryptodome,cryptography,sagewhen allowed)RsaCtfTool,msieve,yafu/pari/gp(factorization)netcat,curl,requests(interact with challenge)- Quick helpers:
xxd,base64,jq,cut,sed
Useful general commands (CTF lab)
# base64 decode/encode
echo 'aGVsbG8=' | base64 -d
echo -n 'hello' | base64
# hex dump / convert
echo -n '68656c6c6f' | xxd -r -p
xxd -p secret.bin
# openssl: AES decrypt (example, ECB or CBC)
openssl enc -aes-128-cbc -d -in cipher.bin -K <keyhex> -iv <ivhex> -nopad
# hash identification
hashid cipher.txt # or use online hash identifiers in CTF labs
# hashcat example (sha1)
hashcat -m 100 -a 0 hash.txt /path/to/wordlist.txt
# john example
john --wordlist=rockyou.txt --format=raw-md5 hash.txt
1) Encoding / classical crypto (easy CTF starters)
- Recognize: base64, hex, ROT13, Caesar, Vigenère, XOR single-byte, XOR repeating key.
- Tools:
cyberchef,xorsearch,rabin2for binaries. - Quick XOR single byte (python):
# find single-byte xor key
from itertools import cycle
data = bytes.fromhex(open('c.hex').read().strip())
for k in range(256):
p = bytes([b ^ k for b in data])
if b'CTF' in p or all(32<=c<127 for c in p[:20]):
print(k, p[:200])
2) Hashes / cracking
- Identify hash type (hashid). Use
hashcatwith correct-m. - Salted vs unsalted: if salted and salt known, craft appropriate input.
- Rainbow tables not common in modern CTFs — brute force / wordlists + rules typical.
- Hashcat example:
hashcat -m 0 -a 0 hashes.txt /usr/share/wordlists/rockyou.txt
- When facing custom hash or multiple rounds, write python to replicate and then brute-force.
3) HMAC / MAC misuse
- Length-extension attack: applies to Merkle–Damgård hashes (MD5, SHA-1, SHA-256) if app uses
hash(secret || message)to authenticate. If challenge revealshash(secret||msg)and allows you to append data, do length extension. - Tools:
hashpumpy(python).
# example: use hashpumpy to forge message and new hash
hashpumpy -s <orig_hash> -d "<orig_msg>" -a "<append_data>" -k <key_len>
- If HMAC (proper) used, length-extension not possible. Look for
md5(secret + msg)vshmac_md5.
4) Symmetric crypto gotchas (AES)
- ECB detection: repeated blocks → identical ciphertext blocks.
detect-ecb.pyor justxxd. - Nonce/IV reuse: CTR/GCM reuse catastrophic — recover keystream or plaintext XORs.
- Padding oracle: CBC with padding errors and distinguishable messages → full decryption via padding oracle attack.
Padding oracle (CTF pattern)
- If server responds differently for valid vs invalid padding, you can brute-force plaintext block-by-block.
- Python skeleton (lab only):
# pseudo-code: query oracle until padding valid
for i in range(256):
test_block = craft_block(i, known_bytes, ...)
resp = requests.get(url_with_cipher(test_block))
if resp.status_code == 200 or 'padding' not in resp.text:
# guessed byte
- Tools:
paddingoraclelibraries exist; implement per-challenge.
5) One-Time Pad / stream cipher reuse
- OTP reuse (same keystream XORed with two plaintexts):
c1 ^ c2 = p1 ^ p2. Known-plaintext/ciphertext crib-dragging recovers texts. - Use
xortoolor write crib-drag script.
6) RSA — the CTF goldmine
Common CTF RSA scenarios and quick edicts:
Recognize from challenge
- Small
e(e = 3,5,17) — potential low exponent attack (if m^e < n → cube root). - Common modulus
nreused across keys with different e — usegcd(n1, n2)to find shared prime. - Small prime factors — factor
nwithyafu,msieve,factordb(CTF offline). pandqvery close — Fermat factorization.dsmall → Wiener's attack (useRsaCtfToolorwiener).- Broadcast attack (same message to multiple recipients with small e).
- Padding oracle / Bleichenbacher (advanced; rare in CTFs but possible).
Commands & tools
# RsaCtfTool (automates many attacks)
git clone https://github.com/Ganapati/RsaCtfTool
python3 RsaCtfTool/RsaCtfTool.py --publickey target.pem --uncipher
# msieve / yafu for factoring
yafu 'factor(<n>)'
msieve <n>
# e=3 small message (use python gmpy2)
# compute integer cube root of ciphertext if m^3 < n
Example: low e (e=3) decrypt if c < n and m^3 < n
from gmpy2 import iroot, mpz
c = mpz(<ciphertext>)
m, exact = iroot(c, 3)
if exact:
print(bytes.fromhex(hex(int(m))[2:]))
7) ECC / ECDSA traps in CTFs
- Reused nonce
kin ECDSA → recover private key. If two signatures sharek, solve linear equations. - Weak curve params or small subgroup attacks sometimes appear.
- Tools:
sageor write small scripts usingecdsalibrary.
8) PRNG / RNG weaknesses
rand()ormt19937predictability is common in CTFs. If key or token generated by predictable RNG, reverse internal state.mt19937: capture 624 outputs, reconstruct state → predict next outputs.pwntoolshas helpers.- Example Python for mt19937 state recovery: use
numpy/randomorpymt19937libs.
9) Signature forgery / length extension / format flaws
- XML Signature wrapping or naive verification often exploited in advanced CTFs.
- If signature verification compares strings instead of verifying with key — trivial bypass.
10) Practical CTF walk-throughs (mini examples)
A. Hash challenge — salted bcrypt
- Given bcrypt hash string
$2b$12$..., usehashcat -m 3200orjohn --format=bcryptwith wordlist.
hashcat -m 3200 bcrypt_hash.txt rockyou.txt
B. RSA small e = 3, ciphertext c, public n
- Try low-exponent attack:
# python3 snippet (gmpy2 recommended)
from gmpy2 import iroot, mpz
c=mpz(int(cipher_hex,16))
m, ok = iroot(c,3)
if ok: print(bytes.fromhex(hex(int(m))[2:]))
C. Padding Oracle (CBC) — block-by-block
- Use padding oracle script (many public PoC scripts). Approach: iterate last byte values until server indicates valid padding. Build plaintext.
D. OTP reuse (two ciphertexts)
c1 ^ c2 = p1 ^ p2. Use crib dragging with common words ("flag","CTF","http").
11) Scripts & snippets (CTF staples)
Length extension (hashpumpy)
pip install hashpumpy
# Example:
hashpumpy -s <orig_hash> -d "<orig_msg>" -a "<append>" -k <keylen>
RSA common modulus gcd quick test (python)
import math
def gcd(a,b): return math.gcd(a,b)
n1 = int(open('n1.txt').read().strip())
n2 = int(open('n2.txt').read().strip())
g = gcd(n1,n2)
if g!=1: print("shared prime:", g)
Crib-dragger simple (XOR)
c1 = bytes.fromhex(open('c1.hex').read().strip())
c2 = bytes.fromhex(open('c2.hex').read().strip())
x = bytes([a^b for a,b in zip(c1,c2)])
# try cribs
cribs = [b'flag', b'CTF', b'http']
for crib in cribs:
for i in range(len(x)-len(crib)):
segment = bytes([x[j]^crib[j-i] for j in range(i,i+len(crib))])
12) Strategy & heuristics in CTF crypto
- Recognize patterns fast. Many challenges reuse the same classes: RSA quirks, padding oracle, length extension, OTP reuse.
- Automate trial transforms. Use CyberChef or scripts to try base64/hex/rot/xor combos quickly.
- Use tooling first, then custom script.
RsaCtfTool,hashcat,hashpumpy,xortoolsave huge time. - Check challenge metadata. CTF description often hints (e.g., “legacy” → likely MD5/SHA1; “embedded” → maybe LFSR/PRNG).
- Read outputs carefully. Error messages, whitespace, line endings often leak useful info (nonce lengths, byte-order).
13) Resources to practice
- Crypto CTF rooms on HackTheBox / TryHackMe
- Crypto challenges on cryptohack.org and old CTF writeups (study patterns)
- Implement small exercises locally: RSA toy keys, MT19937 state recoveries, padding oracle labs.