Hacking the CTF

What a CTF Actually Is

A Capture the Flag (CTF) competition is a security challenge where participants find hidden strings (called "flags") by exploiting vulnerabilities in purpose-built systems. The flag is typically a string like flag{s0m3_r4nd0m_t3xt} that you submit to a scoreboard for points. The team or individual with the most points at the end wins.

CTFs are one of the most efficient ways to build practical security skills. Every challenge is a contained, legal environment designed to teach a specific technique. You cannot accidentally break anything important. You cannot go to prison for poking at it too hard. And when you solve it, you know exactly what you learned.

They are also the primary way that talented security practitioners first get noticed. Google's Project Zero, major security consultancies, and government agencies all actively recruit from competitive CTF teams. A strong CTF track record is often worth more than a certification in technical hiring discussions.[1]

---

CTF Categories

Most CTFs are "jeopardy-style": a grid of challenges across multiple categories, each worth a point value reflecting difficulty. Some are "attack-defense" style where teams simultaneously attack opponents and defend their own systems. For beginners, start with jeopardy-style.

Category	What It Tests	Example Challenge
Web	Web application vulnerabilities (SQLi, XSS, SSRF, SSTI, auth bypass)	Exploit a login form to retrieve the admin flag
Pwn (Binary Exploitation)	Memory corruption, buffer overflows, format strings, ROP chains	Exploit a vulnerable C binary to get a shell
Reverse Engineering	Analyzing compiled binaries, obfuscated code, custom algorithms	Reverse a binary to find what input produces the flag
Cryptography	Breaking weak cryptographic implementations	Decrypt ciphertext encrypted with a broken custom cipher
Forensics	Analyzing disk images, memory dumps, network captures, steganography	Find a flag hidden in a PCAP file or image
OSINT	Open source intelligence gathering	Find a specific piece of information about a real or fictional target from public sources
Miscellaneous	Anything else: programming challenges, trivia, puzzle-based	Decode a series of transformations to recover the flag

---

Getting Started: The Right Sequence

Most beginners try a challenge, get stuck immediately, feel demoralized, and quit. The fix is sequencing.

Step 1: Start With Guided Platforms

Before competitive CTFs, spend time on platforms designed for learning:

PicoCTF (picoctf.org). Run by Carnegie Mellon University. Problems are archived permanently. Difficulty ranges from truly beginner to intermediate. The hints system is generous. Start here if you have never done a CTF.[2]

TryHackMe (tryhackme.com). Structured rooms that guide you through techniques before asking you to apply them. The "Pre-Security" and "Jr Penetration Tester" paths include CTF-style challenges with tutorial content alongside them.

OverTheWire Wargames (overthewire.org/wargames). Text-based challenges, particularly excellent for Linux fundamentals and basic exploitation. Start with Bandit (pure Linux command line) before moving to Natas (web) or Leviathan (basic exploitation).

Step 2: Build Your Toolkit

Before competitions, have these installed and know how to use them:

# Web challenges
burpsuite          # HTTP interception and manipulation
ffuf               # Directory and parameter fuzzing
sqlmap             # SQL injection automation
gobuster           # Directory enumeration

# Forensics and steganography
binwalk            # Extract files from within other files
steghide           # Steganography tool
exiftool           # Metadata extraction
foremost           # File carving from disk images
volatility3        # Memory forensics framework

# Cryptography
CyberChef          # Browser-based data transformation (cyberchef.io)
hashcat / john     # Password cracking
pwntools           # Python library for exploit development

# Reverse engineering
ghidra             # NSA's free reverse engineering tool
gdb / pwndbg       # Debugger with pwndbg extension
ltrace / strace    # Trace library and system calls
strings            # Extract readable strings from binaries

# General
python3            # Scripting everything
nc (netcat)        # Connect to challenge services

Step 3: Enter Beginner-Friendly Competitions

Once you have solved problems on guided platforms, enter live competitions. CTFtime.org lists all upcoming CTFs with difficulty ratings and team size requirements.[3]

Good first competitions:

• picoCTF (annual, beginner)

• DEF CON CTF Qualifier (hard, but just participating is educational)

• Google CTF Beginner's Quest (annual, well-designed beginner track)

• NahamCon CTF (community-run, beginner-friendly)

• HTB Cyber Apocalypse (annual, all difficulties)

---

How to Approach a Challenge

Most people get stuck because they do not have a systematic approach. Here is one that works:

1. Enumerate Everything

Before trying to exploit anything, understand what you have. For web challenges:

# What is this application?
curl -I https://challenge.ctf.io

# What directories exist?
gobuster dir -u https://challenge.ctf.io -w /usr/share/wordlists/dirb/common.txt

# What do the source comments say?
# View → Page Source (Ctrl+U) in browser

# What are the cookies? What headers are returned?
# Burp Suite → HTTP history

For forensics challenges:

# What kind of file is this actually? (don't trust the extension)
file challenge.unknown

# What strings are readable?
strings challenge.unknown | grep -i flag

# Is there something hidden inside?
binwalk -e challenge.unknown

# What metadata does it have?
exiftool challenge.unknown

2. Look for the Obvious First

CTF challenge designers are communicating with you. The challenge name, description, and category are hints. A challenge called "Cookie Monster" in the Web category is almost certainly about cookie manipulation. Start with the obvious interpretation.

3. Identify the Vulnerability Class

In web challenges: is the input reflected anywhere? Is there a login form? Is there file upload? What does the error message say when you send unexpected input? Each of these suggests a vulnerability class to test.

In binary challenges: what does the binary do? Run it. What happens with long inputs? What happens with format strings (%x%x%x)? Use checksec to see what protections are enabled:

checksec --file=./vulnerable_binary
# RELRO:    Partial RELRO
# Stack:    No canary found       ← Stack overflow possible
# NX:       NX disabled           ← Shellcode execution possible
# PIE:      No PIE                ← Fixed addresses, easier exploitation

4. Use Automation for the Boring Parts

Password cracking, directory fuzzing, and encoding detection should be automated. Do not manually try passwords or encodings when tools exist.

# CyberChef (cyberchef.io) recognizes common encodings automatically
# Paste the ciphertext and use "Magic" operation to auto-detect

# For encoding/decoding in Python:
import base64
base64.b64decode("ZmxhZ3tleGFtcGxlfQ==")
# b'flag{example}'

5. When Stuck: Read, Ask, Walk Away

If you are stuck after 30 minutes of active effort:

• Read the challenge description again very carefully

• Look at the point value: is this a 500-point challenge you're trying as a beginner?

• Search for writeups of similar challenges from past CTFs (not this exact challenge)

• Ask your team or the CTF Discord

• Walk away for 20 minutes. The solution often arrives when you stop forcing it.

---

Category Deep Dives

Web: The Most Accessible Start

Web challenges are the most accessible for beginners because every browser is a tool and HTTP is human-readable. The foundational techniques:

SQL Injection:

-- Classic test: does adding a quote break the query?
username: admin'
-- Error? Likely SQL injection.

-- Try authentication bypass:
username: admin'--
password: anything
-- Queries become: WHERE username='admin'--' AND password='anything'
-- Everything after -- is a comment; password check is bypassed

Server-Side Template Injection (SSTI):

-- Test if template expressions are evaluated:
{{ 7*7 }}
-- If the page shows 49 instead of {{ 7*7 }}, SSTI is confirmed
-- Then escalate to code execution based on the template engine

Path Traversal:

-- Does the application serve files based on a parameter?
https://challenge.ctf.io/file?name=report.pdf
-- Try:
https://challenge.ctf.io/file?name=../../../etc/passwd

Forensics: The Art of Finding Things

Forensics challenges hide flags inside files, network captures, memory dumps, and images. The core skill is knowing where to look.

# Network capture analysis with Wireshark
# Look for: HTTP traffic with flags in parameters or responses
# Filter: http contains "flag"

# Memory forensics with Volatility
volatility3 -f memory.dmp windows.pslist  # Running processes
volatility3 -f memory.dmp windows.filescan  # Files in memory
volatility3 -f memory.dmp windows.dumpfiles --virtaddr 0xADDR  # Extract file

# Steganography
steghide extract -sf image.jpg  # Extract hidden data (may need password)
zsteg image.png  # LSB steganography in PNG files

Cryptography: Break the Math

Crypto challenges exploit implementation weaknesses rather than the underlying mathematics. AES and RSA are not broken, but specific modes, small key sizes, reused nonces, and implementation errors are.

Common crypto challenge patterns:

Pattern	What to Try
RSA with small e and small message	Check if m^e < n; modular reduction may not happen, so plaintext is recoverable directly
Repeated XOR key	Detect key length with index of coincidence; recover with frequency analysis
ECB mode block cipher	Detect by looking for repeated 16-byte blocks in ciphertext
CBC with padding oracle	Automate with padbuster or custom script
Predictable random seed	If seed is time-based, brute-force the timestamp range

---

Writing Writeups

A writeup is a technical walkthrough of how you solved a challenge. Writing them is not optional. It is how you consolidate what you learned and build your portfolio.

What a good writeup includes:

1. The challenge name, category, and point value

2. What the challenge presented (screenshot or description)

3. Your initial observation and what vulnerability class you suspected

4. The specific steps you took, including dead ends

5. The actual exploit or solution code

6. The flag

7. What you learned

Writeups go on your GitHub. They demonstrate the ability to reason through a problem systematically and explain findings clearly, which is exactly what penetration test reports require.

The compound effect of writeups: Every writeup you publish becomes searchable. Other practitioners find them when working on similar problems. Your profile grows passively. Recruiters and hiring managers look for candidates who have public evidence of thinking, and a GitHub full of CTF writeups is exactly that.

Some of the most successful security practitioners built their early reputation entirely through CTF writeups and open-source tool contributions, before they had any industry experience at all.

---

Building a CTF Team

Solo CTF is harder. Most top competitors work in teams of four to eight people with complementary skills: one person focusing on web, one on pwn, one on crypto, one on forensics.

Finding a team:

• CTFtime.org has a team-finding feature

• CTF Discord servers (many competitions have them)

• University security clubs

• The community Discord for this book

Starting a team: all you need is two people who are learning together. Document what you solve. Share writeups internally before publishing. Compete in beginner-friendly events. Skill develops faster when you can see how others approach the same problem.

---

The Complete Free Resource Library

CTFs are one entry point into security. The wider universe of free, high-quality training is enormous if you know where to look. Everything below is free at the time of writing. Pick a category, start, finish, and move on. Quality beats quantity.

Free Practice Platforms (browser-based, no install)

• TryHackMe — free rooms include the entire Pre Security path, Complete Beginner path, OWASP Top 10, Linux Fundamentals 1-3, Network Fundamentals, and Web Fundamentals (https://tryhackme.com)

• HackTheBox Starting Point — free tier covering the first six machines with guided walkthroughs (https://app.hackthebox.com/starting-point)

• HackTheBox Academy free modules — Introduction to Networking, Linux Fundamentals, Windows Fundamentals, Intro to Academy (https://academy.hackthebox.com)

• PortSwigger Web Security Academy — 200+ hands-on web labs, completely free, from the creators of Burp Suite (https://portswigger.net/web-security). The single strongest free web-security training available anywhere.

• PicoCTF — Carnegie Mellon's permanent CTF archive (https://picoctf.org)

• OverTheWire Wargames — Bandit, Natas, Narnia, Leviathan, Krypton, and more (https://overthewire.org/wargames/)

• Root Me — 500+ free challenges across all categories (https://www.root-me.org)

• CTFlearn — beginner-friendly community challenges (https://ctflearn.com)

• pwn.college — Arizona State University's free binary exploitation curriculum (https://pwn.college)

• Cryptopals — Matasano's eight-set crypto challenge gauntlet (https://cryptopals.com)

Free Vulnerable VMs (download and run locally)

• VulnHub — hundreds of vulnerable VMs; classics include Mr-Robot, the Kioptrix series, Brainpan, and the DC series (https://www.vulnhub.com)

• OWASP Juice Shop — modern web app intentionally riddled with the OWASP Top 10 (https://owasp.org/www-project-juice-shop/)

• OWASP WebGoat — guided web vulnerability lessons (https://owasp.org/www-project-webgoat/)

• DVWA (Damn Vulnerable Web Application) — PHP/MySQL app for SQLi, XSS, and command injection practice (https://github.com/digininja/DVWA)

• Metasploitable 3 — purposely vulnerable Windows and Linux VMs from Rapid7 (https://github.com/rapid7/metasploitable3)

• HackTheBox Starting Point VMs — free, downloadable machines on the free tier

• VulnLab free machines — newer, well-designed, free entry tier (https://www.vulnlab.com)

Free GitHub Repos (reference and tooling)

• PayloadsAllTheThings — every web payload you will ever need, organized by vulnerability class (https://github.com/swisskyrepo/PayloadsAllTheThings)

• SecLists — Daniel Miessler's wordlists for fuzzing, passwords, usernames, and subdomains (https://github.com/danielmiessler/SecLists)

• Awesome Hacking — meta-list of hacking resources (https://github.com/Hack-with-Github/Awesome-Hacking)

• Awesome CTF — curated CTF tools, frameworks, and wargames (https://github.com/apsdehal/awesome-ctf)

• HackTricks — Carlos Polop's living encyclopedia of pentesting techniques (https://github.com/carlospolop/hacktricks)

• PEASS-ng — privilege escalation enumeration suite including LinPEAS and WinPEAS (https://github.com/peass-ng/PEASS-ng)

• GTFOBins — Unix binaries that can break restricted shells (https://gtfobins.github.io)

• LOLBAS — Windows equivalent for living-off-the-land binaries (https://lolbas-project.github.io)

• CTF Writeups archives — historical writeup repositories (https://github.com/ctfs and forks)

Free YouTube Channels (verified high signal)

• IppSec — the gold standard for HTB walkthroughs; every retired box explained (https://www.youtube.com/@ippsec)

• LiveOverflow — binary exploitation, CTFs, deep technical content (https://www.youtube.com/@LiveOverflow)

• John Hammond — CTF walkthroughs, malware analysis, beginner-friendly (https://www.youtube.com/@_JohnHammond)

• The Cyber Mentor (Heath Adams / TCM Security) — pentesting fundamentals (https://www.youtube.com/@TCMSecurityAcademy)

• NahamSec — bug bounty hunting (https://www.youtube.com/@NahamSec)

• STÖK — bug bounty methodology and storytelling (https://www.youtube.com/@stokfredrik)

• HackerSploit — broad pentesting tutorials (https://www.youtube.com/@HackerSploit)

• Computerphile — cryptography and computer-science fundamentals (https://www.youtube.com/@Computerphile)

• Professor Messer — free Security+ and Network+ courses (https://www.youtube.com/@professormesser)

• David Bombal — networking, Linux, and ethical hacking (https://www.youtube.com/@davidbombal)

• Simply Cyber (Gerald Auger) — GRC, careers, and the human side of security (https://www.youtube.com/@SimplyCyber)

• Gynvael Coldwind — advanced reversing and exploitation streams (https://www.youtube.com/@GynvaelEN)

Free Books and Reading

• OWASP Web Security Testing Guide — the methodology behind modern web pentesting (https://owasp.org/www-project-web-security-testing-guide/)

• OWASP Application Security Verification Standard (ASVS) — the requirements catalog for secure web apps (https://owasp.org/www-project-application-security-verification-standard/)

• OWASP Cheat Sheet Series — fast-reference cheat sheets for every common security topic (https://cheatsheetseries.owasp.org)

• The Hitchhiker's Guide to Online Anonymity — comprehensive OPSEC reference (https://anonymousplanet.org)

• NIST SP 800-115 Technical Guide to Information Security Testing — the federal reference for security testing methodology (https://csrc.nist.gov/pubs/sp/800/115/final)

A recommended starting path for absolute beginners:

1. Pre Security path on TryHackMe (free) — networking and Linux foundations

2. OverTheWire Bandit — Linux command line fluency

3. PortSwigger Web Security Academy — first 30 labs, especially SQL injection and XSS

4. PicoCTF beginner challenges — get used to the CTF format

5. HackTheBox Starting Point — guided machine compromises

6. Pick one YouTube channel (IppSec for HTB, LiveOverflow for pwn) and watch one video per week

This sequence is free, requires only a laptop and internet access, and produces a solid foundation in three to six months at a few hours per week.

---

References

[1] Chung, S. P., Xu, W., & Lee, W. (2013). CTF competitions as cybersecurity education. IEEE Security and Privacy, 2013. doi:10.1109/MSP.2013.6

[2] PicoCTF. (2024). picoCTF: Free Cybersecurity Games for Students. Carnegie Mellon University. Retrieved from https://picoctf.org

[3] CTFtime Team. (2024). CTFtime: All About CTF (Capture The Flag). Retrieved from https://ctftime.org

[4] OverTheWire. (2024). Wargames. Retrieved from https://overthewire.org/wargames/

[5] Peng, P., Xu, L., Quinn, L., Hu, H., Gu, G., & Wang, H. (2014). jTrans: Jump-target-aware transformer for binary code similarity. IEEE S&P 2014. doi:10.1109/SP.2014.21

---

Further Reading

Resource	What It Covers
CTFtime.org	All upcoming CTFs with ratings, writeup archives, and team search
PicoCTF	Best beginner CTF platform; permanently available archive
OverTheWire	Progressive wargames for Linux, web, and exploitation fundamentals
HTB Academy	Structured learning paths with CTF-style challenges
CTF101.org	Introductory guides to each CTF category
LiveOverflow on YouTube	CTF challenge walkthroughs and binary exploitation education

---

Playing CTFs and want teammates, writeup feedback, or challenge hints? Join the community on Discord or reach out on LinkedIn. If this chapter helped, contribute back. This book is open source and your additions are welcome.