Linux Perf Handbook

Performance engineering, debugging, and system tuning reference.

Target Kernel: Linux 6.6+ (EEVDF scheduler, modern eBPF features) Last Updated: 2026-02

How to Use This Handbook

1. Start with the 60-second checklist below for quick triage
2. Use the Quick Navigation to find specific topics
3. Check the cheatsheets for copy-paste commands
4. Refer to curated sources for deep dives on specific areas

For investigation workflow:

Symptom → 60-Second Analysis → Identify resource bottleneck → Deep dive chapter

Quick Navigation

Chapter	Description
01 - Modern CLI Replacements	Rust/Go tools replacing classic Unix utils
02 - System Monitoring	CPU, memory, process monitoring
03 - Network Analysis	Traffic analysis, DNS, HTTP testing
04 - Disk & Storage	I/O benchmarking, filesystem tools
05 - Performance Profiling	perf, flame graphs, profilers
06 - eBPF & Tracing	BCC tools, bpftrace, ftrace, sched_ext
07 - Containers & K8s	Docker, Kubernetes debugging
08 - Kernel Tuning	sysctl, EEVDF scheduler, memory, I/O
09 - Network Tuning	TCP, BBR, io_uring, NUMA networking
10 - Java/JVM	JVM profiling and tuning
11 - GPU & HPC	GPU profiling, MIG, HPC tracing
12 - Observability & Metrics	Prometheus, Grafana, OpenTelemetry
13 - Latency Analysis	Tail latency, coordinated omission, P99
14 - Database Profiling	PostgreSQL, MySQL, query optimization
Database Production Debugging	Hot partitions, cache pollution, admission control, lock analysis
15 - Memory Subsystem	NUMA, huge pages, memory profiling
16 - Scheduler & Interrupts	CPU scheduling, context switching
Scheduler Debugging Deep Dive	CFS bugs, run queue attribution, Perfetto, noisy neighbor detection
17 - Ftrace Production	Function tracing, trace-cmd
18 - VDSO & Clock Source	Time syscalls, TSC, cloud VM performance
18 - Off-CPU Analysis	Wall-clock profiling, blocking detection, load-scaling bottlenecks
19 - Storage Engine Patterns	LMDB, RocksDB, LSM, columnar, vectorized engines
26 - C++ HFT Optimization Patterns	SwissTable/F14, branchless, hugepages, FastQueue, kernel bypass, 30 LLM-actionable levers

Advanced Topics & Production Patterns

Guide	Description
CRDTs: Lock-Free Distributed State	G-Counter, PN-Counter, OR-Set, LWW-Register with delta sync
Coordinated Omission Guide	Load testing correctness, wrk2, timestamp injection
eBPF Performance Overhead	Hook overhead, map types, production deployment
Container Debugging Patterns	cAdvisor, GOMAXPROCS, PSI, cgroup v2 debugging
TCP Edge Cases & Load Balancers	SYN retry, LB buffering, timeout hierarchies
Bryan Cantrill Debugging Methodology	Questions-first, state preservation, systematic elimination

Cheatsheets

Cheatsheet	Description
One-Liners	Quick diagnostic commands by problem type
Sysctl Reference	Key kernel parameters
VDSO/Clock Troubleshooting	Quick detection and fixes for time-related performance

60-Second Analysis

From Brendan Gregg's Linux Performance Analysis in 60,000 Milliseconds:

uptime                           # load averages
dmesg | tail                     # kernel errors
vmstat 1                         # system-wide stats
mpstat -P ALL 1                  # CPU balance
pidstat 1                        # process CPU
iostat -xz 1                     # disk I/O
free -m                          # memory usage
sar -n DEV 1                     # network I/O
sar -n TCP,ETCP 1               # TCP stats
top                              # overview

Tool Categories

Classic -> Modern Replacements

Classic	Modern	Why
ls	eza	Git status, icons, tree view
cat	bat	Syntax highlighting
find	fd	5x faster, simpler syntax
grep	ripgrep	10x faster, .gitignore aware
du	dust	Visual bars
df	duf	Clean tables
top	btop	Dashboard UI
dig	dog/doggo	DoH/DoT, colors
curl	xh	Human-friendly HTTP
sed	sd	Sane regex
cd	zoxide	Frecency-based jump

Performance Stack

Application  ->  async-profiler (Java), py-spy (Python), rbspy (Ruby)
     |
Userspace    ->  perf, valgrind, heaptrack
     |
Syscalls     ->  strace, ltrace
     |
Kernel       ->  eBPF/BCC, bpftrace, ftrace
     |
Hardware     ->  perf stat, turbostat, intel_gpu_top

Network Stack

L7 (HTTP)    ->  httpie, xh, hey, wrk, k6, vegeta
     |
L4 (TCP)     ->  ss, netstat, tcpdump, termshark
     |
L3 (IP)      ->  mtr, traceroute, ping, gping
     |
L2 (Link)    ->  ethtool, ip link

Quick Install (Debian/Ubuntu)

# Modern CLI tools
sudo apt install ripgrep fd-find bat eza fzf btop git-delta zoxide duf gping

# Performance tools
sudo apt install linux-tools-common linux-tools-$(uname -r) bpfcc-tools bpftrace

# Network tools
sudo apt install mtr-tiny tcpdump nmap iperf3 netcat-openbsd

# Monitoring
sudo apt install sysstat htop iotop

Version Requirements

Component	Minimum	Recommended	Notes
Linux Kernel	5.15	6.6+	EEVDF scheduler, modern eBPF
bpftrace	0.16	0.20+	BTF support, newer features
bcc-tools	0.25	0.28+	Latest BPF features
perf	matches kernel	-	Install linux-tools-$(uname -r)
iproute2	6.0	6.7+	netkit, newer tc features

Key Kernel Features by Version

Version	Feature
5.15+	io_uring maturity, BTF by default
6.1+	MGLRU, improved memory management
6.4+	Per-VMA locks, reduced mmap contention
6.6+	EEVDF scheduler (replaces CFS)
6.7+	Netkit stable
6.9+	BPF Arena, new kfuncs
6.12+	sched_ext merged, PREEMPT_RT mainline

Curated Sources

Essential Reading

Source	Focus	Link
Brendan Gregg	Performance methodology, eBPF	brendangregg.com
Julia Evans	Debugging, Linux internals	jvns.ca
Netflix Tech Blog	Production performance	netflixtechblog.com
Cloudflare Blog	Network performance, eBPF	blog.cloudflare.com
Meta Engineering	eBPF at scale, kernel	engineering.fb.com
Dan Luu	Systems analysis, measurement	danluu.com

In This Repository

Curated extracts from these sources with actionable insights:

• Netflix Performance Playbook - ZGC, flame graphs, load shedding
• Cloudflare Network Performance - TCP tuning, XDP, DDoS
• Meta eBPF & Systems Engineering - Katran, Strobelight, sched_ext
• Dan Luu Systems Insights - Latency, measurement, caching
• Julia Evans Systems Debugging - strace, debugging methodology

Tools & References

• BCC Tools - eBPF-based Linux tools
• bpftrace - High-level tracing language
• perf-tools - Performance analysis tools
• FlameGraph - Stack trace visualizer
• sched_ext - BPF schedulers

Resources

• Brendan Gregg's Linux Performance
• Linux Tracing in 15 Minutes
• BCC Tools
• perf-tools