TCP Three-Way Handshake Attacks: How Hackers Exploit the Foundation of Internet Communication

Saif
May 11
3 min read

Every time you browse the web or send an email, the TCP three-way handshake quietly ensures your connection is reliable. It’s a brilliant piece of engineering, but its predictable mechanics are a goldmine for attackers. As offensive security enthusiasts, we’ve spent hours in the lab poking at this protocol, uncovering ways to exploit it and, more importantly, how to lock it down. In this post, we’ll walk you through advanced attacks on the handshake, share real-world lab results, and offer battle-tested defenses—all while staying firmly within ethical boundaries.

What’s the TCP Three-Way Handshake?

At its core, the handshake is a three-step dance that sets up a TCP connection:

The client fires off a SYN packet, signaling, “Hey, let’s talk!”
The server responds with a SYN-ACK, saying, “Got it, I’m ready—here’s my sequence number.”
The client seals the deal with an ACK, and the connection is live.

It’s elegant but not bulletproof. The handshake’s reliance on sequence numbers and resource allocation opens doors for clever attacks, which we’ve tested extensively in controlled environments.

TCP Three-Way Handshake Attacks: Breaking the Protocol

Red teaming the handshake is like finding cracks in a fortress. TCP three-way handshake attacks, such as SYN floods, session hijacking, and split-handshakes, exploit the protocol’s predictable structure to disrupt or manipulate connections. Here are the key attacks we explored, all conducted ethically in a virtual lab.

SYN Flood: Overwhelming the Server

A SYN flood is like jamming a server’s inbox with fake connection requests. By spamming SYN packets, attackers fill the server’s backlog queue with half-open connections, starving it of resources.

Tool in Action: hping3

hping3 -S -p 80 --flood 192.168.1.100

Custom Script: We whipped up a Python script to simulate this chaos:

import socket
import threading

def syn_flood(target_ip, target_port):
    try:
        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        s.connect((target_ip, target_port))
    except:
        pass

target_ip = '192.168.1.100'
target_port = 80
for _ in range(1000):
    threading.Thread(target=syn_flood, args=(target_ip, target_port)).start()

What Happened: In our lab, the target server’s CPU spiked to 90%, and memory usage went through the roof. Legitimate connections either crawled or dropped entirely.

TCP Session Hijacking: Stealing the Conversation

Imagine slipping into a phone call and taking over the conversation. That’s session hijacking, one of the more insidious TCP handshake vulnerabilities. By sniffing or guessing TCP sequence numbers, attackers can inject malicious packets into an active session.

Our Approach: Using Wireshark, we captured a handshake and noted the sequence numbers. With Scapy, we crafted a spoofed packet, injecting data into the session. It worked like a charm in our controlled setup, highlighting how critical sequence number security is.

RST Injection: Cutting the Cord

RST injection is a blunt but effective attack. By forging a TCP packet with the RST (reset) flag, attackers can abruptly kill a connection.

Scapy Example:

from scapy.all import *
ip = IP(src="192.168.1.2", dst="192.168.1.100")
tcp = TCP(sport=12345, dport=80, flags="R", seq=1001)
pkt = ip/tcp
send(pkt)

Results: The connection dropped instantly, as if someone yanked the plug.

Split-Handshake: Sneaking Past Defenses

Some devices choke on non-standard handshakes, like the split-handshake, where the server sends an ACK, then a SYN, instead of a SYN-ACK. This can trick firewalls and IDS/IPS into letting unauthorized connections through.

Lab Story: We set up two VMs—one running Kali Linux, the other an Ubuntu server—behind a pfSense firewall. Using Scapy, we crafted a split-handshake. The firewall didn’t bat an eye, and our attacker VM chatted with the server undetected. Wireshark logs showed the handshake’s odd flow, confirming the bypass.

This attack’s success against certain firewalls underscores why red teams must test beyond standard scenarios.

Defense Strategies

Exploiting the handshake is fun (in a lab, of course), but defending against these attacks is where the real challenge lies. Here’s what we recommend based on our testing:

Cap SYN Packets: Firewalls should limit incoming SYN rates to prevent floods.
Use SYN Cookies: Enable SYN cookies to manage half-open connections without eating up resources.
Randomize Sequence Numbers: Modern TCP stacks should generate unpredictable sequence numbers to foil hijacking.
Tune IDS/IPS: Keep detection signatures sharp for handshake quirks, like split-handshakes.
Test Regularly: Run handshake-based attack simulations in your red team exercises.

Tips for Red Teams

Our lab work taught us a few tricks for offensive security pros:

Write custom scripts to manipulate handshakes during pen tests.
Probe network devices for how they handle handshake oddities.
Share detailed findings with blue teams to tighten defenses.
Keep learning—TCP/IP vulnerabilities evolve fast.

Wrapping Up: Securing TCP

The TCP three-way handshake is a cornerstone of internet communication, but it’s not invincible. Our offensive tests show it’s a target for savvy attackers. By understanding these attacks and deploying layered defenses, organizations can stay one step ahead. Keep testing, keep learning, and keep your networks secure.

THE DAILY PULSE