[Paper review] Design VPN Protocols Understood by WireGuard Paper

Pangyoalto

Mar 8, 2026 • 11 min read

WireGuard is a VPN protocol first introduced in 2015. While VPN protocols like OpenVPN and IPsec already existed, WireGuard leveraged modern cryptographic techniques to achieve a leaner codebase and higher performance. Since being merged into Linux 5.6 as a kernel module, it has seen growing adoption.

You may not be familiar with the name WireGuard, but you might already be using it without knowing. Services like Tailscale and Cloudflare WARP are built on top of WireGuard.

I first became interested in WireGuard when I learned that Tailscale, the networking platform we use at openclaw, is built on it. I wanted to expose my home server so I could use openclaw from anywhere, but at the same time, only I should be able to access it.

WireGuard not only provides an encrypted tunnel but also never responds to unauthenticated packets, effectively hiding the server's existence from external scanners.

In this post, based on the WireGuard whitepaper (NDSS 2017), I explain how WireGuard achieves secure communication in a way that is accessible even to those unfamiliar with VPNs and networking.

Prerequisite: VPN

Understanding WireGuard requires some background on VPNs. Let me briefly cover the basics.

A VPN (Virtual Private Network) is a technology that connects two physically separated devices as if they were on the same internal network. It routes traffic through the public internet, but wraps it in an encrypted tunnel so that intermediate nodes (cafe WiFi, ISPs, routers, etc.) cannot see the traffic.

Without VPN:
[Laptop] ──── public internet ────→ [Web Server]
                    │
            eavesdropping possible

With VPN:
[Laptop] ════ encrypted tunnel ════ [VPN Server] ──── internet ────→ [Web Server]
                │                       │
              secure                  plaintext
    (can't read even if intercepted) (trusted network - internal network, etc.)

The VPN server is the gateway to a network I trust.

Common use cases:

Eavesdrop prevention: Encrypt communication on untrusted networks like cafe WiFi
Remote access: Access internal company servers (10.0.0.x) from home. Since private IPs are not routable on the public internet, traffic is routed to internal IPs through the VPN tunnel

When you connect to a VPN, a virtual network interface (`tun0`) is created on your laptop and assigned an internal IP. The physical interface (`eth0`) remains intact, and the routing table is modified to send only internal subnet traffic through `tun0`.

After VPN connection on my laptop:

Interfaces:
  eth0:  192.168.1.100   ← Home router IP. Used for actual communication with VPN server
  tun0:  10.0.0.2        ← Virtual interface created by VPN. Internal IP

Routing table:
  10.0.0.0/24  → tun0    (internal subnet goes through tunnel)
  0.0.0.0/0    → eth0    (everything else as usual)

Packet flow (when connecting to 10.0.0.5):
  App → tun0 → VPN client encrypts → eth0 → VPN server decrypts → internal 10.0.0.5

Prerequisite: Cryptography Basics

Here are the cryptography concepts needed to understand WireGuard's handshake.

Symmetric Encryption and AEAD

Symmetric encryption uses a single key for both encryption and decryption. It is fast, but both parties must share the same key in advance.

Instead of plain symmetric encryption, WireGuard uses AEAD (Authenticated Encryption with Associated Data). It performs encryption and integrity verification in a single operation.

(key, nonce, plaintext, associated data) → (ciphertext, authentication tag)

During decryption, if the authentication tag does not match, the data has been tampered with or the key is wrong, so the entire message is rejected. Associated data is used for data that should not be encrypted but needs tamper protection (e.g., packet headers).

Diffie-Hellman Key Exchange

An algorithm that allows both parties to create a shared secret without directly transmitting any secrets.

Pre-agreed: prime p, generator g (both public)

Alice                                 Bob
Private key a (random integer)        Private key b (random integer)
Public key A = g^a mod p              Public key B = g^b mod p

──── Send A ──────────────────→
←──────────────────── Send B ────

Shared secret = B^a mod p             Shared secret = A^b mod p
              = g^(ba) mod p                        = g^(ab) mod p
              ↑ Same value!                         ↑ Same value!

Even if an eavesdropper sees g, p, A, and B, computing a or b from them is computationally infeasible.

The Problem WireGuard Aims to Solve

The first sentence of the whitepaper summarizes WireGuard's goal well:

WireGuard is a secure network tunnel, operating at layer 3, implemented as a kernel virtual network interface for Linux, which aims to replace both IPsec for most use cases, as well as popular user space and/or TLS-based solutions like OpenVPN, while being more secure, more performant, and easier to use

Existing VPNs have grown complex over decades of evolution. IPsec has dozens of related RFCs, and OpenVPN consists of over 100,000 lines of code. Larger codebases and complex configurations mean more room for vulnerabilities.

WireGuard's goal is a VPN as simple as SSH. Just as SSH became the de facto standard for remote access, the aim is to build a VPN that is easy to configure, small in codebase, yet strong in security.

Looking at the specific problems with existing VPNs:

IPsec: Relies on Linux's xfrm layer (for ciphersuite and transformation decisions) and the complex IKEv2 protocol (for key exchange). The separation of the transport encryption layer and key exchange layer makes configuration and management complex from the user's perspective.
OpenVPN: A TLS-based user space solution. Simpler configuration than IPsec. The core idea is building a VPN on top of a proven TLS library (OpenSSL) — using TLS handshake for key exchange and a separate channel for encryption. However, frequent packet copying between kernel and user space degrades performance, and dependence on the full TLS stack creates a wide attack surface.

WireGuard solves these problems as follows:

Virtual interface: Provides a virtual interface like `wg0` that can be controlled with standard tools like `ip` or `ifconfig`.

The paper draws an analogy to SSH — just as registering a public key in SSH allows only that user to connect, registering a peer's public key and allowed IPs in WireGuard allows only that peer to communicate within the specified IP range.

ip link add dev wg0 type wireguard
ip address add 10.0.0.1/24 dev wg0
wg set wg0 private-key ./privatekey \
    peer ABCDEF... allowed-ips 10.0.0.2/32 endpoint 203.0.113.5:51820
ip link set wg0 up

Fixed cryptographic primitives: By fixing the available protocols to Curve25519, ChaCha20-Poly1305, etc., vulnerabilities that could arise from cipher negotiation are eliminated at the root.
Key distribution agnostic: Like OpenSSH, it is agnostic to key distribution methods. Whether you use PGP-signed email or LDAP, it works on top of any infrastructure.
Layer 3 operation: Operates at Layer 3 within the kernel, maintaining high performance while being usable across diverse network environments.

CryptoKey Routing

CryptoKey Routing is WireGuard's core design. Each peer has a public key (Curve25519) and allowed IPs, and this mapping serves as the routing table.

The behavior differs depending on packet direction:

Outgoing: When a packet is sent to `wg0`, the peer mapped to the destination IP is found, and the packet is encrypted with that peer's secure session.
Incoming: After decrypting an encrypted packet, the inner packet's source IP is checked against the peer's allowed IPs. If it doesn't match, the packet is dropped.

Packet Encapsulation

WireGuard is a Layer 3 tunnel. It encapsulates the inner packet (the original packet with an internal source IP) as the payload of a UDP packet. The outer UDP header contains the actual public IP (Internet endpoint), while the inner packet contains the internal IP valid only within `wg0`.

Roaming

WireGuard maintains each peer's Internet endpoint in the routing table. When a packet is received, the peer's current location is determined from the outer IP, and the peer's identity from the inner source IP.

In the previous diagram, when host `gN65...z6EA` sends a packet to `HIgo...f8yk` at `192.95.5.69:41414`, `HIgo...f8yk` decrypts it, confirms the peer identity via the inner source IP, and updates `gN65...z6EA`'s Internet endpoint.

Thanks to this, even when the Internet endpoint changes — such as switching from WiFi to a mobile network — the identity is determined by the internal IP, so the connection is seamlessly maintained. Once the receiving side learns the new endpoint, bidirectional communication resumes.

example

wg0 is registered with 10.192.122.3/24, and `ip route add 10.0.0.0/8 dev wg0` directs the 10.0.0.0/8 IP range through wg0 as well.

When a user sends a packet to 10.10.10.230, the packet goes through the wg0 interface, is encrypted with the secure session for public key gN65...z6EA, and attached to a UDP packet sent to 192.95.5.70:54421.

Protocol & Cryptography

Before sending encrypted encapsulated packets, a 1-RTT handshake must be performed to exchange keys. Once the handshake is complete, both sides encrypt data using the shared symmetric keys.

The security properties that WireGuard's protocol aims to achieve are:

Authenticated Key Exchange (AKE): Both peers verify each other's identity while simultaneously generating a shared secret. WireGuard achieves this with a 1-RTT handshake using the Noise "IK" pattern.
Perfect Forward Secrecy (PFS): Even if a peer's long-term static private key is compromised, past traffic cannot be decrypted. This is achieved by generating a new ephemeral key pair for each handshake.
Identity Hiding: A passive observer cannot learn a peer's static public key. This is achieved by encrypting the static key in the first handshake message.
Replay Protection: Even if an attacker captures and retransmits handshake packets, they cannot hijack the session. Messages include an encrypted timestamp, and the responder drops any message with a timestamp equal to or lower than the highest it has received.

Additionally, WireGuard never responds to unauthenticated packets, preventing it from revealing its existence to attackers. This was difficult for other VPN solutions because of their handshake structure. Protocols requiring multiple RTTs must send a response before authentication is complete, which by itself reveals the server's existence.

WireGuard solves this by performing authentication with just the first packet, but this approach introduces a tradeoff of being vulnerable to replay attacks.

WireGuard addresses this tradeoff with (1) replay attack prevention and (2) DoS mitigation mechanisms. I will explain these two first, then describe the core handshake process.

Replay Attack Prevention

The initiator includes an encrypted TAI64N timestamp in the first message. The responder records the highest timestamp received from each peer and drops any message with a timestamp equal to or lower than that.

Even if the responder restarts and loses its timestamp state, this is not a major problem. The purpose of a replay attack is to corrupt the state of an active secure session, but right after a restart there are no active sessions. When the legitimate initiator subsequently attempts a handshake with a more recent timestamp, a valid session is established, and the attacker's replayed messages are invalidated due to their outdated timestamps.

DoS Mitigation

WireGuard's handshake involves Curve25519 point multiplication, which is CPU intensive. If an attacker floods handshake requests, it can exhaust the responder's CPU.

WireGuard mitigates this with a cookie mechanism. When the responder's CPU load is high, instead of proceeding with the handshake, it sends an encrypted cookie to the initiator. The initiator then resends the handshake message with the cookie's MAC included. The responder can verify the MAC without performing expensive DH operations, significantly reducing CPU burden. Since the MAC is bound to the initiator's source IP and port, it also facilitates rate limiting.

What distinguishes WireGuard's cookie mechanism from existing approaches:

Silent until authenticated: Conventional mechanisms respond with a cookie to anyone, but WireGuard only responds with a cookie if the initiator includes a MAC computed with the responder's public key. This keeps the responder hidden from scanners.
Cookie encryption: The cookie is encrypted with the responder's public key before transmission, protecting against man-in-the-middle attacks.
Stronger binding: Additional information is included in the cryptographic process that binds the cookie to the message, preventing attackers from using forged cookies to disrupt legitimate connections.

Noise

WireGuard's handshake is not designed from scratch but built on top of the Noise Protocol Framework (Trevor Perrin, 2016).

To understand Noise, you first need to distinguish between static keys and ephemeral keys. A static key is a long-term key pair generated once during WireGuard installation, stored in the configuration file, with its public key distributed to peers in advance. It serves as the peer's identity. An ephemeral key, on the other hand, is a disposable key pair generated fresh for each handshake and deleted immediately after the session ends. It is the foundation of Forward Secrecy.

WireGuard uses the Noise IK pattern.

K (Known): The responder's static public key is already known. Since WireGuard registers the peer's public key during configuration (`wg set wg0 peer ABCDEF...`), it can perform DH with the responder's public key from the very first message to open an encrypted channel.
I (Immediate): The initiator's static public key is transmitted immediately in the first message. It is AEAD encrypted with the responder's public key, so an eavesdropper cannot determine who the initiator is — only the responder's private key can decrypt it.

Let's see how this IK pattern manifests in the actual handshake messages.

Handshake

The left shows a normal handshake scenario; the right shows a scenario where the responder is under heavy load and sends a cookie reply.

mac1, mac2: The MAC fields described in the DoS mitigation section. If the initiator sends a valid MAC in the mac1 field, the responder responds with a cookie.
timestamp: The field described in the replay attack prevention section.
Ii: A randomly generated value. The session initiated by this message uses this value.

The remaining fields are generated as follows.

In summary:

The initiator sends a random value Ii along with an ephemeral public key, an encrypted static public key, and an encrypted timestamp.

The responder verifies the remaining fields of the message using these values.

The responder generates a random value Ir and similarly sends an ephemeral public key and encrypted payload.

The empty field in the diagram (0̂ bytes) is 0 + 16 bytes. The 16 bytes are the Poly1305 authentication tag.

By sending the authentication tag generated by AEAD, the initiator proves that it has computed the correct session keys.

The initiator can now use the session keys (Transport Data Keys) as symmetric keys.

The detailed field computation formulas are omitted here (they can be found in the paper).

Once the handshake is complete, transport keys can be derived.The initiator and responder now send encapsulated packets encrypted with these keys (the "packet" in the diagram).

Each transport packet includes a counter to prevent replay attacks.

Conclusion

When I first looked into WireGuard, I didn't expect it to have enough depth to write about. But contrary to my expectations, WireGuard's design philosophy became more impressive the deeper I dug.
The most striking aspect is the silence property — never responding to scanners. To achieve this, authentication must be completed within 1-RTT, and that requirement naturally leads to the choice of technologies like the Noise IK pattern, DH, and AEAD. It was fascinating to see how a single design goal determines the structure of the entire protocol.

CryptoKey Routing, which separates the source IP used on the internal network from the Internet endpoint exposed on the public internet, is also impressive. Thanks to this separation, roaming — maintaining a connection even when switching from WiFi to a mobile network — works seamlessly without any additional mechanism.

Existing VPNs grew complex because layers were added as features accumulated. WireGuard took the opposite approach: fixing only modern cryptographic primitives and stripping away everything else (the codebase is only about 4,000 lines).

For those who, like me, need to expose a home server externally, the silence property and simple configuration are a huge help.

The paper also covers automatic key rotation, Linux kernel implementation, and performance benchmarks, but these were omitted from this post for brevity. If you're interested, I recommend reading the original paper.