VXLAN: Extending Layer 2 Across Layer 3

In the previous chapter, we learned how GRE tunnels can transport packets between two endpoints across a routed network.

GRE is useful, simple, and easy to understand. However, it has a major limitation:

GRE creates point-to-point tunnels.

As infrastructure grows, this model becomes difficult to manage.

Imagine a virtualized environment with:

Hundreds of servers
Thousands of virtual machines
Multiple racks
Multiple data centers

Applications often expect machines to be on the same Layer 2 network, even when they are physically separated.

Building a giant Layer 2 network across an entire data center is neither practical nor scalable.

VXLAN was created to solve this problem.

In this chapter, you will learn:

Why VXLAN exists
How VXLAN works
VXLAN encapsulation
VTEPs (VXLAN Tunnel Endpoints)
VXLAN Network Identifiers (VNIs)
How Linux implements VXLAN
How to build a VXLAN network in Linux
Why VXLAN became the foundation of modern cloud networking

The Problem VXLAN Solves

Consider two servers:

+---------+                  +---------+
| ServerA |                  | ServerB |
+---------+                  +---------+
     |                            |
     |                            |
  Rack 1                       Rack 2

Each server hosts virtual machines or containers.

A virtual machine on ServerA:

10.1.1.10

needs to communicate with a virtual machine on ServerB:

10.1.1.20

as if they are connected to the same Ethernet switch.

However, the physical network between servers is routed:

ServerA ---- Router ---- Router ---- ServerB

Routers do not forward Ethernet frames.

They forward IP packets.

The challenge is:

How can we transport Layer 2 traffic across a Layer 3 network?

VXLAN provides the answer.

What Is VXLAN?

VXLAN stands for:

Virtual eXtensible Local Area Network

It is defined in RFC 7348.

VXLAN allows Ethernet frames to be encapsulated inside UDP packets.

Instead of sending an Ethernet frame directly:

Ethernet Frame
      |
      v
 Network

VXLAN wraps the frame:

+----------------------+
| Outer Ethernet       |
+----------------------+
| Outer IP Header      |
+----------------------+
| UDP Header           |
+----------------------+
| VXLAN Header         |
+----------------------+
| Original Frame       |
+----------------------+

The physical network only sees:

Ethernet
IP
UDP

The original Layer 2 frame remains untouched inside the tunnel.

VXLAN Creates an Overlay Network

The physical network is called the:

Underlay Network

The virtual network built on top of it is called the:

Overlay Network

Example:

Overlay Network
--------------------
 VM A      VM B
   |         |
   +---------+
        |
      VXLAN

Underlay Network
--------------------
 ServerA --- Router --- ServerB

Applications only see the overlay network.

The physical infrastructure only handles the underlay.

This separation is one of the reasons cloud networking scales so well.

Understanding VTEPs

A VXLAN Tunnel Endpoint (VTEP) is responsible for:

Encapsulating outgoing frames
Decapsulating incoming frames

Every server participating in VXLAN acts as a VTEP.

Example:

          VXLAN Tunnel

+------+                  +------+
|VTEP A|==================|VTEP B|
+------+                  +------+
    |                         |
   VM1                       VM2

When VM1 sends an Ethernet frame:

VTEP A receives the frame.
VTEP A encapsulates it in VXLAN.
The packet crosses the routed network.
VTEP B removes the VXLAN header.
VM2 receives the original frame.

The virtual machines never see the encapsulation process.

VXLAN Network Identifiers (VNIs)

Traditional VLANs support:

4096 VLANs

This becomes a limitation in large environments.

VXLAN introduces:

VNI (VXLAN Network Identifier)

A VNI is:

24 bits

allowing:

16,777,216

separate virtual networks.

Example:

VNI 100
VNI 200
VNI 300

Each VNI represents an isolated Layer 2 segment.

Think of a VNI as a VLAN that scales to millions of networks.

VXLAN Packet Structure

A VXLAN packet looks like:

+-----------------------+
| Outer Ethernet Header |
+-----------------------+
| Outer IP Header       |
+-----------------------+
| UDP Header            |
+-----------------------+
| VXLAN Header          |
+-----------------------+
| Ethernet Frame        |
+-----------------------+

Notice something important:

The payload is an entire Ethernet frame.

This means VXLAN transports:

ARP
IPv4
IPv6
Broadcasts
Multicasts
Unknown unicasts

Anything that can exist inside an Ethernet frame.

Why UDP?

VXLAN uses:

UDP 4789

by default.

This design provides several advantages:

Easy routing through IP networks
ECMP compatibility
Hardware offload support
Better scalability than protocol-specific tunneling

From the perspective of the physical network, VXLAN traffic is simply UDP traffic.

Linux VXLAN Implementation

Linux includes native VXLAN support in the kernel.

A VXLAN device behaves similarly to:

A bridge port
A virtual interface
A tunnel endpoint

You can view VXLAN support:

ip link help vxlan

or:

man ip-link

No additional software is required.

Lab Topology

We will create two Linux hosts:

Host A                     Host B

192.168.100.1         192.168.100.2
      |                     |
      +---------+-----------+
                |
         Physical Network

VXLAN VTEPs:

Host A vxlan100
Host B vxlan100

VNI:

Overlay network:

10.10.10.0/24

Creating the VXLAN Interface

On Host A:

sudo ip link add vxlan100 \
    type vxlan \
    id 100 \
    local 192.168.100.1 \
    remote 192.168.100.2 \
    dstport 4789

Bring it up:

sudo ip link set vxlan100 up

Assign an address:

sudo ip addr add 10.10.10.1/24 dev vxlan100

Configure the Other Side

On Host B:

sudo ip link add vxlan100 \
    type vxlan \
    id 100 \
    local 192.168.100.2 \
    remote 192.168.100.1 \
    dstport 4789

Bring it up:

sudo ip link set vxlan100 up

Assign an address:

sudo ip addr add 10.10.10.2/24 dev vxlan100

Testing Connectivity

Ping the remote VXLAN endpoint:

ping 10.10.10.2

Expected:

64 bytes from 10.10.10.2

Verify routes:

ip route

You should see:

10.10.10.0/24 dev vxlan100

Linux now treats the VXLAN tunnel as a normal network interface.

Capturing VXLAN Traffic

Start packet capture:

sudo tcpdump -ni any udp port 4789

Generate traffic:

ping 10.10.10.2

Example output:

IP 192.168.100.1.4789 >
   192.168.100.2.4789

The physical network sees only UDP traffic.

The original Ethernet frame is hidden inside the VXLAN packet.

Viewing Encapsulation

Capture with verbose output:

sudo tcpdump -ni any -vv udp port 4789

You will observe:

Outer Ethernet
Outer IP
UDP
VXLAN
Inner Ethernet
Inner IP
ICMP

This is the complete overlay encapsulation process.

VXLAN and Bridges

In practice, VXLAN interfaces are usually attached to bridges.

Example:

VM1
 |
tap0
 |
br0
 |
vxlan100
 |
Network

The bridge performs switching.

The VXLAN interface transports frames between hosts.

This architecture is used extensively by:

OpenStack
Kubernetes networking solutions
VMware NSX
Open vSwitch
Public cloud platforms

Hands-On: Bridge a VXLAN Interface

Create a bridge:

sudo ip link add br0 type bridge
sudo ip link set br0 up

Attach VXLAN:

sudo ip link set vxlan100 master br0

Verify:

bridge link show

Example:

vxlan100 master br0

Linux now treats remote VXLAN endpoints as part of the same Layer 2 domain.

Unknown Traffic and Flooding

Consider an ARP request:

Who has 10.10.10.2?

The sender does not know the destination MAC address.

VXLAN must flood the frame.

In small deployments this can be configured manually.

In large deployments this becomes inefficient.

This challenge led to technologies such as:

EVPN
BGP control planes
SDN controllers

which provide scalable MAC learning and distribution.

MTU Considerations

VXLAN adds additional headers:

Header	Size
Outer Ethernet	14 bytes
Outer IP	20 bytes
UDP	8 bytes
VXLAN	8 bytes

Typical overhead:

50 bytes

A 1500-byte packet may become:

1550 bytes

This can cause:

Fragmentation
Packet loss
Performance degradation

Inspect MTU:

ip link show vxlan100

A common adjustment:

sudo ip link set vxlan100 mtu 1450

The exact value depends on the underlay network.

Static VXLAN vs Dynamic VXLAN

The examples in this chapter use:

remote <ip>

which creates a static tunnel.

This is useful for learning but does not scale.

Modern deployments typically use:

Multicast learning
EVPN
BGP
SDN controllers

to automatically discover remote VTEPs.

Large cloud environments may contain thousands of VTEPs.

Manual configuration would be impossible.

GRE vs VXLAN

Feature	GRE	VXLAN
Encapsulation	IP	UDP
Layer 2 Extension	Limited	Native
Multi-Tenant Support	No	Yes
VNI Segmentation	No	Yes
Cloud Usage	Rare	Very Common
Scalability	Low	High

GRE is an excellent educational tunnel.

VXLAN is the foundation of modern virtual networking.

Real-World Usage

Many platforms use VXLAN internally:

Kubernetes networking solutions such as Flannel and Calico
OpenStack Neutron
VMware NSX
Public cloud networking systems
Multi-tenant data centers

Even when administrators never create a VXLAN interface manually, there is a good chance that VXLAN is carrying traffic somewhere in the infrastructure.

Key Takeaways

VXLAN extends Layer 2 networks across Layer 3 infrastructure.
VXLAN encapsulates Ethernet frames inside UDP packets.
VTEPs perform encapsulation and decapsulation.
VXLAN uses VNIs to create millions of isolated virtual networks.
Linux provides native VXLAN support through the kernel.
VXLAN interfaces can be attached to Linux bridges.
Packet captures reveal VXLAN as UDP traffic, usually on port 4789.
VXLAN introduces additional packet overhead and MTU considerations.
VXLAN scales far better than GRE and has become a standard building block of modern cloud networking.

Networking for Non-Network Engineers