Building a Multi-Host Virtual Network

Throughout this book, we have explored the building blocks of Linux networking:

Network namespaces
Virtual Ethernet devices
Linux bridges
Routing
GRE tunnels
VXLAN overlays
Policy routing
VRFs

Individually, these technologies are useful. However, modern platforms rarely use them in isolation.

Kubernetes, OpenStack, VMware, cloud providers, and SDN platforms combine these primitives to create virtual networks that span multiple physical hosts while appearing as a single network to applications.

In this chapter, we will bring everything together and build a simple multi-host virtual network using Linux networking primitives.

The goal is not to build a production-ready solution.

The goal is to understand the architecture that many modern networking systems are built upon.

By the end of this chapter, you will understand how workloads running on different servers can communicate as if they are connected to the same Ethernet switch.

The Problem

Suppose we have two Linux servers:

+---------+                     +---------+
| Host A  |                     | Host B  |
+---------+                     +---------+
     |                               |
     |                               |
     +----------- Network -----------+

Each host runs workloads.

These workloads might be:

Virtual machines
Containers
Kubernetes Pods
Application namespaces

Example:

Host A: App A
Host B: App B

We want:

App A <-> App B

to communicate directly on the same virtual Layer 2 network.

However, the physical network between hosts is routed.

The workloads cannot simply share a Linux bridge.

We need an overlay network.

Final Topology

By the end of this chapter, we will build:

                 Underlay Network

        192.168.100.1      192.168.100.2
             Host A             Host B
                 |                 |
                 +--------+--------+
                          |
                    Physical Network

------------------------------------------------

                 Overlay Network

       ns-a                         ns-b
     10.10.10.1                  10.10.10.2
          |                           |
        veth                        veth
          |                           |
        br0                         br0
          |                           |
      vxlan100 ================= vxlan100

From the perspective of the namespaces:

10.10.10.1 and 10.10.10.2
appear to be on the same Ethernet network

even though they are running on different hosts.

What We Will Build

Each host contains:

Namespace
    |
   veth
    |
 Linux Bridge
    |
 VXLAN Interface

The bridge performs Layer 2 switching.

The VXLAN interface transports Ethernet frames between hosts.

This architecture closely resembles the design used by:

OpenStack
Flannel VXLAN
VMware NSX
Many cloud networking systems

Step 1: Create a Namespace

On Host A:

sudo ip netns add ns-a

On Host B:

sudo ip netns add ns-b

Verify:

ip netns

Example:

ns-a

or:

ns-b

Step 2: Create a veth Pair

Host A:

sudo ip link add veth-a type veth peer name eth0 netns ns-a

Host B:

sudo ip link add veth-b type veth peer name eth0 netns ns-b

This creates a virtual cable between:

Host Network Namespace

and

Application Namespace

Step 3: Create a Linux Bridge

Host A:

sudo ip link add br0 type bridge
sudo ip link set br0 up

Host B:

sudo ip link add br0 type bridge
sudo ip link set br0 up

Verify:

bridge link show

The bridge currently has no ports.

Step 4: Attach veth Interfaces

Host A:

sudo ip link set veth-a master br0
sudo ip link set veth-a up

Host B:

sudo ip link set veth-b master br0
sudo ip link set veth-b up

The topology now looks like:

Namespace
    |
   veth
    |
   br0

on each host.

Step 5: Configure Namespace Interfaces

Host A:

sudo ip netns exec ns-a \
    ip addr add 10.10.10.1/24 dev eth0

sudo ip netns exec ns-a \
    ip link set lo up

sudo ip netns exec ns-a \
    ip link set eth0 up

Host B:

sudo ip netns exec ns-b \
    ip addr add 10.10.10.2/24 dev eth0

sudo ip netns exec ns-b \
    ip link set lo up

sudo ip netns exec ns-b \
    ip link set eth0 up

At this point the namespaces are configured but cannot yet communicate.

The bridges are isolated from one another.

Step 6: Create the VXLAN Overlay

Host A:

sudo ip link add vxlan100 \
    type vxlan \
    id 100 \
    local 192.168.100.1 \
    remote 192.168.100.2 \
    dstport 4789

Host B:

sudo ip link add vxlan100 \
    type vxlan \
    id 100 \
    local 192.168.100.2 \
    remote 192.168.100.1 \
    dstport 4789

Bring the interfaces up:

sudo ip link set vxlan100 up

on both hosts.

Step 7: Connect VXLAN to the Bridge

Host A:

sudo ip link set vxlan100 master br0

Host B:

sudo ip link set vxlan100 master br0

Now each bridge has:

Bridge
├── Local veth
└── VXLAN tunnel

The bridge can switch traffic between local workloads and remote hosts.

Verifying the Topology

Inspect bridge ports:

bridge link show

Expected output:

veth-a master br0
vxlan100 master br0

or:

veth-b master br0
vxlan100 master br0

depending on the host.

Testing Layer 2 Connectivity

From Host A:

sudo ip netns exec ns-a ping 10.10.10.2

Expected:

64 bytes from 10.10.10.2

The packet path is:

ns-a
  |
veth-a
  |
br0
  |
vxlan100
  |
Physical Network
  |
vxlan100
  |
br0
  |
veth-b
  |
ns-b

This is effectively a virtual Ethernet switch spanning multiple hosts.

Observing ARP

Start a capture:

sudo tcpdump -ni any arp

Run:

sudo ip netns exec ns-a ping 10.10.10.2

Observe:

ARP Request
Who has 10.10.10.2?

The ARP frame crosses the VXLAN tunnel.

This demonstrates that VXLAN transports complete Ethernet frames, not just IP packets.

Observing VXLAN Traffic

Capture VXLAN traffic:

sudo tcpdump -ni any udp port 4789

Generate traffic:

sudo ip netns exec ns-a ping 10.10.10.2

Example:

192.168.100.1.4789 >
192.168.100.2.4789

Notice:

The underlay sees UDP packets.
The overlay sees Ethernet frames.

This is the essence of overlay networking.

Inspecting the Bridge Forwarding Database

Linux bridges maintain a MAC address table.

View it:

bridge fdb show

Example:

52:54:00:aa:bb:cc dev veth-a
52:54:00:dd:ee:ff dev vxlan100

The bridge learns:

MAC Address -> Port

just like a physical switch.

Unknown Traffic and Flooding

When the bridge does not know a destination MAC:

Unknown Unicast

it floods the frame.

The frame is sent:

veth
vxlan

and potentially every VXLAN peer.

This is acceptable in small environments but becomes inefficient at scale.

Large deployments solve this using:

EVPN
BGP
SDN controllers
Open vSwitch control planes

Scaling Beyond Two Hosts

Our lab contains:

Host A
Host B

Real environments may contain:

100 Hosts
1000 Hosts

The architecture remains similar:

Namespace / VM
        |
      Bridge
        |
      VXLAN
        |
   Routed Network

The control plane becomes more sophisticated, but the underlying concepts remain the same.

How Kubernetes Uses Similar Concepts

Many Kubernetes networking solutions use a similar architecture.

A Pod:

Pod
 |
veth
 |
Host Namespace
 |
Overlay Network

Depending on the CNI implementation:

VXLAN
Geneve
IP-in-IP
Native routing

may be used to transport traffic between nodes.

Although the implementation details differ, the networking primitives are largely the same ones we have used throughout this book.

Troubleshooting Checklist

When a multi-host virtual network is not working, verify:

VXLAN Interface

ip link show vxlan100

Bridge Configuration

bridge link show

Forwarding Database

bridge fdb show

Underlay Connectivity

ping <remote-host-ip>

VXLAN Traffic

tcpdump -ni any udp port 4789

Namespace Configuration

ip netns exec ns-a ip addr

ARP Resolution

ip netns exec ns-a ip neigh

These commands solve most overlay-network troubleshooting problems.

What We Built

Let's review what we created:

Application Namespace
        |
       veth
        |
     Bridge
        |
      VXLAN
        |
   Physical Network
        |
      VXLAN
        |
     Bridge
        |
       veth
        |
Application Namespace

This architecture combines nearly every major networking concept introduced throughout this book.

You have effectively built a simplified version of the networking layer used by many cloud and virtualization platforms.

Key Takeaways

Multi-host virtual networks are built by combining Linux networking primitives.
Network namespaces provide workload isolation.
veth pairs connect workloads to the host network.
Linux bridges provide Layer 2 switching.
VXLAN transports Ethernet frames across routed networks.
The physical network acts as the underlay network.
The virtual network acts as the overlay network.
ARP, broadcasts, and Ethernet frames can traverse VXLAN tunnels.
Linux bridges learn MAC addresses just like physical switches.
Modern cloud networking systems are built on the same fundamental concepts demonstrated in this chapter.

Networking for Non-Network Engineers