Why Traditional Networks Don't Scale
In the previous chapters, we built networks using Linux bridges, virtual Ethernet devices, routing, and packet filtering. Everything worked well because all systems existed within a relatively small environment.
However, modern infrastructure is very different.
A Kubernetes cluster may contain hundreds of nodes. A cloud provider may operate tens of thousands of servers. Even a medium-sized enterprise can have hundreds of virtual machines spread across multiple racks, buildings, or data centers.
The networking techniques that work perfectly inside a small network begin to show limitations as the network grows.
This chapter explains:
- Why traditional Layer 2 networks become problematic at scale
- Why large networks rely heavily on Layer 3 routing
- Why overlays and tunnels exist
- The challenges that technologies like VXLAN are designed to solve
Understanding these problems is more important than understanding the solutions. Once you understand the limitations of traditional networking, technologies like GRE, VXLAN, EVPN, and SDN become much easier to understand.
The Traditional Ethernet Network
A typical Ethernet network looks like this:
+---------+
| Switch |
+---------+
| | |
| | |
Host Host Host
Every device belongs to the same Layer 2 domain.
For example:
| Host | IP Address |
|---|---|
| Host A | 10.10.10.11 |
| Host B | 10.10.10.12 |
| Host C | 10.10.10.13 |
All devices:
- Share the same Ethernet segment
- Use ARP to discover MAC addresses
- Can communicate directly without routing
This model is simple and efficient.
For small networks, it works extremely well.
Understanding the Layer 2 Broadcast Domain
One of the most important concepts in networking is the broadcast domain.
A broadcast domain is the set of devices that receive Layer 2 broadcast traffic.
For example:
ARP Request:
Who has 10.10.10.12?
Tell 10.10.10.11
This packet is sent to:
ff:ff:ff:ff:ff:ff
which means:
Deliver this frame to everybody.
The switch floods the frame to all ports.
Broadcast
|
v
+------------------+
| Switch |
+------------------+
| | |
v v v
Host Host Host
A B C
Every host receives the frame.
Only Host B responds.
As the network grows, the amount of broadcast traffic grows as well.
Demonstrating Broadcast Traffic
You can observe this behavior directly.
On a Linux machine:
sudo tcpdump -i eth0 arp
Open another terminal:
ping 192.168.1.100
If the MAC address is unknown, you'll see something similar:
ARP, Request who-has 192.168.1.100 tell 192.168.1.10
This request is visible to every device in the Layer 2 network.
Even systems that are completely unrelated to the communication must receive and process the frame.
The Broadcast Problem
Imagine a network with:
- 10 hosts
- 100 hosts
- 1,000 hosts
- 10,000 hosts
ARP traffic grows.
Neighbor table updates grow.
Unknown unicast flooding grows.
Control traffic grows.
The network spends increasing amounts of time distributing packets that are not actual application traffic.
This is one reason large Layer 2 domains become inefficient.
MAC Address Tables Also Grow
Ethernet switches maintain MAC address tables.
A switch learns:
MAC Address -> Port
Example:
00:11:22:33:44:55 -> Port 1
aa:bb:cc:dd:ee:ff -> Port 2
When a frame arrives, the switch looks up the destination MAC address and forwards it.
As networks become larger:
- More devices exist
- More MAC addresses exist
- More table entries must be maintained
Large switching domains require larger forwarding tables and more control-plane resources.
Layer 2 Loops Become Dangerous
Consider two switches connected incorrectly:
+---------+
|Switch A |
+---------+
| |
| |
+---------+
|Switch B |
+---------+
A loop exists.
Ethernet frames have no built-in mechanism that prevents indefinite forwarding.
A broadcast frame may circulate forever.
The result can be:
- Broadcast storms
- MAC table instability
- Network outages
This is why protocols such as STP (Spanning Tree Protocol) exist.
STP intentionally disables some links to prevent loops.
Hands-On: Observe a Bridge Learning MAC Addresses
Create a bridge:
sudo ip link add br0 type bridge
sudo ip link set br0 up
Create a veth pair:
sudo ip link add veth1 type veth peer name veth2
Attach one side:
sudo ip link set veth1 master br0
sudo ip link set veth1 up
sudo ip link set veth2 up
Inspect the bridge forwarding database:
bridge fdb show
Example output:
52:54:00:ab:cd:ef dev veth1 master br0
This is effectively the software equivalent of a switch MAC table.
As more devices join, more entries appear.
Why Routing Scales Better
Large networks are usually built around Layer 3 routing rather than huge Layer 2 domains.
Consider:
Subnet A
10.1.0.0/24
Subnet B
10.2.0.0/24
Subnet C
10.3.0.0/24
Host A
|
Router
/ | \
B C D
Each subnet becomes its own broadcast domain.
An ARP request in one subnet stays inside that subnet.
It is not flooded across the entire network.
This significantly reduces unnecessary traffic.
Broadcast Containment
Suppose Host A sends:
Who has 10.1.0.20?
Only systems inside:
10.1.0.0/24
receive the request.
Hosts in:
10.2.0.0/24
10.3.0.0/24
never see it.
This isolation is one of the key reasons routed networks scale much better than flat Layer 2 networks.
The Data Center Problem
Now consider a modern data center.
Rack 1 Rack 2 Rack 3 Rack 4
Each rack contains:
- Physical servers
- Virtual machines
- Containers
Applications may require systems in different racks to appear as if they belong to the same Layer 2 network.
Examples:
- VM migration
- Legacy applications
- Virtualized environments
- Multi-tenant cloud platforms
Unfortunately, extending a giant Layer 2 network across an entire data center recreates all the scaling problems we just discussed.
This creates a dilemma:
We want Layer 2 connectivity, but we need Layer 3 scalability.
Why Overlays Exist
Overlay technologies solve this problem.
The basic idea is:
Application Network
|
v
Overlay Network
|
v
Physical Routed Network
Instead of extending Ethernet directly, we encapsulate traffic and transport it across a routed network.
The physical network only needs IP routing.
The virtual Layer 2 network exists on top of it.
This provides:
- Better scalability
- Better isolation
- Easier multi-tenancy
- Simpler physical network design
A Simple Mental Model
Imagine two buildings connected through the Internet.
Without an overlay:
Building A ---- Internet ---- Building B
Only Layer 3 connectivity exists.
With an overlay:
Building A
|
Virtual Switch
|
==== Tunnel ====
|
Virtual Switch
|
Building B
The tunnel makes distant systems appear as though they share the same local network.
This concept is the foundation of:
- GRE
- VXLAN
- Geneve
- Many SDN technologies
- Modern cloud networking
What Happens Next
The remainder of this part focuses on the technologies that allow Linux networks to scale beyond a single Ethernet segment.
We will start with GRE, the simplest form of Linux tunneling, then move to VXLAN and Open vSwitch, and finally combine these technologies to build a multi-host virtual network.
By the end of this section, you will understand how modern platforms such as Kubernetes, OpenStack, VMware, public clouds, and SDN systems connect workloads across large routed networks while preserving the illusion of local connectivity.
Key Takeaways
- Traditional Ethernet networks rely heavily on Layer 2 broadcasts.
- Large broadcast domains generate unnecessary traffic and become difficult to manage.
- Switches must maintain increasingly large MAC address tables.
- Layer 2 loops can cause severe network outages.
- Routing scales better because it divides networks into smaller broadcast domains.
- Modern infrastructure often requires Layer 2 connectivity across Layer 3 networks.
- Overlay technologies solve this problem by encapsulating traffic and transporting it across routed networks.
- GRE and VXLAN are examples of overlay technologies that extend networking beyond a single Layer 2 segment.