MTU, Fragmentation and Path MTU Discovery

Throughout this book, we have treated packets as if they could be any size.

In reality, every network has limits.

A network interface cannot transmit an infinitely large frame. Ethernet, tunnels, VPNs, VXLAN overlays, cloud networks, and routers all impose size constraints on the packets they carry.

One of the most common causes of mysterious network problems is a mismatch between packet sizes and network MTUs.

Symptoms often include:

  • Slow applications
  • Hanging SSH sessions
  • Broken HTTPS connections
  • Intermittent packet loss
  • VPN connectivity issues
  • Kubernetes networking problems
  • Overlay network failures

In this chapter, you will learn:

  • What MTU is
  • Why packet size matters
  • How fragmentation works
  • Why fragmentation is often undesirable
  • How Path MTU Discovery (PMTUD) works
  • Common MTU-related failures
  • How to troubleshoot MTU issues on Linux

Understanding MTU is essential for anyone working with Linux networking, virtualization, containers, cloud infrastructure, or overlay networks.

What Is MTU?

MTU stands for:

Maximum Transmission Unit

The MTU defines the largest packet that can be transmitted through a network interface without fragmentation.

For Ethernet, the most common MTU is:

1500 bytes

Example:

+----------------------+
| Ethernet Frame       |
| Payload <= 1500B     |
+----------------------+

If a packet exceeds the MTU, something must happen:

  • Fragment the packet
  • Drop the packet
  • Signal the sender to use smaller packets

Which behavior occurs depends on the protocol and network configuration.

Viewing MTU on Linux

Inspect network interfaces:

ip link show

Example:

2: eth0: mtu 1500

Or:

ip addr show eth0

Output:

mtu 1500

You can also use:

ip -br link

Example:

eth0     UP    mtu 1500

Why Ethernet Uses 1500 Bytes

Historically, Ethernet standardized on:

1500 bytes

for the Layer 3 payload.

This became the default MTU for most networks.

As a result, many protocols and applications implicitly assume:

MTU = 1500

Even today, most cloud providers and data centers continue using this value.

Packet Size vs Frame Size

A common source of confusion is the distinction between:

  • IP packet size
  • Ethernet frame size

Consider:

Ethernet Header      14 bytes
IP Header            20 bytes
TCP Header           20 bytes
Application Data

When we say:

MTU = 1500

we mean:

IP Payload <= 1500 bytes

The actual Ethernet frame is larger because Ethernet headers are added separately.

A Packet That Fits

Suppose an application generates:

1400 bytes

of TCP payload.

The packet might look like:

TCP Data       1400
TCP Header       20
IP Header        20
-------------------
Total          1440

Since:

1440 < 1500

the packet fits.

No fragmentation is required.

A Packet That Does Not Fit

Now consider:

TCP Data       2000
TCP Header       20
IP Header        20
-------------------
Total          2040

The packet exceeds:

1500 bytes

The network must deal with the oversized packet somehow.

IPv4 Fragmentation

Historically, IPv4 routers could fragment packets.

A large packet:

2040 bytes

might become:

Fragment 1
Fragment 2

Example:

Original Packet
     |
     v

+-------------+
| 2040 Bytes  |
+-------------+

     |

Fragmentation

     |

+-------------+
| Fragment 1  |
+-------------+

+-------------+
| Fragment 2  |
+-------------+

The destination host reassembles the fragments.

Observing Fragmentation

You can observe fragmented packets using tcpdump.

Start a capture:

sudo tcpdump -ni eth0

Generate oversized traffic.

Captured packets may contain:

frag

or:

offset

Example:

IP fragment id 12345 offset 1480

indicating that fragmentation has occurred.

Why Fragmentation Is Bad

Fragmentation sounds convenient.

However, it introduces several problems.

Increased Overhead

Every fragment requires:

  • Additional headers
  • Additional processing
  • Additional memory

Packet Loss Amplification

Suppose a packet becomes:

4 fragments

If one fragment is lost:

Entire packet lost

The receiver cannot reconstruct the original packet.

Reduced Performance

Fragmentation creates more packets.

More packets means:

  • More interrupts
  • More CPU work
  • More memory usage

Firewall Complications

Many security devices handle fragmented packets differently.

Fragmentation can complicate filtering and inspection.

IPv6 and Fragmentation

IPv6 changed the model significantly.

Routers no longer fragment packets.

Instead:

Router drops packet

and sends an ICMP message back to the sender.

The sender must reduce packet size.

This approach avoids many fragmentation-related problems.

The Don't Fragment (DF) Bit

IPv4 supports a special flag:

DF = Don't Fragment

When set:

Router must not fragment packet

If the packet is too large:

Drop packet

and return:

ICMP Fragmentation Needed

This mechanism is fundamental to Path MTU Discovery.

Understanding Path MTU

The MTU of a path is determined by the smallest MTU along that path.

Consider:

Host A
   |
1500
   |
Router
   |
1400
   |
Router
   |
1500
   |
Host B

The path MTU becomes:

1400

because the smallest link determines the maximum packet size.

Path MTU Discovery (PMTUD)

Path MTU Discovery allows hosts to determine the largest packet size that can traverse a path.

The process is:

  1. Sender transmits a packet with DF set.
  2. Router cannot forward packet.
  3. Router drops packet.
  4. Router sends ICMP "Fragmentation Needed".
  5. Sender reduces packet size.
  6. Communication continues.

Visualization:

Sender
   |
1500-byte Packet
   |
   v

Router

Packet Too Large

   |
   v

ICMP:
Fragmentation Needed

   |
   v

Sender retries
with smaller packet

Viewing the DF Flag

Capture traffic:

sudo tcpdump -ni eth0

You may see:

Flags [DF]

Example:

IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF])

This indicates that fragmentation is prohibited.

Testing MTU with Ping

Linux ping can generate packets of specific sizes.

Send:

ping -M do -s 1472 8.8.8.8

Explanation:

1472 bytes ICMP payload
20 bytes IP header
8 bytes ICMP header
----------------------
1500 bytes total

The option:

-M do

sets:

Don't Fragment

Finding the Maximum MTU

Start large:

ping -M do -s 1472 <host>

If it fails:

Frag needed

Reduce size:

ping -M do -s 1400 <host>

Continue until packets succeed.

This allows you to estimate the path MTU manually.

A Common Tunnel Problem

Suppose we create a VXLAN tunnel.

VXLAN adds approximately:

50 bytes

of overhead.

Physical network:

MTU = 1500

Overlay network:

1500-byte packet
+ 50-byte VXLAN header
-----------------------
1550 bytes

The packet no longer fits.

This often causes:

  • Kubernetes networking issues
  • Overlay connectivity failures
  • VPN performance problems

Inspecting MTU on Tunnel Interfaces

Check interface MTU:

ip link show vxlan100

Example:

mtu 1450

GRE example:

ip link show gre1

Output:

mtu 1476

Tunnel interfaces often use reduced MTUs to account for encapsulation overhead.

Changing MTU

Temporarily change MTU:

sudo ip link set eth0 mtu 1400

Verify:

ip link show eth0

Example:

mtu 1400

Be careful when changing MTU remotely, especially over SSH.

Incorrect values can disconnect your session.

PMTUD Failure: The Black Hole Problem

PMTUD depends on ICMP.

Consider:

Sender
  |
Router
  |
Firewall
  |
Receiver

If the firewall blocks:

ICMP Fragmentation Needed

messages:

PMTUD breaks

The sender never learns the correct MTU.

Symptoms include:

  • HTTPS hangs
  • SSH freezes after login
  • Random connection stalls
  • Large file transfers failing

These are often called:

MTU Black Holes

Diagnosing MTU Black Holes

Useful tools:

ping -M do

tracepath

tcpdump

Example:

tracepath google.com

Output:

pmtu 1500

or:

pmtu 1450

The tool automatically attempts to discover path MTU.

Hands-On Lab: Simulating MTU Issues

Create two namespaces:

sudo ip netns add ns1
sudo ip netns add ns2

Create a veth pair:

sudo ip link add veth1 type veth peer name veth2

Assign:

sudo ip link set veth1 netns ns1
sudo ip link set veth2 netns ns2

Configure:

sudo ip netns exec ns1 ip addr add 10.0.0.1/24 dev veth1
sudo ip netns exec ns2 ip addr add 10.0.0.2/24 dev veth2

Bring interfaces up:

sudo ip netns exec ns1 ip link set lo up
sudo ip netns exec ns1 ip link set veth1 up

sudo ip netns exec ns2 ip link set lo up
sudo ip netns exec ns2 ip link set veth2 up

Reduce MTU:

sudo ip netns exec ns1 ip link set veth1 mtu 1200
sudo ip netns exec ns2 ip link set veth2 mtu 1200

Test:

sudo ip netns exec ns1 ping \
    -M do \
    -s 1300 \
    10.0.0.2

Observe the failure.

Then:

sudo ip netns exec ns1 ping \
    -M do \
    -s 1100 \
    10.0.0.2

Observe the success.

This demonstrates MTU limits directly.

Jumbo Frames

Some networks use larger MTUs:

9000 bytes

commonly called:

Jumbo Frames

Benefits:

  • Fewer packets
  • Lower CPU overhead
  • Higher throughput

Common environments:

  • Storage networks
  • HPC clusters
  • Data center fabrics

However, every device along the path must support the larger MTU.

A single device configured for 1500 bytes becomes the limiting factor.

Real-World Examples

You will encounter MTU considerations in:

  • VXLAN
  • GRE
  • WireGuard
  • IPsec
  • OpenVPN
  • Kubernetes
  • OpenStack
  • Cloud networking
  • Storage networks

Many networking incidents eventually trace back to an MTU mismatch somewhere in the path.

Key Takeaways

  • MTU defines the maximum packet size a network can carry.
  • Ethernet commonly uses an MTU of 1500 bytes.
  • Oversized packets may be fragmented or dropped.
  • Fragmentation introduces overhead and reduces efficiency.
  • IPv6 routers do not perform fragmentation.
  • Path MTU is determined by the smallest MTU along a path.
  • Path MTU Discovery uses ICMP messages to determine the correct packet size.
  • Blocking ICMP can break PMTUD and create MTU black holes.
  • Tunnel technologies such as GRE and VXLAN reduce the effective MTU.
  • Tools such as ping -M do, tracepath, tcpdump, and ip link are essential for MTU troubleshooting.