Introduction

Modern software systems are built on networks.

Whether you're deploying applications, operating Kubernetes clusters, managing cloud infrastructure, building platforms, troubleshooting production incidents, or writing backend services, almost everything eventually becomes a networking problem. Yet networking remains one of the most misunderstood areas of engineering. Many engineers learn enough commands to get by, but struggle when things stop working because they lack a mental model of what is actually happening underneath.

This book was created to bridge that gap.

Networking for Non-Network Engineers is a practical guide to understanding computer networking from the perspective of engineers who build and operate systems rather than design enterprise networks. The goal is not to prepare you for a certification exam or teach vendor-specific technologies. Instead, the focus is on developing a deep understanding of how networks work, especially on Linux systems, through explanations, examples, and hands-on exploration.

Throughout this book, you'll learn how packets move through networks, how Linux makes routing decisions, how virtual networking works, how packets are processed by the kernel, and how modern networking technologies are built on top of these foundations. The concepts covered here form the basis for understanding containers, Kubernetes, service meshes, cloud networking, VPNs, and many other technologies used in modern infrastructure.

This book assumes curiosity, not expertise. Whether you're a DevOps engineer, Platform engineer, SRE, Backend developer, Systems administrator, student, or simply someone who wants to understand networking better, the material is designed to build intuition first and details second.

Open Source and Community Driven

This book is an open-source project and is developed in public. The source code, content, and examples are available on GitHub:

• https://github.com/kavehmoradian/Networking-for-Non-Network-Engineers

If you find an error, discover an unclear explanation, have suggestions for improvements, or want to contribute new content, pull requests and issues are welcome. Networking is a vast field, and community feedback helps make the book more accurate, practical, and useful for everyone.

A Practical Philosophy

The book follows a simple principle:

Learn the fundamentals well enough that the tools become obvious.

Networking tools, commands, and platforms change over time. Fundamental concepts do not. By understanding the underlying mechanics, you'll be able to troubleshoot unfamiliar systems, learn new technologies faster, and reason about complex infrastructure with confidence.

Let's begin by exploring why networking knowledge has become essential for modern engineers.

Why Networking Matters

Many engineers think networking is a specialized field reserved for network administrators or CCNA-certified professionals. In reality, almost every modern application depends on the network. Whether you're deploying a service to production, configuring a reverse proxy, debugging a failed API call, or exposing an application to the Internet, you're working with networking—even if you don't realize it.

Consider a simple web request. A user opens a browser and visits your application. Within milliseconds, the request passes through DNS servers, routers, firewalls, load balancers, reverse proxies, the Linux kernel, and finally reaches your application. If any component along that path fails or is misconfigured, the application becomes unavailable. Understanding that journey is often the difference between solving a production incident in minutes instead of hours.

Networking Is Everywhere

As a DevOps or Platform engineer, networking is part of your daily work.

You configure:

• Reverse proxies such as NGINX or Traefik
• Load balancers
• Firewalls and security groups
• VPNs
• Linux routing tables
• DNS records
• Virtual machines
• Cloud networks
• Kubernetes Services and Ingresses

Even when you're writing automation with Ansible or Terraform, many of the resources you create are networking resources.

You may not have "Network Engineer" in your job title, but networking is already part of your job.

Most Production Issues Are Network-Related

Many production incidents that initially appear to be application bugs are actually networking problems.

For example:

• A service cannot reach its database.
• A pod cannot communicate with another pod.
• A server has Internet access but cannot resolve hostnames.
• An API works from one machine but times out from another.
• A firewall silently drops packets.
• A route sends traffic to the wrong network.
• A load balancer reports unhealthy backends.
• An MTU mismatch causes intermittent failures.

Without a basic understanding of networking, these problems can be difficult to diagnose. Engineers often resort to restarting services or changing configurations without knowing the root cause.

Networking Is Not Magic

One reason networking feels intimidating is that it is often presented as a long list of protocols, acronyms, and standards.

OSI. TCP. UDP. ARP. ICMP. NAT. VXLAN. GRE. VLAN.

It can seem like an overwhelming collection of unrelated technologies.

In reality, networking follows a small number of simple principles. Every protocol exists to solve a specific problem, and most modern networking technologies are built by combining a handful of fundamental building blocks.

Once you understand those building blocks, networking becomes far less mysterious.

This Book Takes a Different Approach

This is not a certification guide, and it assumes no prior networking expertise.

Instead of asking you to memorize protocols or commands, we'll answer questions such as:

• How does a packet leave my machine?
• How does Linux decide where to send it?
• What happens before a TCP connection is established?
• Why does NAT work?
• How can multiple isolated networks exist on a single Linux host?
• How does a Linux bridge behave like a physical switch?

We'll build these concepts step by step using Linux itself as our laboratory.

By the end of this book, you'll be able to look beyond commands like ping, ip, ss, or tcpdump and understand the mechanisms that make them work. That knowledge will not only make you better at troubleshooting Linux systems but will also prepare you to understand container and Kubernetes networking, where the same Linux networking primitives are used extensively.

How Computers Actually Communicate

Before we dive into protocols, routing, or Linux networking, let's answer a much more fundamental question:

How does one computer send information to another?

Whether you're opening a website, connecting to a database, or sending a message over SSH, every network operation begins with one computer sending information to another. While the technologies involved can become complex, the basic idea is surprisingly simple: computers exchange data by transmitting a sequence of electrical, optical, or radio signals over a communication medium.

Everything Starts with Data

Computers don't understand text, images, or videos.

They understand only binary data—a sequence of 0s and 1s, known as bits.

For example, the text:

Hello

is stored as binary data:

01001000 01100101 01101100 01101100 01101111

Fortunately, humans never work directly with these bits. Software converts them into formats that computers can store, process, and transmit.

From Bits to Bytes

A single bit can represent only two values: 0 or 1.

To represent more complex information, bits are grouped together.

The most common grouping is the byte, which consists of 8 bits.

For example:

Everything stored or transmitted by a computer—documents, images, videos, and network traffic—is ultimately represented as bytes.

Sending Data Across a Network

Imagine Computer A wants to send the word:

Hello

to Computer B.

The application first converts the text into bytes.

Those bytes must then travel across some form of communication medium, such as:

• An Ethernet cable
• Fiber optic cable
• Wi-Fi
• A cellular network

Regardless of the medium, the goal is always the same:

Convert bytes into signals that another computer can receive and convert back into bytes.

At the lowest level, networks don't send "messages" or "files."

They send signals that represent bits.

Why Data Is Broken Into Pieces

Suppose you want to send a 2 GB file.

Sending it as one enormous block would be inefficient.

If a single bit were corrupted during transmission, the entire file would have to be sent again.

Instead, computers divide data into many smaller pieces before transmitting it.

Each piece can be transmitted independently and, if necessary, retransmitted without affecting the rest of the data.

These pieces are known by different names depending on where they are in the networking stack:

• Frame – a unit of data at the data-link layer.
• Packet – a unit of data at the network layer.
• Segment – a unit of data at the transport layer.

You'll learn what each of these means in the following chapters. For now, it's enough to understand that large amounts of data are always divided into smaller units before being transmitted.

A Journey Across the Network

Imagine you're opening a web page.

Although it appears to be a single action, a series of events takes place behind the scenes:

1. Your browser generates a request.
2. The request is converted into bytes.
3. The bytes are divided into smaller pieces.
4. Those pieces are transmitted across the network.
5. Routers and switches forward them toward the destination.
6. The destination computer receives the pieces.
7. The original data is reconstructed.
8. The web server processes the request and sends a response using the same process.

This entire sequence usually completes in just a few milliseconds.

Building on This Foundation

At this point, you only need to remember three key ideas:

• Computers communicate by exchanging binary data.
• That data is transmitted as electrical, optical, or radio signals.
• Large amounts of data are divided into smaller pieces before being sent across a network.

The next chapter introduces the TCP/IP stack, which defines how those pieces are organized, addressed, transmitted, and reassembled so that computers around the world can communicate reliably.

The TCP/IP Stack

In the previous chapter, we learned that computers exchange binary data by breaking it into smaller pieces and transmitting those pieces across a network. But one important question remains:

How do computers know how to create, send, receive, and interpret those pieces of data?

The answer is protocols.

A protocol is simply a set of rules that defines how computers communicate. Just as two people must speak the same language to understand each other, computers must follow the same protocols to exchange data successfully.

Modern computer networks are built on a collection of protocols known as the TCP/IP protocol suite, or simply the TCP/IP stack.

Why Do We Need a Stack?

Imagine you're mailing a package to someone in another country.

The process involves multiple independent steps:

• You write the message.
• You place it in an envelope.
• You write the destination address.
• A postal service transports it.
• The recipient receives and opens it.

Each step has a different responsibility.

Computer networking works in exactly the same way.

Instead of having one enormous protocol responsible for everything, networking is divided into multiple layers. Each layer performs one specific job and passes its data to the next layer.

This design makes networking simpler, more flexible, and easier to extend.

The Four Layers of TCP/IP

The TCP/IP model consists of four layers.

Each layer provides services to the layer above it while relying on the layer below it.

For example, a web browser doesn't need to know how Ethernet works. It simply asks the transport layer to deliver data to another computer.

Encapsulation

Suppose you visit:

https://google.com

Your browser creates an HTTP request.

As the request moves down the TCP/IP stack, each layer adds its own information.

Application Layer
┌──────────────────────────┐
│ HTTP Request             │
└──────────────────────────┘
             │
             ▼
Transport Layer
┌──────────────────────────┐
│ TCP Header               │
│ HTTP Request             │
└──────────────────────────┘
             │
             ▼
Internet Layer
┌──────────────────────────┐
│ IP Header                │
│ TCP Header               │
│ HTTP Request             │
└──────────────────────────┘
             │
             ▼
Link Layer
┌──────────────────────────┐
│ Ethernet Header          │
│ IP Header                │
│ TCP Header               │
│ HTTP Request             │
└──────────────────────────┘

Each layer wraps the data produced by the layer above it.

This process is called encapsulation.

When the data reaches the destination, the opposite happens.

Each layer removes its own header until the original HTTP request reaches the web server.

This reverse process is called decapsulation.

What Does Each Layer Actually Do?

Application Layer

This is where applications live.

Your browser, SSH client, email client, or database client all operate here.

Application protocols define the meaning of the data being exchanged.

Examples include:

• HTTP
• HTTPS
• DNS
• SSH
• SMTP
• FTP

The application layer does not worry about routing, packet delivery, or network hardware.

Its only concern is the application data.

Transport Layer

The transport layer enables communication between two applications running on different computers.

Its responsibilities include:

• Identifying applications using ports
• Breaking large messages into smaller pieces
• Detecting lost data
• Reordering data if necessary
• Ensuring reliable delivery (TCP)
• Providing lightweight delivery (UDP)

The two most common transport protocols are:

• TCP
• UDP

We'll explore both in detail in the next chapter.

Internet Layer

The Internet layer is responsible for moving data between networks.

It answers questions such as:

• Where should this packet go?
• Which network is the destination on?
• Which router should receive it next?

The primary protocol here is IP (Internet Protocol).

Unlike TCP, IP makes no guarantees that packets will arrive or arrive in order.

Its job is simply to move packets toward their destination.

Link Layer

The Link layer is responsible for communication within a local network.

This includes technologies such as:

• Ethernet
• Wi-Fi

At this layer, computers communicate using hardware addresses (MAC addresses) instead of IP addresses.

We'll revisit this layer when we discuss Ethernet, Linux bridges, VLANs, and virtual networking.

A Packet's Journey

Imagine you run:

curl https://google.com

Here's what happens:

1. curl creates an HTTP request.
2. The transport layer adds a TCP header.
3. The Internet layer adds an IP header.
4. The Link layer creates an Ethernet frame.
5. The frame is transmitted across the network.
6. The receiving computer removes the Ethernet header.
7. The IP layer processes the packet.
8. TCP reassembles the data if necessary.
9. The HTTP request reaches the web server.

Each layer performs exactly one job before passing the data to the next layer.

This layered approach is one of the primary reasons the Internet can scale to billions of devices.

TCP/IP vs. OSI

You may have heard of the OSI model, which divides networking into seven layers.

While the OSI model is useful as a conceptual framework, modern operating systems and Internet protocols are based on the TCP/IP model.

Throughout this book, we'll use the TCP/IP model because it more accurately reflects how Linux networking is implemented.

Looking Ahead

Understanding the TCP/IP stack is essential because nearly every networking technology you'll encounter fits somewhere within it.

As you continue through this book, you'll learn how:

• TCP and UDP operate at the Transport layer.
• IP and routing operate at the Internet layer.
• Ethernet, bridges, and VLANs operate at the Link layer.
• DNS, HTTP, and SSH operate at the Application layer.

By understanding the responsibilities of each layer, you'll be able to reason about networking problems systematically instead of treating the network as a black box.

TCP, UDP, Ports and Sockets

In the previous chapter, we learned that the Transport layer is responsible for communication between applications running on different computers. But how does that communication actually work?

Imagine a server running multiple applications:

• A web server listening for HTTP requests
• An SSH server accepting remote logins
• A database waiting for queries

All of these applications share the same network interface and the same IP address. When a packet arrives, how does Linux know which application should receive it?

The answer lies in four closely related concepts:

• TCP
• UDP
• Ports
• Sockets

Understanding these concepts is essential because almost every network application depends on them.

TCP vs. UDP

The two most common transport protocols are TCP (Transmission Control Protocol) and UDP (User Datagram Protocol).

Although both transport data between applications, they are designed for different purposes.

The choice between TCP and UDP depends on what the application needs.

TCP: Reliable Communication

TCP is designed for situations where every byte of data matters.

Before any data is exchanged, the two computers establish a connection.

During communication, TCP:

• Detects lost packets
• Retransmits missing data
• Delivers data in the correct order
• Prevents duplicate data
• Adjusts transmission speed based on network conditions

These guarantees make TCP ideal for applications such as:

• Web browsing (HTTP/HTTPS)
• SSH
• Email
• Databases
• File transfers

Reliability comes at a cost.

TCP requires additional processing and control messages, making it slightly slower than UDP.

UDP: Fast and Lightweight

UDP takes a much simpler approach.

It sends data without first establishing a connection.

There is no guarantee that:

• Packets arrive
• Packets arrive only once
• Packets arrive in order

If a packet is lost, UDP does nothing.

This simplicity makes UDP extremely fast and efficient.

Applications that use UDP either tolerate packet loss or implement their own reliability mechanisms.

Common examples include:

• DNS
• Voice over IP (VoIP)
• Video conferencing
• Online gaming
• Streaming
• DHCP

For these applications, receiving data quickly is often more important than receiving every packet.

Ports

An IP address identifies a computer.

A port identifies an application running on that computer.

Think of an apartment building.

The building's street address identifies the building.

The apartment number identifies the individual resident.

Likewise:

IP Address → Computer

Port → Application

For example:

192.168.1.10:22

means:

• Computer: 192.168.1.10
• Application listening on port 22 (typically SSH)

A single computer can have thousands of ports open simultaneously, allowing many applications to communicate at the same time.

Well-Known Ports

Many common services use standardized port numbers.

Using standard ports allows clients to locate services without additional configuration.

Applications are free to use different ports, but these defaults are widely recognized.

What Is a Socket?

A socket is the endpoint of a network connection.

It combines:

• An IP address
• A port number
• A transport protocol (TCP or UDP)

For example:

192.168.1.10:443/TCP

identifies a specific TCP socket.

Applications create sockets to send and receive network data.

From the application's perspective, networking is performed through sockets rather than directly through TCP or UDP.

Client and Server Sockets

When you open a website, two sockets are involved.

The server listens on a well-known port:

203.0.113.10:443

Your computer creates a temporary client socket:

192.168.1.25:51432

The complete connection is identified by four values:

Client IP
Client Port
Server IP
Server Port

For example:

192.168.1.25:51432
        ↓
203.0.113.10:443

This allows thousands of clients to communicate with the same server simultaneously, even though they are all connecting to port 443.

Each connection has a unique combination of client and server addresses and ports.

Listening and Established Connections

Before a server can accept connections, it must create a listening socket.

For example:

ss -lnt

might show:

State   Local Address:Port

LISTEN  0.0.0.0:22
LISTEN  0.0.0.0:80
LISTEN  0.0.0.0:443

TODO: more detail about sockets When a client connects, Linux creates a new socket representing that specific connection.

This is why a web server can serve thousands of clients at the same time while listening on a single port.

Putting It All Together

Suppose you run:

curl https://google.com

Behind the scenes:

1. curl creates a TCP socket.
2. Linux assigns a temporary source port.
3. A connection is established with the server's port 443.
4. HTTP data is exchanged through the socket.
5. When communication is complete, the socket is closed.

From the application's point of view, everything happens through the socket. The operating system handles the underlying TCP communication.

Looking Ahead

So far we've focused on communication between applications running on different computers.

But one important question remains:

How does Linux know where to send a packet once it leaves a socket?

To answer that, we need to understand routing—the process Linux uses to decide where every packet should go next.

Routing

Every time an application sends data across a network, Linux must answer one simple question:

Where should this packet go?

The answer is determined by routing.

Routing is the process of deciding where a packet should be sent next. Every packet that leaves your computer goes through this process, whether it's destined for another machine on your local network or a server on the other side of the Internet.

Before we can understand how routing works, we first need to understand what a network is and how a computer determines whether another IP address is local or remote.

IP Addresses, Networks, and Hosts

An IPv4 address consists of 32 bits.

For example:

192.168.1.10

By itself, this address is incomplete.

A computer also needs to know which part of the address identifies the network and which part identifies the host.

For example:

192.168.1.10/24

The /24 tells us that:

• The first 24 bits identify the network
• The remaining 8 bits identify the host

Visually:

192.168.1.10/24

11000000.10101000.00000001.00001010

|-------- Network --------|-- Host --|
       24 bits            8 bits

The network portion identifies the subnet.

The host portion identifies an individual device inside that subnet.

Netmask (Subnet Mask)

Before CIDR notation became common, networks were represented using a subnet mask (or netmask).

For example:

IP Address
192.168.1.10
11000000.10101000.00000001.00001010

Subnet Mask
255.255.255.0
11111111.11111111.11111111.00000000


|-------- Network --------|-- Host --|
       num of 1 bits     

Today, the same network is usually written as:

192.168.1.10/24

These two notations mean exactly the same thing.

Linux accepts both formats, although CIDR notation is used almost everywhere today.

How Does a Netmask Work?

A netmask is simply another 32-bit value.

Every 1 bit belongs to the network.

Every 0 bit belongs to the host.

For example:

IP Address

192.168.1.10

11000000.10101000.00000001.00001010


Subnet Mask

255.255.255.0

11111111.11111111.11111111.00000000

The mask separates the address into two parts:

Network
192.168.1

Host
10

The computer performs a bitwise AND between the IP address and the subnet mask to determine the network address.

Fortunately, you almost never need to perform this calculation manually. Understanding what the mask represents is much more important than memorizing the binary arithmetic.

Network Address

The network address identifies the subnet itself.

Suppose a computer has:

192.168.1.42/24

The network address is:

192.168.1.0

because the host bits become zero.

Likewise:

Notice that many different hosts can belong to the same network.

What Does "Same Network" Mean?

Consider a computer configured as:

IP Address
192.168.1.10/24

Now suppose it wants to communicate with the following destinations.

The first two destinations belong to the same subnet:

192.168.1.0/24

Linux knows these devices are directly reachable.

The last two belong to different networks.

Linux knows it cannot reach them directly.

Instead, it must send the packets to a router.

Directly Connected Networks

Every network interface defines a directly connected network.

For example:

eth0

192.168.1.10/24

means:

Everything inside 192.168.1.0/24 is directly reachable through eth0.

Linux does not need another router to reach these hosts.

It can communicate with them directly.

What Is Routing?

Routing begins only after Linux determines that the destination is not on one of its directly connected networks.

Suppose our computer wants to reach:

8.8.8.8

Since this address is not part of:

192.168.1.0/24

Linux must ask:

Which router should receive this packet?

The answer comes from the routing table.

The Routing Table

Linux stores its routing information in a routing table.

You can display it with:

ip route

Example:

default via 192.168.1.1 dev eth0

192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.10

10.10.0.0/16 via 192.168.1.254 dev eth0

Each entry tells Linux how to reach a particular network.

Notice something important:

The routing table contains networks, not individual hosts.

Routing almost always happens at the network level.

Reading a Route

Let's examine one entry:

10.10.0.0/16 via 192.168.1.254 dev eth0

This means:

• To reach the network 10.10.0.0/16
• Send the packet to 192.168.1.254
• Use the interface eth0

The address 192.168.1.254 is called the next hop or gateway.

It is the next device responsible for forwarding the packet.

The Default Route

Eventually you'll encounter this route:

default via 192.168.1.1

or equivalently:

0.0.0.0/0

This is the default route.

It simply means:

If no other route matches the destination, send the packet to this gateway.

Almost every computer connected to the Internet has a default route.

Without one, the computer could communicate only with devices on its directly connected networks.

Longest Prefix Match

Sometimes multiple routes match the same destination.

For example:

10.0.0.0/8

10.10.0.0/16

10.10.10.0/24

Now suppose the destination is:

10.10.10.42

All three routes match.

Linux chooses the most specific route.

This rule is called the Longest Prefix Match.

Since /24 is more specific than /16, and /16 is more specific than /8, Linux selects:

10.10.10.0/24

This rule is fundamental to IP routing.

Putting It All Together

Whenever Linux sends a packet, it follows roughly this process:

Application
        │
        ▼
Destination IP
        │
        ▼
Is the destination on one of my directly connected networks?
        │
   Yes ─┴─ No
    │        │
    ▼        ▼
Send      Search the routing table
Directly       │
               ▼
      Select the best matching route
               │
               ▼
        Determine the next hop
               │
               ▼
        Send packet to the gateway

At this point, Linux knows where the packet should go.

However, it still has another problem.

It knows the IP address of the next hop, but Ethernet doesn't send frames to IP addresses—it sends them to MAC addresses.

So how does Linux discover the MAC address of the next hop?

That's the job of the Address Resolution Protocol (ARP), which we'll explore in the next chapter.

ARP: Finding the Next Hop

In the previous chapter, we learned that applications communicate using IP addresses and ports. But this raises an important question:

If I know the destination IP address, how do I actually send data to it?

The answer is that you can't—at least not directly.

On an Ethernet network, computers do not send data to IP addresses. They send Ethernet frames to MAC addresses.

So before Linux can transmit a packet, it must first answer another question:

Which MAC address belongs to the next IP address?

This is the job of the Address Resolution Protocol (ARP).

IP Addresses vs. MAC Addresses

Every network interface has two important addresses.

An IP address identifies a device on a network and is used for routing packets between networks.

A MAC address identifies a network interface on the local network and is used to deliver Ethernet frames.

For example:

Think of an IP address as a person's postal address, while the MAC address is the label on their mailbox. The postal system uses the address to find the correct house, but once the delivery truck arrives, it still needs the correct mailbox.

The Problem

Suppose your computer wants to send a packet to:

192.168.1.20

Your routing table says that this destination is on the same local network.

Linux now knows where the packet should go, but it still doesn't know how to deliver the Ethernet frame.

It needs the destination MAC address.

At this point, Linux checks whether it already knows the answer.

The ARP Cache

Linux keeps a small table of recently discovered IP-to-MAC mappings.

This table is called the ARP cache, or more generally in Linux, the neighbor table.

You can view it with:

ip neigh

Example:

192.168.1.20 dev eth0 lladdr 00:11:22:33:44:55 REACHABLE
192.168.1.1  dev eth0 lladdr aa:bb:cc:dd:ee:ff STALE

If Linux finds the destination in this table, it immediately sends the Ethernet frame to the corresponding MAC address.

If not, it must discover it.

ARP Request

When Linux doesn't know the destination MAC address, it sends an ARP Request.

In simple terms, the request says:

"Who has IP address 192.168.1.20? Please tell me your MAC address."

Because Linux doesn't yet know where that device is, the request is sent as a broadcast.

Every device on the local Ethernet network receives the request.

ARP Reply

Every computer checks the requested IP address.

Only the computer that owns the address replies.

The reply says something like:

"192.168.1.20 is at 00:11:22:33:44:55."

Linux stores this mapping in its neighbor table for future use.

Now it can finally construct an Ethernet frame and send the packet.

Putting It Together

Suppose you run:

ping 192.168.1.20

If the MAC address is unknown, the sequence looks like this:

Application
      │
      ▼
Create ICMP packet
      │
      ▼
Routing decides:
Destination is on the local network
      │
      ▼
Check neighbor table
      │
      ▼
No entry found
      │
      ▼
Broadcast ARP Request
      │
      ▼
Receive ARP Reply
      │
      ▼
Store IP → MAC mapping
      │
      ▼
Transmit Ethernet frame

Notice that the ICMP packet isn't sent until ARP has completed successfully.

What If the Destination Is on Another Network?

Suppose your computer wants to reach:

8.8.8.8

Linux knows this address is not on the local network.

Instead of looking for the MAC address of 8.8.8.8, Linux sends the packet to its default gateway.

That means ARP resolves the MAC address of the router, not the final destination.

For example:

Destination IP:
8.8.8.8

Next Hop:
192.168.1.1

MAC Needed:
Router's MAC address

This is why the chapter is called "Finding the Next Hop."

ARP never tries to find the MAC address of a device on another network. It always resolves the MAC address of the next device that will receive the Ethernet frame.

Key Takeaways

Remember these four ideas:

• IP addresses identify where packets should go.
• MAC addresses identify where Ethernet frames should be delivered.
• ARP translates an IP address into a MAC address on the local network.
• ARP always resolves the next hop, not necessarily the final destination.

Understanding ARP explains many networking behaviors that initially seem mysterious. It also prepares us for the next chapter, where we'll explore routing and learn how Linux decides what the next hop actually is.

ICMP: The Network's Control Protocol

So far, we've learned how Linux decides where a packet should go (routing) and how it discovers the MAC address of the next hop (ARP).

But what happens when something goes wrong?

What if:

• The destination doesn't exist?
• A router can't forward the packet?
• A packet loops endlessly through the network?
• The packet is too large to pass through a link?

Unlike TCP or UDP, which transport application data, the Internet Control Message Protocol (ICMP) is responsible for communicating information about the network itself.

Think of ICMP as the network's control and diagnostic protocol.

What Is ICMP?

ICMP is part of the Internet Protocol (IP) suite and operates alongside IP.

Instead of carrying application data, ICMP carries control messages.

These messages help computers and routers:

• Report errors
• Diagnose connectivity problems
• Exchange network information

For example, a router may send an ICMP message to tell a sender:

"I can't reach that destination."

or

"Your packet is too large."

Common ICMP Messages

ICMP defines many different message types.

Some of the most common are:

Each message helps computers understand what happened to a packet after it left the sender.

Ping Uses ICMP

The most familiar use of ICMP is the ping command.

For example:

ping 8.8.8.8

ping sends an ICMP Echo Request.

If the destination is reachable, it responds with an ICMP Echo Reply.

Computer A         Computer B
     │                 │
     │  Echo Request   │
     │────────────────►│
     │                 │
     │  Echo Reply     │
     │◄────────────────│

This simple exchange tells us:

• The destination is reachable.
• The network path is working.
• The host is responding to ICMP.

It does not tell us whether a particular application, such as a web server or SSH server, is available.

A server can ignore ICMP requests while still serving web traffic normally.

Time to Live (TTL)

Every IP packet contains a field called Time to Live (TTL).

Despite its name, TTL is not measured in seconds.

Instead, it is a counter.

Every router that forwards a packet decreases the TTL by one.

If the value reaches zero, the router discards the packet and sends an ICMP Time Exceeded message back to the sender.

This prevents packets from circulating forever if a routing loop occurs.

Without TTL, a single routing mistake could flood the network indefinitely.

How Traceroute Works

The traceroute command is built on the TTL mechanism.

Instead of sending one packet, it sends many packets with increasing TTL values.

For example:

TTL = 1

The first router decreases the TTL to zero and returns:

ICMP Time Exceeded

Now traceroute knows the address of the first router.

It repeats the process:

TTL = 2

The second router responds.

Then:

TTL = 3

And so on until the destination is reached.

This is how traceroute discovers every hop between your computer and the destination.

ICMP Is Not an Application Protocol

A common misconception is that ICMP works like TCP or UDP.

It doesn't.

ICMP:

• Has no ports.
• Does not establish connections.
• Does not transport application data.

Instead, it exists to support the operation of IP networks.

Applications like ping and traceroute simply use ICMP to gather information about the network.

Looking Ahead

By now, we've seen how a computer:

• Chooses the next hop using the routing table.
• Discovers the next hop's MAC address using ARP.
• Receives status and error information using ICMP.

One important question remains:

How do we find the IP address of a server in the first place?

Humans prefer names like:

example.com

Computers communicate using IP addresses.

In the next chapter, we'll explore the Domain Name System (DNS) and learn how those names are translated into the IP addresses that make communication possible.

How DNS Really Works

Imagine opening your browser and visiting:

https://google.com

Within a few milliseconds, your browser connects to the correct web server somewhere on the Internet.

But computers don't communicate using names like google.com.

They communicate using IP addresses.

So before a connection can be established, one important question must be answered:

What is the IP address of google.com?

The answer comes from the Domain Name System (DNS).

DNS is often described as the Internet's phone book, but it's better thought of as a distributed database that translates human-readable names into IP addresses.

Without DNS, you would need to remember IP addresses instead of names for every website and service you use.

Why Do We Need DNS?

Humans are good at remembering names.

Computers are good at using numbers.

For example, which is easier to remember?

google.com

or

93.184.216.34

Names are also flexible.

If a company moves its website to another server, the IP address may change, but users can continue visiting the same domain name.

DNS provides this layer of indirection.

Applications use names.

Networks use IP addresses.

DNS connects the two.

Domain Names

A domain name is organized as a hierarchy.

For example:

api.google.com

can be broken into three parts:

api      example      com
 │           │         │
Host     Domain      Top-Level Domain (TLD)

Reading from right to left:

• .com is the Top-Level Domain (TLD).
• example is the domain registered under .com.
• api is a subdomain (often representing a specific service).

This hierarchical structure allows the Internet to scale without requiring a single central database.

DNS Records

DNS stores different types of information called records.

Some of the most common record types are:

For example:

google.com.     A      93.184.216.34

This record tells clients that google.com resolves to the IPv4 address 93.184.216.34.

DNS Resolution

Suppose you run:

curl https://google.com

Before TCP can establish a connection, Linux must first discover the server's IP address.

The process looks like this:

Application
      │
      ▼
Need IP address
      │
      ▼
Query DNS Resolver
      │
      ▼
Receive IP address
      │
      ▼
Create TCP connection

Notice that DNS happens before any TCP connection to the web server is created.

Without the destination IP address, the operating system has nowhere to send the packet.

Recursive Resolvers

Your computer usually doesn't contact the authoritative DNS server directly.

Instead, it asks a recursive resolver.

This resolver may belong to:

• Your ISP
• Your company
• A public DNS provider such as Google Public DNS or Cloudflare DNS

The resolver performs the work of finding the answer and returns the final IP address to your computer.

From the client's perspective, DNS appears to be a simple question-and-answer service.

Where Does the Resolver Find the Answer?

If the recursive resolver doesn't already know the answer, it performs a series of queries.

Suppose we're looking up:

api.google.com

The resolver follows the DNS hierarchy.

1. Ask a Root Name Server:

"Who is responsible for .com?"

The root server replies with the name servers for .com.

↓

2. Ask a .com Name Server:

"Who is responsible for google.com?"

The .com server replies with the authoritative name servers for google.com.

↓

3. Ask the Authoritative Name Server:

"What is the IP address of api.google.com?"

The authoritative server returns the requested DNS record.

↓

4. Return the answer to the client.

The client now knows the server's IP address and can begin the TCP connection.

DNS Caching

Performing the full lookup for every request would be slow.

Instead, DNS uses caching extensively.

Your operating system may cache results.

Your browser may cache results.

Your company's resolver may cache results.

Your ISP may cache results.

Each DNS record contains a value called TTL (Time To Live).

TTL specifies how long a cached answer may be reused before it should be queried again.

For example:

google.com. 300 IN A 93.184.216.34

A TTL of 300 seconds means the result may be cached for five minutes.

Caching dramatically reduces DNS traffic and improves performance across the Internet.

DNS Uses Both UDP and TCP

One interesting detail about DNS is that it uses both UDP and TCP.

Most DNS queries use UDP port 53 because requests and responses are usually small, making UDP fast and efficient.

However, DNS switches to TCP port 53 in situations such as:

• Large responses
• Zone transfers between DNS servers
• When a UDP response is too large

Most users never notice this transition because DNS software handles it automatically.

Common DNS Tools

Linux provides several utilities for querying DNS.

To look up a hostname:

dig google.com

or

host google.com

or

nslookup google.com

Among these tools, dig is generally preferred because it provides the most detailed output.

We'll use these commands later in the Linux networking section of this book.

Putting It All Together

When you visit:

https://google.com

the following events occur:

Browser
    │
    ▼
DNS Lookup
    │
    ▼
Receive IP Address
    │
    ▼
Routing
    │
    ▼
ARP
    │
    ▼
TCP Connection
    │
    ▼
HTTPS Request
    │
    ▼
Web Server Response

Notice that everything we've learned in Part 1 works together.

• DNS translates a name into an IP address.
• Routing determines where packets should go.
• ARP discovers the next hop's MAC address.
• TCP establishes a reliable connection.
• The application finally exchanges data.

By this point, you have the fundamental concepts needed to understand how two computers communicate across a network. The rest of this book builds on these foundations by exploring how Linux implements networking internally and how these same concepts are used to build modern virtual networks.

Linux Network Interfaces

In Part 1, we learned how computers communicate using IP addresses, routing, ARP, TCP, and DNS.

Now it's time to see how Linux implements these concepts.

Everything begins with the network interface.

Whether you're configuring a physical Ethernet card, creating a bridge, attaching a container to a virtual network, or debugging Kubernetes networking, you're ultimately working with Linux network interfaces.

Understanding interfaces is one of the most important skills in Linux networking because almost every networking operation starts from one.

What Is a Network Interface?

A network interface is Linux's representation of a network connection.

From the kernel's perspective, every packet enters and leaves the system through a network interface.

An interface might represent:

• A physical Ethernet card
• A Wi-Fi adapter
• A virtual Ethernet device
• A bridge
• A VLAN
• A tunnel
• A loopback device

Linux treats all of these as network interfaces.

This unified design is one of the reasons Linux networking is so powerful.

Listing Network Interfaces

The modern way to list interfaces is with the ip command.

ip link

Example output:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT

A simpler view:

ip -br link

Example:

lo               UNKNOWN        00:00:00:00:00:00
ens18            UP             52:54:00:12:34:56

The -br (brief) option is often easier to read, especially on systems with many interfaces.

Interface Names

Older Linux systems used names like:

eth0
eth1
wlan0

Modern distributions typically use predictable interface names, such as:

ens18
ens160
enp3s0
eno1

These names are generated based on hardware location, making them more stable across reboots.

You'll still encounter eth0 in:

• Containers
• Virtual machines
• Documentation
• Older systems

Throughout this book, we'll use whatever interface names exist on the system.

Viewing Interface Details

To inspect an interface:

ip addr show ens18

Example:

2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP>
    inet 192.168.1.10/24
    link/ether 52:54:00:12:34:56

Notice that one interface contains several pieces of information.

• Interface name
• Administrative state
• MAC address
• IP address
• Prefix length

Unlike many networking devices, Linux stores all of these properties as part of the interface itself.

Administrative State

Every interface has an administrative states. Whether the interface has been enabled.

View:

ip link

Bring an interface up:

sudo ip link set ens18 up

Bring it down:

sudo ip link set ens18 down

An interface that is administratively down cannot send or receive traffic.

The Loopback Interface

Every Linux system has an interface called:

lo

View it:

ip addr show lo

Example:

1: lo
inet 127.0.0.1/8

The loopback interface never leaves the computer.

Traffic sent to:

127.0.0.1

never reaches a physical network.

Instead, it is delivered directly back into the kernel.

Many applications use the loopback interface to communicate with services running on the same machine.

MAC Addresses

Every Ethernet interface has a MAC address.

View it:

ip link show ens18

Example:

link/ether 52:54:00:12:34:56

Remember from Part 1:

• IP addresses identify hosts on a network.
• MAC addresses identify interfaces on a local Ethernet network.

When Linux sends an Ethernet frame, it places the destination MAC address in the frame header.

IP Addresses

Interfaces can have one or more IP addresses.

Show only IPv4 addresses:

ip -4 addr

Show only IPv6:

ip -6 addr

Assign a temporary address:

sudo ip addr add 192.168.100.10/24 dev ens18

Remove it:

sudo ip addr del 192.168.100.10/24 dev ens18

These changes affect the running system only.

After a reboot, persistent configuration depends on your network management software, such as NetworkManager or systemd-networkd.

Interface Statistics

Linux tracks packet counters for every interface.

Display them:

ip -s link

Example:

RX:
packets  bytes

TX:
packets  bytes

These counters are invaluable when troubleshooting.

They can help answer questions such as:

• Is traffic reaching the server?
• Is the interface transmitting packets?
• Are packets being dropped?
• Are errors increasing?

Every Packet Has an Interface

One of the most important concepts in Linux networking is this:

Every packet enters through one interface and leaves through another.

Whether the interface represents:

• A physical NIC
• A bridge
• A veth pair
• A VLAN
• A VXLAN tunnel

the Linux kernel processes them in the same way.

This consistent abstraction is what makes Linux networking both flexible and powerful.

In the following chapters, we'll create new interfaces ourselves and use them to build entirely virtual networks.

Inspecting Interface Capabilities with ethtool

While the ip command manages network interfaces and their addresses, it does not expose hardware-specific information.

For that, Linux provides the ethtool utility.

ethtool allows you to inspect and configure properties of network interfaces, such as:

• Link speed
• Driver information
• Supported features
• Offloading capabilities
• Ring buffer sizes
• Interface statistics

When troubleshooting network performance, ethtool is often one of the first tools you'll use.

Viewing Interface Information

Display basic information about an interface:

sudo ethtool ens18

Example output:

Settings for ens18:
    Speed: 1000Mb/s
    Auto-negotiation: on
    Port: Twisted Pair
    Link detected: yes

Some important fields are:

If Link detected is no, the interface is not connected to another device, regardless of whether it has an IP address configured.

Viewing Interface Statistics

Many network drivers expose additional statistics beyond the standard packet counters shown by ip.

Display them with:

sudo ethtool -S ens18

Depending on the hardware, you may see counters for:

• CRC errors
• Dropped packets
• Missed packets
• Queue statistics
• Interrupt counts
• Driver-specific metrics

These statistics are invaluable when diagnosing hardware or driver problems.

Changing Interface Settings

ethtool can also modify interface settings.

For example, to disable auto-negotiation and set a fixed speed:

sudo ethtool -s ens18 speed 1000 duplex full autoneg off

Changing these settings is uncommon on modern systems because interfaces usually negotiate the optimal parameters automatically.

Unless you have a specific operational requirement, it's generally best to leave these settings unchanged.

When Should You Use ethtool?

Use ethtool when you need to answer questions such as:

• Is the Ethernet cable connected?
• What speed did the interface negotiate?
• Are there hardware-level errors or dropped packets?
• Is this a physical NIC or a virtual one?

While ip tells you how Linux sees an interface, ethtool tells you how the interface itself is operating.

Summary

In this chapter, you learned that:

• Linux represents every network connection as a network interface.
• Interfaces can be physical or virtual.
• Every interface has a name, state, MAC address, and optionally one or more IP addresses.
• ip and ethtool commands are used for inspecting and managing interfaces.
• Every packet entering or leaving the system passes through a network interface.

With this foundation in place, we're ready to explore how Linux decides which interface should be used when sending a packet by examining the system's routing tables.

How Linux Makes Routing Decisions

In Part 1, we learned the concepts behind IP routing: networks, routing tables, longest prefix match, and the default gateway.

In this chapter, we'll look at how the Linux kernel actually makes routing decisions. We'll inspect the routing table, understand how routes are selected, and learn the tools used to troubleshoot routing problems.

By the end of this chapter, you should be able to answer one of the most common networking questions:

Why did Linux send this packet through that interface?

The Linux Routing Table

The Linux kernel maintains one or more routing tables that describe how packets should be forwarded.

Each route represents a destination network and tells the kernel how to reach it.

Display the routing table:

ip route

Example output:

default via 192.168.1.1 dev ens18 proto dhcp metric 100

192.168.1.0/24 dev ens18 proto kernel scope link src 192.168.1.10

10.10.0.0/16 via 192.168.1.254 dev ens18

Each line is a route that the kernel may use when forwarding packets.

Understanding a Route

Consider the following route:

10.10.0.0/16 via 192.168.1.254 dev ens18

It can be read as:

In other words:

To reach the 10.10.0.0/16 network, send packets through interface ens18 to the router at 192.168.1.254.

Not every route contains a gateway.

For example:

192.168.1.0/24 dev ens18

This is a directly connected route.

Linux already knows that every host in this network is reachable through ens18, so no intermediate router is required.

Where Do Routes Come From?

Routes can be installed in several ways.

Automatically

When an interface receives an IP address, Linux automatically creates a route for the directly connected network.

For example:

192.168.1.10/24

automatically creates:

192.168.1.0/24 dev ens18

By DHCP

DHCP servers commonly install a default gateway.

For example:

default via 192.168.1.1

This allows the system to communicate with networks outside the local subnet.

Statically

Administrators can manually add routes.

Example:

sudo ip route add 10.20.0.0/16 via 192.168.1.254

Delete it:

sudo ip route del 10.20.0.0/16

Unless persisted through your network management software, these changes are temporary and disappear after a reboot.

Longest Prefix Match

One of the most important routing rules is Longest Prefix Match.

Suppose the routing table contains:

10.0.0.0/8
10.10.0.0/16
10.10.10.0/24

Now the destination is:

10.10.10.42

All three routes match.

Linux chooses the most specific route:

10.10.10.0/24

The kernel always prefers the route with the longest matching network prefix.

This behavior is independent of the order in which the routes appear in the routing table.

Route Metrics

Sometimes multiple routes have exactly the same destination prefix.

For example:

default via 192.168.1.1 dev ens18 metric 100

default via 10.0.0.1 dev ens19 metric 200

Since both routes are equally specific, Linux compares their metrics.

A lower metric is preferred.

In this example:

metric 100

is selected over:

metric 200

Metrics are commonly assigned automatically by DHCP clients, NetworkManager, or systemd-networkd, but they can also be configured manually.

Route Scope

Some routes include a scope.

Example:

192.168.1.0/24 dev ens18 scope link

The scope describes where a route is considered valid.

The most common values are:

Most administrators rarely configure scope manually, but understanding it helps when reading routing tables.

Finding the Selected Route

One of the most useful commands in Linux networking is:

ip route get <destination>

For example:

ip route get 8.8.8.8

Example output:

8.8.8.8 via 192.168.1.1 dev ens18 src 192.168.1.10

This command shows exactly how Linux would route traffic to the specified destination.

It answers several important questions:

• Which interface will be used?
• Which gateway will receive the packet?
• Which source IP address will be chosen?

When troubleshooting routing issues, this command is often more useful than simply viewing the routing table.

Source Address Selection

A Linux system may have multiple IP addresses.

For example:

ens18
    192.168.1.10/24
    192.168.1.20/24

or even multiple interfaces:

ens18
    192.168.1.10

ens19
    10.10.0.10

When sending a packet, Linux must also choose the source IP address.

The selected source address usually belongs to the outgoing interface and may be influenced by the route itself.

You can see the selected source address with:

ip route get 8.8.8.8

The output includes:

8.8.8.8 via 192.168.1.1 dev ens18 src 192.168.1.10

This is the address remote systems will see as the sender.

We'll revisit source address selection later when discussing policy routing and VRFs.

The Local Routing Table

Linux maintains a special routing table for addresses that belong to the local system.

Display it with:

ip route show table local

Example:

local 192.168.1.10 dev lo
broadcast 192.168.1.255 dev ens18

These routes are managed automatically by the kernel and ensure that packets destined for local addresses are delivered internally instead of being transmitted on the network.

Monitoring Routing Changes

To observe routing updates in real time:

ip monitor route

Now make a routing change or disconnect a network interface.

You'll immediately see updates as the kernel modifies its routing tables.

This command is particularly useful when troubleshooting DHCP, VPNs, or dynamic routing changes.

Common Troubleshooting Commands

The following commands should become part of your daily toolkit.

Display all routes:

ip route

Display detailed interface addresses:

ip addr

Determine how Linux would reach a destination:

ip route get 1.1.1.1

Show the local routing table:

ip route show table local

Monitor routing changes:

ip monitor route

These commands answer the majority of routing questions encountered on Linux systems.

Looking Ahead

By the end of this chapter, Linux has determined:

• The destination network.
• The best matching route.
• The outgoing interface.
• The next-hop gateway.
• The source IP address.

However, one question still remains before the packet can leave the machine:

How does Linux actually transmit the packet through the selected interface?

The answer lies in the neighbor table, where Linux resolves IP addresses to MAC addresses before constructing the Ethernet frame. In the next chapter, we'll begin observing packets on a running Linux system and watch this process happen in real time.

Observing Network Traffic on Linux

In the previous chapter, we learned how Linux decides where a packet should go by consulting its routing table.

But routing decisions are only part of the story.

To truly understand Linux networking, you need to observe packets as they move through the system.

One of the most valuable networking skills is the ability to observe what is actually happening on the wire.

When diagnosing networking problems, experienced engineers often follow a simple principle:

Don't guess. Capture the traffic.

In this chapter, we'll learn how to observe network traffic on Linux using industry-standard tools and develop a systematic approach to troubleshooting network issues.

The Linux Networking Toolbox

Linux provides several tools for observing different parts of the networking stack.

Rather than using these tools independently, we'll combine them to understand exactly what Linux is doing.

Packet Capture

Every packet that enters or leaves a Linux network interface can be captured and inspected.

Packet capturing allows you to observe:

• Which hosts are communicating
• Which protocols are in use
• Source and destination IP addresses
• TCP and UDP ports
• DNS queries
• HTTP requests
• TLS handshakes
• ICMP messages
• ARP requests and replies

Instead of relying on assumptions, you can examine the packets themselves.

tcpdump

The most widely used packet capture tool on Linux is tcpdump.

Although graphical tools like Wireshark are excellent for packet analysis, tcpdump is available on nearly every Linux server and is the first tool most administrators reach for when troubleshooting production systems.

Listing Available Interfaces

Before capturing traffic, identify the available interfaces.

tcpdump -D

Example:

1.eth0
2.lo
3.any

You can also use:

ip -br link

to identify interface names.

Capturing Traffic

Capture packets on an interface:

sudo tcpdump -i ens18

Immediately you'll begin seeing packets similar to:

14:35:12 IP 192.168.1.10.51234 > 8.8.8.8.53: UDP
14:35:12 IP 8.8.8.8.53 > 192.168.1.10.51234: UDP

Each line represents a packet captured from the network.

By default, tcpdump resolves hostnames and service names, which can make the output harder to read.

For troubleshooting, it's usually better to disable name resolution.

sudo tcpdump -n -i ens18

The -n option tells tcpdump to display numeric IP addresses instead of performing DNS lookups.

Capturing on All Interfaces

Sometimes you're not sure which interface is carrying the traffic.

Linux provides a special pseudo-interface called any.

sudo tcpdump -i any

This captures traffic from every interface on the system.

It's particularly useful on servers with multiple NICs or virtual networking devices.

Limiting the Number of Packets

Capturing traffic indefinitely isn't always practical.

Capture only ten packets:

sudo tcpdump -c 10 -i ens18

After ten packets, tcpdump exits automatically.

Packet Capture Filters

Without filters, a busy server may generate thousands of packets every second.

Fortunately, tcpdump supports powerful capture filters.

Capture only ICMP:

sudo tcpdump -i ens18 icmp

Only ARP:

sudo tcpdump -i ens18 arp

Only DNS:

sudo tcpdump -i ens18 port 53

Only SSH:

sudo tcpdump -i ens18 port 22

Only traffic to a specific host:

sudo tcpdump -i ens18 host 192.168.1.20

Only incoming traffic:

sudo tcpdump -i ens18 dst host 192.168.1.20

Only outgoing traffic:

sudo tcpdump -i ens18 src host 192.168.1.20

Capture only TCP:

sudo tcpdump -i ens18 tcp

Capture only UDP:

sudo tcpdump -i ens18 udp

Filters can also be combined.

sudo tcpdump -i ens18 tcp port 443

Learning a handful of filters dramatically improves the usefulness of packet captures.

Reading Packet Summaries

Consider this packet:

192.168.1.10.45132 > 8.8.8.8.53: UDP

Reading from left to right:

Understanding this summary is often enough to determine whether traffic is flowing correctly.

Capturing Packet Contents

To inspect packet payloads:

sudo tcpdump -X -i ens18

or

sudo tcpdump -XX -i ens18

These options display both hexadecimal and ASCII representations of the packet.

This is useful when examining protocols such as HTTP or DNS.

Keep in mind that encrypted protocols such as HTTPS or SSH will not display readable application data.

The Neighbor Table

In Part 1, we learned that ARP translates IP addresses into MAC addresses.

Linux stores these mappings in the neighbor table.

Display it:

ip neigh

Example:

192.168.1.1 dev ens18 lladdr 52:54:00:aa:bb:cc REACHABLE
192.168.1.20 dev ens18 lladdr 52:54:00:11:22:33 STALE

Each entry associates an IP address with a MAC address on the local network.

Before transmitting an Ethernet frame, Linux checks this table.

If the required entry already exists, Linux immediately knows the destination MAC address.

Otherwise, it must perform ARP.

Watching ARP in Action

Let's observe this process.

First, clear the neighbor table.

sudo ip neigh flush all

Verify that it is empty.

ip neigh

Now start a packet capture:

sudo tcpdump -n -i ens18 arp

In another terminal, run:

ping 192.168.1.1

You should observe packets similar to:

ARP, Request who-has 192.168.1.1 tell 192.168.1.10

ARP, Reply 192.168.1.1 is-at 52:54:00:aa:bb:cc

Stop the capture and inspect the neighbor table again.

ip neigh

Example:

192.168.1.1 dev ens18 lladdr 52:54:00:aa:bb:cc REACHABLE

Notice the relationship:

1. The neighbor table was empty.
2. Linux transmitted an ARP Request.
3. The router replied.
4. Linux stored the mapping in the neighbor table.
5. The ICMP Echo Request was finally transmitted.

This is exactly the ARP process we studied in Part 1, now observed on a live Linux system.

Observing a DNS Lookup

Let's combine several tools.

Start a packet capture:

sudo tcpdump -n -i ens18 port 53

In another terminal:

dig example.com

You'll observe DNS queries and replies.

Observing a Web Request

Now let's observe a complete TCP connection.

Start:

sudo tcpdump -n -i ens18 tcp port 443

Generate traffic:

curl https://example.com

You'll observe:

• TCP three-way handshake
• TLS handshake packets
• Application data
• Connection teardown

At this point, you're observing the concepts introduced throughout Part 1 in a real Linux system.

Writing Packet Captures to a File

In production environments, packet captures are often saved for later analysis.

Write packets to a file:

sudo tcpdump -i ens18 -w capture.pcap

Stop the capture with Ctrl+C.

The resulting .pcap file can later be analyzed using:

• Wireshark
• tcpdump
• tshark

Saving captures allows you to investigate problems without remaining connected to the affected system.

Reading Capture Files

To inspect a previously captured file:

tcpdump -r capture.pcap

All of the filtering and display options can also be applied when reading capture files.

Common Troubleshooting Examples

Is DNS Working?

sudo tcpdump -n -i ens18 port 53

If you see outgoing DNS queries but no replies, the problem is likely between the client and the DNS server.

Is ARP Working?

sudo tcpdump -n -i ens18 arp

You should observe ARP requests followed by ARP replies.

Repeated ARP requests with no replies usually indicate a Layer 2 connectivity problem.

Is the Server Receiving Requests?

sudo tcpdump -n -i ens18 tcp port 443

If no packets appear, the traffic is probably being blocked or never reaching the server.

Is Traffic Leaving the Server?

sudo tcpdump -n -i ens18 host 192.168.1.20

Watching both requests and responses helps determine whether the problem is inbound, outbound, or both.

Understanding What You're Capturing

One important point is often overlooked.

tcpdump captures packets at the interface where you are listening.

If you capture on:

lo

you'll see loopback traffic.

If you capture on:

ens18

you'll see packets entering and leaving that physical interface.

Later in this book, we'll capture traffic on bridges, veth interfaces, network namespaces, and VXLAN devices.

The same packet may appear on multiple interfaces as it moves through the Linux networking stack.

Understanding where you capture is just as important as understanding what you capture.

Building a Troubleshooting Workflow

Rather than capturing packets randomly, develop a repeatable workflow.

1. Verify the correct interface.
2. Capture traffic with tcpdump.
3. Confirm that packets leave the source.
4. Confirm that replies return.
5. Narrow the capture using filters.
6. Save captures for later analysis if necessary.

Following the same process every time makes troubleshooting faster and more reliable.

Looking Ahead

So far, we've focused on packets moving through network interfaces.

The next question is:

Which application owns these network connections?

In the next chapter, we'll explore Linux sockets and learn how to inspect active TCP and UDP connections using tools such as ss, allowing us to connect observed network traffic to the processes generating it.

Understanding Linux Network Connections

So far in this book, we've focused on the network itself.

We've learned how Linux selects routes, resolves MAC addresses, and transmits packets through network interfaces.

But networking doesn't exist just for the kernel—it exists for applications.

When a web server listens on port 443 or a browser connects to a website, they don't interact with network interfaces directly. Instead, they communicate through an operating system abstraction called a socket.

Understanding sockets is essential for troubleshooting Linux systems because almost every networking problem eventually comes down to answering questions like:

• Which process is listening on this port?
• Is the application accepting connections?
• Is data flowing?
• Why is the connection stuck?
• Is the application keeping up with incoming traffic?

In this chapter, we'll explore how Linux represents network connections and learn how to inspect them using modern Linux tools.

From Packets to Sockets

Consider a web server.

          Client
             │
             │ TCP
             ▼
      Network Interface
             │
             ▼
      Linux Networking Stack
             │
             ▼
          Socket
             │
             ▼
        Web Server

Packets arrive through a network interface.

The Linux networking stack processes them.

Eventually, the payload is delivered to a socket owned by an application.

Similarly, when an application sends data, it writes to a socket.

The kernel handles everything else.

What Is a Socket?

A socket is an endpoint for network communication.

Applications do not send Ethernet frames or IP packets directly.

Instead, they perform operations such as:

• socket()
• bind()
• listen()
• accept()
• connect()
• send()
• recv()

These system calls interact with sockets managed by the Linux kernel.

You don't need to be a programmer to troubleshoot sockets, but understanding this abstraction makes many networking tools easier to understand.

Socket Types

Linux supports several socket families and types.

The most common are:

In this chapter, we'll focus primarily on TCP and UDP sockets.

Inspecting Connections with ss

The standard tool for inspecting sockets is ss.

Display all TCP connections:

ss -t

Display all UDP sockets:

ss -u

Display both listening and established sockets:

ss -tun

Unlike the older netstat utility, ss communicates directly with the kernel and is significantly faster on busy systems.

Understanding ss Output

Example:

State      Recv-Q Send-Q Local Address:Port      Peer Address:Port

ESTAB      0      0      192.168.1.10:22         192.168.1.20:52144

Each column has a specific meaning.

Understanding these fields is often enough to diagnose many networking problems.

Listening Sockets

Servers create listening sockets while waiting for clients.

Display listening sockets:

ss -ltn

Example:

LISTEN 0 4096 0.0.0.0:443

This means:

• TCP socket
• Listening
• Port 443
• Accepting connections on every IPv4 interface

To include process information:

sudo ss -ltnp

Example:

LISTEN 0 4096 0.0.0.0:443 users:(("nginx",pid=2145))

Now we know exactly which process owns the socket.

Receive Queue (Recv-Q)

One of the most misunderstood fields in ss is Recv-Q.

Recv-Q represents data that has already arrived from the network but has not yet been read by the application.

For example:

State     Recv-Q 
ESTAB     20480  

This indicates:

• Packets have successfully reached the server.
• The kernel has buffered the data.
• The application has not consumed it yet.

A growing receive queue often suggests that:

• The application is slow.
• The application is blocked.
• The application is overloaded.

The network is usually not the problem.

Send Queue (Send-Q)

Send-Q represents data that the application has written to the socket but has not yet been fully transmitted or acknowledged.

Possible causes include:

• Network congestion
• Slow receivers
• Packet loss
• Small TCP receive windows

A constantly increasing Send-Q deserves investigation.

Unlike Recv-Q, a large Send-Q often points toward network conditions rather than application performance.

Suppose we observe:

State     Recv-Q    Send-Q 
ESTAB     524288    0

Interpretation:

• Clients are sending data.
• Linux is receiving it.
• The application is not reading it fast enough.

Now consider:

State     Recv-Q    Send-Q 
ESTAB     0        524288

Interpretation:

• The application is producing data.
• The peer is not receiving it quickly enough.
• The network or receiver may be the bottleneck.

Learning to interpret these two numbers is an essential troubleshooting skill.

Displaying Detailed Socket Information

For more information:

ss -ti

Example output includes:

• Congestion control algorithm
• RTT (Round Trip Time)
• Retransmissions
• Receive window
• Send window
• MSS
• Congestion window

These statistics are extremely useful when diagnosing TCP performance issues.

Monitoring Connections

Watch connections in real time:

watch -n 1 ss -tun

This refreshes the connection table every second.

It's useful for observing:

• Incoming client connections
• Connection spikes
• Short-lived connections
• Idle sessions

Identifying Which Process Owns a Connection

One of the most common troubleshooting tasks is determining which process is using a port.

sudo ss -tunp

Example:

ESTAB

users:(("postgres",pid=1523))

Now you know:

• Process name
• Process ID
• Connection details

This is often faster than searching through process lists manually.

Inspecting UNIX Sockets

Not all sockets communicate over a network.

Many Linux services communicate locally using UNIX domain sockets.

Display them:

ss -xl

Example:

u_str LISTEN /run/systemd/private

UNIX sockets avoid network overhead and are widely used by:

• Docker
• systemd
• PostgreSQL
• MySQL
• Redis
• containerd

Although they don't use IP addresses, they share the same socket abstraction.

Common Troubleshooting Questions

Is the application listening?

ss -ltn

Which process owns port 443?

sudo ss -ltnp | grep :443

Is the connection established?

ss -tan

Is the application reading data?

Check Recv-Q.

Is the network preventing data from being delivered?

Check Send-Q.

Is the connection closing correctly?

Inspect the TCP state.

A Practical Troubleshooting Workflow

When investigating an application connectivity problem:

1. Verify the service is listening.
2. Confirm the correct process owns the socket.
3. Verify the TCP connection reaches the ESTABLISHED state.
4. Inspect Recv-Q and Send-Q.
5. Capture packets with tcpdump if necessary.
6. Verify routing and neighbor resolution if packets are missing.

Notice how every chapter in this part builds upon the previous one.

Looking Ahead

So far we've explored Linux networking on a single host.

We've seen interfaces, routing tables, neighbor tables, packet captures, and sockets.

In the next chapter, we'll bring these concepts together by building a small Linux network from scratch. We'll create multiple hosts, configure addressing and routing, generate traffic, and use the tools we've learned to observe every step of the communication process.

Build a Small Linux Network

Throughout Part 2, we've explored the fundamental building blocks of Linux networking.

We've learned how to inspect network interfaces, understand routing decisions, observe packets, and inspect network connections.

Now it's time to put everything together.

In this chapter, we'll build a small Linux network using virtual machines (or physical hosts) and use the tools from previous chapters to observe every step of the communication process.

The goal is not simply to make the network work, but to understand how Linux behaves while it is working.

The Network Topology

We'll build a simple network consisting of two Linux hosts connected through a router.

                  10.0.1.0/24
+---------+      +---------+      +---------+
| Host A  |------| Router  |------| Host B  |
|10.0.1.10|      |         |      |10.0.2.10|
+---------+      |         |      +---------+
                 |         |
                 |         |
                 +---------+
                10.0.2.0/24

This topology is intentionally simple.

It allows us to observe nearly every networking concept introduced in Part 2.

What You'll Need

You can complete this chapter using:

• A virtualization platform such as VMware, VirtualBox, KVM, or Hyper-V
• Three virtual machines

The Linux distribution is not important.

The examples use the iproute2 tools available on virtually every modern Linux distribution.

Configure the Network

Assign the following addresses.

Configure the addresses:

sudo ip addr add 10.0.1.10/24 dev eth0

Adjust the commands for each system as needed.

Verify:

ip addr

Configure the Default Gateway

Host A:

sudo ip route add default via 10.0.1.1

Host B:

sudo ip route add default via 10.0.2.1

Verify:

ip route

The output should include a default route pointing to the router.

Enable Packet Forwarding

By default, Linux does not forward packets between interfaces.

Enable forwarding on the router:

sudo sysctl -w net.ipv4.ip_forward=1

Verify:

sysctl net.ipv4.ip_forward

Expected output:

net.ipv4.ip_forward = 1

At this point, the router is capable of forwarding packets between the two networks.

Verify Local Connectivity

Before testing routing, verify that each host can reach its local gateway.

From Host A:

ping 10.0.1.1

From Host B:

ping 10.0.2.1

If these tests fail, troubleshoot the local network before proceeding.

Verify End-to-End Connectivity

Now test communication across both networks.

From Host A:

ping 10.0.2.10

If everything is configured correctly, the ping should succeed.

At this point, Linux has performed every operation we've discussed so far:

• Selected a route.
• Determined the outgoing interface.
• Resolved the router's MAC address.
• Generated ICMP packets.
• Forwarded packets through the router.
• Delivered the reply back to the source.

Observe the Routing Decision

Before sending traffic, ask Linux how it plans to reach Host B.

On Host A:

ip route get 10.0.2.10

Example:

10.0.2.10 via 10.0.1.1 dev eth0 src 10.0.1.10

Notice that Linux does not attempt to reach Host B directly.

Instead, it sends packets to the router.

Observe the Neighbor Table

Before the first ping:

ip neigh

You may see no entry for the router.

Now execute:

ping 10.0.2.10

Check again:

ip neigh

You'll now find an entry similar to:

10.0.1.1 dev eth0 lladdr 52:54:00:aa:bb:cc REACHABLE

Notice that the neighbor table contains the MAC address of the router, not Host B.

This reinforces an important concept from Part 1:

Ethernet frames are always addressed to the next hop, not necessarily the final destination.

Observe the Packets

Start a packet capture on Host A.

sudo tcpdump -n -i eth0

Run another ping.

You'll observe:

• ARP (if necessary)
• ICMP Echo Request
• ICMP Echo Reply

Repeat the capture on the router.

Notice that the router receives the packet on one interface and forwards it through another.

Finally, capture traffic on Host B.

Watching the same packet traverse multiple systems is one of the best ways to understand packet forwarding.

Observe Interface Statistics

Before generating traffic:

ip -s link

Generate several pings.

Run the command again.

Notice how:

• RX counters increase.
• TX counters increase.

On the router, both interfaces should show activity because packets arrive on one interface and leave through the other.

Observe Connections

Although ping uses ICMP instead of TCP, we can still generate TCP traffic.

Start a simple web server on Host B.

python3 -m http.server 8080

From Host A:

curl http://10.0.2.10:8080

Inspect the socket:

ss -tn

Observe:

• Listening socket
• Established connection
• Local and remote addresses

Everything we've learned about sockets now becomes visible.

Trace the Complete Packet Journey

Let's summarize what happened when Host A connected to Host B.

Application
      │
      ▼
Socket
      │
      ▼
Routing Table Lookup
      │
      ▼
Neighbor Table Lookup
      │
      ▼
Ethernet Frame Created
      │
      ▼
Outgoing Interface
      │
      ▼
Router
      │
      ▼
Forwarding Decision
      │
      ▼
Host B
      │
      ▼
Application

This is the complete path that every packet follows through the Linux networking stack at a high level.

The remaining chapters of this book will examine each stage in greater detail.

Troubleshooting Checklist

If connectivity fails, work through the following checklist.

1. Is the interface up?

ip link

2. Does the interface have the correct IP address?

ip addr

3. Is the routing table correct?

ip route

4. Does the neighbor table contain the expected MAC address?

ip neigh

5. Are packets reaching the interface?

tcpdump -n -i eth0

6. Are interface counters increasing?

ip -s link

7. Is packet forwarding enabled on the router?

sysctl net.ipv4.ip_forward

This workflow should become second nature. It follows the same methodology introduced throughout Part 2 and applies equally well to simple laboratory environments and production systems.

Summary

In this chapter, we combined the concepts from Part 2 into a working Linux network.

You observed how Linux:

• Configures network interfaces.
• Chooses routes.
• Resolves neighbors using ARP.
• Generates and captures packets.
• Forwards traffic through a router.
• Tracks interface statistics.
• Creates and manages network connections.

At this point, you have a solid understanding of Linux networking on a single network and across multiple subnets.

In Part 3, we'll begin replacing physical cables and switches with virtual networking components. You'll learn how Linux builds entirely virtual networks using namespaces, virtual Ethernet devices, bridges, and other kernel networking primitives—the same technologies that underpin containers, virtualization platforms, and many cloud networking solutions.

Why Linux Needs Virtual Networking

Modern Linux systems rarely communicate using only physical network cards. Containers, virtual machines, Kubernetes clusters, cloud platforms, VPNs, and software-defined networks all depend on virtual networking. Before learning how Linux implements these technologies, we first need to understand why they exist.

Introduction

So far in this book, we've focused on how a Linux host communicates with the outside world using physical network interfaces. Every example assumed a simple model:

Application
      │
 TCP/IP Stack
      │
Network Interface (eth0)
      │
 Ethernet
      │
   Physical Network

This model works perfectly for traditional servers.

However, modern infrastructure rarely looks like this.

A Kubernetes node may run dozens of containers. A virtualization host may run hundreds of virtual machines. A cloud server may have multiple isolated tenant networks. A laptop may simultaneously run Docker, Podman, VirtualBox, WireGuard, and a VPN client.

None of these workloads own a physical network card.

Yet every one of them behaves as if it has its own network.

How is that possible?

The answer is virtual networking.

The Problem

Imagine a Linux server with one physical NIC:

          +----------------------+
          |      Linux Host      |
          |                      |
          |        eth0          |
          +----------+-----------+
                     |
               Physical Switch

Now suppose you want to run five applications that should be isolated from one another.

For example:

• Production API
• Development API
• Testing environment
• Monitoring stack
• Database

If every application shares the same network namespace, then:

• every process sees the same interfaces
• every process shares the same routing table
• every process shares the same firewall
• every process shares the same ARP cache

This becomes difficult to manage.

Worse, applications may accidentally interfere with one another.

Containers would not be isolated.

Virtual machines would not be isolated.

Multi-tenant environments would be impossible.

Linux needed a better solution.

The Old Approach: Physical Separation

Before virtualization became common, isolation usually meant buying more hardware.

Server A
  eth0
    │
 Switch

Server B
  eth0
    │
 Switch

Server C
  eth0
    │
 Switch

Every workload received:

• its own CPU
• its own RAM
• its own storage
• its own network card

This worked, but it was:

• expensive
• inefficient
• difficult to scale

Most servers used only a fraction of their available resources.

Virtualization changed everything.

One Physical Machine, Many Virtual Machines

Instead of buying five servers, we can run five virtual machines.

             Linux Host

        +--------------------+
        |      Hypervisor    |
        +--------------------+
        | VM1                |
        | VM2                |
        | VM3                |
        | VM4                |
        | VM5                |
        +--------------------+
                |
             eth0

Each VM believes it owns:

• a CPU
• memory
• disks
• network adapters

But none of those network adapters are physical.

They are entirely virtual.

Linux creates software devices that behave exactly like real Ethernet adapters.

Containers Need the Same Thing

Containers are lighter than virtual machines.

They share the host kernel.

But they still require isolation.

Imagine running:

nginx
redis
postgres
prometheus
grafana

If every container shared one network stack:

• identical ports could not be reused
• firewall rules would affect everyone
• routing changes would affect everyone

Instead, Linux gives each container its own networking environment.

To the container, it looks like this:

+--------------------------------+
|          Linux Kernel          |
|                                |
|  Interfaces                    |
|    lo                          |
|    eth0                        |
|                                |
|  Routing Table                 |
|  Neighbor Table                |
|  Firewall Rules                |
|  Listening Sockets             |
+--------------------------------+

Even though everything exists only inside one kernel.

Virtual Networking Is Everywhere

Many engineers think virtual networking only matters for Docker.

In reality, nearly every modern platform depends on it.

• Docker
• Podman
• Kubernetes
• LXC/LXD
• KVM
• QEMU
• VMware
• Hyper-V
• Cloud platforms
• VPN software

If you work with DevOps, cloud infrastructure, or Kubernetes, you interact with virtual networking every day—even if you never configure it directly.

Linux Doesn't Emulate Networking

One surprising fact is that Linux doesn't create a "fake" networking system for containers and virtual machines.

Instead, Linux uses the exact same networking stack you've already learned.

• TCP is still TCP.
• IP is still IP.
• ARP still works.
• Routing still works.
• Netfilter still works.

The kernel simply creates more network devices.

From the kernel's perspective, a virtual Ethernet interface and a physical Ethernet interface are both network devices.

• They both send packets.
• They both receive packets.
• They both appear in the routing table.

This design is one of Linux's greatest strengths.

Everything you've learned so far continues to apply.

The Building Blocks of Linux Virtual Networking

Linux virtual networking is built from a surprisingly small set of primitives.

             +-----------------------+
             | Network Namespace     |
             +-----------+-----------+
                         |
                   Virtual Ethernet
                         |
                 Linux Bridge / OVS
                         |
                  Routing & Netfilter
                         |
                      Physical NIC

Throughout this part of the book, we'll study each piece individually.

Network Namespaces

Namespaces create independent networking environments.

Each namespace has its own:

• interfaces
• routing table
• neighbor table
• firewall rules
• sockets

To applications inside the namespace, it appears to be an independent Linux machine.

Virtual Ethernet (veth)

A virtual Ethernet pair behaves like an Ethernet cable.

vethA  <==========>  vethB

Anything entering one side immediately exits the other.

Linux uses veth pairs to connect:

• containers
• namespaces
• virtual switches
• routers

Linux Bridge

A Linux bridge acts like a software Ethernet switch.

        Bridge
     +-----------+
      |   |   |
     v1  v2  eth0

It learns MAC addresses exactly like a physical switch.

This allows multiple virtual machines or containers to communicate as if connected to the same Layer 2 network.

Routing

Sometimes workloads belong to different networks.

In that case, Linux routes packets between them.

Exactly the same routing principles from earlier chapters apply.

The only difference is that many interfaces are virtual.

Netfilter

Every packet may also pass through Netfilter.

This allows Linux to:

• filter traffic
• perform NAT
• track connections
• implement Kubernetes networking
• enforce container isolation

Again, this is the same Netfilter framework you'll study later in this book.

Why Learn These Pieces Separately?

When beginners first encounter Docker or Kubernetes networking, it often feels like magic.

A single command creates a container:

docker run nginx

Suddenly:

• an interface appears
• an IP address exists
• DNS works
• routing works
• NAT works
• internet access works

It looks like Docker invented an entire networking stack.

It didn't.

Docker simply automated the assembly of Linux networking primitives.

The same is true for:

• Podman
• Kubernetes
• containerd
• CRI-O
• libvirt
• LXC

Once you understand the underlying Linux primitives, these higher-level platforms become much easier to understand and troubleshoot.

What You'll Build in This Part

Unlike the previous parts of the book, this section is highly practical.

You'll build everything yourself.

Starting with empty Linux namespaces, you'll gradually assemble increasingly sophisticated virtual networks.

By the end of this part, you will have built:

• isolated network namespaces
• virtual Ethernet links
• software switches
• Linux routers
• VLAN-aware networks
• TUN/TAP interfaces
• complete multi-host virtual topologies

Most importantly, you'll understand why each component exists and how packets move through the system.

Key Takeaways

• Modern Linux systems rely heavily on virtual networking to support containers, virtual machines, and cloud infrastructure.
• Virtual networking provides isolation without requiring additional physical hardware.
• Linux uses the same networking stack for physical and virtual interfaces; there is no separate "container networking" stack.
• A small set of kernel primitives—network namespaces, virtual Ethernet devices, bridges, routing, and Netfilter—forms the foundation of most Linux networking technologies.
• Tools such as Docker, Kubernetes, Podman, and KVM build on these primitives rather than replacing them.
• Understanding these building blocks makes complex networking platforms significantly easier to understand, configure, and debug.

Coming Next

In the next chapter, we'll begin with the most fundamental building block of Linux virtual networking: Network Namespaces. You'll learn how a single Linux kernel can host multiple isolated network stacks, each with its own interfaces, routing table, neighbor table, and sockets, making one machine behave like many independent systems.

Network Namespaces: Multiple Networks on One Linux Host

Network namespaces are one of the most powerful features in the Linux kernel. They allow a single Linux system to host multiple independent network stacks, making one machine behave like many separate computers. Every container runtime relies on this feature.

Learning Objectives

By the end of this chapter, you will be able to:

• Understand what a network namespace is
• Explain why network namespaces exist
• Create and delete network namespaces
• Execute commands inside namespaces
• Move interfaces between namespaces
• Understand what resources are isolated
• Observe how namespaces change the kernel's view of networking
• Troubleshoot namespaces using standard Linux tools

The Problem

Consider a normal Linux system.

+--------------------------------+
|          Linux Kernel          |
|                                |
|  Interfaces                    |
|    lo                          |
|    eth0                        |
|                                |
|  Routing Table                 |
|  Neighbor Table                |
|  Firewall Rules                |
|  Listening Sockets             |
+--------------------------------+

Every process on the system shares the same network stack.

If one process adds a route:

ip route add 10.100.0.0/24 via 192.168.1.1

every process immediately sees that route.

If one process starts listening on port 80:

python3 -m http.server 80

no other process can also listen on TCP port 80 using the same IP address.

Everything shares the same networking environment.

For traditional servers, this is exactly what we want.

For containers and virtual environments, it is not.

Why Containers Need Isolation

Imagine running three web applications.

Host

Application A
Application B
Application C

Each application expects to listen on:

TCP port 80

Without isolation:

Application A → Port 80 ✓

Application B → Port 80 ✗
Address already in use

Application C → Port 80 ✗
Address already in use

Only one application can own the port.

Now imagine each application has its own network stack.

Namespace A
------------
eth0
lo
Port 80

Namespace B
------------
eth0
lo
Port 80

Namespace C
------------
eth0
lo
Port 80

Now every application can use port 80 simultaneously because they are listening inside different network namespaces.

This is exactly how containers work.

What Is a Network Namespace?

A network namespace is an isolated networking environment managed by the Linux kernel.

Each namespace has its own:

• network interfaces
• routing table
• neighbor (ARP/NDP) table
• firewall rules
• connection tracking table
• sockets
• /proc/net
• network-related sysctls (many are namespace-specific)

A process inside one namespace cannot directly see the networking resources of another namespace.

Think of a network namespace as giving a process its own private networking world.

One Kernel, Multiple Network Stacks

The important thing to understand is that namespaces do not create multiple kernels.

There is still only one Linux kernel.

                    Linux Kernel

       +-------------------------------+
       |                               |
       | Namespace A                   |
       | Namespace B                   |
       | Namespace C                   |
       | Namespace D                   |
       |                               |
       +-------------------------------+

The kernel simply maintains multiple independent networking contexts.

Every packet is processed by the same kernel, but inside the correct namespace.

The Default Namespace

When Linux boots, it automatically creates one network namespace.

This is called the initial namespace (sometimes informally called the root namespace).

Unless you explicitly create another namespace, every process runs here.

+--------------------------------+
|       Initial Namespace        |
|                                |
|  Interfaces                    |
|    lo                          |
|    eth0                        |
|                                |
|  Routing Table                 |
|  Neighbor Table                |
|  Firewall Rules                |
|  Listening Sockets             |
+--------------------------------+

Everything you've learned so far in this book has taken place inside this initial namespace.

Listing Network Namespaces

Most Linux systems use the ip command to manage namespaces.

List all named namespaces:

ip netns list

A new installation usually returns nothing:

<no output>

That does not mean no namespace exists.

It simply means no additional named namespaces have been created.

The initial namespace is always present but is not shown by ip netns list.

Creating a Namespace

Create a namespace called web:

sudo ip netns add web

Now list namespaces:

ip netns list

Output:

web

Create another:

sudo ip netns add database

Now:

web
database

Each namespace has its own completely independent networking stack.

Where Are Namespaces Stored?

After creating a namespace:

ip netns add web

look inside:

ls -l /var/run/netns

Example:

web
database

These are bind mounts that allow the namespace to be referenced by name.

The actual namespace object exists inside the Linux kernel.

Running Commands Inside a Namespace

To execute a command inside a namespace:

sudo ip netns exec web ip addr

Output:

1: lo: <LOOPBACK>

Notice something surprising.

There is no eth0.

Only the loopback interface exists.

Every new namespace starts almost empty.

Looking at Interfaces

Compare the host:

ip addr

Example:

lo
eth0

Now compare the namespace:

ip netns exec web ip addr

Example:

lo

The namespace cannot see the host's interfaces.

This is the first demonstration that namespaces isolate networking resources.

Looking at Routes

Host:

ip route

Example:

default via 192.168.1.1
192.168.1.0/24 dev eth0

Namespace:

ip netns exec web ip route

Output:

<no output>

No interfaces means no routes.

Every namespace owns its own routing table.

Looking at Neighbor Entries

Host:

ip neigh

Example:

192.168.1.1 dev eth0 lladdr 00:11:22:33:44:55

Namespace:

ip netns exec web ip neigh

Output:

<no output>

Neighbor tables are isolated as well.

Looking at Listening Sockets

Host:

ss -lnt

Namespace:

ip netns exec web ss -lnt

Initially:

State Recv-Q Send-Q Local Address:Port

No listening sockets exist because no applications are running inside that namespace.

The Loopback Interface

Every namespace contains its own loopback interface.

Initially it is down.

Verify:

ip netns exec web ip addr

Example:

lo: <LOOPBACK>

Bring it up:

sudo ip netns exec web ip link set lo up

Verify:

ip netns exec web ip addr

Now:

lo: <LOOPBACK,UP,LOWER_UP>

Most container runtimes automatically enable loopback.

When working manually, you must remember to do it yourself.

Creating a Process Inside a Namespace

Start a shell:

sudo ip netns exec web bash

Your shell is now running inside the namespace.

Everything executed from this shell stays inside the namespace.

Check:

ip addr

Output:

lo

Exit normally:

exit

Demonstration: Independent Routing Tables

Create another namespace.

sudo ip netns add test

Bring up loopback:

sudo ip netns exec test ip link set lo up

Add a route:

sudo ip netns exec test ip route add blackhole 10.10.10.0/24

Check routes:

ip netns exec test ip route

Output:

blackhole 10.10.10.0/24

Now check the host:

ip route

The route does not exist.

The routing table is completely independent.

Namespace Isolation in Practice

Suppose we have two namespaces.

                 Linux Kernel

+---------------------------------------------+

Namespace A                Namespace B

eth0                       eth0

10.0.0.2                   192.168.10.2

Route A                    Route B

Sockets A                  Sockets B

Firewall A                 Firewall B

+---------------------------------------------+

Although both namespaces run inside the same kernel, they behave like two independent Linux machines.

Neither knows the other's interfaces exist.

What Is Shared?

Namespaces isolate networking, not the entire operating system.

Processes still share:

• the Linux kernel
• CPU
• physical memory
• filesystem (unless other namespaces are used)
• kernel modules
• scheduler

Only networking resources are separated.

What Is Not Shared?

Each namespace has its own:

• interfaces
• IP addresses
• routing table
• neighbor table
• ARP cache
• sockets
• firewall rules
• connection tracking
• multicast memberships
• /proc/net

This isolation is what makes containers practical.

Cleaning Up

Delete namespaces:

sudo ip netns delete web
sudo ip netns delete database
sudo ip netns delete test

Verify:

ip netns list

Output:

<no output>

Common Mistakes

Forgetting to Enable Loopback

Many commands fail because lo is still down.

Always run:

ip netns exec <namespace> ip link set lo up

Expecting eth0 to Exist

A newly created namespace contains no physical interfaces.

Interfaces must either be:

• moved into the namespace, or
• created inside it.

We'll learn both approaches in the next chapters.

Expecting Internet Access

A namespace has no connectivity by default.

Creating a namespace does not connect it to anything.

It is simply an isolated networking environment.

Key Takeaways

• A network namespace provides an isolated network stack inside the Linux kernel.
• The initial namespace is created automatically during boot and contains the host's normal networking configuration.
• Every namespace has its own interfaces, routing table, neighbor table, sockets, firewall rules, and connection tracking state.
• Newly created namespaces contain only a disabled loopback interface.
• Processes running in different namespaces can safely use the same IP addresses and port numbers because they belong to different networking contexts.
• Creating a namespace does not provide connectivity. Interfaces and links must be added explicitly.

Coming Next

A network namespace is an isolated networking environment, but by itself it cannot communicate with anything—not even another namespace.

In the next chapter, we'll introduce the Virtual Ethernet (veth) pair, one of the most fundamental Linux networking primitives. A veth pair acts like an Ethernet cable inside the kernel, allowing two namespaces to exchange packets as if they were connected by a physical wire. Nearly every container platform relies on this mechanism to connect isolated network namespaces to the outside world.

Virtual Ethernet (veth): The Cable Inside Linux

A network namespace is like a computer with no network cable plugged in. A virtual Ethernet (veth) pair is that cable. It is the simplest and most important building block of Linux virtual networking, forming the foundation of Docker, Kubernetes, Podman, LXC, and many other technologies.

Learning Objectives

By the end of this chapter, you will be able to:

• Understand what a virtual Ethernet (veth) pair is
• Explain how packets travel through a veth pair
• Create and inspect veth interfaces
• Connect two network namespaces using a veth pair
• Assign IP addresses and exchange packets
• Observe packet counters and interface states
• Troubleshoot common veth issues

The Problem

In the previous chapter, we created network namespaces.

Host

Namespace A

lo

Host

Namespace B

lo

• Each namespace had its own network stack.
• Each namespace had its own routing table.
• Each namespace had its own sockets.

But there was one major problem.

They couldn't communicate with anything.

They were like two computers sitting on a desk with no Ethernet cables.

What Is a veth Pair?

A virtual Ethernet pair consists of two virtual network interfaces permanently connected together.

Think of it as an Ethernet cable with exactly two ends.

+---------+======================+---------+
| vethA   |                      | vethB   |
+---------+======================+---------+

Anything transmitted into one end immediately appears at the other end.

Unlike a physical Ethernet cable:

• there is no switch
• there is no hub
• there is no routing
• there is no packet modification

The Linux kernel simply copies frames from one interface to its peer.

Understanding the "Pair"

A veth interface can never exist by itself.

Creating one interface automatically creates its peer.

vethA  <----------------->  vethB

Delete one interface:

ip link delete vethA

Both interfaces disappear.

The pair is treated as one logical object by the kernel.

How Packets Flow

Suppose a process sends an ICMP Echo Request.

Namespace A

Application
      │
      ▼
   TCP/IP Stack
      │
      ▼
    vethA
      │
====================
Virtual Cable
====================
      │
    vethB
      │
      ▼
   TCP/IP Stack
      │
Application

Namespace B

The packet never leaves the Linux kernel.

No physical NIC is involved.

No switch is involved.

No external network is involved.

Everything happens entirely in memory.

This makes veth communication extremely fast.

Creating a veth Pair

Create a pair:

sudo ip link add vethA type veth peer name vethB

Verify:

ip link show

Example:

...
10: vethA@if11: <BROADCAST,MULTICAST>
11: vethB@if10: <BROADCAST,MULTICAST>
...

Notice the peer notation:

vethA@if11
vethB@if10

Each interface knows the index of its peer.

Inspecting the Interfaces

Display detailed information:

ip -d link show vethA

Example:

vethA@if11
    link/ether 8a:5d:7d:...
    veth

Notice the interface type:

veth

Unlike eth0, which is driven by a hardware device driver, these interfaces exist entirely in software.

Bringing the Interfaces Up

Initially the interfaces are down.

Check:

ip link show vethA

Bring them up:

sudo ip link set vethA up
sudo ip link set vethB up

Verify:

ip link show vethA

Example:

UP,LOWER_UP

Unlike physical interfaces, LOWER_UP simply means the peer is present and operational.

Assigning IP Addresses

Assign addresses:

sudo ip addr add 10.10.10.1/24 dev vethA
sudo ip addr add 10.10.10.2/24 dev vethB

Verify:

ip addr show vethA

Example:

inet 10.10.10.1/24

Testing Connectivity

Ping the peer:

ping 10.10.10.2

Output:

64 bytes from 10.10.10.2

The packet never touched a physical network.

It traveled only through the kernel.

Watching Packet Counters

Display statistics:

ip -s link show vethA

Example:

RX:
    packets 120

TX:
    packets 120

Run another ping.

Display the counters again.

The numbers increase immediately.

Packet counters are one of the easiest ways to verify traffic is flowing.

Connecting Network Namespaces

Now we'll connect two isolated namespaces.

Create them:

sudo ip netns add blue
sudo ip netns add red

Enable loopback:

sudo ip netns exec blue ip link set lo up
sudo ip netns exec red ip link set lo up

Create the veth Pair

sudo ip link add blue-veth type veth peer name red-veth

Current layout:

Host

blue-veth <-------> red-veth

Both interfaces still belong to the host namespace.

Move Each Interface

Move one interface into each namespace.

sudo ip link set blue-veth netns blue
sudo ip link set red-veth netns red

Now the topology becomes:

Namespace blue

blue-veth
      ||
====================
Virtual Ethernet
====================
      ||
red-veth

Namespace red

This is one of the most common operations performed by container runtimes.

Configure the Interfaces

Inside blue:

sudo ip netns exec blue ip addr add 10.0.0.1/24 dev blue-veth
sudo ip netns exec blue ip link set blue-veth up

Inside red:

sudo ip netns exec red ip addr add 10.0.0.2/24 dev red-veth
sudo ip netns exec red ip link set red-veth up

Verify:

sudo ip netns exec blue ip addr

Example:

lo
blue-veth

Test the Connection

From blue:

sudo ip netns exec blue ping 10.0.0.2

Output:

64 bytes from 10.0.0.2

Reverse direction:

sudo ip netns exec red ping 10.0.0.1

Communication works.

You have connected two isolated Linux systems using nothing but kernel networking primitives.

Observe the Neighbor Table

Before pinging:

sudo ip netns exec blue ip neigh

Output:

<no output>

Generate traffic:

sudo ip netns exec blue ping -c 1 10.0.0.2

Check again:

sudo ip netns exec blue ip neigh

Example:

10.0.0.2 dev blue-veth lladdr 5a:18:df:3b:17:91 REACHABLE

Even though everything is virtual, ARP still works exactly as it does on a physical Ethernet network.

This is an important observation.

The Linux networking stack does not distinguish between physical and virtual Ethernet devices when performing Layer 2 communication.

Observe Packet Counters

Inside the namespace:

sudo ip netns exec blue ip -s link show blue-veth

Generate more traffic:

sudo ip netns exec blue ping -c 5 10.0.0.2

Check again:

sudo ip netns exec blue ip -s link show blue-veth

Both RX and TX counters increase.

Packet accounting works exactly as it does on physical interfaces.

Capturing Traffic

Because a veth interface behaves like a normal Ethernet interface, tools such as tcpdump work without modification.

Capture packets:

sudo ip netns exec blue tcpdump -i blue-veth

From another terminal:

sudo ip netns exec red ping 10.0.0.1

You will observe:

• ARP requests
• ARP replies
• ICMP Echo Requests
• ICMP Echo Replies

From tcpdump's perspective, this is just another Ethernet interface.

The Peer Relationship

The peer relationship is visible through ip link.

Example:

blue-veth@if8

The @if8 indicates the interface index of its peer.

Likewise:

red-veth@if7

This makes it easy to identify connected interfaces while troubleshooting.

What Happens If One Side Goes Down?

Bring one interface down:

sudo ip netns exec red ip link set red-veth down

Now inspect the other side:

sudo ip netns exec blue ip link show blue-veth

You'll notice that carrier state changes.

Attempting to ping now fails because the virtual cable is effectively unplugged.

This behavior closely matches a physical Ethernet cable.

Cleaning Up

Delete the namespaces:

sudo ip netns delete blue
sudo ip netns delete red

Notice that the veth interfaces disappear automatically.

Why?

Because they belonged to the namespaces that were deleted.

No manual cleanup is required.

Where veth Is Used

The veth driver is one of the most widely used networking components in Linux.

Examples include:

• Docker
• Podman
• Kubernetes
• containerd
• CRI-O
• LXC/LXD
• CNI plugins

Whenever a container receives an eth0 interface, one end of a veth pair is typically inside the container's network namespace, while the other end remains in the host namespace or is attached to a software switch.

Common Mistakes

Forgetting to Bring Interfaces Up

Creating a veth pair does not automatically enable it.

Always execute:

ip link set <interface> up

Forgetting Loopback

A namespace usually needs its loopback interface enabled.

ip netns exec <namespace> ip link set lo up

Moving the Wrong Interface

Once an interface is moved into another namespace, it disappears from the host's interface list.

To inspect or configure it afterward, use:

ip netns exec <namespace> ...

Assuming veth Performs Routing

A veth pair is only a Layer 2 point-to-point link.

It does not:

• route packets
• switch frames
• perform NAT
• filter traffic

Those responsibilities belong to other Linux networking components.

Key Takeaways

• A veth pair is a pair of virtual Ethernet interfaces permanently connected inside the Linux kernel.
• Packets transmitted on one interface immediately appear on its peer without leaving the host.
• Veth interfaces behave like normal Ethernet interfaces and fully participate in ARP, IP, ICMP, and other networking protocols.
• One end of a veth pair is commonly placed inside a network namespace while the other remains in the host namespace or connects to a software switch.
• Container runtimes such as Docker, Podman, Kubernetes, and LXC use veth pairs to connect isolated network namespaces.
• A veth pair provides only a direct Layer 2 connection; it does not perform switching, routing, or firewalling.

Coming Next

A veth pair connects exactly two endpoints. But what if you need to connect ten containers, twenty virtual machines, or hundreds of workloads to the same Layer 2 network?

In the next chapter, we'll introduce the Linux bridge, a software implementation of an Ethernet switch. You'll learn how Linux forwards Ethernet frames between multiple interfaces, builds MAC address tables, and forms the virtual networks used by containers and virtualization platforms.

Linux Bridges: Building a Software Switch

A virtual Ethernet (veth) pair can connect exactly two endpoints. But modern systems rarely contain only two network devices. Containers, virtual machines, and network namespaces often need to communicate with many other devices on the same Layer 2 network. This is the job of a Linux bridge—a software implementation of an Ethernet switch.

Learning Objectives

By the end of this chapter, you will be able to:

• Understand what a Linux bridge is
• Explain how a bridge differs from a router
• Create and manage Linux bridges
• Connect multiple network namespaces to a bridge
• Observe MAC address learning and frame forwarding
• Inspect the bridge forwarding database (FDB)
• Troubleshoot bridge connectivity issues
• Understand how Docker, KVM, and Kubernetes use Linux bridges

The Problem

In the previous chapter, we connected two namespaces using a veth pair.

Namespace A
    |
 vethA
==================
 vethB
    |
Namespace B

This works well for two endpoints.

Now suppose we have three namespaces.

Namespace A

Namespace B

Namespace C

Can we connect them using veth pairs?

A <------> B

A <------> C

B <------> C

Technically, yes.

But notice what happens as the number of devices increases.

Four namespaces require six veth pairs.

Five namespaces require ten.

Ten namespaces require forty-five.

The number of required links grows rapidly.

This is not how real Ethernet networks are built.

How Physical Networks Solve This Problem

Imagine connecting ten physical servers.

You would never run an Ethernet cable between every pair of servers.

Instead, you connect each server to a switch.

              Ethernet Switch

          +----------------------+
          |                      |
          +--+--+--+--+--+--+----+
             |  |  |  |  |
             |  |  |  |  |
            PC1 PC2 PC3 PC4 PC5

Each device has one cable.

The switch decides where each frame should go.

Linux provides the exact same concept in software.

What Is a Linux Bridge?

A Linux bridge is a software implementation of a Layer 2 Ethernet switch.

Instead of connecting physical Ethernet ports, it connects Linux network interfaces.

             Linux Bridge

        +-------------------+
        |       br0         |
        +---+----+----+-----+
            |    |    |
          veth1 veth2 eth0

Every connected interface is called a bridge port.

The bridge receives Ethernet frames on one port and forwards them to another based on the destination MAC address.

Bridge vs Router

A bridge operates at Layer 2.

A router operates at Layer 3.

The difference is fundamental.

A bridge extends an Ethernet network.

A router connects different IP networks.

Building Our First Bridge

Create a bridge named br0.

sudo ip link add br0 type bridge

Verify:

ip link show br0

Example:

7: br0: <BROADCAST,MULTICAST>

Bring it up.

sudo ip link set br0 up

Now Linux has a software Ethernet switch.

At this point, however, it has no ports connected.

Creating Two Namespaces

Create two namespaces.

sudo ip netns add blue
sudo ip netns add green

Enable loopback.

sudo ip netns exec blue ip link set lo up

sudo ip netns exec green ip link set lo up

Create Two veth Pairs

Each namespace needs one connection to the bridge.

blue-veth <------> blue-host

green-veth <------> green-host

Create them.

sudo ip link add blue-veth type veth peer name blue-host

sudo ip link add green-veth type veth peer name green-host

Move One End into Each Namespace

Move the namespace side.

sudo ip link set blue-veth netns blue

sudo ip link set green-veth netns green

Current topology:

Namespace blue          Host

blue-veth -------- blue-host

Namespace green

green-veth ------ green-host

The host-side interfaces remain in the initial namespace.

Attach Interfaces to the Bridge

Attach the host interfaces.

sudo ip link set blue-host master br0

sudo ip link set green-host master br0

Verify.

bridge link

Example:

blue-host
green-host

These interfaces are now bridge ports.

Bring Everything Up

Host interfaces:

sudo ip link set blue-host up

sudo ip link set green-host up

Namespace interfaces:

sudo ip netns exec blue ip link set blue-veth up

sudo ip netns exec green ip link set green-veth up

Assign IP Addresses

Blue:

sudo ip netns exec blue ip addr add 192.168.100.10/24 dev blue-veth

Green:

sudo ip netns exec green ip addr add 192.168.100.20/24 dev green-veth

Topology:

          +-------------+
          |    br0      |
          +------+------+ 
                 |
       +---------+---------+
       |                   |
 blue-host           green-host
       |                   |
================   ================
 blue-veth         green-veth
       |                   |
 Namespace blue     Namespace green

Testing Connectivity

Ping from blue.

sudo ip netns exec blue ping 192.168.100.20

Output:

64 bytes from 192.168.100.20

The bridge forwards the Ethernet frames exactly like a physical switch.

What Actually Happened?

Let's follow the first packet.

Application

↓

TCP/IP Stack

↓

blue-veth

↓

blue-host

↓

Linux Bridge

↓

green-host

↓

green-veth

↓

TCP/IP Stack

↓

Application

Notice something important.

The bridge never looked at the IP address.

It forwarded the frame using only Ethernet information.

The Forwarding Database (FDB)

Physical switches maintain a MAC address table.

Linux bridges do the same.

Display it.

bridge fdb show

Example:

52:54:00:11:22:33 dev blue-host

52:54:00:44:55:66 dev green-host

This is called the Forwarding Database (FDB).

It tells the bridge which MAC addresses are reachable through each port.

How MAC Learning Works

Suppose the bridge has just been created.

Initially, its forwarding table is empty.

Bridge

FDB

(empty)

Blue sends the first frame.

Source MAC:

52:54:00:11:22:33

The bridge learns:

52:54:00:11:22:33

↓

blue-host

Later, Green replies.

52:54:00:44:55:66

↓

green-host

Now the table becomes:

MAC Address              Port

52:54:00:11:22:33   blue-host

52:54:00:44:55:66   green-host

From this point onward, the bridge knows exactly where each destination is located.

This process is called MAC learning.

Unknown Unicast Traffic

What happens if the bridge receives a frame for an unknown MAC address?

It floods the frame.

            Unknown Destination

                  Bridge

         +-----+-----+-----+

        Port1 Port2 Port3

Every port except the incoming one receives the frame.

Once the destination replies, the bridge learns where it is.

This behavior is identical to a physical Ethernet switch.

Broadcast Traffic

Broadcast frames are always flooded.

Examples include:

• ARP Requests
• DHCP Discover
• IPv6 Neighbor Discovery (multicast)

When Blue sends an ARP request:

Who has 192.168.100.20?

The bridge forwards that frame to every bridge port except the sender.

Green receives the request and responds.

The bridge then learns Green's MAC address.

Capturing Traffic

Observe the bridge in action.

Terminal 1:

sudo tcpdump -i blue-host

Terminal 2:

sudo tcpdump -i green-host

Terminal 3:

sudo ip netns exec blue ping 192.168.100.20

Observe:

• ARP Request
• ARP Reply
• ICMP Echo Request
• ICMP Echo Reply

The bridge simply forwards Ethernet frames.

Bridge Ports

Show bridge ports.

bridge link

Example:

blue-host

green-host

Detailed information:

bridge -d link

This displays bridge-specific information such as state and flags.

Interfaces Attached to a Bridge

Check interface relationships.

ip link show master br0

Example:

blue-host

green-host

This is useful when troubleshooting large systems.

Assigning an IP Address to the Bridge

A bridge is itself a network interface.

It can have an IP address.

sudo ip addr add 192.168.100.1/24 dev br0

Now the host becomes another device on the same Layer 2 network.

                Host

          192.168.100.1

               br0
                |
     +----------+-----------+
     |                      |
 Namespace blue     Namespace green

This is commonly used when the host must communicate with connected containers or virtual machines.

Where Linux Bridges Are Used

Linux bridges are used extensively throughout modern infrastructure.

• Docker (default bridge network)
• KVM/libvirt
• LXC/LXD
• Network emulation labs
• Some CNI plugins

Although some environments now use Open vSwitch or eBPF-based networking, the Linux bridge remains one of the most important networking components in the kernel.

Common Mistakes

Forgetting to Enable Interfaces

Creating interfaces does not automatically enable them.

Always bring interfaces up.

ip link set <interface> up

Forgetting to Attach Interfaces

Simply creating a veth pair does not connect it to the bridge.

Use:

ip link set <interface> master br0

Assigning IP Addresses to the Wrong Interface

When using a bridge, IP addresses are usually assigned to:

• the namespace interface, or
• the bridge itself.

The host-side veth interfaces normally do not require IP addresses.

Confusing Bridges with Routers

A bridge does not route packets between different IP networks.

If two namespaces belong to different subnets, a router is required.

We'll build one later in this book.

Cleaning Up

Delete the namespaces.

sudo ip netns delete blue

sudo ip netns delete green

Delete the bridge.

sudo ip link delete br0

Deleting the bridge automatically detaches its ports.

Key Takeaways

• A Linux bridge is a software implementation of an Ethernet switch.
• Bridge ports can be physical interfaces, veth interfaces, TAP devices, or other supported network interfaces.
• The bridge forwards Ethernet frames based on destination MAC addresses using its Forwarding Database (FDB).
• Unknown unicast and broadcast traffic are flooded to all bridge ports except the incoming port.
• Linux bridges operate entirely at Layer 2 and do not perform routing or NAT.
• A bridge is itself a network interface and can have an IP address assigned when the host needs to participate in the connected network.
• Docker, KVM, LXC, and many virtualization platforms rely on Linux bridges to interconnect virtual workloads.

Coming Next

So far, we've learned the three fundamental building blocks of Linux virtual networking:

• Network namespaces provide isolation.
• veth pairs provide point-to-point connectivity.
• Linux bridges provide Layer 2 switching.

In the next chapter, we'll combine these primitives to build a complete virtual network. Starting from an empty Linux system, we'll create multiple isolated hosts, connect them to a bridge, assign IP addresses, and observe packets flowing exactly as they would across a physical Ethernet network.

Building a Complete Virtual Network

So far, we've learned the individual building blocks of Linux virtual networking: network namespaces, virtual Ethernet (veth) pairs, and Linux bridges. In this chapter, we'll combine them to build a complete virtual Ethernet network from scratch. By the end, you'll have a network that behaves just like a small physical LAN—all running inside a single Linux kernel.

Learning Objectives

By the end of this chapter, you will be able to:

• Build a complete virtual Layer 2 network from scratch
• Connect multiple network namespaces using a Linux bridge
• Verify connectivity using common Linux networking tools
• Observe ARP, MAC learning, and ICMP traffic
• Understand how containers are connected to host networks
• Troubleshoot common virtual networking problems

What We're Building

Throughout this chapter, we'll build the following topology.

                    Linux Host

                  +-------------+
                  |     br0     |
                  +------+------+ 
                         |
        +----------------+----------------+
        |                |                |
      host1            host2           host3
        |                |                |
================  ================  ================
      veth1            veth2            veth3
        |                |                |
     Namespace A     Namespace B     Namespace C

Each namespace will behave like an independent Linux machine.

Each machine will:

• have its own network interface
• have its own IP address
• communicate through a software Ethernet switch
• exchange ARP and ICMP packets just like physical hosts

Nothing leaves the Linux kernel.

Step 1 — Create the Network Namespaces

Create three namespaces.

sudo ip netns add ns1
sudo ip netns add ns2
sudo ip netns add ns3

Verify:

ip netns list

Example:

ns1
ns2
ns3

Enable loopback inside each namespace.

sudo ip netns exec ns1 ip link set lo up
sudo ip netns exec ns2 ip link set lo up
sudo ip netns exec ns3 ip link set lo up

At this point, each namespace contains only a loopback interface.

Step 2 — Create the Linux Bridge

Create a software switch.

sudo ip link add br0 type bridge

Bring it online.

sudo ip link set br0 up

Verify:

ip link show br0

Step 3 — Create Three veth Pairs

Each namespace needs one Ethernet cable.

Create three veth pairs.

sudo ip link add ns1-veth type veth peer name ns1-host

sudo ip link add ns2-veth type veth peer name ns2-host

sudo ip link add ns3-veth type veth peer name ns3-host

Current topology:

ns1-veth <------> ns1-host

ns2-veth <------> ns2-host

ns3-veth <------> ns3-host

All interfaces currently belong to the initial namespace.

Step 4 — Move Interfaces into the Namespaces

Move one end of each pair.

sudo ip link set ns1-veth netns ns1

sudo ip link set ns2-veth netns ns2

sudo ip link set ns3-veth netns ns3

Now the topology becomes:

Namespace ns1

ns1-veth

Host

ns1-host

Repeat for the remaining namespaces.

Step 5 — Connect the Host Interfaces to the Bridge

Attach every host-side interface to the bridge.

sudo ip link set ns1-host master br0

sudo ip link set ns2-host master br0

sudo ip link set ns3-host master br0

The bridge now has three ports.

Verify:

bridge link

Example:

ns1-host

ns2-host

ns3-host

Step 6 — Bring Every Interface Up

Host-side interfaces:

sudo ip link set ns1-host up

sudo ip link set ns2-host up

sudo ip link set ns3-host up

Namespace interfaces:

sudo ip netns exec ns1 ip link set ns1-veth up

sudo ip netns exec ns2 ip link set ns2-veth up

sudo ip netns exec ns3 ip link set ns3-veth up

Every cable is now connected.

Step 7 — Assign IP Addresses

Configure the namespace interfaces.

Namespace 1:

sudo ip netns exec ns1 ip addr add 192.168.50.11/24 dev ns1-veth

Namespace 2:

sudo ip netns exec ns2 ip addr add 192.168.50.12/24 dev ns2-veth

Namespace 3:

sudo ip netns exec ns3 ip addr add 192.168.50.13/24 dev ns3-veth

The network now looks like this.

                     br0

        +--------------+--------------+

    ns1-host       ns2-host      ns3-host

        |              |              |

================ ================ ================

   ns1-veth      ns2-veth      ns3-veth

192.168.50.11 192.168.50.12 192.168.50.13

Step 8 — Verify the Configuration

Check interfaces inside Namespace 1.

sudo ip netns exec ns1 ip addr

Check routes.

sudo ip netns exec ns1 ip route

Output:

192.168.50.0/24 dev ns1-veth

Repeat for the remaining namespaces.

Step 9 — Test Connectivity

From Namespace 1:

sudo ip netns exec ns1 ping 192.168.50.12

Then:

sudo ip netns exec ns1 ping 192.168.50.13

Everything should respond successfully.

Repeat from the other namespaces.

At this point, we have built a fully functional Ethernet LAN.

Understanding the First Ping

Let's examine what happens when Namespace 1 pings Namespace 2.

ping 192.168.50.12

The following sequence occurs.

Application

↓

ICMP Echo Request

↓

IP

↓

Ethernet

↓

ns1-veth

↓

ns1-host

↓

Linux Bridge

↓

ns2-host

↓

ns2-veth

↓

Namespace 2

The bridge forwards the Ethernet frame exactly like a physical switch.

Step 10 — Observe the ARP Process

Clear the neighbor table.

sudo ip netns exec ns1 ip neigh flush all

Verify:

sudo ip netns exec ns1 ip neigh

Output:

<no output>

Now send one ping.

sudo ip netns exec ns1 ping -c1 192.168.50.12

Display the neighbor table again.

sudo ip netns exec ns1 ip neigh

Example:

192.168.50.12 dev ns1-veth lladdr 1a:3b:62:8d:91:ef REACHABLE

ARP behaved exactly as it would on a physical Ethernet network.

Step 11 — Observe the Bridge Learning MAC Addresses

Display the forwarding database.

bridge fdb show br br0

Example:

8a:1d:2e:3f:44:55 dev ns1-host

3e:81:90:71:ac:29 dev ns2-host

72:b0:f4:18:5d:90 dev ns3-host

The bridge learned which MAC addresses are reachable through each bridge port.

No manual configuration was required.

Step 12 — Capture Traffic

Open a terminal.

sudo tcpdump -i ns1-host

Open another terminal.

sudo ip netns exec ns1 ping 192.168.50.12

Observe:

• ARP Request
• ARP Reply
• ICMP Echo Request
• ICMP Echo Reply

The packets are identical to those seen on a physical Ethernet network.

Step 13 — Test Broadcast Traffic

Send an ARP request again.

sudo ip netns exec ns1 ping -c1 192.168.50.13

The ARP request is broadcast.

The bridge floods it to every port except the incoming one.

                br0

         +------+------+------+

      ns1    ns2    ns3

          Broadcast Frame

Namespace 2 ignores the request.

Namespace 3 responds because the IP address belongs to it.

Step 14 — Add the Host to the Network

The bridge itself is a network interface.

Assign it an IP address.

sudo ip addr add 192.168.50.1/24 dev br0

Now the host can communicate with every namespace.

Test from the host.

ping 192.168.50.11

Or from Namespace 1.

sudo ip netns exec ns1 ping 192.168.50.1

The host is now another machine on the same Ethernet segment.

A Familiar Architecture

If this topology looks familiar, that's because it is.

Docker's default bridge network is conceptually very similar.

                 docker0

          +---------+---------+

         veth      veth

          |         |

    Container   Container

KVM, libvirt, LXC, and many virtualization platforms build similar topologies using the same Linux networking primitives.

Troubleshooting the Network

When the network doesn't work, check the following in order.

Is the namespace present?

ip netns list

Are interfaces up?

Host:

ip link

Namespace:

ip netns exec ns1 ip link

Look for:

UP
LOWER_UP

Is the interface connected to the bridge?

bridge link

Does the namespace have an IP address?

ip netns exec ns1 ip addr

Does the namespace have the expected route?

ip netns exec ns1 ip route

Is ARP working?

ip netns exec ns1 ip neigh

Did the bridge learn MAC addresses?

bridge fdb show

Can you see packets?

tcpdump -i ns1-host

or

ip netns exec ns1 tcpdump -i ns1-veth

Cleaning Up

Delete the namespaces.

sudo ip netns delete ns1
sudo ip netns delete ns2
sudo ip netns delete ns3

Delete the bridge.

sudo ip link delete br0

The remaining host-side veth interfaces disappear automatically because their peers no longer exist.

What We've Built

Without Docker...

Without Kubernetes...

Without libvirt...

Without any container runtime...

...we built an entire virtual Ethernet network using only Linux kernel primitives.

Specifically, we combined:

• Network namespaces for isolation
• veth pairs for point-to-point links
• A Linux bridge for Layer 2 switching

This is the foundation upon which many modern container and virtualization platforms are built.

Key Takeaways

• A complete virtual Ethernet network can be built using only network namespaces, veth pairs, and a Linux bridge.
• Each namespace behaves like an independent Linux host with its own interfaces, routing table, neighbor table, and sockets.
• A Linux bridge forwards Ethernet frames between connected interfaces using MAC address learning.
• ARP, ICMP, and Ethernet behave exactly as they do on a physical network.
• The bridge itself can be assigned an IP address, allowing the host to participate in the virtual network.
• The architecture built in this chapter closely resembles the default networking model used by Docker, libvirt, and many other virtualization platforms.

Coming Next

The virtual network we built behaves like a single Layer 2 Ethernet segment. Every namespace belongs to the same IP subnet, so they can communicate directly without a router.

In the next chapter, we'll transform the Linux host into a router. We'll create multiple independent networks, enable IP forwarding, and learn how Linux routes packets between subnets—the same principles used by physical routers, cloud gateways, and Kubernetes nodes.

Linux as a Router

A Linux bridge connects devices within the same Layer 2 network. But real networks are rarely limited to a single subnet. Sooner or later, packets must travel between different networks. That's the job of a router. In this chapter, we'll transform a Linux host into a fully functional IP router and see exactly how packets move between independent networks.

Learning Objectives

By the end of this chapter, you will be able to:

• Understand the difference between switching and routing
• Configure Linux to forward packets
• Build multiple isolated IP networks
• Route packets between subnets
• Observe routing decisions in action
• Understand the role of the default gateway
• Troubleshoot routing problems
• Recognize how Kubernetes nodes, cloud VMs, and physical routers perform the same function

The Problem

In the previous chapter, we built a virtual Ethernet network.

                br0

      +---------+---------+

    Host1     Host2     Host3

192.168.50.11
192.168.50.12
192.168.50.13

All hosts belonged to the same subnet.

192.168.50.0/24

Communication was simple.

Every host could reach every other host directly using ARP.

But what happens when hosts belong to different networks?

Host A

192.168.10.10/24

-------------

Host B

192.168.20.10/24

Host A cannot send Ethernet frames directly to Host B because they are in different IP networks.

Something must connect those networks.

That something is a router.

What Is a Router?

A router is a device that forwards IP packets between different networks.

Unlike a bridge:

• it operates at Layer 3
• it makes decisions using IP addresses
• it maintains a routing table
• it rewrites Ethernet headers for every hop

Every packet that crosses between IP networks passes through a router.

Linux can perform exactly the same job.

The Network We'll Build

We'll create two independent Ethernet networks connected by one Linux router.

                    Linux Router

             left-br          right-br
                |                 |
         +------+                 +------+
         |                               |
      router-l                      router-r
         |                               |
===================             ===================
        |                               |
      hostA                          hostB
        |                               |
 Namespace A                     Namespace B

Network A

192.168.10.0/24

Network B

192.168.20.0/24

The Linux host will connect both networks together.

Step 1 — Create the Namespaces

Create two namespaces.

sudo ip netns add hostA
sudo ip netns add hostB

Enable loopback.

sudo ip netns exec hostA ip link set lo up

sudo ip netns exec hostB ip link set lo up

Step 2 — Create Two Linux Bridges

Each bridge represents an independent Ethernet network.

sudo ip link add left-br type bridge

sudo ip link add right-br type bridge

Bring them up.

sudo ip link set left-br up

sudo ip link set right-br up

Step 3 — Create the veth Pairs

Each namespace needs one Ethernet connection.

sudo ip link add hostA-veth type veth peer name hostA-host

sudo ip link add hostB-veth type veth peer name hostB-host

Move one side into the namespaces.

sudo ip link set hostA-veth netns hostA

sudo ip link set hostB-veth netns hostB

Step 4 — Connect the Hosts to Their Bridges

Attach the host-side interfaces.

sudo ip link set hostA-host master left-br

sudo ip link set hostB-host master right-br

Bring them up.

sudo ip link set hostA-host up

sudo ip link set hostB-host up

Step 5 — Configure the Host Interfaces

Namespace A

sudo ip netns exec hostA ip addr add 192.168.10.10/24 dev hostA-veth

sudo ip netns exec hostA ip link set hostA-veth up

Namespace B

sudo ip netns exec hostB ip addr add 192.168.20.10/24 dev hostB-veth

sudo ip netns exec hostB ip link set hostB-veth up

Current topology

Host A

192.168.10.10

        |

     left-br


right-br

        |

192.168.20.10

Host B

There is still no connection between the two bridges.

Step 6 — Give the Router an Interface on Each Network

Remember that a Linux bridge is itself a network interface.

Assign an IP address to each bridge.

sudo ip addr add 192.168.10.1/24 dev left-br

sudo ip addr add 192.168.20.1/24 dev right-br

The Linux host now belongs to both networks.

                Linux Router

left-br                  right-br

192.168.10.1          192.168.20.1

At this point, Linux has two interfaces connected to two different IP networks.

Step 7 — Test Local Connectivity

Namespace A should reach its local gateway.

sudo ip netns exec hostA ping 192.168.10.1

Namespace B should also reach its gateway.

sudo ip netns exec hostB ping 192.168.20.1

These pings stay inside their local Ethernet networks.

Step 8 — Try to Reach the Other Network

From Namespace A:

sudo ip netns exec hostA ping 192.168.20.10

The result:

Network is unreachable

Why?

Because Namespace A only knows about:

192.168.10.0/24

It has no route for:

192.168.20.0/24

The packet never leaves the namespace.

Step 9 — Configure the Default Gateway

Tell each namespace where to send packets destined for other networks.

Namespace A:

sudo ip netns exec hostA ip route add default via 192.168.10.1

Namespace B:

sudo ip netns exec hostB ip route add default via 192.168.20.1

Verify.

sudo ip netns exec hostA ip route

Example:

default via 192.168.10.1

192.168.10.0/24 dev hostA-veth

Now the packet reaches the Linux router.

But communication still fails.

Why Doesn't It Work Yet?

The packet now arrives at the Linux host.

However, Linux does not forward packets between interfaces by default.

Instead, it silently drops them.

This behavior protects ordinary desktop and server systems from unintentionally acting as routers.

Step 10 — Enable IP Forwarding

Check the current setting.

cat /proc/sys/net/ipv4/ip_forward

Output:

0

Enable forwarding.

sudo sysctl -w net.ipv4.ip_forward=1

Or:

echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward

Verify.

cat /proc/sys/net/ipv4/ip_forward

Output:

1

Linux is now acting as a router.

Step 11 — Test Again

From Namespace A:

sudo ip netns exec hostA ping 192.168.20.10

Now the ping succeeds.

Congratulations.

You have built an IP router using only the Linux kernel.

Following the Packet

Let's follow the packet step by step.

Namespace A creates an ICMP Echo Request.

Destination

192.168.20.10

Namespace A checks its routing table.

Destination not local

↓

Use default gateway

192.168.10.1

An ARP request resolves the gateway's MAC address.

The Ethernet frame is sent.

Destination MAC

Linux Router

The Linux router receives the frame.

Ethernet Header

↓

Removed

Linux now examines the IP packet.

Destination IP

192.168.20.10

Linux searches its routing table.

192.168.20.0/24

↓

right-br

A new Ethernet header is created.

Source MAC

right-br

Destination MAC

Host B

The packet leaves through the second interface.

Notice what changed.

• Ethernet headers were replaced.
• MAC addresses changed.
• The IP packet stayed the same.

This is exactly how every IP router operates.

Observe the Routing Table

Display the router's routing table.

ip route

Example:

192.168.10.0/24 dev left-br

192.168.20.0/24 dev right-br

Linux automatically installed these connected routes when IP addresses were assigned.

No manual routing configuration was required.

Observe TTL Changes

Capture packets.

sudo tcpdump -i right-br icmp

Run:

sudo ip netns exec hostA ping 192.168.20.10

Notice that the packet's TTL has decreased by one.

Every router decrements the TTL before forwarding the packet.

This prevents routing loops from lasting forever.

Observe ARP

Namespace A never learns Host B's MAC address.

Check:

sudo ip netns exec hostA ip neigh

You'll only see:

192.168.10.1

The namespace only knows the MAC address of its default gateway.

Likewise, Host B only knows the router's MAC address.

This is one of the most important concepts in IP networking.

Hosts do not ARP for remote destinations.

They ARP only for the next hop.

Common Routing Problems

IP Forwarding Disabled

Check:

cat /proc/sys/net/ipv4/ip_forward

Must be:

1

Missing Default Route

Verify:

ip netns exec hostA ip route

Expected:

default via 192.168.10.1

Wrong Subnet Mask

Ensure both hosts and router interfaces use compatible prefixes.

Incorrect subnet masks often cause confusing routing behavior.

Interfaces Down

Verify:

ip link

and

ip netns exec hostA ip link

Firewall Rules

Linux may be forwarding packets correctly, but firewall rules can still block them.

Later in this book, we'll study Netfilter, iptables, and nftables in detail.

Real-World Examples

Linux routers are everywhere.

Examples include:

Once IP forwarding is enabled, Linux behaves exactly like a dedicated router.

Cleaning Up

Delete the namespaces.

sudo ip netns delete hostA

sudo ip netns delete hostB

Delete the bridges.

sudo ip link delete left-br

sudo ip link delete right-br

Disable forwarding if desired.

sudo sysctl -w net.ipv4.ip_forward=0

Key Takeaways

• A router forwards IP packets between different Layer 3 networks.
• Linux becomes a router when it has interfaces in multiple networks and IP forwarding is enabled.
• Hosts use their routing tables to determine whether a destination is local or remote.
• Remote packets are sent to the default gateway, not directly to the destination host.
• A router removes the incoming Ethernet header, makes a routing decision based on the destination IP address, and creates a new Ethernet header for the outgoing interface.
• The router decrements the packet's TTL before forwarding it.
• Linux automatically installs connected routes for directly attached networks.

Coming Next

So far, every virtual network we've built has used a single Layer 2 broadcast domain per bridge. In production environments, however, a single physical network is often divided into multiple logical networks using VLANs, while technologies like TUN and TAP create virtual interfaces for VPNs, virtualization, and userspace networking.

In the next chapter, we'll explore these additional Linux networking primitives and see how they extend the virtual networking toolbox beyond bridges and veth pairs.

VLANs and TUN/TAP: Two More Linux Networking Primitives

Throughout this part of the book, we've focused on three fundamental Linux networking primitives: network namespaces, veth pairs, and Linux bridges. Together, they allow us to build complete virtual Ethernet networks. However, Linux provides additional networking primitives that solve different problems. In this chapter, we'll explore two of the most important: VLAN interfaces and TUN/TAP devices.

Learning Objectives

By the end of this chapter, you will be able to:

• Understand why VLANs exist
• Understand what TUN and TAP devices are
• Create and experiment with TUN/TAP interfaces
• Recognize where VLANs and TUN/TAP are used in real-world systems

Not Every Network Needs More Hardware

Suppose a company has three departments:

• Engineering
• Finance
• Operations

Initially, every computer is connected to the same Ethernet switch.

          Ethernet Switch

      +------+------+------+

    Eng1   Fin1   Ops1

Everyone shares the same broadcast domain.

Every ARP request reaches every device.

Every unknown unicast is flooded everywhere.

As the network grows, this becomes inefficient.

More importantly, different departments may need to be isolated from one another.

Buying three physical switches works, but it is expensive and difficult to manage.

Instead, we can divide one physical switch into multiple logical switches.

This is the purpose of VLANs.

What Is a VLAN?

A Virtual Local Area Network (VLAN) divides one physical Ethernet network into multiple independent Layer 2 networks.

Consider a switch with eight ports.

Without VLANs:

+--------------------+
|      Switch        |
+--------------------+

Every port belongs to the
same Ethernet network.

With VLANs:

+--------------------+
|       Switch       |
+--------------------+

Ports 1-4 → VLAN 10

Ports 5-8 → VLAN 20

Even though every device is connected to the same physical switch, devices in different VLANs cannot communicate directly.

From the devices' perspective, they appear to be connected to different switches.

VLANs Are Layer 2 Isolation

A VLAN does not create a new IP network.

It creates a new Ethernet broadcast domain.

Typically, each VLAN is assigned its own IP subnet.

For example:

The separation happens at Layer 2.

Routing between VLANs requires a router.

How VLANs Work

The IEEE 802.1Q standard introduces a VLAN tag inside an Ethernet frame.

A normal Ethernet frame looks like this:

Destination MAC

Source MAC

EtherType

Payload

A VLAN-tagged frame inserts an additional header.

Destination MAC

Source MAC

802.1Q VLAN Tag

EtherType

Payload

The tag contains a VLAN Identifier (VID).

For example:

VLAN 10

or

VLAN 20

Switches use this tag to decide which logical network the frame belongs to.

VLANs and Linux Bridges

A bridge can contain VLAN interfaces.

For example:

             br0

        +-----+------+

    eth0.10   veth1

Only VLAN 10 traffic reaches that bridge.

Similarly:

             br1

        +-----+------+

    eth0.20   veth2

Linux virtualization platforms frequently use this design to connect virtual machines or containers to different VLANs.

What Is TUN/TAP?

Unlike VLANs, TUN and TAP do not divide Ethernet networks.

Instead, they create virtual network devices that exchange packets with user-space applications.

This makes them ideal for VPN software, emulators, virtual machines, and networking tools.

TUN vs TAP

Although they are often mentioned together, they operate at different layers.

Remember this simple rule:

TUN works with IP packets. TAP works with Ethernet frames.

Understanding TUN

A TUN interface behaves like a point-to-point IP interface.

Application

↓

IP Packet

↓

tun0

↓

User Space Program

Notice what is missing.

There is no Ethernet header.

The application reads and writes raw IP packets.

Understanding TAP

A TAP interface behaves like an Ethernet interface.

Application

↓

Ethernet Frame

↓

tap0

↓

User Space Program

Applications receive complete Ethernet frames, including:

• destination MAC
• source MAC
• EtherType
• payload

To Linux, a TAP interface looks almost identical to a physical Ethernet device.

Why User Space?

Unlike veth pairs, packets entering a TUN or TAP device are delivered to a user-space program.

That application decides what happens next.

For example, it may:

• encrypt the packet
• compress it
• forward it over TCP
• encapsulate it inside UDP
• inspect or modify it
• discard it

This flexibility is why VPN software uses TUN/TAP devices.

Where TAP Is Used

Many virtualization platforms connect guest operating systems using TAP devices.

Guest VM

↓

tap0

↓

Linux Bridge

↓

Physical Network

From the guest's perspective, it owns a normal Ethernet adapter.

The guest never knows the interface is virtual.

Where TUN Is Used

TUN devices are common in VPN software.

Examples include:

• WireGuard
• OpenVPN (TUN mode)
• Tailscale
• ZeroTier
• Cloudflare WARP

Applications exchange IP packets through the TUN interface and encapsulate them for transport across another network.

Comparing Linux Networking Primitives

At this point, we've covered six major Linux networking primitives.

These primitives can be combined to build almost any Linux networking architecture.

Common Mistakes

Expecting VLANs to Route Traffic

VLANs provide Layer 2 isolation.

Communication between VLANs still requires a router.

Confusing TUN and TAP

If an application expects Ethernet frames, use TAP.

If it expects IP packets, use TUN.

Choosing the wrong device type usually prevents communication.

Forgetting User-Space Ownership

A TUN or TAP interface by itself does not forward packets.

A user-space application must open the device and read or write packets.

Summary of Part 3

In this part of the book, we've built a complete virtual networking environment using only Linux kernel features.

We learned how:

• Network namespaces isolate networking resources.
• veth pairs create virtual Ethernet cables.
• Linux bridges implement Layer 2 switching.
• Linux routes packets between different IP networks.
• VLANs divide one physical network into multiple logical Ethernet networks.
• TUN and TAP devices connect the kernel networking stack to user-space applications.

These are the same building blocks used by container runtimes, hypervisors, VPN software, cloud platforms, and Kubernetes networking.

Understanding these primitives is far more valuable than memorizing the commands of a specific platform because the platforms themselves are built on top of these concepts.

Key Takeaways

• VLANs provide Layer 2 isolation by inserting an IEEE 802.1Q tag into Ethernet frames.
• Communication between VLANs requires Layer 3 routing.
• A TUN device carries IP packets, while a TAP device carries complete Ethernet frames.
• TUN/TAP devices connect the kernel networking stack with user-space applications, enabling VPNs, virtualization, and network emulation.
• Modern networking platforms combine namespaces, veth pairs, bridges, routing, VLANs, and TUN/TAP devices to create flexible virtual networks.

Coming Next

So far, we've explored Linux networking from the perspective of network devices and topology. We know how packets move between namespaces, bridges, routers, and virtual interfaces.

In the next part of the book, we'll shift our focus inside the Linux kernel and follow a packet through its entire processing path. We'll examine where routing decisions occur, how Netfilter intercepts traffic, how connection tracking maintains state, and how Linux implements firewalling and Network Address Translation (NAT).

Following a Packet Through the Linux Kernel

In the previous chapters, we focused on the visible parts of Linux networking: interfaces, routes, sockets, and traffic observation. Now we go one layer deeper.

This chapter follows a packet as it enters and leaves the Linux kernel. The goal is not to memorize every kernel function, but to build a mental model of where packets go, what decisions are made, and which tools let you inspect those decisions.

By the end of this chapter, you should be able to answer questions like:

• Why does a packet arrive on one interface but get delivered to a process on another?
• Why is a packet dropped before it reaches an application?
• What does Linux do after a packet hits a network card?
• Why do tools like tcpdump, ss, ip route get, nft, and conntrack show different parts of the same story?

The Big Picture

A packet does not go directly from the network card to your application.

Linux processes it through a pipeline:

1. The network card receives the frame.

2. The kernel driver hands the data to the networking stack.

3. The kernel decides whether the packet is:

   • destined for the local host,
   • forwarded to another host,
   • or dropped.

4. If the packet is local, Linux delivers it to the correct socket.

5. If the packet is outgoing, Linux builds the packet, applies routing decisions, and transmits it.

The important lesson is this:

Linux networking is not a single path. It is a set of decision points.

At each step, the kernel may:

• accept the packet,
• rewrite it,
• queue it,
• forward it,
• or discard it.

A Packet’s Journey: Ingress and Egress

Every packet can be viewed from two directions.

Ingress path

This is what happens when a packet enters the machine from the network.

Typical flow:

• NIC receives the frame
• driver places data into memory
• kernel creates an skb
• packet moves up the stack
• routing decision is made
• packet is delivered locally or forwarded

Egress path

This is what happens when a process sends data out.

Typical flow:

• application calls send()
• kernel creates packet data
• routing selects the output interface
• packet may be modified by firewall/NAT
• driver transmits the frame

The same host may be both a sender and a receiver at the same time, and Linux handles both paths simultaneously.

The skb: Linux’s Packet Container

Inside the kernel, packets are represented by a structure called sk_buff, usually shortened to skb.

Think of an skb as the kernel’s working envelope for a packet. It contains:

• packet data,
• metadata,
• protocol information,
• interface information,
• and pointers used by the network stack.

The skb is not just payload. It is also the kernel’s way of tracking:

• where the packet came from,
• where it is going,
• how much of it has been parsed,
• and which subsystem should handle it next.

You do not need to know the C structure to understand Linux networking, but you should understand the idea:

The kernel does not pass “raw bytes” between networking stages. It passes a packet object with metadata.

That metadata is what allows Linux to make routing, firewall, and delivery decisions efficiently.

From NIC to Kernel

When a packet arrives on the wire, the network interface card receives the Ethernet frame and hands it to the kernel through the driver.

At a high level:

• the NIC stores the frame in memory,
• the driver notices the arrival,
• the kernel reads the data,
• and the packet is inserted into the network stack.

Modern Linux systems often use interrupt moderation and NAPI to reduce overhead. The exact driver behavior depends on the hardware, but the conceptual model stays the same:

The hardware detects the packet, and the kernel turns it into an skb.

For our purposes, the important question is not how many interrupts happened but what the kernel does after it gets the packet.

The First Decision: Local Delivery or Forwarding

After a packet enters the stack, Linux asks a basic question:

Is this packet meant for this host, or should it be forwarded somewhere else?

This decision depends on:

• destination IP address,
• routing table,
• interface configuration,
• and kernel forwarding settings.

There are two broad outcomes.

1. Local delivery

If the destination IP belongs to the local machine, Linux tries to deliver the packet to a socket.

This is the case when:

• you ssh into the host,
• a web server is listening on port 80,
• or a process is bound to the destination address.

2. Forwarding

If the packet is not for the local host but the machine has routing enabled, Linux forwards the packet to another interface.

This is what happens on routers, gateways, Kubernetes nodes, and Linux machines acting as network intermediaries.

Forwarding is one of the key differences between an ordinary host and a router.

Local Delivery: From Packet to Socket

When a packet is meant for the local machine, Linux still does not hand it directly to the application.

The kernel must match the packet to a socket.

That involves checks such as:

• destination IP address,
• destination port,
• protocol (TCP or UDP),
• socket binding state,
• and connection state.

For example:

• A packet destined to 10.0.0.10:80/tcp may be delivered to an nginx process.
• A packet destined to 127.0.0.1:5432/tcp may go to a local PostgreSQL server.
• A packet destined to a closed port may trigger an ICMP error or a TCP RST.

This is why open ports matter: they are sockets that are ready to receive traffic.

Outgoing Packets: From Socket to Wire

When an application sends data, the path goes in the opposite direction.

A simplified egress flow looks like this:

1. application calls send()
2. kernel writes data into socket buffers
3. TCP or UDP builds transport-layer headers
4. IP layer chooses a destination route
5. link layer resolves the next hop
6. frame is sent to the NIC
7. NIC transmits it on the wire

Each layer adds information:

• TCP or UDP adds port information.
• IP adds source and destination addresses.
• Ethernet adds source and destination MAC addresses.

Linux does not guess where to send the packet. It consults routing, neighbor resolution, and interface state.

Routing Is Still Central

Even in packet processing, routing remains one of the main decision points.

Before Linux sends a packet out, it answers:

• Which destination network does this packet belong to?
• Which interface should be used?
• What is the next hop?
• What source address should be selected?

You already saw routing in the routing chapter. Here, routing becomes operational:

Routing is the decision step that chooses the packet’s next movement in the kernel.

A useful command for understanding this is:

ip route get 8.8.8.8

Example output:

8.8.8.8 via 192.168.1.1 dev eth0 src 192.168.1.50 uid 1000
    cache

This tells you:

• where Linux will send the packet,
• which interface it will use,
• and which source address it will choose.

That is not a theoretical answer. It is the kernel’s actual decision.

What Happens Before a Packet Reaches a Process

Many people assume a packet arrives, then the application reads it.

In reality, Linux does much more work before the application ever sees it.

For incoming traffic, the kernel may:

• verify the packet format,
• check routing rules,
• consult firewall rules,
• update connection-tracking state,
• perform NAT rewrites,
• and only then deliver the packet to the socket.

This means a packet can be:

• accepted by the NIC but dropped by firewall rules,
• routed correctly but never delivered because no process is listening,
• or translated by NAT before reaching the application.

The application sees only the final result.

A Practical Example: Watching a TCP Connection

Let’s build a simple mental model using an SSH or HTTP connection.

Suppose a client connects to a server on port 22.

On the client

• The application calls connect().
• Linux selects a source port.
• Linux finds a route to the server.
• The packet leaves the host.

On the server

• The NIC receives the SYN packet.
• Linux checks whether the destination IP belongs to the host.
• The kernel looks for a listening socket on port 22.
• If SSH is listening, the packet is delivered to that socket.
• The SSH daemon responds with a SYN-ACK.

This is the moment where protocol state matters. TCP is not just packet exchange; it is a coordinated state machine implemented across kernel and application.

You will see more of that in the connection-tracking chapter, but the basic idea belongs here:

The packet is not “for an app” until the kernel has matched it to a socket.

Tools for Observing the Path

You cannot see every internal kernel step directly, but you can infer a lot.

ip route get

Shows routing decisions.

ip route get 1.1.1.1

Useful when you want to know:

• which interface Linux will use,
• which gateway it prefers,
• and which source address it picks.

ss

Shows socket state in the kernel.

ss -tulpen

Useful when you want to know:

• what is listening,
• which process owns the socket,
• and whether a connection exists.

This connects packet delivery to application state.

tcpdump

Shows packets at the interface boundary.

tcpdump -i eth0 -nn

Useful when you want to know:

• whether packets entered or left the interface,
• what ports and addresses are involved,
• and whether the wire sees the traffic at all.

Important limitation:

tcpdump sees packets on the interface. It does not tell you everything the kernel does after that.

ip -s link

Shows interface statistics.

ip -s link show dev eth0

Useful when you want to inspect:

• received/transmitted packets,
• drops,
• errors,
• and queue-level counters.

This helps you distinguish:

• “the packet never arrived,” from
• “the packet arrived but was dropped later.”

Hands-On Lab 1: Check the Route the Kernel Will Use

Pick any reachable destination, for example a public DNS server or your gateway.

ip route get 8.8.8.8

Look at the output and identify:

• the outgoing interface,
• the next hop,
• the source IP address.

Now compare that with:

ip addr show
ip route show

You should see how the kernel’s routing decision matches the machine’s interface and route configuration.

What to learn from this

The packet path begins before transmission. Linux chooses the route first, then sends the packet.

Hands-On Lab 2: Watch a Packet Leave the Host

Open one terminal and run:

tcpdump -i eth0 -nn icmp

In another terminal, run:

ping 8.8.8.8

You should see ICMP echo requests and replies.

Now ask yourself:

• Which interface carried the packet?
• Did the packet leave the host?
• Did the reply come back?

This lab shows the boundary between local kernel processing and network transmission.

Hands-On Lab 3: Observe a Listening Socket

Start a simple service, such as a Python HTTP server:

python3 -m http.server 8080

In another terminal:

ss -tulpen | grep 8080

You should see the listening socket.

Now test it locally:

curl http://127.0.0.1:8080

Then test it from another host if possible.

What matters here is the chain:

• packet arrives,
• kernel matches port 8080,
• socket receives the traffic,
• server process reads it.

The application does not receive random network data. It receives traffic that has already passed kernel matching.

Localhost Is Still Networking

A common mistake is to think 127.0.0.1 is “not real networking.”

It is real networking inside the kernel.

When a process sends traffic to localhost:

• the packet still enters the networking stack,
• routing still happens,
• socket lookup still happens,
• and the packet is still delivered through kernel networking logic.

The difference is that the traffic never leaves the machine.

This is useful because it proves a key idea:

Networking is not defined by distance. It is defined by the kernel path the packet takes.

Why Some Packets Disappear

From the application’s point of view, packets can vanish for many reasons:

• no route exists,
• the firewall dropped them,
• no process is listening,
• the destination address is wrong,
• the interface is down,
• or the packet was malformed.

This is why packet debugging is often about narrowing the layer where failure occurs.

A good debugging sequence is:

1. Is the packet leaving the host?
2. Is the packet reaching the host?
3. Is Linux routing it correctly?
4. Is a socket listening?
5. Is a firewall or NAT rule altering it?
6. Is the application handling it?

That sequence is the basis of most Linux network troubleshooting.

The Kernel Is the Traffic Cop

At this point, the right mental model should be clear.

Linux is not just a passive conduit. It is an active traffic cop.

For every packet, the kernel may:

• inspect it,
• classify it,
• route it,
• filter it,
• translate it,
• queue it,
• forward it,
• or deliver it.

That is why Linux networking is powerful. It is also why networking problems often require kernel-level thinking.

A packet’s life inside Linux is a sequence of decisions, not a straight line.

What Comes Next

In this chapter, we followed packets through the Linux kernel at a conceptual level. We did not yet examine all the subsystems that can influence packet flow.

In the next chapters, we will go deeper into the pieces that modify or intercept packets:

• netfilter, the framework behind Linux packet filtering,
• connection tracking, which remembers flows,
• NAT, which rewrites addresses and ports,
• and firewall tooling such as iptables and nftables.

Once you understand the packet path, these tools stop feeling magical. They become specific control points in a known pipeline.

Summary

• Packets enter and leave Linux through a defined kernel path.
• The skb is the kernel’s packet container.
• Linux decides whether a packet is local, forwarded, or dropped.
• Routing, firewalling, NAT, and socket lookup all happen inside the kernel.
• tcpdump, ss, ip route get, and ip -s link help you observe different parts of the path.
• The most important habit is to think in stages, not in a single “packet arrived” event.

The next chapter will focus on the first major decision framework in that path: netfilter.

Netfilter: The Linux Packet Processing Framework

In the previous chapter, we followed a packet through the Linux kernel and saw that packet handling is a sequence of decisions, not a single event. One of the most important decision layers in that sequence is netfilter.

Netfilter is the Linux kernel framework that handles packet filtering, NAT, packet mangling, and packet inspection at specific points in the packet path. It is the mechanism behind Linux firewalling and packet rewriting. Tools such as iptables and nftables are user-space interfaces for configuring netfilter, but netfilter itself lives in the kernel.

This chapter focuses on the framework itself: where it sits in the packet path, what it can do, and how to reason about it when troubleshooting.

What Netfilter Is

Netfilter is not a firewall application. It is the in-kernel packet-processing framework that provides hook points where policy can be applied.

At a high level, netfilter allows Linux to:

• inspect packets,
• allow or deny traffic,
• rewrite addresses and ports,
• track flow state,
• and apply policy to traffic entering, leaving, or passing through the host.

The important distinction is:

Netfilter is the kernel framework. iptables and nftables are configuration interfaces for it.

That distinction matters because later chapters will focus on rule syntax, while this chapter focuses on where packet processing happens inside the kernel.

Why Netfilter Matters

A Linux system can route packets and deliver them to sockets without netfilter rules at all. But netfilter adds the packet-policy layer that makes Linux useful as:

• a firewall,
• a router,
• a NAT gateway,
• a container host,
• and a general-purpose packet-control platform.

In practice, netfilter is often invisible until traffic stops working. Then it becomes one of the first places to inspect.

If a packet is:

• arriving but not reaching the application,
• being forwarded unexpectedly,
• translated into a different source or destination address,
• or silently dropped,

netfilter is often part of the explanation.

Netfilter Hooks

Netfilter works by placing hook points at key locations in the packet path. When a packet reaches one of those points, the kernel can pass it through netfilter logic before continuing.

The main hooks are:

• PREROUTING
• INPUT
• FORWARD
• OUTPUT
• POSTROUTING

These names describe where the packet is in the kernel pipeline.

PREROUTING

This is one of the first hook points for incoming packets, before the kernel fully decides whether the packet is for the local host or should be forwarded.

It is commonly associated with early packet classification and destination NAT.

INPUT

Packets destined for the local machine pass through this hook before they are delivered to local sockets.

This is where host-level inbound filtering usually matters.

FORWARD

Packets that are not for the local host but are being routed through the machine pass through this hook.

This matters on routers, gateways, and systems with forwarding enabled.

OUTPUT

Packets generated locally by applications or services pass through this hook before they leave the host.

This affects traffic created on the local machine itself.

POSTROUTING

This is one of the last hook points before a packet leaves the system.

It is commonly associated with source NAT and final packet adjustments.

The Netfilter Mental Model

You do not need the internal implementation details to use netfilter effectively. A useful mental model is:

1. A packet reaches a hook.
2. Netfilter evaluates the applicable rules.
3. A rule may allow the packet to continue.
4. A rule may drop the packet.
5. A rule may rewrite the packet.
6. The packet then continues to the next stage.

That is the core pattern.

When you are debugging, the real question is not “is netfilter on?” The real question is:

• which hook saw the packet,
• which rule matched it,
• and what action was taken?

What Netfilter Can Do

Netfilter rules can lead to several kinds of outcomes.

Accept

The packet is allowed to continue through the stack.

Drop

The packet is discarded and does not continue.

Reject

The packet is discarded and the kernel may send a response explaining that it was refused.

Rewrite

Packet headers can be modified, which is how NAT and similar transformations work.

Log or trace

The packet can be recorded for diagnostics or auditing.

These are the practical operations that matter. Everything else is implementation detail.

Netfilter Is Stateful

Netfilter is closely tied to connection tracking, which gives the kernel memory of flows.

That means packets are not always evaluated as isolated events. The kernel can know whether a packet is:

• part of a new flow,
• part of an established flow,
• related to an existing flow,
• or invalid.

This is important because many firewall policies are written around connection state rather than individual packets.

You do not need to master connection tracking in this chapter, but you should already understand that netfilter is not purely stateless.

Netfilter and Packet Direction

A clean way to reason about netfilter is to separate traffic into three categories.

Inbound to a local service

A packet arrives on an interface and is destined for a service on the host.

Typical path:

• packet enters the host,
• PREROUTING is evaluated,
• routing decides the packet is local,
• INPUT is evaluated,
• the packet reaches the socket.

Transit traffic

A packet arrives and must be routed to another host.

Typical path:

• packet enters the host,
• PREROUTING is evaluated,
• routing decides the packet must be forwarded,
• FORWARD is evaluated,
• POSTROUTING is evaluated,
• packet leaves the host.

Locally generated traffic

A process on the host sends a packet.

Typical path:

• application creates traffic,
• OUTPUT is evaluated,
• routing decides where it should go,
• POSTROUTING is evaluated,
• packet leaves the host.

This is one of the most practical ways to think about packet handling on Linux.

Netfilter Is Not Routing

Routing and netfilter are related, but they solve different problems.

Routing answers:

• Where should this packet go?

Netfilter answers:

• Should this packet be allowed?
• Should it be rewritten?
• Should it be logged?

A packet can be routed correctly and still be dropped by netfilter.

A packet can also be allowed by netfilter and still fail because routing does not know where to send it.

That distinction is important in troubleshooting.

Netfilter and NAT

NAT is one of netfilter’s major jobs.

When Linux rewrites source or destination addresses or ports, that rewrite is typically happening through netfilter hooks.

Common NAT use cases include:

• allowing private IP ranges to access the internet through one public IP,
• redirecting traffic to another host or service,
• translating container traffic,
• and building gateways or edge devices.

You do not need to implement NAT in this chapter. You only need to know that NAT is not a separate subsystem floating next to Linux networking. It is part of the netfilter pipeline.

Observing Netfilter Without Writing Rules

Because this chapter is about the framework, not rule syntax, the goal is to observe behavior rather than build a firewall.

View the current ruleset

On systems using iptables:

sudo iptables -L -n -v

On systems using nftables:

sudo nft list ruleset

These commands show what is already configured on the machine. They are useful because they let you see how packet-processing policy is represented in the kernel.

Check whether forwarding is enabled

sysctl net.ipv4.ip_forward

If forwarding is disabled, the machine cannot behave as a router for IPv4 traffic.

Watch packets at the interface boundary

tcpdump -i eth0 -nn

This shows whether packets are actually entering or leaving the interface. It does not tell you which netfilter rule acted on them, but it helps you separate “packet never arrived” from “packet arrived and was dropped later.”

Check routing decisions

ip route get 8.8.8.8

This shows the route the kernel would use before netfilter and socket delivery complete the story.

How This Fits Into the Larger Packet Path

Netfilter is not the first thing a packet sees, and it is not the last.

A packet typically passes through this larger sequence:

1. NIC and driver receive or send data.
2. The kernel builds or consumes the packet object.
3. Routing decides where the packet should go.
4. Netfilter evaluates policy at relevant hooks.
5. Connection tracking may associate the packet with a flow.
6. NAT may rewrite packet headers.
7. The packet is delivered to a socket or transmitted on an interface.

That is why packet troubleshooting is often layered.

You do not ask only “is the firewall broken?” You ask:

• did the packet arrive,
• did routing choose the right path,
• did netfilter allow it,
• and did the socket receive it?

Common Misunderstandings

“Netfilter is the firewall”

Not quite. Netfilter is the framework. Firewall rules are one use of it.

“iptables and nftables are different kernels”

No. They are different user-space interfaces to the same general kernel packet-processing framework.

“If a packet is on the wire, Linux must have accepted it”

Not necessarily. A packet can be visible on the interface and still be dropped later in the kernel.

“Routing and firewalling are the same thing”

They are separate decisions. Routing selects a path; netfilter enforces policy on that path.

Summary

• Netfilter is the Linux kernel packet-processing framework.
• It provides hooks such as PREROUTING, INPUT, FORWARD, OUTPUT, and POSTROUTING.
• iptables and nftables are configuration interfaces, not the framework itself.
• Netfilter can accept, drop, reject, log, or rewrite packets.
• It is closely tied to connection tracking and NAT.
• Routing decides where packets go; netfilter decides what is allowed to happen to them.
• tcpdump, ip route get, iptables -L -n -v, nft list ruleset, and sysctl net.ipv4.ip_forward are useful observational tools.

In the next chapter, we will go deeper into the state that makes netfilter powerful: connection tracking.

Connection Tracking: The Hidden State Machine

In the previous chapter, we learned that netfilter is the framework responsible for packet filtering, NAT, and packet manipulation inside the Linux kernel.

One important feature of netfilter is that it is stateful. Instead of treating every packet as an independent event, the kernel remembers ongoing network conversations. This memory is provided by a subsystem called connection tracking, often abbreviated as conntrack.

Connection tracking is one of the most important—and least visible—parts of Linux networking. It enables stateful firewalls, makes NAT work correctly, and allows Linux to understand the relationship between packets.

In this chapter, we'll explore how connection tracking works, how to inspect it, and why understanding it is essential when troubleshooting Linux networks.

What Is Connection Tracking?

Despite its name, connection tracking does not only track TCP connections.

Instead, it tracks network flows.

A flow is identified by information such as:

• Source IP address
• Destination IP address
• Transport protocol (TCP, UDP, ICMP, ...)
• Source port (if applicable)
• Destination port (if applicable)

This combination is often called the 5-tuple.

For example:

Once Linux sees the first packet belonging to this flow, it creates an entry in the connection tracking table.

Future packets matching the same flow can be recognized immediately.

Why Does Linux Need Connection Tracking?

Imagine a firewall without memory.

Every packet would have to be evaluated independently.

Consider a TCP connection:

Client                     Server

SYN -------------------->

      <---------------- SYN-ACK

ACK -------------------->

HTTP Request ----------->

      <---------------- HTTP Response

Without remembering previous packets, Linux cannot easily determine whether an incoming packet belongs to an existing connection or is an unexpected packet.

Connection tracking solves this problem.

It allows Linux to recognize that:

• this packet belongs to an already established flow,
• this packet is a reply,
• this packet is part of an existing NAT mapping,
• this packet is invalid.

The Hidden State Machine

Connection tracking internally maintains a state machine for every tracked flow.

The most commonly encountered states are:

• NEW
• ESTABLISHED
• RELATED
• INVALID

These are connection tracking states, not TCP states.

That distinction is extremely important.

NEW

NEW means the packet is attempting to start a new flow.

For TCP, this is typically the initial SYN packet.

Example:

Client ---- SYN ----> Server

The flow has never been seen before.

ESTABLISHED

Once traffic is flowing in both directions, the connection becomes established.

Client <=====> Server

Packets belonging to this conversation match the existing conntrack entry.

Most firewall rules simply allow:

• established
• related

traffic while applying stricter rules only to new connections.

RELATED

Some protocols create additional connections that logically belong to an existing one.

Examples include:

• FTP data channels
• Some ICMP error messages

Although these packets belong to a different flow, they are related to an existing tracked connection.

INVALID

Packets may be marked invalid for many reasons:

• malformed packets
• packets that cannot be associated with any flow
• corrupted headers
• unexpected sequence of packets

Many firewall configurations simply drop invalid packets.

Conntrack State vs TCP State

A very common misunderstanding is confusing conntrack states with TCP states.

TCP itself has many protocol states:

• LISTEN
• SYN_SENT
• SYN_RECEIVED
• ESTABLISHED
• FIN_WAIT
• TIME_WAIT
• CLOSE_WAIT

Connection tracking does not replace TCP.

Instead, it provides a higher-level view used by the networking stack.

Think of it like this:

Application
      │
 TCP State Machine
      │
Connection Tracking
      │
Netfilter
      │
IP Layer

TCP handles reliable communication.

Connection tracking remembers flows for packet processing.

Following a New Connection

Let's follow an HTTPS connection.

Client                     Linux Server

SYN ------------------------>

          conntrack creates entry

SYN-ACK <--------------------

ACK ------------------------->

HTTPS Request --------------->

HTTPS Response <--------------

After the first packet, Linux already knows:

• who initiated the flow,
• both endpoints,
• transport protocol,
• ports,
• current tracking state.

Every later packet can immediately match this entry.

Where Are Connections Stored?

The Linux kernel stores tracked flows inside the connection tracking table.

This table exists entirely inside the kernel.

Each entry contains information such as:

• source address
• destination address
• ports
• protocol
• timeout
• current state
• NAT information (if applicable)

The table is constantly updated as packets arrive.

When a flow becomes inactive, its entry eventually expires.

Inspecting the Connection Tracking Table

The easiest way to inspect tracked connections is with the conntrack utility.

On Debian-based systems:

sudo apt install conntrack

List current entries:

sudo conntrack -L

Example output:

tcp      6 431999 ESTABLISHED src=192.168.1.10 dst=10.0.0.5 sport=53000 dport=443 \
src=10.0.0.5 dst=192.168.1.10 sport=443 dport=53000

Although the output looks intimidating, notice that it simply describes:

• original direction
• reply direction
• protocol
• state
• timeout

Watch Connections Appear

Open one terminal:

sudo conntrack -E

This command listens for connection tracking events.

Open another terminal:

curl https://example.com

You should see events similar to:

NEW
UPDATE
DESTROY

This demonstrates that connection tracking is constantly updating its internal state.

Compare ss and conntrack

Run:

ss -tn

You may see:

ESTAB 192.168.1.10:53000 10.0.0.5:443

Now compare:

sudo conntrack -L

Although both commands show active communication, they answer different questions.

ss shows:

• sockets
• processes
• transport-layer information

conntrack shows:

• packet flows
• flow state
• NAT information
• kernel tracking entries

They complement each other.

Observe UDP Tracking

Many people think only TCP is tracked.

Let's prove otherwise.

Start a DNS lookup:

dig example.com

Immediately inspect:

sudo conntrack -L | grep udp

You should see a UDP flow.

Although UDP has no handshake, Linux still tracks the exchange for a limited amount of time.

Connection tracking is therefore about flows, not protocol reliability.

Connection Tracking and NAT

Connection tracking and NAT are tightly coupled.

Consider a home router.

Laptop
192.168.1.20
      │
      ▼
Linux Router
Public IP
      │
      ▼
Internet

When the laptop sends a packet:

192.168.1.20:53000
        ↓
203.0.113.15:41000

The router changes the source address.

But when the reply returns:

203.0.113.15:41000

How does Linux know which internal machine should receive it?

Connection tracking remembers the translation.

Without conntrack, NAT would fail after the first packet.

This is why NAT depends heavily on connection tracking.

We'll examine NAT in detail in the next chapter.

Timeouts

Entries cannot remain in memory forever.

Each protocol has its own timeout values.

Examples:

• TCP established connections
• UDP flows
• ICMP exchanges

Once the timeout expires, the kernel removes the entry.

You can inspect some timeout-related settings:

sysctl net.netfilter.nf_conntrack_tcp_timeout_established

Example:

net.netfilter.nf_conntrack_tcp_timeout_established = 432000

This value represents the timeout (in seconds) for established TCP connections.

Connection Tracking Capacity

Since every active flow consumes kernel memory, the tracking table has a maximum size.

Check the current limit:

cat /proc/sys/net/netfilter/nf_conntrack_max

Check current usage:

cat /proc/sys/net/netfilter/nf_conntrack_count

Example:

Current entries : 1854
Maximum entries : 262144

If the table becomes full, Linux cannot track additional flows.

This can cause:

• connection failures
• dropped packets
• NAT failures
• firewall issues

Monitoring conntrack usage is therefore an important operational task on busy systems.

When Connection Tracking Causes Problems

Although conntrack is extremely useful, it can also become a bottleneck.

Common issues include:

• exhausted conntrack table
• incorrect timeout values
• asymmetric routing
• hardware offloading interactions
• extremely high connection rates

Large Kubernetes clusters, load balancers, and busy NAT gateways often require conntrack tuning.

When Is Connection Tracking Not Used?

Not every packet must be tracked.

Examples include:

• some high-performance routers
• specialized packet forwarding
• certain DDoS mitigation scenarios

Skipping connection tracking reduces memory usage and CPU overhead, but you lose stateful firewalling and NAT support.

For most Linux servers, leaving connection tracking enabled is the correct choice.

Debugging Checklist

When packets do not behave as expected, ask:

1. Does the flow appear in conntrack -L?
2. Is the state what you expect?
3. Is the conntrack table full?
4. Are timeout values appropriate?
5. Is NAT depending on this connection?

Answering these questions often narrows the problem quickly.

Summary

• Connection tracking is a kernel subsystem that remembers network flows.
• It is implemented as part of the netfilter framework.
• It tracks flows, not just TCP connections.
• Common conntrack states are NEW, ESTABLISHED, RELATED, and INVALID.
• Conntrack states are different from TCP protocol states.
• NAT relies heavily on connection tracking to map reply traffic back to the correct flow.
• The conntrack utility allows you to inspect, monitor, and troubleshoot tracked flows.
• Every tracked flow consumes kernel memory, making table sizing and monitoring important on busy systems.

In the next chapter, we'll build on this foundation by exploring Network Address Translation (NAT) and see how connection tracking enables address and port rewriting while preserving end-to-end communication.

Network Address Translation (NAT)

In the previous chapter, we learned that connection tracking gives Linux memory. Instead of treating every packet independently, the kernel remembers network flows and their state.

This memory enables one of the most widely used networking technologies in the world: Network Address Translation (NAT).

Today, almost every network uses NAT in some form. Your home router uses it. Cloud providers use it. Kubernetes nodes use it. Virtualization platforms use it. Firewalls and gateways use it.

Despite its widespread use, NAT is often misunderstood. Many engineers think of it as simply "changing IP addresses." While that is technically true, it misses the bigger picture.

NAT is a mechanism that rewrites packet headers while preserving communication between two endpoints.

In this chapter, we'll learn why NAT exists, how it works, why it depends on connection tracking, and how Linux performs address translation.

Why NAT Exists

Imagine the Internet without NAT.

Every device would need its own globally unique public IP address.

Laptop      Public IP
Phone       Public IP
TV          Public IP
Printer     Public IP
Camera      Public IP

That worked when the Internet was small.

As millions—and later billions—of devices came online, the available IPv4 address space became insufficient.

IPv4 provides approximately 4.3 billion addresses.

Although that sounds like a large number, many addresses are reserved, and the remaining addresses are far fewer than the number of internet-connected devices today.

The networking community needed a solution.

Two technologies emerged:

• IPv6, which dramatically increases the address space.
• NAT, which allows many devices to share fewer public addresses.

While IPv6 is the long-term solution, NAT remains one of the most common technologies in IPv4 networks.

Public and Private Addresses

To understand NAT, you first need to understand the difference between public and private IP addresses.

Public addresses are globally routable on the Internet.

Private addresses are not.

The most common private IPv4 ranges are:

If your laptop has the address:

192.168.1.50

that address only has meaning inside your local network.

Internet routers will not forward packets with that source address.

Therefore, before a packet leaves your home network, something must replace the private address with a public one.

That "something" is usually NAT.

What NAT Actually Does

At its core, NAT modifies packet headers.

Suppose your laptop sends:

Source      : 192.168.1.20
Destination : 8.8.8.8

The packet cannot travel across the Internet with that private source address.

Before the packet leaves your router, Linux changes it:

Before NAT

Src: 192.168.1.20
Dst: 8.8.8.8

↓

After NAT

Src: 203.0.113.15
Dst: 8.8.8.8

Only the source address changed.

Everything else about the communication remains the same.

The destination server never sees the private address.

Instead, it believes the packet originated from:

203.0.113.15

This process is called translation because Linux translates one network address into another.

NAT Is Transparent

One of NAT's most useful properties is transparency.

Neither the application nor the remote server knows that translation occurred.

The application simply calls:

connect()

The server simply receives packets.

Neither side explicitly requests NAT.

The kernel performs translation while forwarding packets.

This transparency explains why NAT is so widely deployed. Applications usually require no modification.

Types of NAT

Although NAT is often discussed as a single technology, there are several forms.

The two most important are:

• Source NAT (SNAT)
• Destination NAT (DNAT)

Another common form is Port Address Translation (PAT), often called masquerading.

Source NAT (SNAT)

Source NAT changes the source address of outgoing packets.

Original packet:

Src: 192.168.1.20
Dst: 8.8.8.8

After translation:

Src: 203.0.113.15
Dst: 8.8.8.8

The destination remains unchanged.

This is the type of NAT used by most home routers.

Thousands of internal devices can share a single public address.

Typical Uses of SNAT

SNAT is commonly used for:

• Home networks
• Enterprise gateways
• Cloud virtual machines
• Kubernetes worker nodes
• Virtual machines accessing external networks

Whenever private hosts need internet access, SNAT is usually involved.

Destination NAT (DNAT)

Destination NAT changes the destination address instead.

Imagine a packet arriving from the Internet:

Src: 198.51.100.20
Dst: 203.0.113.15

Linux rewrites the destination:

Src: 198.51.100.20
Dst: 192.168.1.10

The packet is then forwarded to the internal server.

DNAT is commonly used for:

• Port forwarding
• Reverse proxies
• Load balancers
• Publishing internal services

Port Address Translation (PAT)

What happens if hundreds of internal devices all use one public IP?

Changing only the IP address is not enough.

Consider two laptops:

Laptop A
192.168.1.20:53000

Laptop B
192.168.1.30:53000

Both connect to the same website.

If Linux translated only the IP address:

203.0.113.15:53000
203.0.113.15:53000

the replies would become ambiguous.

Linux solves this by translating ports as well.

Before

192.168.1.20:53000

↓

After

203.0.113.15:41001

Another connection becomes:

192.168.1.30:53000

↓

203.0.113.15:41002

Now every flow is unique.

This technique is called Port Address Translation (PAT).

Many Linux systems refer to this as masquerading, although masquerading is actually a specific implementation of source NAT that automatically uses the outgoing interface's address.

Why Connection Tracking Is Required

A common question is:

If Linux changes the source address, how does it know where to send the reply?

Suppose the outgoing packet becomes:

203.0.113.15:41001

Later the server replies:

Dst: 203.0.113.15:41001

How does Linux know that this reply actually belongs to:

192.168.1.20:53000

The answer is connection tracking.

When the first packet leaves the machine, conntrack creates an entry similar to:

Original

192.168.1.20:53000
↓

8.8.8.8:443

Translated

203.0.113.15:41001
↓

8.8.8.8:443

When the reply returns, Linux performs a lookup in the conntrack table.

The stored translation tells the kernel exactly:

• which internal machine originated the flow,
• which original source port was used,
• and how to restore the packet before delivery.

Without this state information, NAT would only work for the first outgoing packet.

The return traffic would have nowhere to go.

Following a Packet Through NAT

Let's follow a complete packet exchange.

Laptop
192.168.1.20

↓

Linux Router
Public IP:
203.0.113.15

↓

Internet

The laptop sends:

192.168.1.20:53000

↓

8.8.8.8:443

The router performs source NAT:

203.0.113.15:41001

↓

8.8.8.8:443

The server replies:

8.8.8.8:443

↓

203.0.113.15:41001

The router consults connection tracking.

It restores:

8.8.8.8:443

↓

192.168.1.20:53000

The laptop receives exactly what it expects.

Neither endpoint knows translation occurred.

Where NAT Happens

NAT is part of the netfilter framework.

Conceptually, translation happens at two different stages of the packet path.

Destination NAT

Destination NAT happens before the kernel makes its forwarding or local-delivery decision.

Why?

Because after changing the destination address, Linux must route the packet using its new destination.

Source NAT

Source NAT happens just before the packet leaves the machine.

At this point, Linux already knows:

• which interface will transmit the packet,
• which route it will use,
• and which source address should be translated.

This ordering keeps routing decisions consistent while allowing the packet to leave with the correct public address.

In later chapters, you'll see exactly how iptables and nftables attach NAT rules to these stages.

Common NAT Scenarios

Home Router

Many private devices share one public address.

Laptop
Phone
TV
Game Console

↓

Router (NAT)

↓

Internet

Cloud Virtual Machine

Private cloud instances access the Internet through a public gateway performing source NAT.

Port Forwarding

External users access an internal web server through destination NAT.

Internet

↓

Public IP

↓

Internal Web Server

Container Platforms

Container runtimes frequently use NAT so containers can communicate outside their private network.

Although the exact implementation differs between platforms, the underlying concept is the same.

Limitations of NAT

Although NAT solved the IPv4 address shortage, it also introduced complexity.

Some drawbacks include:

• breaks true end-to-end addressing,
• makes troubleshooting more difficult,
• complicates peer-to-peer communication,
• requires additional kernel state,
• increases dependence on connection tracking.

For these reasons, IPv6 was designed so that widespread NAT would not be necessary.

Even so, NAT remains essential in today's IPv4 networks.

Observe NAT on a Home Network

If your Linux machine is behind a typical home router, compare your local IP address with your public IP.

First, inspect your local address:

ip addr show

Then determine your public address using any public IP lookup service from a web browser.

You will notice that they are different.

This difference exists because your router is performing source NAT before packets reach the Internet.

Observe Connection Tracking During NAT

Generate some outbound traffic:

curl https://example.com

Then inspect the connection tracking table:

sudo conntrack -L

Look for an entry matching your connection.

Although you have not configured any NAT rules yourself, this demonstrates that connection tracking is maintaining flow state that NAT depends upon.

Compare Private and Public Routes

Inspect your routing table:

ip route

Notice that your machine routes traffic toward a gateway using private addresses.

The gateway—not your host—is responsible for translating packets before they enter the public Internet.

This reinforces an important idea:

Routing decides where packets go. NAT changes how they appear while they travel.

NAT Is Not Routing

Routing and NAT are closely related, but they solve different problems.

Routing answers:

Where should this packet go?

NAT answers:

Should this packet's addresses or ports be rewritten before it continues?

A packet may be routed without NAT.

A packet may also undergo NAT while following the same route.

They are independent operations that happen at different stages of packet processing.

Summary

• NAT rewrites packet headers while preserving communication between endpoints.
• It was introduced primarily to allow many devices to share a limited number of IPv4 addresses.
• Source NAT modifies source addresses; destination NAT modifies destination addresses.
• Port Address Translation (PAT) allows many simultaneous connections to share a single public IP address by translating ports.
• Connection tracking is essential because it remembers how translated packets map back to their original flows.
• Source NAT and destination NAT occur at different points in the packet-processing pipeline.
• NAT is widely used in home networks, cloud platforms, virtualization systems, and container environments.
• NAT is not routing; routing determines the path, while NAT rewrites packet headers along that path.

In the next chapter, we will finally begin configuring packet-processing policy directly by learning iptables, the traditional userspace interface for the Linux netfilter framework.

iptables: Building Firewalls on Linux

In the previous chapter, we learned that netfilter is the Linux kernel framework responsible for packet filtering, NAT, and packet manipulation. However, the kernel itself does not provide a convenient way to configure these rules.

For many years, that role was filled by iptables.

Although modern Linux distributions are gradually adopting nftables, iptables remains widely used. Many production servers, cloud images, embedded systems, appliances, and existing automation still rely on it. Understanding iptables is therefore an important skill, even if you primarily use nftables in new deployments.

In this chapter, you'll learn how iptables is organized, how packets are matched against rules, and how to build simple but practical firewalls.

What is iptables?

iptables is a userspace utility that configures the IPv4 packet filtering rules inside the Linux kernel.

It allows administrators to define rules that decide whether packets should be:

• accepted,
• dropped,
• rejected,
• logged,
• modified,
• or passed to another chain for additional processing.

Remember:

iptables is not the firewall itself. It is the tool used to configure the netfilter firewall inside the Linux kernel.

Checking Your System

Before working with iptables, determine whether your system uses it directly or through the newer nftables backend.

iptables --version

Example:

iptables v1.8.11 (nf_tables)

or

iptables v1.8.11 (legacy)

• legacy means the classic kernel backend.
• nf_tables means the iptables command is translating rules into nftables.

Fortunately, the command syntax remains nearly identical.

The iptables Architecture

iptables organizes rules into three levels:

Tables
    └── Chains
            └── Rules

For example:

filter
├── INPUT
├── OUTPUT
└── FORWARD

Each rule is evaluated in order until a decision is made.

Think of it like this:

Incoming packet
       │
       ▼
Rule 1 ? No
       │
Rule 2 ? No
       │
Rule 3 ? Yes
       │
       ▼
 DROP

Rule order is extremely important.

Tables

Most administrators only interact with a few tables.

filter

The default table.

Responsible for:

• firewall rules
• allowing traffic
• blocking traffic

You'll spend most of your time here.

nat

Responsible for:

• Source NAT
• Destination NAT
• Port forwarding
• Masquerading

mangle

Used to modify packet headers.

Examples include:

• changing TTL
• modifying DSCP
• packet marking

raw

Allows special packet processing before connection tracking.

Mostly used in advanced networking environments.

Chains

Inside each table are chains.

The filter table contains:

INPUT
OUTPUT
FORWARD

Each corresponds to a netfilter hook you learned previously.

INPUT

Packets destined for the local machine.

Example:

SSH connections.

OUTPUT

Packets generated locally.

Example:

A web request made by curl.

FORWARD

Packets routed through the machine.

Example:

Linux acting as a router.

Rule Matching

A rule contains two parts:

• match conditions
• target (action)

Example:

iptables -A INPUT -p tcp --dport 22 -j ACCEPT

Breaking it apart:

-A INPUT

Append to INPUT chain.

-p tcp

Match TCP packets.

--dport 22

Destination port 22.

-j ACCEPT

Accept the packet.

A packet that does not match simply continues to the next rule.

Targets

The most common targets are:

Listing Rules

Display every rule:

sudo iptables -L

Show numeric addresses:

sudo iptables -L -n

Show packet counters:

sudo iptables -L -v -n

Example:

Chain INPUT

pkts bytes target
245 18340 ACCEPT
12   720 DROP

The counters are extremely useful when debugging.

Rule Order Matters

Consider these rules:

1 ACCEPT tcp port 22
2 DROP all

SSH works.

Now reverse them:

1 DROP all
2 ACCEPT tcp port 22

SSH never reaches rule 2.

iptables always evaluates rules from top to bottom.

The first matching rule wins.

Default Policies

Every chain has a default policy.

Check it:

sudo iptables -L

Example:

Chain INPUT (policy ACCEPT)

or

Chain INPUT (policy DROP)

If no rule matches, the default policy determines what happens.

Hands-on Lab: Create a Simple Firewall

Warning

Perform these labs on a virtual machine. An incorrect firewall rule can lock you out of a remote server.

Step 1

Display current rules.

sudo iptables -L -n -v

Initially you may see:

Chain INPUT (policy ACCEPT)

Step 2

Allow SSH.

sudo iptables -A INPUT -p tcp --dport 22 -j ACCEPT

Step 3

Allow ICMP.

sudo iptables -A INPUT -p icmp -j ACCEPT

Step 4

Drop everything else.

sudo iptables -A INPUT -j DROP

Your rule set becomes:

INPUT

Allow SSH

Allow ICMP

Drop everything else

Test:

ping server

Should succeed.

Then:

curl http://server

Should fail.

Hands-on Lab: Logging

Sometimes you don't want to drop traffic immediately.

Instead, log it.

sudo iptables -A INPUT -j LOG --log-prefix "DROP: "

Logs can be viewed using:

journalctl -k

or

dmesg

Logging is invaluable during firewall development.

Hands-on Lab: Packet Counters

Generate traffic:

ping localhost

Now inspect:

sudo iptables -L -v -n

Notice how packet and byte counters increase.

These counters help answer questions like:

• Is the rule matching?
• How often?
• Is this rule ever used?

Inserting Rules

Appending is not always enough.

Suppose you already have:

1 DROP all

Adding:

iptables -A INPUT -p tcp --dport 22 -j ACCEPT

doesn't help.

Instead:

sudo iptables -I INPUT 1 -p tcp --dport 22 -j ACCEPT

The rule is inserted at position 1.

Always remember:

• -A appends
• -I inserts

Deleting Rules

List rules with numbers:

sudo iptables -L --line-numbers

Example:

1 ACCEPT tcp 22

2 DROP all

Delete rule 2:

sudo iptables -D INPUT 2

Flushing Rules

Remove every rule:

sudo iptables -F

This flushes the current table.

It does not change default policies.

Be careful on production servers.

Saving Rules

Unlike configuration files, iptables rules exist only in kernel memory.

After a reboot, they disappear unless saved.

Different Linux distributions use different mechanisms.

Examples include:

• iptables-save
• iptables-restore
• netfilter-persistent
• distribution-specific startup scripts

We'll keep persistence outside the scope of this chapter because it varies across distributions.

Common Mistakes

Forgetting Loopback

Blocking loopback breaks many applications.

Always allow it.

Blocking Established Connections

Without stateful rules, replies to existing connections may be blocked.

We'll improve our firewall after learning connection tracking.

Testing Through SSH

Always be careful when modifying firewall rules over SSH.

A single incorrect rule can disconnect your session.

Whenever possible:

• use a VM,
• keep another console open,
• or have out-of-band access.

Best Practices

• Keep rules simple.
• Comment complex rules.
• Put specific rules before general ones.
• Monitor packet counters.
• Test incrementally.
• Understand every rule before deploying it.

Remember:

A firewall should be predictable. Complexity often creates security problems rather than solving them.

Looking Ahead

The examples in this chapter intentionally avoided stateful filtering. Every rule treated packets independently.

In real production environments, almost every firewall relies on connection tracking.

Instead of writing rules that allow both directions of every connection, the firewall can recognize packets that belong to an already established flow.

That makes rule sets both simpler and more secure.

We'll explore that in the next chapter on Connection Tracking, and later revisit firewall design using stateful rules.

Summary

• iptables is the traditional userspace interface for configuring netfilter.
• Rules are organized into tables, chains, and rules.
• Packets are evaluated from top to bottom.
• The first matching rule determines the outcome.
• Rule order is critical.
• The filter table is used for firewalling.
• Packet counters are valuable for troubleshooting.
• Rules are stored in kernel memory and must be saved separately if they should survive a reboot.
• Understanding iptables makes it much easier to understand existing Linux systems, even if new deployments use nftables.

nftables: Modern Linux Firewalling

For nearly two decades, iptables was the standard way to configure the Linux firewall. While it is still widely used, modern Linux distributions are increasingly adopting nftables as the preferred packet filtering framework.

Unlike iptables, which evolved over many years through multiple utilities and extensions, nftables was designed as a clean replacement with a simpler architecture, better performance, and greater flexibility.

If you're starting with a modern Linux distribution such as Debian 12, Ubuntu 24.04, Fedora, or RHEL 9, there is a good chance that your firewall is already powered by nftables, even if you still use the iptables command.

In this chapter, you'll learn how nftables is organized, how to write firewall rules, and how to build practical firewalls using its native syntax.

Why nftables?

As Linux networking evolved, the limitations of iptables became increasingly apparent.

Some of its challenges included:

• separate tools for IPv4 and IPv6,
• duplicated rule sets,
• multiple independent utilities (iptables, ip6tables, ebtables, arptables),
• inefficient rule evaluation in large firewalls,
• inconsistent syntax between features.

To address these issues, the Linux kernel introduced nftables.

The goals were:

• one framework for all protocol families,
• simpler syntax,
• more flexible rule matching,
• better performance,
• easier maintenance.

Today, nftables is considered the future of Linux firewalling.

nftables Architecture

Like iptables, nftables configures the netfilter framework.

However, its internal organization is more flexible.

Instead of predefined tables and chains, you create them yourself.

The hierarchy is:

Tables
    └── Chains
            └── Rules

Unlike iptables, chains are explicitly connected to netfilter hooks when they are created.

Tables

A table groups related chains.

Create a table:

sudo nft add table inet firewall

List tables:

sudo nft list tables

Example:

table inet firewall

Notice the protocol family:

• ip
• ip6
• inet
• arp
• bridge
• netdev

The inet family is especially useful because one rule set can filter both IPv4 and IPv6 traffic.

Chains

Unlike iptables, chains are not predefined.

You create them and specify:

• which hook they attach to,
• their priority,
• and their default policy.

Example:

sudo nft add chain inet firewall input \
'{ type filter hook input priority 0; policy accept; }'

This creates a chain attached to the INPUT hook.

The important fields are:

• type — filter, nat, route
• hook — input, output, forward, prerouting, postrouting
• priority — execution order relative to other chains
• policy — default action

Rules

Rules are added to chains.

Example:

sudo nft add rule inet firewall input tcp dport 22 accept

Breaking it down:

• tcp
• destination port 22
• action accept

The syntax is intentionally close to natural language.

Another example:

sudo nft add rule inet firewall input icmp accept

Listing the Ruleset

Display the complete configuration:

sudo nft list ruleset

Example:

table inet firewall {
    chain input {
        type filter hook input priority 0;
        policy accept;

        tcp dport 22 accept
        icmp accept
    }
}

Unlike iptables, a single command displays the entire ruleset.

Rule Evaluation

Packets are evaluated sequentially.

For example:

tcp dport 22 accept
icmp accept
drop

If an SSH packet arrives:

• rule 1 matches
• evaluation stops
• packet is accepted

If an HTTP packet arrives:

• rule 1 fails
• rule 2 fails
• final rule drops the packet

Just like iptables, rule order matters.

Matches

Rules can match many packet attributes.

Examples include:

Match protocol:

tcp

Match source address:

ip saddr 192.168.1.100

Match destination address:

ip daddr 10.0.0.5

Match source port:

tcp sport 443

Match destination port:

tcp dport 80

Multiple conditions can be combined naturally.

Hands-on Lab: Create Your First Firewall

Warning

Perform these exercises on a virtual machine. Incorrect firewall rules can interrupt remote access.

Step 1

Create a table.

sudo nft add table inet firewall

Step 2

Create an input chain.

sudo nft add chain inet firewall input \
'{ type filter hook input priority 0; policy drop; }'

Notice that the default policy is now drop.

Step 3

Allow loopback traffic.

sudo nft add rule inet firewall input iif lo accept

Many applications rely on loopback communication.

Step 4

Allow established connections.

sudo nft add rule inet firewall input ct state established,related accept

This rule depends on connection tracking, which you learned in the previous chapter.

Without it, replies to existing connections would be blocked.

Step 5

Allow SSH.

sudo nft add rule inet firewall input tcp dport 22 accept

Step 6

Allow ICMP.

sudo nft add rule inet firewall input icmp accept

Display the final configuration:

sudo nft list ruleset

Your ruleset should resemble:

table inet firewall {
    chain input {
        policy drop;

        iif lo accept
        ct state established,related accept
        tcp dport 22 accept
        icmp accept
    }
}

You now have a simple stateful firewall.

Sets

One of the most powerful features of nftables is sets.

Instead of writing multiple nearly identical rules:

tcp dport 22 accept
tcp dport 80 accept
tcp dport 443 accept

Create a single rule:

sudo nft add rule inet firewall input tcp dport {22,80,443} accept

This is easier to read and scales much better.

Sets are one of the biggest advantages of nftables over iptables.

Counters

Rules can maintain packet counters.

Example:

sudo nft add rule inet firewall input counter tcp dport 22 accept

Display them:

sudo nft list ruleset

Example:

counter packets 120 bytes 9120

Counters are useful for verifying whether a rule is actually matching traffic.

Logging

To log matching packets:

sudo nft add rule inet firewall input log prefix "INPUT: "

Kernel logs can be viewed using:

journalctl -k

Logging is invaluable when diagnosing firewall behavior.

Deleting Rules

Each rule has a handle number.

Display handles:

sudo nft -a list ruleset

Example:

tcp dport 22 accept # handle 12

Delete it:

sudo nft delete rule inet firewall input handle 12

Flushing Rules

Remove every rule from a table:

sudo nft flush table inet firewall

Delete the table completely:

sudo nft delete table inet firewall

Saving the Configuration

Like iptables, nftables rules exist in kernel memory.

To save them:

sudo nft list ruleset > firewall.nft

Load them again:

sudo nft -f firewall.nft

Most Linux distributions provide a system service that automatically loads a saved configuration during boot. The exact file location and service name vary between distributions.

nftables vs iptables

The concepts remain the same because both configure the same netfilter framework.

The difference lies in usability and architecture.

Common Mistakes

Forgetting Loopback

Applications frequently communicate through 127.0.0.1.

Blocking loopback can cause unexpected failures.

Always allow it explicitly.

Omitting Established Connections

Without:

ct state established,related accept

return traffic is often blocked.

This is one of the most common mistakes when creating a firewall from scratch.

Forgetting the Default Policy

If your chain policy is drop, only explicitly allowed traffic will pass.

This is generally a good security practice, but it requires careful testing.

Editing Production Firewalls Blindly

Always keep an alternative management path, such as:

• a virtual machine console,
• out-of-band management,
• or a second SSH session.

This reduces the risk of locking yourself out.

Best Practices

• Prefer the inet family whenever possible.
• Build stateful firewalls using connection tracking.
• Use sets instead of repetitive rules.
• Keep rules ordered from most specific to most general.
• Monitor counters during troubleshooting.
• Test firewall changes incrementally.
• Store firewall rules in version control.

A firewall should be predictable, readable, and easy to audit.

Looking Ahead

So far, we have explored how Linux processes packets, how netfilter provides hook points inside the kernel, and how both iptables and nftables configure packet filtering.

The remaining challenge is understanding why packets are accepted or dropped when something goes wrong.

In the next chapter, we will learn how to trace packets through the Linux networking stack, identify where they are being processed, and use debugging tools to troubleshoot complex networking problems.

Summary

• nftables is the modern userspace interface for configuring the Linux netfilter framework.
• It replaces the older iptables family with a unified and more flexible design.
• Rules are organized into tables, chains, and rules, but chains are explicitly attached to netfilter hooks.
• The inet family allows a single ruleset to filter both IPv4 and IPv6 traffic.
• Stateful firewalls rely on connection tracking through ct state.
• Sets make rule definitions more compact and efficient.
• Rule counters and logging are valuable tools for debugging.
• nftables is the recommended choice for new Linux deployments, while understanding iptables remains important for maintaining existing systems.

Tracing Packet Flow

So far in this part of the book, we have learned how packets move through the Linux kernel, how netfilter processes them, how connection tracking maintains flow state, how NAT rewrites addresses, and how firewalls influence packet handling.

The next challenge is troubleshooting.

When a network problem occurs, the question is usually not:

"Is networking broken?"

The real question is:

"Where did the packet stop?"

Did it leave the source host?

Did it reach the destination?

Was it dropped by a firewall?

Did routing send it somewhere unexpected?

Did NAT rewrite it?

Did the application receive it?

Packet tracing is the process of answering these questions by following a packet through the network stack and identifying where things go wrong.

This chapter introduces a systematic approach to tracing packet flow on Linux.

The Most Important Troubleshooting Principle

When debugging networking problems, avoid making assumptions.

Many engineers immediately jump to conclusions:

• "DNS is broken."
• "The firewall is blocking it."
• "The application isn't listening."
• "The network team changed something."

Instead, follow the packet.

Networking troubleshooting becomes much easier when you answer one question at a time:

1. Was the packet created?
2. Did it leave the source?
3. Did it reach the destination?
4. Was it routed correctly?
5. Was it filtered?
6. Was it delivered to the application?
7. Did the response return?

The goal is to find the exact point where the packet disappears.

A Packet's Journey

Consider a simple HTTP request:

Client
  |
  | TCP SYN
  v
Router
  |
  v
Server
  |
  | TCP SYN-ACK
  v
Client

A failure can occur at any point.

For example:

Client
  |
  | SYN
  v
Server
  |
  X Firewall drops packet

Or:

Client
  |
  | SYN
  v
Server
  |
  X No process listening

Or:

Client
  |
  X Incorrect route

The purpose of packet tracing is to identify which failure actually occurred.

Step 1: Verify the Application

Before investigating the network, verify that an application is actually listening.

Check listening sockets:

ss -tulpen

Example:

tcp LISTEN 0 128 0.0.0.0:80

Or search for a specific port:

ss -tulpen | grep :80

If nothing is listening, the packet may reach the host successfully and still fail.

Step 2: Verify Routing

Before a packet can leave a host, Linux must decide where to send it.

Check the routing decision:

ip route get 8.8.8.8

Example:

8.8.8.8 via 192.168.1.1 dev eth0 src 192.168.1.100

This shows:

• selected route,
• outgoing interface,
• source address,
• next hop.

If routing is incorrect, packet tracing should stop here because the packet will never reach its destination.

Step 3: Observe the Interface

The next question is:

Did the packet actually leave the machine?

Use tcpdump.

Open a terminal:

sudo tcpdump -i eth0 -nn

In another terminal:

curl http://example.com

You should see packets appear.

Example:

IP 192.168.1.100.53422 > 93.184.216.34.80

If no packets appear:

• application problem,
• routing problem,
• firewall problem,
• or local networking issue.

The packet never reached the wire.

Step 4: Observe Both Ends

When possible, capture traffic on both systems.

Client:

tcpdump -i eth0 -nn host SERVER_IP

Server:

tcpdump -i eth0 -nn host CLIENT_IP

Possible outcomes:

Seen on both hosts

Client -> Server

Network connectivity exists.

Investigate firewalls or applications.

Seen only on client

Client -> X

Problem exists somewhere in the network path.

Seen only on server

Rare but possible.

Usually indicates:

• asymmetric routing,
• capture mistake,
• or incorrect filtering.

Step 5: Check Firewall Counters

If packets reach the host but connections fail, inspect firewall activity.

iptables:

sudo iptables -L -v -n

nftables:

sudo nft list ruleset

Look for:

• packet counters increasing,
• unexpected drops,
• unexpected rejects.

Counters often reveal exactly which rule processed the packet.

Step 6: Inspect Connection Tracking

Connection tracking records flow state.

View current entries:

sudo conntrack -L

Example:

tcp 6 431999 ESTABLISHED

Search for a specific host:

sudo conntrack -L | grep 10.0.0.10

Useful questions:

• Did conntrack create a flow?
• Is the connection established?
• Is NAT being tracked?

If no entry exists, the packet may never have reached connection tracking.

Step 7: Inspect NAT

NAT frequently causes confusion because packet headers change during transit.

Consider:

Client 10.0.0.5
       |
       |
       v
Gateway
       |
       |
       v
Internet

The internet never sees:

10.0.0.5

It sees:

203.0.113.10

When tracing packets through NAT:

• capture before translation,
• capture after translation,
• compare source and destination addresses.

Remember:

A packet may be correct before NAT and different afterward.

Step 8: Verify Kernel Forwarding

When Linux acts as a router, forwarding must be enabled.

Check:

sysctl net.ipv4.ip_forward

Expected:

net.ipv4.ip_forward = 1

If forwarding is disabled:

net.ipv4.ip_forward = 0

Packets may arrive correctly and still never leave the host.

Using nft Trace

One of the most powerful debugging features in modern Linux is nftables tracing.

Enable tracing:

sudo nft add rule inet firewall input meta nftrace set 1

Monitor trace events:

sudo nft monitor trace

Example output:

trace id ...
packet:
rule:
verdict accept

This allows you to see:

• which chain processed the packet,
• which rule matched,
• which verdict was returned.

It is one of the closest things Linux has to a packet debugger.

Tracing a Failed SSH Connection

Suppose SSH does not work.

Start systematically.

Is SSH listening?

ss -tulpen | grep :22

Is the packet arriving?

tcpdump -i eth0 port 22

Is the firewall dropping it?

iptables -L -v -n

or

nft list ruleset

Is routing correct?

ip route get CLIENT_IP

Following this sequence usually identifies the problem quickly.

Using Traceroute

Sometimes the packet disappears between hosts.

Use:

traceroute 8.8.8.8

or:

tracepath 8.8.8.8

Example:

1 192.168.1.1
2 10.1.0.1
3 203.0.113.1

These tools help identify where packets stop in the network path.

Remember:

They show the network path, not the internal Linux kernel path.

Common Packet Tracing Workflow

A practical workflow is:

Application
    ↓
Socket
    ↓
Route
    ↓
Interface
    ↓
Firewall
    ↓
Conntrack
    ↓
NAT
    ↓
Network
    ↓
Destination

At each step ask:

Did the packet reach here?

If yes:

Move to the next layer.

If no:

Investigate that layer.

Common Mistakes

Looking at Only One Host

Always inspect both ends whenever possible.

A packet visible on one host may never reach the other.

Assuming Firewall Problems

Many connectivity problems are actually:

• routing issues,
• application issues,
• DNS issues,
• incorrect addresses.

Follow evidence instead of assumptions.

Forgetting NAT

Many engineers capture packets on both sides of a gateway and become confused because the addresses differ.

Always consider NAT when tracing traffic.

Ignoring Counters

Firewall and interface counters often reveal the answer immediately.

Check them before making changes.

Summary

• Packet tracing is the process of identifying where packets stop or change.
• Effective troubleshooting follows the packet instead of making assumptions.
• ss helps verify application sockets.
• ip route get shows routing decisions.
• tcpdump reveals packets entering and leaving interfaces.
• Firewall counters help identify filtering decisions.
• conntrack shows connection state.
• NAT can change packet headers and must be considered during tracing.
• nft monitor trace provides detailed packet-processing visibility.
• The most effective troubleshooting approach is to validate one layer at a time.

At this point, you have completed the packet-processing section of the book and have the tools needed to understand, observe, filter, translate, and troubleshoot packet flow on Linux.

Why Traditional Networks Don't Scale

In the previous chapters, we built networks using Linux bridges, virtual Ethernet devices, routing, and packet filtering. Everything worked well because all systems existed within a relatively small environment.

However, modern infrastructure is very different.

A Kubernetes cluster may contain hundreds of nodes. A cloud provider may operate tens of thousands of servers. Even a medium-sized enterprise can have hundreds of virtual machines spread across multiple racks, buildings, or data centers.

The networking techniques that work perfectly inside a small network begin to show limitations as the network grows.

This chapter explains:

• Why traditional Layer 2 networks become problematic at scale
• Why large networks rely heavily on Layer 3 routing
• Why overlays and tunnels exist
• The challenges that technologies like VXLAN are designed to solve

Understanding these problems is more important than understanding the solutions. Once you understand the limitations of traditional networking, technologies like GRE, VXLAN, EVPN, and SDN become much easier to understand.

The Traditional Ethernet Network

A typical Ethernet network looks like this:

        +---------+
        | Switch  |
        +---------+
         |   |   |
         |   |   |
      Host Host Host

Every device belongs to the same Layer 2 domain.

For example:

All devices:

• Share the same Ethernet segment
• Use ARP to discover MAC addresses
• Can communicate directly without routing

This model is simple and efficient.

For small networks, it works extremely well.

Understanding the Layer 2 Broadcast Domain

One of the most important concepts in networking is the broadcast domain.

A broadcast domain is the set of devices that receive Layer 2 broadcast traffic.

For example:

ARP Request:
Who has 10.10.10.12?
Tell 10.10.10.11

This packet is sent to:

ff:ff:ff:ff:ff:ff

which means:

Deliver this frame to everybody.

The switch floods the frame to all ports.

          Broadcast
              |
              v

    +------------------+
    |      Switch      |
    +------------------+
      |     |      |
      v     v      v

    Host  Host   Host
      A     B      C

Every host receives the frame.

Only Host B responds.

As the network grows, the amount of broadcast traffic grows as well.

Demonstrating Broadcast Traffic

You can observe this behavior directly.

On a Linux machine:

sudo tcpdump -i eth0 arp

Open another terminal:

ping 192.168.1.100

If the MAC address is unknown, you'll see something similar:

ARP, Request who-has 192.168.1.100 tell 192.168.1.10

This request is visible to every device in the Layer 2 network.

Even systems that are completely unrelated to the communication must receive and process the frame.

The Broadcast Problem

Imagine a network with:

• 10 hosts
• 100 hosts
• 1,000 hosts
• 10,000 hosts

ARP traffic grows.

Neighbor table updates grow.

Unknown unicast flooding grows.

Control traffic grows.

The network spends increasing amounts of time distributing packets that are not actual application traffic.

This is one reason large Layer 2 domains become inefficient.

MAC Address Tables Also Grow

Ethernet switches maintain MAC address tables.

A switch learns:

MAC Address  -> Port

Example:

00:11:22:33:44:55 -> Port 1
aa:bb:cc:dd:ee:ff -> Port 2

When a frame arrives, the switch looks up the destination MAC address and forwards it.

As networks become larger:

• More devices exist
• More MAC addresses exist
• More table entries must be maintained

Large switching domains require larger forwarding tables and more control-plane resources.

Layer 2 Loops Become Dangerous

Consider two switches connected incorrectly:

      +---------+
      |Switch A |
      +---------+
        |     |
        |     |
      +---------+
      |Switch B |
      +---------+

A loop exists.

Ethernet frames have no built-in mechanism that prevents indefinite forwarding.

A broadcast frame may circulate forever.

The result can be:

• Broadcast storms
• MAC table instability
• Network outages

This is why protocols such as STP (Spanning Tree Protocol) exist.

STP intentionally disables some links to prevent loops.

Hands-On: Observe a Bridge Learning MAC Addresses

Create a bridge: