Following a Packet Through the Linux Kernel

In the previous chapters, we focused on the visible parts of Linux networking: interfaces, routes, sockets, and traffic observation. Now we go one layer deeper.

This chapter follows a packet as it enters and leaves the Linux kernel. The goal is not to memorize every kernel function, but to build a mental model of where packets go, what decisions are made, and which tools let you inspect those decisions.

By the end of this chapter, you should be able to answer questions like:

Why does a packet arrive on one interface but get delivered to a process on another?
Why is a packet dropped before it reaches an application?
What does Linux do after a packet hits a network card?
Why do tools like tcpdump, ss, ip route get, nft, and conntrack show different parts of the same story?

The Big Picture

A packet does not go directly from the network card to your application.

Linux processes it through a pipeline:

The network card receives the frame.
The kernel driver hands the data to the networking stack.
The kernel decides whether the packet is:
- destined for the local host,
- forwarded to another host,
- or dropped.
If the packet is local, Linux delivers it to the correct socket.
If the packet is outgoing, Linux builds the packet, applies routing decisions, and transmits it.

The important lesson is this:

Linux networking is not a single path. It is a set of decision points.

At each step, the kernel may:

accept the packet,
rewrite it,
queue it,
forward it,
or discard it.

A Packet’s Journey: Ingress and Egress

Every packet can be viewed from two directions.

Ingress path

This is what happens when a packet enters the machine from the network.

Typical flow:

NIC receives the frame
driver places data into memory
kernel creates an skb
packet moves up the stack
routing decision is made
packet is delivered locally or forwarded

Egress path

This is what happens when a process sends data out.

Typical flow:

application calls send()
kernel creates packet data
routing selects the output interface
packet may be modified by firewall/NAT
driver transmits the frame

The same host may be both a sender and a receiver at the same time, and Linux handles both paths simultaneously.

The `skb`: Linux’s Packet Container

Inside the kernel, packets are represented by a structure called sk_buff, usually shortened to skb.

Think of an skb as the kernel’s working envelope for a packet. It contains:

packet data,
metadata,
protocol information,
interface information,
and pointers used by the network stack.

The skb is not just payload. It is also the kernel’s way of tracking:

where the packet came from,
where it is going,
how much of it has been parsed,
and which subsystem should handle it next.

You do not need to know the C structure to understand Linux networking, but you should understand the idea:

The kernel does not pass “raw bytes” between networking stages. It passes a packet object with metadata.

That metadata is what allows Linux to make routing, firewall, and delivery decisions efficiently.

From NIC to Kernel

When a packet arrives on the wire, the network interface card receives the Ethernet frame and hands it to the kernel through the driver.

At a high level:

the NIC stores the frame in memory,
the driver notices the arrival,
the kernel reads the data,
and the packet is inserted into the network stack.

Modern Linux systems often use interrupt moderation and NAPI to reduce overhead. The exact driver behavior depends on the hardware, but the conceptual model stays the same:

The hardware detects the packet, and the kernel turns it into an skb.

For our purposes, the important question is not how many interrupts happened but what the kernel does after it gets the packet.

The First Decision: Local Delivery or Forwarding

After a packet enters the stack, Linux asks a basic question:

Is this packet meant for this host, or should it be forwarded somewhere else?

This decision depends on:

destination IP address,
routing table,
interface configuration,
and kernel forwarding settings.

There are two broad outcomes.

1. Local delivery

If the destination IP belongs to the local machine, Linux tries to deliver the packet to a socket.

This is the case when:

you ssh into the host,
a web server is listening on port 80,
or a process is bound to the destination address.

2. Forwarding

If the packet is not for the local host but the machine has routing enabled, Linux forwards the packet to another interface.

This is what happens on routers, gateways, Kubernetes nodes, and Linux machines acting as network intermediaries.

Forwarding is one of the key differences between an ordinary host and a router.

Local Delivery: From Packet to Socket

When a packet is meant for the local machine, Linux still does not hand it directly to the application.

The kernel must match the packet to a socket.

That involves checks such as:

destination IP address,
destination port,
protocol (TCP or UDP),
socket binding state,
and connection state.

For example:

A packet destined to 10.0.0.10:80/tcp may be delivered to an nginx process.
A packet destined to 127.0.0.1:5432/tcp may go to a local PostgreSQL server.
A packet destined to a closed port may trigger an ICMP error or a TCP RST.

This is why open ports matter: they are sockets that are ready to receive traffic.

Outgoing Packets: From Socket to Wire

When an application sends data, the path goes in the opposite direction.

A simplified egress flow looks like this:

application calls send()
kernel writes data into socket buffers
TCP or UDP builds transport-layer headers
IP layer chooses a destination route
link layer resolves the next hop
frame is sent to the NIC
NIC transmits it on the wire

Each layer adds information:

TCP or UDP adds port information.
IP adds source and destination addresses.
Ethernet adds source and destination MAC addresses.

Linux does not guess where to send the packet. It consults routing, neighbor resolution, and interface state.

Routing Is Still Central

Even in packet processing, routing remains one of the main decision points.

Before Linux sends a packet out, it answers:

Which destination network does this packet belong to?
Which interface should be used?
What is the next hop?
What source address should be selected?

You already saw routing in the routing chapter. Here, routing becomes operational:

Routing is the decision step that chooses the packet’s next movement in the kernel.

A useful command for understanding this is:

ip route get 8.8.8.8

Example output:

8.8.8.8 via 192.168.1.1 dev eth0 src 192.168.1.50 uid 1000
    cache

This tells you:

where Linux will send the packet,
which interface it will use,
and which source address it will choose.

That is not a theoretical answer. It is the kernel’s actual decision.

What Happens Before a Packet Reaches a Process

Many people assume a packet arrives, then the application reads it.

In reality, Linux does much more work before the application ever sees it.

For incoming traffic, the kernel may:

verify the packet format,
check routing rules,
consult firewall rules,
update connection-tracking state,
perform NAT rewrites,
and only then deliver the packet to the socket.

This means a packet can be:

accepted by the NIC but dropped by firewall rules,
routed correctly but never delivered because no process is listening,
or translated by NAT before reaching the application.

The application sees only the final result.

A Practical Example: Watching a TCP Connection

Let’s build a simple mental model using an SSH or HTTP connection.

Suppose a client connects to a server on port 22.

On the client

The application calls connect().
Linux selects a source port.
Linux finds a route to the server.
The packet leaves the host.

On the server

The NIC receives the SYN packet.
Linux checks whether the destination IP belongs to the host.
The kernel looks for a listening socket on port 22.
If SSH is listening, the packet is delivered to that socket.
The SSH daemon responds with a SYN-ACK.

This is the moment where protocol state matters. TCP is not just packet exchange; it is a coordinated state machine implemented across kernel and application.

You will see more of that in the connection-tracking chapter, but the basic idea belongs here:

The packet is not “for an app” until the kernel has matched it to a socket.

Tools for Observing the Path

You cannot see every internal kernel step directly, but you can infer a lot.

`ip route get`

Shows routing decisions.

ip route get 1.1.1.1

Useful when you want to know:

which interface Linux will use,
which gateway it prefers,
and which source address it picks.

`ss`

Shows socket state in the kernel.

ss -tulpen

Useful when you want to know:

what is listening,
which process owns the socket,
and whether a connection exists.

This connects packet delivery to application state.

`tcpdump`

Shows packets at the interface boundary.

tcpdump -i eth0 -nn

Useful when you want to know:

whether packets entered or left the interface,
what ports and addresses are involved,
and whether the wire sees the traffic at all.

Important limitation:

tcpdump sees packets on the interface. It does not tell you everything the kernel does after that.

`ip -s link`

Shows interface statistics.

ip -s link show dev eth0

Useful when you want to inspect:

received/transmitted packets,
drops,
errors,
and queue-level counters.

This helps you distinguish:

“the packet never arrived,” from
“the packet arrived but was dropped later.”

Hands-On Lab 1: Check the Route the Kernel Will Use

Pick any reachable destination, for example a public DNS server or your gateway.

ip route get 8.8.8.8

Look at the output and identify:

the outgoing interface,
the next hop,
the source IP address.

Now compare that with:

ip addr show
ip route show

You should see how the kernel’s routing decision matches the machine’s interface and route configuration.

What to learn from this

The packet path begins before transmission. Linux chooses the route first, then sends the packet.

Hands-On Lab 2: Watch a Packet Leave the Host

Open one terminal and run:

tcpdump -i eth0 -nn icmp

In another terminal, run:

ping 8.8.8.8

You should see ICMP echo requests and replies.

Now ask yourself:

Which interface carried the packet?
Did the packet leave the host?
Did the reply come back?

This lab shows the boundary between local kernel processing and network transmission.

Hands-On Lab 3: Observe a Listening Socket

Start a simple service, such as a Python HTTP server:

python3 -m http.server 8080

In another terminal:

ss -tulpen | grep 8080

You should see the listening socket.

Now test it locally:

curl http://127.0.0.1:8080

Then test it from another host if possible.

What matters here is the chain:

packet arrives,
kernel matches port 8080,
socket receives the traffic,
server process reads it.

The application does not receive random network data. It receives traffic that has already passed kernel matching.

Localhost Is Still Networking

A common mistake is to think 127.0.0.1 is “not real networking.”

It is real networking inside the kernel.

When a process sends traffic to localhost:

the packet still enters the networking stack,
routing still happens,
socket lookup still happens,
and the packet is still delivered through kernel networking logic.

The difference is that the traffic never leaves the machine.

This is useful because it proves a key idea:

Networking is not defined by distance. It is defined by the kernel path the packet takes.

Why Some Packets Disappear

From the application’s point of view, packets can vanish for many reasons:

no route exists,
the firewall dropped them,
no process is listening,
the destination address is wrong,
the interface is down,
or the packet was malformed.

This is why packet debugging is often about narrowing the layer where failure occurs.

A good debugging sequence is:

Is the packet leaving the host?
Is the packet reaching the host?
Is Linux routing it correctly?
Is a socket listening?
Is a firewall or NAT rule altering it?
Is the application handling it?

That sequence is the basis of most Linux network troubleshooting.

The Kernel Is the Traffic Cop

At this point, the right mental model should be clear.

Linux is not just a passive conduit. It is an active traffic cop.

For every packet, the kernel may:

inspect it,
classify it,
route it,
filter it,
translate it,
queue it,
forward it,
or deliver it.

That is why Linux networking is powerful. It is also why networking problems often require kernel-level thinking.

A packet’s life inside Linux is a sequence of decisions, not a straight line.

What Comes Next

In this chapter, we followed packets through the Linux kernel at a conceptual level. We did not yet examine all the subsystems that can influence packet flow.

In the next chapters, we will go deeper into the pieces that modify or intercept packets:

netfilter, the framework behind Linux packet filtering,
connection tracking, which remembers flows,
NAT, which rewrites addresses and ports,
and firewall tooling such as iptables and nftables.

Once you understand the packet path, these tools stop feeling magical. They become specific control points in a known pipeline.

Summary

Packets enter and leave Linux through a defined kernel path.
The skb is the kernel’s packet container.
Linux decides whether a packet is local, forwarded, or dropped.
Routing, firewalling, NAT, and socket lookup all happen inside the kernel.
tcpdump, ss, ip route get, and ip -s link help you observe different parts of the path.
The most important habit is to think in stages, not in a single “packet arrived” event.

The next chapter will focus on the first major decision framework in that path: netfilter.

Networking for Non-Network Engineers