Tracing Packet Flow

So far in this part of the book, we have learned how packets move through the Linux kernel, how netfilter processes them, how connection tracking maintains flow state, how NAT rewrites addresses, and how firewalls influence packet handling.

The next challenge is troubleshooting.

When a network problem occurs, the question is usually not:

"Is networking broken?"

The real question is:

"Where did the packet stop?"

Did it leave the source host?

Did it reach the destination?

Was it dropped by a firewall?

Did routing send it somewhere unexpected?

Did NAT rewrite it?

Did the application receive it?

Packet tracing is the process of answering these questions by following a packet through the network stack and identifying where things go wrong.

This chapter introduces a systematic approach to tracing packet flow on Linux.

The Most Important Troubleshooting Principle

When debugging networking problems, avoid making assumptions.

Many engineers immediately jump to conclusions:

  • "DNS is broken."
  • "The firewall is blocking it."
  • "The application isn't listening."
  • "The network team changed something."

Instead, follow the packet.

Networking troubleshooting becomes much easier when you answer one question at a time:

  1. Was the packet created?
  2. Did it leave the source?
  3. Did it reach the destination?
  4. Was it routed correctly?
  5. Was it filtered?
  6. Was it delivered to the application?
  7. Did the response return?

The goal is to find the exact point where the packet disappears.

A Packet's Journey

Consider a simple HTTP request:

Client
  |
  | TCP SYN
  v
Router
  |
  v
Server
  |
  | TCP SYN-ACK
  v
Client

A failure can occur at any point.

For example:

Client
  |
  | SYN
  v
Server
  |
  X Firewall drops packet

Or:

Client
  |
  | SYN
  v
Server
  |
  X No process listening

Or:

Client
  |
  X Incorrect route

The purpose of packet tracing is to identify which failure actually occurred.

Step 1: Verify the Application

Before investigating the network, verify that an application is actually listening.

Check listening sockets:

ss -tulpen

Example:

tcp LISTEN 0 128 0.0.0.0:80

Or search for a specific port:

ss -tulpen | grep :80

If nothing is listening, the packet may reach the host successfully and still fail.

Step 2: Verify Routing

Before a packet can leave a host, Linux must decide where to send it.

Check the routing decision:

ip route get 8.8.8.8

Example:

8.8.8.8 via 192.168.1.1 dev eth0 src 192.168.1.100

This shows:

  • selected route,
  • outgoing interface,
  • source address,
  • next hop.

If routing is incorrect, packet tracing should stop here because the packet will never reach its destination.

Step 3: Observe the Interface

The next question is:

Did the packet actually leave the machine?

Use tcpdump.

Open a terminal:

sudo tcpdump -i eth0 -nn

In another terminal:

curl http://example.com

You should see packets appear.

Example:

IP 192.168.1.100.53422 > 93.184.216.34.80

If no packets appear:

  • application problem,
  • routing problem,
  • firewall problem,
  • or local networking issue.

The packet never reached the wire.

Step 4: Observe Both Ends

When possible, capture traffic on both systems.

Client:

tcpdump -i eth0 -nn host SERVER_IP

Server:

tcpdump -i eth0 -nn host CLIENT_IP

Possible outcomes:

Seen on both hosts

Client -> Server

Network connectivity exists.

Investigate firewalls or applications.

Seen only on client

Client -> X

Problem exists somewhere in the network path.

Seen only on server

Rare but possible.

Usually indicates:

  • asymmetric routing,
  • capture mistake,
  • or incorrect filtering.

Step 5: Check Firewall Counters

If packets reach the host but connections fail, inspect firewall activity.

iptables:

sudo iptables -L -v -n

nftables:

sudo nft list ruleset

Look for:

  • packet counters increasing,
  • unexpected drops,
  • unexpected rejects.

Counters often reveal exactly which rule processed the packet.

Step 6: Inspect Connection Tracking

Connection tracking records flow state.

View current entries:

sudo conntrack -L

Example:

tcp 6 431999 ESTABLISHED

Search for a specific host:

sudo conntrack -L | grep 10.0.0.10

Useful questions:

  • Did conntrack create a flow?
  • Is the connection established?
  • Is NAT being tracked?

If no entry exists, the packet may never have reached connection tracking.

Step 7: Inspect NAT

NAT frequently causes confusion because packet headers change during transit.

Consider:

Client 10.0.0.5
       |
       |
       v
Gateway
       |
       |
       v
Internet

The internet never sees:

10.0.0.5

It sees:

203.0.113.10

When tracing packets through NAT:

  • capture before translation,
  • capture after translation,
  • compare source and destination addresses.

Remember:

A packet may be correct before NAT and different afterward.

Step 8: Verify Kernel Forwarding

When Linux acts as a router, forwarding must be enabled.

Check:

sysctl net.ipv4.ip_forward

Expected:

net.ipv4.ip_forward = 1

If forwarding is disabled:

net.ipv4.ip_forward = 0

Packets may arrive correctly and still never leave the host.

Using nft Trace

One of the most powerful debugging features in modern Linux is nftables tracing.

Enable tracing:

sudo nft add rule inet firewall input meta nftrace set 1

Monitor trace events:

sudo nft monitor trace

Example output:

trace id ...
packet:
rule:
verdict accept

This allows you to see:

  • which chain processed the packet,
  • which rule matched,
  • which verdict was returned.

It is one of the closest things Linux has to a packet debugger.

Tracing a Failed SSH Connection

Suppose SSH does not work.

Start systematically.

Is SSH listening?

ss -tulpen | grep :22

Is the packet arriving?

tcpdump -i eth0 port 22

Is the firewall dropping it?

iptables -L -v -n

or

nft list ruleset

Is routing correct?

ip route get CLIENT_IP

Following this sequence usually identifies the problem quickly.

Using Traceroute

Sometimes the packet disappears between hosts.

Use:

traceroute 8.8.8.8

or:

tracepath 8.8.8.8

Example:

1 192.168.1.1
2 10.1.0.1
3 203.0.113.1

These tools help identify where packets stop in the network path.

Remember:

They show the network path, not the internal Linux kernel path.

Common Packet Tracing Workflow

A practical workflow is:

Application
    ↓
Socket
    ↓
Route
    ↓
Interface
    ↓
Firewall
    ↓
Conntrack
    ↓
NAT
    ↓
Network
    ↓
Destination

At each step ask:

Did the packet reach here?

If yes:

Move to the next layer.

If no:

Investigate that layer.

Common Mistakes

Looking at Only One Host

Always inspect both ends whenever possible.

A packet visible on one host may never reach the other.

Assuming Firewall Problems

Many connectivity problems are actually:

  • routing issues,
  • application issues,
  • DNS issues,
  • incorrect addresses.

Follow evidence instead of assumptions.

Forgetting NAT

Many engineers capture packets on both sides of a gateway and become confused because the addresses differ.

Always consider NAT when tracing traffic.

Ignoring Counters

Firewall and interface counters often reveal the answer immediately.

Check them before making changes.

Summary

  • Packet tracing is the process of identifying where packets stop or change.
  • Effective troubleshooting follows the packet instead of making assumptions.
  • ss helps verify application sockets.
  • ip route get shows routing decisions.
  • tcpdump reveals packets entering and leaving interfaces.
  • Firewall counters help identify filtering decisions.
  • conntrack shows connection state.
  • NAT can change packet headers and must be considered during tracing.
  • nft monitor trace provides detailed packet-processing visibility.
  • The most effective troubleshooting approach is to validate one layer at a time.

At this point, you have completed the packet-processing section of the book and have the tools needed to understand, observe, filter, translate, and troubleshoot packet flow on Linux.