Troubleshooting Asymmetric Routing on Dual-Interface Linux Servers

Overview

This guide details the diagnosis and resolution of connectivity timeouts (e.g., SSH timeouts) on a Linux server configured with two network interfaces:
1. Internal Interface (eth0): Connected to a private local area network (LAN).
2. External Interface (eth1): Exposed to the public Internet (WAN).

Symptom

When attempting an inbound connection (such as SSH) to the server's public IP address on eth1:
* Client-side error:
text kex_exchange_identification: read: Connection timed out banner exchange: Connection to <PUBLIC_IP> port 22: Connection timed out
* Server-side packet capture (sudo tcpdump -i eth1 -n port 22):
text IP <CLIENT_IP>.62646 > <SERVER_PUBLIC_IP>.22: Flags [S], seq 1352716039, length 0
Only incoming TCP SYN ([S]) packets are observed on eth1. No outgoing TCP SYN-ACK ([S.]) responses are sent back on eth1.
* Firewall checks: Local firewall (iptables / ufw) rules permit traffic and show zero dropped packets.

Root Cause Analysis

The issue is caused by asymmetric routing (reverse path misconfiguration):

Incoming Request: A client sends a TCP SYN packet to eth1 (<PUBLIC_IP>). The packet arrives successfully at the eth1 physical/virtual network card.
Kernel Route Lookup: When the Linux kernel generates the TCP SYN-ACK response destined for <CLIENT_IP>, it queries its routing table (ip route).
Route Misdirection: If the default gateway (default via ...) is configured on eth0 (the internal network) instead of eth1:
The kernel attempts to send the response via eth0 using the internal gateway.
The internal gateway or upstream router drops the packet because it carries a public source IP or cannot route external traffic out.
As a result, tcpdump on eth1 never sees the response, and the client times out waiting for the TCP handshake to complete.

Diagnostic Steps

1. Verify Host Packet Ingestion

Run tcpdump specifically on the public interface to confirm packets arrive at the OS layer:

sudo tcpdump -i eth1 -n port 22

If Flags [S] appears, the upstream hypervisor/firewall (e.g., Proxmox / cloud security group) is correctly forwarding traffic to the virtual machine.

2. Inspect Firewall Rules

Verify that local host firewalls are not dropping ingress/egress packets:

sudo iptables -L -n -v
# or
sudo ufw status verbose

3. Check Routing Table and Netplan Configuration

Display the active routing table:

ip route

Inspect the interface configuration (e.g., Netplan on Ubuntu/Debian):

cat /etc/netplan/50-cloud-init.yaml

Problematic scenario example:

# INCORRECT / PROBLEMATIC CONFIGURATION
network:
  version: 2
  ethernets:
    eth0:
      addresses:
        - "10.0.0.5/24"
      routes:
        - to: "default"
          via: "10.0.0.1"        # <--- Default gateway set to internal interface!
    eth1:
      addresses:
        - "203.0.113.10/32"       # <--- Public IP has no route or gateway configured!

Solution

Approach A: Single Default Gateway on External Interface (Recommended)

If the server should route all general Internet traffic out via the public interface (eth1), set the default gateway on eth1 and remove the default route from eth0.

[!NOTE]
For cloud providers (e.g., OVH / Proxmox host routed IPs) where the public gateway resides outside the /32 subnet, use on-link: true in Netplan.

/etc/netplan/50-cloud-init.yaml

network:
  version: 2
  ethernets:
    eth0:
      match:
        macaddress: "00:11:22:33:44:55"
      addresses:
        - "10.0.0.5/24"
      nameservers:
        addresses:
          - 1.1.1.1
      set-name: "eth0"
      # No default route defined here. Traffic within 10.0.0.0/24 routes automatically.

    eth1:
      match:
        macaddress: "00:11:22:33:44:66"
      addresses:
        - "203.0.113.10/32"
      nameservers:
        addresses:
          - 1.1.1.1
      set-name: "eth1"
      routes:
        - to: "default"
          via: "203.0.113.1"      # Public Gateway IP
          on-link: true

Approach B: Policy-Based / Source Routing (Dual Gateway Setup)

If the default server traffic must exit via eth0 (10.0.0.1), but any traffic arriving specifically at <PUBLIC_IP> (203.0.113.10) on eth1 must reply via eth1 (203.0.113.1):

/etc/netplan/50-cloud-init.yaml

network:
  version: 2
  ethernets:
    eth0:
      match:
        macaddress: "00:11:22:33:44:55"
      addresses:
        - "10.0.0.5/24"
      set-name: "eth0"
      routes:
        - to: "default"
          via: "10.0.0.1"

    eth1:
      match:
        macaddress: "00:11:22:33:44:66"
      addresses:
        - "203.0.113.10/32"
      set-name: "eth1"
      routes:
        - to: "default"
          via: "203.0.113.1"
          table: 100             # Custom routing table ID
          on-link: true
      routing-policy:
        - from: "203.0.113.10"   # Force packets originating from public IP to use table 100
          table: 100

Applying and Validating Changes

Apply the new netplan configuration:
bash sudo netplan apply
Confirm routing rules:
bash ip route show ip rule show # If using Policy Routing (Approach B)
Test connection from client:
bash ssh -v [email protected]
The TCP 3-way handshake completes successfully.

Key Takeaways

tcpdump showing incoming SYN without SYN-ACK responses usually indicates a routing issue or an active local firewall drop.
Dual-interface systems must have explicit routing definitions to avoid sending traffic out of the wrong physical interface.
For cloud/virtualization environments with /32 public IP assignments, ensure on-link: true is set if the gateway address is outside the IP's immediate subnet mask.