Linux Command line Tools for Detecting networking issues in layers 2,3,4

Dealing with networking issues can be a very challenging task.

A packet sent from a client to a server (or vice versa) might be dropped in various locations and from various reasons. some of the most common cases are -

• A firewall that is blocking the packet. it can be a firewall on the server side, on the client side or any firewall along the way between the server and the client. the blocking may be based on an ip (source and/or destination), ports (in case of tcp/udp) or any other layer 3,4 criteria.
• lack of layer 2 connectivity.
• routing configurations that are not set properly.

fortunately, Linux command line tools supply significant information that might help us narrow down the problem.

from my experience as a networking engineer those are my top 8 tools that will help you overcome networking issues.

1. ping
   This is the most fundamental command when troubleshooting a networking issue. the ping command will test the network connectivity between your host and the server. this is definitely the first command to use when suspecting no network connectivity.

2. ifconfig
   ifconfig will allow you to view and configure your network interfaces.
   you can view your network interfaces and the ip addresses assigned to them.
   you can view the network statistics for each network interface (worth looking on dropped or error packets counters).
   you can view the state of the network interface (UP/DOWN).
   

3. tcpdump
   tcpdump will print the content of the packets that are being received or transmitted by a network interface (or by all network interfaces). this is a powerful util due to its ability to give a full view of the network traffic and its filtering capabilities that allow us to filter traffic by layer 3 and 4 properties.
   for example to view all the traffic on eth0 transmitted to port 80 or received by port 80 -
   

4. netstat
   netstat will print network connections, routing tables, interface statistics, masquerade connections, and multicast memberships.
   I want to focus on netstat's ability to provide information about listening sockets. when establishing a tcp connection from a client to a server on server's specific port. the server must have a running program that "listens" on that specific port. using netstat's command (on the server side) we can verify that indeed a program has a listening socket on that specific port and on which ip addresses it accepts connections from.
   

5. ip rule, ip route
   i intentionally put those 2 commands together since they are related. usually when we want to have a look on our configured routing table, we type the ip route command -
   

the output seen on the screen is the main routing table. Linux has several built in routing tables including user defined routing tables that can be configured. to understand the packet flow of the system we should be aware of the routing policy database. ip rule gives us that information -

from the output we see that in this system there are 4 routing tables.
the first routing table to be searched for a matching route is the local table, the second is table 3 and the third is the main table. in this example, a source based routing was defined. all packets whose source ip is 2.2.2.2 and did not have a route match in the local table will be routed based on table 3 and not based on the main table! .the below screenshot shows the content of routing table number 3.

so when searching routing issues, always examine first the routing database using ip rule and then examine the proper routing table using ip route show table [name]

6. iptables
   iptables is an administration tool for IPv4 packet filtering and NAT. This basically serves as a firewall. iptables has 4 tables with predefined chains. usually we will inspect the filter table. the filter table is the default table shown when typing iptables -L and all the filtering is defined in this table.
   

its worth spending a time learning iptables from one of the many tutorials online. using iptables one can inspect specific rules that prevent from a connection to be established ( in the above screenshot connections that are coming from source ip 1.2.3.4 with source port 80 are being dropped). iptables also gives a logging infrastructure, so a rule can be added with an action of LOG (rather then DROP or ACCEPT), so if we expect to see a packet in a certain chain we can just log it and verify that the packet indeed arrived to where it should be.

7. ip neighbour
   this command prints to the screen the arp table. always remember that before sending an ip packet from one point to another, the destination's mac address should be known to the sender (its actually the gateway's mac address in most cases where the 2 endpoints are not directly connected). so its always worth checking out the arp table for layer 2 connectivity issues before checking the network or transport layer.
   

8. conntrack
   conntrack will give you the information of all the system's connection, their state and some other useful information.

to see all the tcp connections

each conntrack line entry shows the 2 directions of a packet: the original direction (the packet that initiated the connection) and the reply direction. the conntrack tool can be very useful in debugging natted connections, in examining a tcp connection's state, examine the number of connections in the system (reaching the max number of connections will prevent from future connections to be established) and much more.

In conclusion, there are almost endless different types of networking issues you can face, solving those issues is not always easy, however mastering those tools i have mentioned will dramatically increase your chances to solve many of those issues.