Unix networking command line tools I use to do my job

I write software for routers that shuttle packets between datacenters. The nitty gritty of exactly how packets are crafted, forwarded and received by different brands of routers and their respective OSes matters a lot in my day to day. So, I’ve spent a lot of time in the past few months working with command line networking tools. 

ping sends an ICMP echo request message to a destination host, with the hope of getting an echo reply back. ICMP is a protocol that sits right above IP, just like TCP and UDP. However, it’s considered to be a network layer just as IP is, because you can’t really call it a transport protocol. It’s not concerned with shuttling application data back and forth between hosts, but rather network information.

I find that this sloppiness exists everywhere in computer networking. Oh look, we have this nice OSI model for the layers of networking!! Layering and separation of concerns is a cornerstone of building scalable, maintainable systems and of modern computer science! Oh but ICMP and IP are both network layer protocols, and ICMP packets are encapsulated in IP packets. Sorry not sorry.

Last thing: ICMP is usually meant for sending error messages, but can also provide information like for ping. For example, if you send UDP packets to a port on a host that is accessible (not blocked by firewall) but isn’t active – no application is listening on it – then you will get a “port unreachable” message. Note that ICMP itself has no concept of ports, so you can’t ping a certain port. Not every protocol on top of IP has to use port numbers! TCP/UDP does and TCP/UDP rules the world, so you’d easily be forgiven for thinking that.

https://en.wikipedia.org/wiki/Internet_Control_Message_Protocol

https://stackoverflow.com/questions/19218596/is-icmp-a-transport-layer-protocol

traceroute shows the path an IP packet would take through the network to reach another device. Given a hostname or IP address, traceroute attempts to find the intermediary hops. 

First, it will send a packet to the destination with a Time to Live (TTL) of 1. The packet should be routed to the first hop, but then dropped because the TTL will be decremented to 0. An ICMP message should be sent back to the source host telling it that the packet’s time was exceeded, and now it knows what the first hop is! This procedure is repeated with a TTL of 2, 3, 4 and so on until the destination is reached. In that case, the ICMP message sent back will be different. Using the default UDP method, it should be “port unreachable” because the destination port will be chosen to be one unlikely to be listened on by an application. 

What does it look like when traceroute fails? 

  • If the destination is not reachable from the source, then the last hop within the network reachable from source may return a “destination unreachable” ICMP message. This could mean the destination is down, doesn’t exist, or because the router is configured to drop packets destined for it. Because traceroute sends 3 probes per hop, it should have seen up to date IP routing tables eventually and so the destination really is unreachable.
  • The destination may be reachable, but only through protocol and port combinations that either you as the user have not specified through traceroute, or are inaccessible to traceroute entirely. A firewall on any device in the path – including the source and destination – could be blocking incoming or outgoing traffic, including your traceroute packets.
  • On its way to the source, an ICMP time exceeded message may get dropped by a device configured to block all incoming ICMP traffic. This device could be in the path between source and destination, or not – the reply may not retrace the hops the probe took. Traceroute will not be able to show that hop because it didn’t get a reply, but the destination is reachable.

There are also options for TCP, ICMP and UDP with constant destination ports as well. These are can be needed to bypass firewalls that block traffic on uncommonly used ports or based on other port and protocol combination rules. For TCP, a handshake is initiated but then canceled by the source as to not confuse destination host applications listening for connections; ICMP likewise does not interfere with listening applications. However, the UDP method could confuse an application listening on the specified destination port – it will assume the data is meant for them to process.

https://linux.die.net/man/8/traceroute

https://www.lumen.com/help/en-us/network/traceroute/timeouts-and-unreachables-in-traceroute.html

tcpdump listens for packets that fit the specifications passed in as command line arguments, and prints information about them out. This is not limited to just TCP packets. 

Some things you can specify include: the packet protocol (IPv4, IPv6, TCP, UDP, ICMP…), packet header field values, source and destination IPs or ports, and interface to listen on. Both incoming and outgoing packets will be shown by default. Something else you can do is have the packet printed out in hex, and copy paste those bytes into a packet decoder like this one: https://hpd.gasmi.net/.

At work this was an invaluable tool for seeing if packets I was sending out were 0) well-formed, 1) sent out at all, 2) being received by a middle hop device, 3) were being forwarded from the middle hop, and 4) being received by the destination device. When you send a packet expecting a reply and don’t get one back, you don’t know what is really going on until you find out which of the above desired events is missing. Once you do, it’s easier to know where to look for issues: the application code? The router firewalls? The OS adding an extra network layer header?

A side note is that working with software that goes on routers has forced me to hold more complex abstractions about what packets do when they reach a computer. We love thinking about computers as nodes in a graph, and links as bidirectional edges between two such nodes. If a packet travels from one node to its neighbor, that means the packet has arrived successfully and will be seen by applications running on the second node listening for it, right?

Haha! Not unless you deliberately made way for your weary traveler! It may get blocked by a firewall. It may get forwarded by the OS, efficiently shutting it on its way without any user applications ever seeing it. It may get dropped by a BPF filter, which also drops packets at the OS level. Or maybe you’re using emulated hardware that behaves subtly differently than real hardware, and it simply decides that the control plane never needs to know about forwarded packets because it’s the data plane’s business anyway. Finding a harmonious configuration becomes is non-trival task in routerland.

https://www.tcpdump.org/manpages/tcpdump.1.html

https://en.wikipedia.org/wiki/Berkeley_Packet_Filter

iptables allows you to manipulate the packet filter rules in the Linux kernel. The context I used this in was adding NAT rules that would allow an emulated device to send packets to a jumpbox situated in a different subnet.

The emulated device was spun up in a container. Even though it is directly connected to the jumpbox, and there existed a /31 pair of IPs for that connection, it does not know about the jumpbox and how to route packets to it. So when I sent packets to the jumpbox end of that /31, nothing happened. 

However, what the device DOES know about is a /31 pair of IPs between it and the container. 

The solution was to direct packets to the container, and with NAT rules on the container, redirect those to the jumpbox instead. These were the two rules:

  1. In the prerouting NAT table, look for packets coming from the device and going to the container on the port set by my application. Change the destination IP to the jumpbox IP.
  2. In the postrouting NAT table, look for packets coming from the device headed to that same port, and change the source IP to the device end of the /31 shared with the jumpbox.

And voila, with tcpdump I could see those packets flowing to the jumpbox! One thing that caught me by surprise when I was figuring out these commands was that UDP packet flows mean something to routers. As in, when you add a new NAT rule, you need to stop and restart an application that’s currently spitting out UDP packets, and only after that will they hit the new filters. Since the UDP protocol doesn’t need to establish a connection or anything between sender and receiver, I didn’t think that’d be the case! Spent an embarrassing amount of time stuck on that one.

Some other tools I haven’t used quite as much, but also seem useful:

netcat can open UDP or TCP connections, listen for them, scan ports, and apparently a ton of other things too. For example, you can script HTTP clients or servers with it.

dig will resolve a domain name to a server IP address, doing a DNS lookup.

nslookup allows you to query for DNS configuration information, and contact different types of servers used for DNS.