Today I decided to focus on something I take for granted, that possibly many other backend developers may not have ever used to help troubleshoot problems: tcpdump and wireshark. Coming from a systems and networking background into a software development role, I have used capture for troubleshooting networked applications I have developed. Today I will try to explain how you can setup a development environment that will allow you to use these tools to troubleshoot your application.
tcpdump is a command-line packet analyzer, which will dump traffic on a network. tcpdump takes in, amongst other things, a network interface, and a BPF expression. BPF stands for Berkeley Packet Filter and is a language specifically designed for expression of network traffic as filters.
Docker is a project that automates the deployment of applications inside of Linux
Containers. Part of the automation revolves around creation of a bridged network
interface which all of the containers running are attached to. This is handy
in that we are able to use tcpdump on the
docker0 interface and watch all the
network traffic that is going from container to container, as well as host
Wireshark is a very popular GUI network protocol analyzer which allows you to view and analyze a wide variety of protocols.
In order to start watching all the container network traffic you can run the following command on the container host:
What this command is doing is performing a capture on the
and filtering with the BPF expression
tcp port 80 which filters just the
HTTP protocol traffic from the result set and then writes the Packet CAPture
data to a file called test.pcap.
Pcap is a pretty simple file format, explained in depth here, but high level contains a global header, and headers followed by corresponding data chunks. The Pcap headers contain information about endianess, timestamp in seconds and microseconds, the size of the following data and the length of the captured packet. The data section following the header is the raw data bytes captured.
Now that we are running our tcp command to capture the port 80 traffic on the
docker0 bridged network interface, we perform some action in the web
application that we want to troubleshoot. When we perform the task we wish
to troubleshoot, we stop tcpdump by signalling and interupt, and our pcap
file is written to disk.
Note: If you have an entire stack of services running, using this process will actually capture ALL the requests/responses throughout your whole stack in the order written to the network interface. This is extremely helpful to figure out problems with many services running. Very helpful.
Now that we have a capture we can open said capture file in a wireshark so we can easily follow the http conversation:
By selecting “Analyze > Follow > HTTP Stream” in the menus you will be able to just see the particular HTTP text protocol for this HTTP conversation which would look like this:
This is very helpful in figuring out “who said what” in a more complicated containerized development environment.