My mentor, CW Rogers, used to talk about people who were so busy paying attention to the swarm of ants at their feet that they didn’t see the elephants stampeding toward them from behind. As IT professionals spend time focusing on shaping network traffic, streamlining the use of bandwidth and adding bigger pipes to the desktop, they may be missing the point—these improvements may have minimal effect on network performance.
The best way to determine the cause of poor network performance is to tap in and listen to the traffic.
The best way to determine the cause of poor network performance is to tap in and listen to the traffic. This is the entire purpose of Wireshark (formerly Ethereal). Listening to the traffic is much like the doctor taking an X-ray of your mother when you take her to the emergency room because she’s complaining of a pain in her arm. Is the arm broken? If so, where is the break? Listen to network traffic to identify problems related to protocols, applications, links, processing time, latency and more.
Note: In July 2006, Gerald Combs (the creator of Ethereal) and the entire Ethereal development team moved over to Wireshark. No updates have been released for Ethereal, while Wireshark is constantly undergoing improvements and updates. Visit wireshark.org to get the latest revision of Wireshark. Visit wiresharkU.com to purchase basic through advanced troubleshooting and security training for Wireshark.
> Tapping In To the Traffic
Before you start listening to your own traffic, make sure you have appropriate authorization to tap into the network traffic. Ideally your network-use policy explicitly states that you have permission to listen in on network traffic for the purpose of troubleshooting, optimizing and securing the network.
When the user complains of performance, try to tap in as close to the user as possible. It’s best if you can watch the user’s screen and examine performance issues while you capture their traffic. Typically, hosts are connected to the network through a switch. This causes the network analyst some grief, because switches don’t forward all traffic everywhere like hubs do. Switches only forward four types of traffic:
- Traffic to and from the connected system’s MAC address
- Traffic to an unknown MAC address
To address this issue, I bring along a simple four-port hub to connect to half-duplex lines and a small network tap to connect to full-duplex lines.
> Wireshark’s Expert Info Composite Window
Wireshark now includes Expert notification of possible problems in a trace file. Select Analyze > Expert Info Composite to view a summary of Expert alerts. In Figure 1, the Expert Info Composite window indicates there are 53 different Expert notifications in the trace file download-bad.pcap. (This trace file is available on the FIN BIT page at wiresharkU.com.)
Click the + to open a sequence and list the packets associated with each Expert notification. In Figure 1, packet 363 is tagged as a Window is Full packet. Wireshark automatically sets the focus in the Packet Detail window to the packet selected in the Expert Info Composite window.
> What to Look For
Most of the interesting and crucial applications on your network run over TCP, not UDP, to take advantage of TCP’s connection-oriented nature and recovery capabilities.
TCP-based communication problems can manifest themselves in a variety of ways: a system appears to lock up, long delays during file downloads or even corrupted files.
Watch for the following problems when analyzing TCP-based applications using Wireshark:
- Zero Window
- Window is Full
- Window Update
- Previous Segment Lost
- Retransmissions/Fast Retransmissions
- Duplicate ACKs
These issues are displayed with a black background and red foreground as shown in Figure 2, because they are linked to expert notifications. The Expert Info notifications are documented in the source code found at anonsvn.wireshark.org/wireshark/trunk/epan/dissectors/packet-tcp.c.
This Expert notification indicates that the packet’s TCP header contains the value zero in the window size field and neither the RST nor FIN bits are set in the header (which would have indicated a connection shutdown process). This is an indication that the sender’s TCP receive buffer is full. No more data can be received until receive buffer space is available again (as noted through a Window Update packet). Watch this one! I’m seeing it more and more often as people load processor-intensive applications on their systems. The application that should pick up the data from the TCP receive buffer space does not get the processing time to pick up the data and the buffer fills up. As you can see in the downloadbad. pcap trace file, this full window condition causes a 32+ second delay in receiving data. The user will most definitely feel this performance problem and likely blame the network—“the network is slow.” For more information on the Zero Window issue, refer to the FIN BIT page at wiresharkU.com.
Window is Full
This Expert notification indicates that the packet will completely fill the remaining TCP buffer space on the receiver’s side of the communications. For example, in packet 361 in download-bad.pcap, the client (10.0.52.164) indicates that its TCP window can accept 2920 bytes of data. Packet 362 from the server contains 1460 bytes of data—now only 1460 bytes of space are in the receive buffer of the client). Packet 363 also contains 1460 bytes of data; Wireshark knows this packet will fill the remaining space in the receive buffer of the client.
This Expert notification indicates that one side of a communication is trying to maintain the TCP connection—in the downloadbad. pcap trace file, the server tries to maintain the connection immediately following the zero window notifications from the client. Notice that the server increases the interval between the keep-alives; it becomes more patient over time.
This Expert notification indicates that a new TCP window size is being advertised. Typically this packet contains no data, uses the same sequence and acknowledgment number as the previous packet and contains a new window size field value. These packets are not a problem unless the communications also contains a Window is Full notification.
Previous Segment Lost (Common at Capture Start)
This Expert notification indicates a previous TCP segment is missing. If you started a capture in the middle of a file download then this notification is expected. If, as in the case of download-bad.pcap, you started the capture before the file transfer began, you should see the entire TCP communication starting with the TCP handshake. You should not see the Previous Segment Lost notification unless there is actual packet loss. The normal recovery process for this problem would include duplicate ACKs and retransmissions.
The Retransmission Expert notification indicates that the packet is considered a retransmission. It is defined by a packet that contains data, has advanced the sequence number value and follows two or more duplicate ACKs in the opposite direction. Those duplicate ACKs must contains the retransmission’s sequence number in their ACK number field. If the retransmission occurs within 20 milliseconds of the last duplicate ACK, it is considered a fast retransmission. If the ACKs are lost in the other direction, Wireshark sees packets that contain data using the same sequence number and notes that these are retransmissions. In the download-bad.pcap file, there are numerous retransmission that are triggered by Duplicate ACKs.
This Expert notification indicates that the sender has noted a missing packet in an inbound data transfer. Receivers track the sequence number of the packets coming from the opposite side of the communication. Those sequence numbers should increment by the number of bytes of data contained in each packet. If a packet arrives with sequence number 10,000 and 1460 bytes of data, the next expected sequence number is 11,460. But if the next packets contain sequence numbers 12,920 then 14,380 and so on, the receiver notifies the sender that it is missing sequence number 11,460. It does this by sending multiple ACKs with that sequence number in the Acknowledgment Number field. Note that the receiver sends a normal ACK first; the duplicate ACKs follow this normal ACK. A sender must receive three identical ACKs (the normal ACK and two duplicates) before retransmitting the packet. Note that a high number of duplicate ACKs may indicate high latency problems; it takes so long to get the third identical ACK to the target to trigger the retransmission. A continuous stream of duplicate ACKs are sent until the retransmission is received. In the download-bad.pcap file, packet 2175 is the forty-eighth duplicate ACK—a definite indication of high latency.
Tapping into Full Duplex Networks
Sometimes referred to as “walkie-talkie” style communications, the simple half-duplex environment supports traffic moving in one direction at a time, transmit or receive, but never both simultaneously. Alternatively, fullduplex networks support two communications channels for simultaneous transmit and receive. A simple hub doesn’t support full-duplex communications, but a full-duplex tap does.
Full duplex taps are placed inline—typically acting as passive devices. Taps are simple to set up. Let’s say, for example, that you want to tap into a full-duplex link that uses CAT5e cable between two network routers. One CAT5e cable runs from the first router to port A. A second CAT5e cable runs from port B to the second router. Monitor ports connect to the analyzer allowing you to see a copy of all traffic.
There are two flavors of full-duplex taps—aggregating and non-aggregating taps. Aggregating taps combine the data from the transmit and receive channels into a single monitor port allowing you to connect a single analyzer to listen to both traffic channels. Non-aggregating taps as shown in Figure 3, do not combine the transmit and receive streams. You must connect the monitor ports to two separate analyzers or an analyzer with two NICs installed.
Hint: If you use non-aggregating taps and two separate analyzers, you should time sync the two analyzers using NTP (Network Time Protocol) to ensure data in your streams can be merged into proper order.
I’ve been listening into networks for years now—since the late 1980s (hiding gray hairs). Tapping into the communications in a passive manner enables you to identify communication problems. Mastering analysis of TCP/IP communications is critical when identifying the source of those problems and differentiates TCP issues from application issues. Knowing how to set up a TCP/IP network and being able to interpret the communications on it are entirely different skill sets, much like reading a foreign language differs from being able to speak that language. Network analysis is one of the key skill sets all IT and security professionals should master. Begin taking your traces and learn how the protocols and applications interact.
Through Wireshark University, Laura Chappell researches, develops and delivers self-paced and instructor-led training on network troubleshooting, optimization and security. For more information on Wireshark University courses, visit wiresharkU.com.