Network Troubleshooting Tools
The following are useful troubleshooting tools for analyzing and understanding network traffic.
ROS 2 CLI
The ROS 2 Command Line Interface (CLI) is a very useful initial set of commands for troubleshooting ROS 2 systems. The following are some of the commonly used commands:
ros2 node list- Check that the nodes are discovered and runningros2 topic list- Check that the topics are discovered and have at least one publisher or subscriber (does not need to be both)ros2 topic echo <topic_name>- Subscribe to a topic and see what is being publishedros2 topic info <topic_name> --verbose- See the identity of each publisher and subscriberros2 topic hz <topic_name>- Check the frequency of messages received on a topicros2 topic bw <topic_name>- Check the bandwidth being used for a topic (given the current topic frequency)ros2 topic delay <topic_name>- Calculates the message delay using timestamps (is only as accurate as the time sync between the computers)
iPerf3
iPerf3 is an open source tool designed to measure the available bandwidth between two devices on a network. It can test both TCP and UDP communications. This is useful for evaluating network quality, and determining if the network is capable of supporting the traffic required for a given system. This tool can also be used to evaluate the impact of distances and obstacles on wireless network connection strength.
iPerf3 uses a server-client model:
- Server: Runs on a computer and listens for connections.
- Client: Runs on the other computer, initiates the test and specifies the details of the test.
It is not advised to use iPerf3 over metered networks (such as cellular or satellite networks) because the test works by transferring large amounts of data.
Installation
sudo apt update
sudo apt install iperf3
Usage
Before beginning the test, identify the network that is being tested and ensure that the two computers are connected on that network and only on that network. Take note of any other traffic happening on the network that may impact the results.
Start the server on one computer
iperf3 -s
On the other computer, launch the client and direct it to the server with the options set for a given test.
For a simple test of how much bandwidth is available run:
iperf3 -c <server-ip>Where
<server-ip>is replaced with the IP address of the computer running the server.This test sends the maximum possible amount of data from the client to the server using TCP. The
-Roption can be used to reverse the traffic and test data flow from the server back to the client.Given that most ROS RMW Implementations use UDP messages, it can be useful to run this test using UDP as well:
iperf3 -c <server_ip> -u -l <packet_size> -b <bandwidth>Where
<server-ip>is replaced with the IP address of the computer running the server,<packet_size>is replaced with the desired UDP packet size in bytes, and<bandwidth>is replaced with the target bandwidth to test for in bits/sec. The size of the UDP packets can be adjusted to emulate traffic consisting of different ROS 2 message sizes. Around 100-200 bytes is a small UDP packet and around 1400 bytes is a large UDP packet:There are many other options documented in the (iPerf manual)[https://iperf.fr/iperf-doc.php].
The most significant result to look at is the bitrate which indicated the available bandwidth. For UDP, it is also useful to look at the percentage of lost packets.
Repeat the test multiple times with any possible variations in the system. Many things impact bandwidth including interference from other wireless networks, physical distance, and obstacles relative to antennas.
bmon
bmon or Bandwidth Monitor is a command-line tool for monitoring current bandwidth usage. While iPerf3 tests the maximum possible bandwidth, bmon is used to monitor an active system to analyze traffic generated through operation.
Installation
sudo apt update
sudo apt install bmon
Usage
To start bmon run
bmon
The Interfaces section lists all of the network interfaces and the traffic being transmitted (TX) and received (RX) on each. The traffic is shown in bps (bits per second) and in pps (packets per second). The lower section shows a graphical representation of the bandwidth usage over time for the selected network. Finally, pressing d opens a detailed statistics section including error rates and dropped packets.
Wireshark
Wireshark is a powerful network protocol analyzer that captures and inspects packets on a network in real-time. It can be used to analyze ROS 2 communication to identify problems and optimize data flow. This can involve identifying the ports in use, analyzing the number of packets or bandwidth being used for different types of messages or investigating individual messages that are not successfully transmitted.
Installation
sudo apt update
sudo apt install wireshark
During installation, it may prompt you to allow non-root users to capture packets. You can accept this option for convenience. You may need to log out and log back in for the change to apply.
Usage
- Open Wireshark and select the network interface(s) connected to the ROS 2 system. If you did not give permissions to non-root users then you will need to launch with sudo.
- Start traffic capture by clicking the
Startbutton. All of the traffic will be displayed in real-time. - Optionally save or export the data.
To maximize utility, make sure to capture the ROS 2 discovery process. This allows Wireshark to interpret DDS (RTPS) messages and makes more information such as topics names available for use in filters and for analysis. In Simple Discovery the discovery process will be captured in the background. For Discovery Server, ensure that the ROS 2 daemon is restarted at the beginning of the capture on the computer where the capture is being taken, to record the discovery process.
Features
Display Filters
Filters can be used to isolate specific types of traffic.
Filtering for ROS DDS messages only (assumes no other RTPS systems on the network):
rtps
Filtering for ROS 2 DDS messages to or from a particular device (for example, a device with IP address 192.168.186.131):
rtps && ip.addr == 192.168.186.131
Filtering for ROS 2 DDS messages based on topic name or message type (for this to work, the discovery process must have been recorded in the same Wireshark capture). Replace the topic string and/or message type as necessary:
rtps && rtps.param.topicName contains "ffmpeg"
or
rtps && rtps.param.typeName == "ffmpeg_image_transport_msgs::msg::dds_::FFMPEGPacket_"
Do note that this will not accurately filter and reflect messages that were fragmented using the IP protocol. Only the message where the IP fragments are reassembled will show up using this filter.
These types of filters can be combined into complex and specific filters to effectively isolate messages of interest.
Statistics
One very useful view for analyzing traffic over time or for optimizing systems is the I/O Graphs view. This is found under the Statistics dropdown menu. The I/O Graphs view allows lines to be added to the graph using different display filters.
In addition to the display filters, take note of the y-axis (bits vs packets) to determine what data is graphed. The Interval setting at the bottom of the window (not shown) controls the timeframe over which each datapoint is calculated.
Additional statistics views that can be useful include Source and Destination Addresses to see which devices have sent/received how many packets and identify outliers (accessed through Statistics -> IPv4 Statistics -> Source and Destination Addresses). Similarly, Destination and Ports can be used to identify which ports are being used on each device (accessed through Statistics -> IPv4 Statistics -> Destination and Ports). These are not the only useful views, but are a good place to start.
tshark
tshark is the command line version of Wireshark. This is useful for Ubuntu server based systems, to be able to enable the use of Wireshark functionality without requiring a display for capture.
Installation
sudo apt update
sudo apt install tshark
During installation, it may prompt you to allow non-root users to capture packets. You can accept this option for convenience. You may need to log out and log back in for the change to apply.
Usage
To capture packets on a specific network interface, use:
tshark -i <interface>
Replace <interface> with your network interface (such as eth0 or wlan0). The -i option can be repeated multiple times to capture from more than one interface simultaneously. Use the -D option to list the available interfaces:
tshark -D
The captured data can be analyzed in the command line, or it can be saved to a file using -W for later analysis in Wireshark:
tshark -i <interface> -w capture.pcap
Capture filters can be used to reduce the log sizes if necessary or when analyzing directly in the command line.
Nmap
Nmap (Network Mapper) is a command line tool for network exploration and management. It can be used to identify connected devices and open ports among other uses.
Installation
sudo apt update
sudo apt install nmap
Usage
A small selection of functionality is discussed here, for full documentation see the Nmap manual.
To detect the other hosts on your network, run a ping scan as follows but replace the IP address with one on the network that is to be scanned:
nmap -sn 10.27.0.1/24
This will return a list of IP addresses where hosts were found that responded to the ping and the associated ping latency.
To do a port scan run:
nmap -p <ports> <ip-address>
Where <ports> is replaced by the port number(s) or range to be checked and <ip-address> is replaced with the target IP address of the device to be checked. This will give a report of the status of all of the ports requested.
To do a host scan run:
nmap <ip-address>
Where <ip-address> is replaced with the target IP address to be inspected. This will give summary information about a particular device on the network including the currently open ports.
Finally, to try and identify what operating system is running at a particular IP address, run:
nmap -O <ip-address>
Where <ip-address> is replaced with the target IP address to be inspected.