As I mentioned in an earlier post, I’ve been experimenting a bit with WireGuard lately. In the process, I’ve been learning about a number of network troubleshooting commands, and before I forget what they all are and do, I’m going to capture them here so I (and you!) can find them again. Here’s my network troubleshooting toolbox.
ssh, dig, telnet, netcat
I covered these in an earlier post, “Troubleshooting a Home Networking Issue,” so I’m not going to go over them again. But I didn’t want to leave them out entirely because they’re must-haves in my toolbox.
Display the Routing Table
What IPv4 routes are currently in the routing table?
On macOS, you can use netstat to get the routing table.
> netstat -rn -f inet
Destination Gateway Flags Netif
default 192.168.1.1 UGScg en0
127 127.0.0.1 UCS lo0
192.168.1 link#15 UCS en0
10.1.1/24 utun5 USc utun5
The arguments tell netstat to show the routing table (-r), show numerical addresses instead of host names (-n), and limit the output to the IPv4 address family (-f inet).
The “Destination” is the network or host the route applies to. The default destination covers anything that doesn’t have a more specific route.
The “Gateway” is the next hop used to reach the destination. It could be a router’s IP address, but it can show link or interface-related information as well (like utun5 in the example).
The “Netif” column tells you which network interface the traffic will actually use.
The example output here is saying that, by default, traffic will go to the router at 192.168.1.1 through interface en0. Loopback traffic (127) goes to the lo0 interface. Local network traffic (192.168.1.x) is directly connected through en0 at the link layer. And finally, traffic for the 10.1.1.x subnet will be routed through the utun5 network tunnel.
The modern Linux tool for getting the routing table is ip:
> ip -4 route
default via 192.168.1.1 dev ens18 proto static
192.168.1.0/24 dev ens18 proto kernel scope link src 192.168.1.5
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
Route for a Specific IP Address
What is the route the OS would use for this specific IPv4 address?
On macOS you’d answer this question with:
> route -n get 8.8.8.8
route to: 8.8.8.8
destination: default
mask: default
gateway: 192.168.1.1
interface: en0
flags: <UP,GATEWAY,DONE,STATIC,PRCLONING,GLOBAL>
recvpipe sendpipe ssthresh rtt,msec rttvar hopcount mtu expire
0 0 0 0 0 0 1500 0
And on Linux:
> ip route get 8.8.8.8
8.8.8.8 via 192.168.1.1 dev ens18 src 192.168.1.5 uid 1000
cache
DNS Configuration
What DNS servers and DNS routing rules is my system using?
On macOS you can use scutil --dns to answer this question:
> scutil --dns
DNS configuration
resolver #1
search domain[0] : home
nameserver[0] : 192.168.1.1
if_index : 14 (en0)
flags : Request A records
reach : 0x00000002 (Reachable)
resolver #2
domain : internal.example.com
nameserver[0] : 10.20.30.40
nameserver[1] : 10.20.30.41
flags : Request A records
reach : 0x00000002 (Reachable)
order : 101400
resolver #3
domain : local
options : mdns
timeout : 5
flags : Request A records
reach : 0x00000000 (Not Reachable)
order : 300000
DNS configuration (for scoped queries)
resolver #1
search domain[0] : home
nameserver[0] : 192.168.1.1
if_index : 14 (en0)
flags : Scoped, Request A records
reach : 0x00000002 (Reachable)
Among other things, this shows the default resolver uses 192.168.1.1 for DNS lookups. It also shows a domain-specific resolver for internal.example.com, which will use 10.20.30.40 and 10.20.30.41 as the nameservers.
On a Linux system using systemd-resolved, the closest equivalent is:
> resolvectl status
Global
Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub
Link 2 (ens18)
Current Scopes: DNS
Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS
DNSSEC=no/unsupported
Current DNS Server: 8.8.8.8
DNS Servers: 8.8.8.8 8.8.4.4
Default Route: yes
Link 3 (wg0)
Current Scopes: none
Protocols: -DefaultRoute -LLMNR -mDNS -DNSOverTLS
DNSSEC=no/unsupported
Default Route: no
Resolving a domain name
What does this specific domain name resolve to on this machine?
On macOS:
> dscacheutil -q host -a name example.com
name: example.com
ip_address: 104.20.23.154
ip_address: 172.66.147.243
On a Linux system using systemd-resolved:
> resolvectl query example.com
example.com: 104.20.23.154 -- link: ens18
172.66.147.243 -- link: ens18
2606:4700:10::6814:179a -- link: ens18
2606:4700:10::ac42:93f3 -- link: ens18
These commands use the OS and whatever configuration it has in place to resolve a domain name. If you want to know what a particular DNS server thinks the address for a domain is, that’s when you’d reach for dig.
> dig @8.8.8.8 example.com
;; ANSWER SECTION:
example.com. 300 IN A 104.20.23.154
example.com. 300 IN A 172.66.147.243
IP Address
What IPv4 addresses are assigned to this computer’s network interfaces?
You can use these commands, whether you need the address assigned to an interface, or if you just want a list of all of the network interfaces.
On macOS:
> ifconfig
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=400<CHANNEL_IO>
ether 3c:22:fb:aa:bb:cc
inet6 fe80::1234:5678:9abc:def0%en0 prefixlen 64 secured scopeid 0xe
inet 192.168.1.5 netmask 0xffffff00 broadcast 192.168.1.255
nd6 options=201<PERFORMNUD,DAD>
media: autoselect
status: active
The inet line is the one we’re looking for – it shows that for the en0 interface, the assigned IP address is 192.168.1.5.
On Linux:
> ip -4 addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
altname enp6s18
altname enx020000000464
inet 192.168.1.5/24 metric 100 brd 192.168.1.255 scope global dynamic ens18
valid_lft 490698sec preferred_lft 490698sec
3: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
inet 10.253.0.1/24 scope global wg0
valid_lft forever preferred_lft forever
This example, in addition to the ens18 interface with 192.168.1.5, also has a WireGuard interface with an address of 10.253.0.1.
Who’s Using this Port?
Which process is using this port?
lsof can be used on both macOS and Linux to see what process is using a specific port:
> sudo lsof -nP -iTCP:53 -iUDP:53
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
mDNSRespo 662 _mdnsresponder 101u IPv4 0x2a64a950e77351c5 0t0 UDP *:53
mDNSRespo 662 _mdnsresponder 102u IPv6 0xe54d626bb199bdc5 0t0 UDP *:53
mDNSRespo 662 _mdnsresponder 103u IPv4 0x64cc0f001d5c41a0 0t0 TCP *:53 (LISTEN)
mDNSRespo 662 _mdnsresponder 104u IPv6 0x1da999b124c99a24 0t0 TCP *:53 (LISTEN)
The -nP options keep addresses and ports as numbers, instead of resolving them to host and service names. The -iTCP:53 tells it to show TCP network sockets using port 53, and -iUDP:53 says to also show UDP network sockets using port 53.
Monitoring Traffic
What network traffic is actually passing through this machine or interface?
If you’re trying to answer that question, tcpdump is the tool to use. It’s a command-line packet analyzer used to capture, display, and analyze network traffic in real time. It’s available on macOS and Linux.
tcpdump is incredibly powerful and flexible, and I’ve only scratched the surface of what it can do. But I’ll list a few use cases I’ve actually used it for. In each of these examples, -n prevents name and service lookups, and -i specifies the interface to capture from. Everything after that is a packet-filter expression.
sudo tcpdump -ni any udp port 51820– capture UDP packets using port 51820 seen on any network interfacesudo tcpdump -ni ens18 port 53– capture TCP or UDP packets using port 53 on ens18sudo tcpdump -ni utun5– capture all packets seen on the utun5 interfacesudo tcpdump -ni en7 'host 192.168.1.15 or port 12300'– capture packets seen on en7 where either the source or destination IP is 192.168.1.15, or the source or destination port is 12300
Conclusion
All of the tools I’ve listed have a ton of flags and options, and can do so much more than the few things I’ve discussed. But I hope that, if you’re trying to troubleshoot a networking problem, these can at least get you started.