Solaris: Network Troubleshooting
Overview
One of the first signs of trouble on the network is a loss of communications by one or more hosts. If a host refuses to come up at all the first time it is added to the network, the problem might lie...
- In one of the configuration files
- In the network interface
If a single host suddenly develops a problem, the network interface might be the cause. If the hosts on a network can communicate with each other but not with other networks, the problem could lie with the router, or it could lie in another network.
Use ifconfig program to obtain information on network interfaces
Use netstat to display routing tables and protocol statistics.
Use tools like ping to quantify problems like the loss of packets by a host.
Running Software Checks
If the network has trouble, some actions that you can take to diagnose and fix software-related problems include:
- Using the netstat command to display network information.
- Checking the hosts database (and ipnodes if you are using IPv6) to make sure that the entries are correct and up to date.
- If you are running RARP, checking the Ethernet addresses in the ethers database to make sure that the entries are correct and up to date.
- Trying to connect by telnet to the local host.
- Ensuring that the network daemon inetd is running. To do this, log in as superuser and type:
# ps -ef | grep inetd
Here is an example of output displayed if the inetd daemon is running:
root 57 1 0 Apr 04 ? 3:19 /usr/sbin/inetd -s root 4218 4198 0 17:57:23 pts/3 0:00 grep inetd
ping Command
Use the ping command to find out whether there is IP connectivity to a particular host. The basic syntax is:
/usr/sbin/ping host [timeout]where host is the host name of the machine in question. The optional timeout argument indicates the time in seconds for ping to keep trying to reach the machine-20 seconds by default. The ping(1M) man page describes additional syntaxes and options. When you run ping, the ICMP protocol sends a datagram to the host you specify, asking for a response. (ICMP is the protocol responsible for error handling on a TCP/IP network. See ICMP Protocol for details.)
How to Determine if a Host Is Running
On the command line, type the following command.% ping hostname
If host hostname is up, this message is displayed:
hostname is alive
This indicates that hostname responded to the ICMP request. However, if hostname is down or cannot receive the ICMP packets, you receive the following response from ping:
no answer from hostname
How to Determine if a Host Is Losing Packets
If you suspect that a machine might be losing packets even though it is running, you can use the s option of ping to try to detect the problem. On the command line, type the following command.
% ping -s hostname
ping continually sends packets to hostname until you send an interrupt character or a timeout occurs. The responses on your screen will resemble:
PING elvis: 56 data bytes 64 bytes from 129.144.50.21: icmp_seq=0. time=80. ms 64 bytes from 129.144.50.21: icmp_seq=1. time=0. ms 64 bytes from 129.144.50.21: icmp_seq=2. time=0. ms 64 bytes from 129.144.50.21: icmp_seq=3. time=0. ms . . . ----elvis PING Statistics---- 4 packets transmitted, 4 packets received, 0% packet loss round-trip (ms) min/avg/max = 0/20/80
The packet-loss statistic indicates whether the host has dropped packets.
If ping fails, check the status of the network reported by ifconfig and netstat, as described in ifconfig Command and netstat Command
ifconfig Command
The ifconfig command displays information about the configuration of an interface that you specify. (Refer to the ifconfig(1M) man page for details.) The syntax of ifconfig is:
ifconfig interface-name [protocol_family]
How to Get Information About a Specific Interface
- Become superuser.
- On the command line, type the following command.
# ifconfig interface
For an le0 interface, your output resembles the following:
le0: flags=863
The flags section just given shows that the interface is configured "up," capable of broadcasting, and not using "trailer" link level encapsulation. The mtu field tells you that this interface has a maximum transfer size of 1500 octets. Information on the second line includes the IP address of the host you are using, the netmask being currently used, and the IP broadcast address of the interface. The third line gives the machine address (Ethernet, in this case) of the host.
How to Get Information About All Interfaces on a Network
A useful ifconfig option is -a, which provides information on all interfaces on your network.- Become superuser.
- On the command line, type the following command.
# ifconfig -a interface
This produces, for example:le0: flags=49
Output that indicates an interface is not running might mean a problem with that interface. In this case, see the ifconfig(1M) man page.
netstat Command
The netstat command generates displays that show network status and protocol statistics. You can display the status of TCP and UDP endpoints in table format, routing table information, and interface information.
netstat displays various types of network data depending on the command line option selected. These displays are the most useful for system administration. The syntax for this form is:
netstat [-m] [-n] [-s] [-i | -r] [-f address_family]
The most frequently used options for determining network status are: s, r, and i. See the netstat(1M) man page for a description of the options.
How to Display Statistics by Protocol
The netstat -s option displays per protocol statistics for the UDP, TCP, ICMP, and IP protocols.
On the command line, type the following command.
% netstat -s
The result resembles the display shown in the example below. (Parts of the output have been truncated.) The information can indicate areas where a protocol is having problems. For example, statistical information from ICMP can indicate where this protocol has found errors.
UDP
udpInDatagrams = 39228 udpOutDatagrams = 2455
udpInErrors = 0
TCP
tcpRtoAlgorithm = 4 tcpMaxConn = -1
tcpRtoMax = 60000 tcpPassiveOpens = 2
tcpActiveOpens = 4 tcpEstabResets = 1
tcpAttemptFails = 3 tcpOutSegs = 315
.
.
IP
ipForwarding = 2 ipDefaultTTL = 255
ipInReceives = 4518 ipInHdrErrors = 0
.
.
ICMP
icmpInMsgs = 0 icmpInErrors = 0
icmpInCksumErrs = 0 icmpInUnknowns = 0
.
.
IGMP:
0 messages received
0 messages received with too few bytes
0 messages received with bad checksum
0 membership queries received
0 membership queries received with invalid field(s)
0 membership reports received
0 membership reports received with invalid field(s)
0 membership reports received for groups to which we belong
0 membership reports sent
How to Display Network Interface Status
The i option of netstat shows the state of the network interfaces that are configured with the machine where you ran the command. On the command line, type the following command:
% netstat -i
Here is a sample display produced by netstat -i:
Name Mtu Net/Dest Address Ipkts Ierrs Opkts Oerrs Collis Queue le0 1500 b5-spd-2f-cm tatra 14093893 8492 10174659 1119 2314178 0 lo0 8232 loopback localhost 92997622 5442 12451748 0 775125 0
Using this display, you can find out how many packets a machine thinks it has transmitted and received on each network. For example, the input packet count (Ipkts) displayed for a server can increase each time a client tries to boot, while the output packet count (Opkts) remains steady. This suggests that the server is seeing the boot request packets from the client, but does not realize it is supposed to respond to them. This might be caused by an incorrect address in the hosts, ipnodes, or ethers database.
On the other hand, if the input packet count is steady over time, it means that the machine does not see the packets at all. This suggests a different type of failure, possibly a hardware problem.
How to Display Routing Table Status
The -r option of netstat displays the IP routing table. On the command line, type the following command.
% netstat -r
Here is a sample display produced by netstat -r run on machine tenere:
Routing tables
Destination Gateway Flags Refcnt Use Interface
temp8milptp elvis UGH 0 0
irmcpeb1-ptp0 elvis UGH 0 0
route93-ptp0 speed UGH 0 0
mtvb9-ptp0 speed UGH 0 0
.
mtnside speed UG 1 567
ray-net speed UG 0 0
mtnside-eng speed UG 0 36
mtnside-eng speed UG 0 558
mtnside-eng tenere U 33 190248 le0
The first column shows the destination network, the second the router through which packets are forwarded. The U flag indicates that the route is up; the G flag indicates that the route is to a gateway. The H flag indicates that the destination is a fully qualified host address, rather than a network.
The Refcnt column shows the number of active uses per route, and the Use column shows the number of packets sent per route. Finally, the Interface column shows the network interface that the route uses.
How to Log Network Problems
- Become superuser.
- Create a log file of routing daemon actions by typing the following command at a command line prompt.
# /usr/sbin/in.routed /var/logfilename
Caution: On a busy network, this can generate almost continuous output.
Displaying Packet Contents
You can use snoop to capture network packets and display their contents. Packets can be displayed as soon as they are received, or saved to a file. When snoop writes to an intermediate file, packet loss under busy trace conditions is unlikely. snoop itself is then used to interpret the file. For information about using the snoop command, refer to the snoop(1M) man page.
The snoop command must be run by root (#) to capture packets to and from the default interface in promiscuous mode. In summary form, only the data pertaining to the highest-level protocol is displayed. For example, an NFS packet only displays NFS information. The underlying RPC, UDP, IP, and Ethernet frame information is suppressed but can be displayed if either of the verbose options is chosen.
The snoop capture file format is described in RFC 1761. To access, use your favorite web browser with the URL: http://ds.internic.net/rfc/rfc1761.txt.
rstatd collects RPC traffic between a client and server.
How to Check All Packets from Your System
- Become superuser.
- Type the following command at the command line prompt to find the interfaces attached to the system.
# netstat -i
Snoop normally uses the first non-loopback device (le0).
- Type snoop. Use Ctl-C to halt the process.
# snoop Using device /dev/le (promiscuous mode) maupiti -> atlantic-82 NFS C GETATTR FH=0343 atlantic-82 -> maupiti NFS R GETATTR OK maupiti -> atlantic-82 NFS C GETATTR FH=D360 atlantic-82 -> maupiti NFS R GETATTR OK maupiti -> atlantic-82 NFS C GETATTR FH=1A18 atlantic-82 -> maupiti NFS R GETATTR OK maupiti -> (broadcast) ARP C Who is 120.146.82.36, npmpk17a-82 ?
- Interpret the results.
In the example, client maupiti transmits to server atlantic-82 using NFS file handle 0343. atlantic-82 acknowledges with OK. The conversation continues until maupiti broadcasts an ARP request asking who is 120.146.82.36?
This example demonstrates the format of snoop. The next step is to filter snoop to capture packets to a file.
Interpret the capture file using details described in RFC 1761. To access, use your favorite web browser with the URL: http://ds.internic.net/rfc/rfc1761.txt
How to Capture snoop Results to a File
- Become superuser.
On the command line, type the following command.
# snoop -o filename
For example:
# snoop -o /tmp/cap Using device /dev/le (promiscuous mode) 30 snoop: 30 packets captured
This has captured 30 packets in a file /tmp/cap. The file can be anywhere with enough disk space. The number of packets captured is displayed on the command line, enabling you to press Ctl-C to abort at any time.
snoop creates a noticeable networking load on the host machine, which can distort the results. To see reality at work, run snoop from a third system, (see the next section).
On the command line, type the following command to inspect the file.
# snoop -i filename
For example:
# snoop -i /tmp/cap 1 0.00000 frmpk17b-082 -> 224.0.0.2 IP D=224.0.0.2 S=129.146.82.1 LEN=32, ID=0 2 0.56104 scout -> (broadcast) ARP C Who is 129.146.82.63, grail ? 3 0.16742 atlantic-82 -> (broadcast) ARP C Who is 129.146.82.76, honeybea ? 4 0.77247 scout -> (broadcast) ARP C Who is 129.146.82.63, grail ? 5 0.80532 frmpk17b-082 -> (broadcast) ARP C Who is 129.146.82.92, holmes ? 6 0.13462 scout -> (broadcast) ARP C Who is 129.146.82.63, grail ? 7 0.94003 scout -> (broadcast) ARP C Who is 129.146.82.63, grail ? 8 0.93992 scout -> (broadcast) ARP C Who is 129.146.82.63, grail ? 9 0.60887 towel -> (broadcast) ARP C Who is 129.146.82.35, udmpk17b-82 ? 10 0.86691 nimpk17a-82 -> 129.146.82.255 RIP R (1 destinations)
Refer to specific protocol documentation for detailed analysis and recommended parameters for ARP, IP, RIP and so forth. Searching the Web is a good place to look at RFCs.
How to Check Packets Between Server and Client
Establish a snoop system off a hub connected to either the client or server.The third system (the snoop system) sees all the intervening traffic, so the snoop trace reflects reality on the wire.
- Become superuser.
- On the command line, type snoop with options and save to a file.
- Inspect and interpret results.
Look at RFC 1761 for details of the snoop capture file. To access, use your favorite web browser with the URL: http://ds.internic.net/rfc/rfc1761.txt
Use snoop frequently and consistently to get a feel for normal system behavior. For assistance in analyzing packets, look for recent white papers and RFCs, and seek the advice of an expert in a particular area, such as NFS or YP. For details on using snoop and its options, refer to the snoop(1M) man page.
Displaying Routing Information
Use the traceroute utility to trace the route an IP packet follows to some internet host. The traceroute utility utilizes the IP protocol time(to live) ttl field and attempts to elicit an ICMP TIME_EXCEEDED response from each gateway along the path, and the response PORT_UNREACHABLE (or ECHO_REPLY) from the destination host. The traceroute utility starts sending probes with a ttl of one and increases by one until it gets to the intended host or has passed through a maximum number of intermediate hosts.
The traceroute utility is especially useful for determining routing misconfiguration and routing path failures. If a particular host is unreachable, you can use the traceroute utility to see what path the packet follows to the intended host and where possible failures might occur.
The traceroute utility also displays the round trip time for each gateway along the path to the target host. This information can be useful for analyzing where traffic is slow between the two hosts.
How to Run the Traceroute Utility
On the command line, type the following command.
% traceroute destination-hostname
Example--traceroute Utility
The following sample of the traceroute command shows the 7-hop path a packet follows from the host istanbul to the host sanfrancisco along with the times for a packet to traverse each hop.
istanbul% traceroute sanfrancisco traceroute: Warning: Multiple interfaces found; using 172.31.86.247 @ le0 traceroute to sanfrancisco (172.29.64.39), 30 hops max, 40 byte packets 1 frbldg7c-86 (172.31.86.1) 1.516 ms 1.283 ms 1.362 ms 2 bldg1a-001 (172.31.1.211) 2.277 ms 1.773 ms 2.186 ms 3 bldg4-bldg1 (172.30.4.42) 1.978 ms 1.986 ms 13.996 ms 4 bldg6-bldg4 (172.30.4.49) 2.655 ms 3.042 ms 2.344 ms 5 ferbldg11a-001 (172.29.1.236) 2.636 ms 3.432 ms 3.830 ms 6 frbldg12b-153 (172.29.153.72) 3.452 ms 3.146 ms 2.962 ms 7 sanfrancisco (172.29.64.39) 3.430 ms 3.312 ms 3.451 ms
