Article

(View Disclaimer)

Contents:

• Environment
• Problem
• Solution
• Server configuration
• Client configuration
• Alternative solutions
• Conclusion
• External links

Environment

SUSE Linux Enterprise Server 10 Service Pack 1

Problem

Kernel boot messages need to be saved after a server crash/hang but access to a serial console is not available.

Solution

Netconsole can send boot kernel printk messages to another host on the network using UDP pakets.

Limitations

• Only IP networks, UDP pakets and ethernet devices are supported.
• The network card driver has to support the netpoll api.
• Early kernel panics might not be captured.

Test systems

Server configuration

1. Determine client MAC address

An easy way to do this is by pinging the client from the server with:

ping -c 1 10.0.0.2; clear; /sbin/arp -n |awk '/10.0.0.2 /{ print $3}'

Note: In case the remote logging client is on a separate subnet than the sender, you will need to know the MAC address of the default gateway.

route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.0.0.0        0.0.0.0         255.0.0.0       U     0      0        0 eth0
0.0.0.0         10.0.0.254      0.0.0.0         UG    0      0        0 eth0

The default gateway is the one where "Destination" is 0.0.0.0. Determine the MAC address by using the ping command in step 1 with the IP of the "Gateway", shown here as 10.0.0.254.

2. Load the netconsole module at boot time

   Edit /etc/sysconfig/kernel and change:

   MODULES_LOADED_ON_BOOT="netconsole"
   

   Run:

   SuSEconfig

3. Netconsole module options

   Edit /etc/modprobe.conf.local and add:

   options netconsole netconsole=4444@10.0.0.1/eth0,6666@10.0.0.2/00:1A:A0:D2:55:66
   

   Explanation

   4444              => Server source port
   10.0.0.1          => Server IP
   eth0              => Server NIC
   6666              => Client listening port
   10.0.0.2          => Client IP
   00:1A:A0:D2:55:66 => Client MAC
   

Client configuration

1. Open firewall port:

Edit /etc/sysconfig/SuSEfirewall2 and change:

FW_ALLOW_INCOMING_HIGHPORTS_UDP="6666"

The firewall needs to be restarted for the changes to take effect.

rcSuSEfirewall2 restart

If the client is already behind a firewall and security is not a concern,
you can alternatively disable SuSEfirewall2:

rcSuSEfirewall2 stop

2. Receive kernel messages

   There are several methods how to read the kernel messages on the client.

   2.1 On screen reading:

   netcat -l -u -p 6666
   

   2.2 Log to a file:

   netcat -l -p 6666 -u 2>&1 | tee /var/log/netconsole &
   

   2.3 Syslog-ng log:

   Edit /etc/syslog-ng/syslog-ng and add:

   source net { udp(ip("0.0.0.0") port(6666)); };
   destination netconsole { file("/var/log/$HOST-netconsole.log"); };
   log { source(net); destination(netconsole); };
   

   Restart syslog-ng:

   service syslog restart
   

   You will find the logs under /var/log/$hostname-netconsole.log

   ls /var/log |grep netconsole
   10.0.0.1-netconsole.log
   

   A sample of the boot messages would look similar to this:

   head -15 /var/log/10.0.0.1-netconsole.log
   May  2 19:00:00 10.0.0.1 Linux version 2.6.16.54-0.2.5-bigsmp (geeko@buildhost) (gcc version 4.1.2 20070115 (prerelease) (SUSE Linux)) #1 SMP Mon Jan 21 13:29:51 UTC 2008
   May  2 19:00:00 10.0.0.1 BIOS-provided physical RAM map:
   May  2 19:00:00 10.0.0.1 BIOS-e820: 0000000000000000 - 00000000000a0000 (usable)
   May  2 19:00:00 10.0.0.1 BIOS-e820: 0000000000100000 - 00000000efff0000 (usable)
   May  2 19:00:00 10.0.0.1 BIOS-e820: 00000000efff0000 - 00000000efffec00 (ACPI data)
   May  2 19:00:00 10.0.0.1 BIOS-e820: 00000000efffec00 - 00000000effff000 (reserved)
   May  2 19:00:00 10.0.0.1 BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
   May  2 19:00:00 10.0.0.1 BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)
   May  2 19:00:00 10.0.0.1 BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
   May  2 19:00:00 10.0.0.1 2943MB HIGHMEM available.
   May  2 19:00:00 10.0.0.1 895MB LOWMEM available.
   May  2 19:00:00 10.0.0.1 found SMP MP-table at 000fe710
   May  2 19:00:00 10.0.0.1 DMI 2.3 present.
   May  2 19:00:00 10.0.0.1 Using APIC driver default
   May  2 19:00:00 10.0.0.1 ACPI: PM-Timer IO Port: 0x808
   
   

Alternative solutions

Kernel messages can be redirected to a serial console if a serial port and a null modem cable are available.

See knowledge base article "Configuring a Remote Serial Console for SLES".

Conclusion

Netconsole is a helpful tool for debugging kernel issues where you need to capture and analyze kernel messages from a crashed or hanging server when disk logging or serial consoles are unavailable.

External links

Netconsole kernel documentation

Disclaimer: As with everything else at Cool Solutions, this content is definitely not supported by Novell (so don't even think of calling Support if you try something and it blows up).

It was contributed by a community member and is published "as is." It seems to have worked for at least one person, and might work for you. But please be sure to test, test, test before you do anything drastic with it.