HOWTO: Configure lkcd to capture a kernel core dump

1. Install the lkcdutils RPM.
2. Make sure boot.lkcd is active at boot time:
   
   chkconfig boot.lkcd on
3. Modify the /etc/sysconfig/dump configuration file with the following settings:
   
   DUMP_ACTIVE="1"
   DUMPDEV="/dev/vmdump"
   DUMPDIR="/var/log/dump"
   DUMP_LEVEL="2"
   DUMP_COMPRESS="2"
   DUMP_FLAGS="0x80000000"
   DUMP_SAVE="1"
   PANIC_TIMEOUT="5"
   BOUNDS_LIMIT="10"
   KEXEC_IMAGE="/boot/vmlinuz"
   KEXEC_CMDLINE="root console=tty0"
   DUMP_MAX_CONCURRENT=4
4. Make sure the DUMPDEV /dev/vmdump is linked to the correct dump device (usually this will be a swap partition, but it can be any block device whose contents may be overwritten):
   
   ls -l /dev/vmdump
   lrwxrwxrwx 1 root root 9 Aug 10 08:33 /dev/vmdump ->/dev/sda1
   
   NOTE: When you run the lkcd command in the next step, it will create a/dev/vmdumpsymbolic link to the first swap partition found in /etc/fstab. If you do not want to use the first swap partition, then you must manually create the DUMPDEV symbolic link.
5. Enable core dump capturing on the server:
   
   lkcd config
   lkcd query
   

   Configured dump device: 0x801
   Configured dump flags: KL_DUMP_FLAGS_DISKDUMP
   Configured dump level: KL_DUMP_LEVEL_HEADER|KL_DUMP_LEVEL_KERN
   Configured dump compression method: KL_DUMP_COMPRESS_GZIP
6. Enable Magic Keys (This is useful in the event of a server hang. Only a kernel oops or panic will automatically create a core dump. However, for security reasons, you will want to disable Magic SysRq keys when there is no longer a need.):
   
   • Persistent across reboots:
     yast | Security and Users | Security settings | Miscellaneous Settings (Page 6), select Enable Magic SysRq Keys, select Finish, Quit
   • Temporary change:
     
     echo 1>/proc/sys/kernel/sysrq
7. Following a system crash, create a tarball of the current kernel image:
   
   tar jcvf kernelcore.tar.bz2 /var/log/dump/n/* # where n represents the current core image

• Server name = core, Server IP = 10.0.0.10, Description = This is the server that will initiate a kernel core dump
• Server name = netdump, Server IP = 10.0.0.20, MAC Addr = aa:bb:cc:dd:ee:ff, Description = This server is the netdump-server and will save the kernel image received from the core server

1. Install the lkcdutils and lkcdutils-netdump-server RPMs
2. Make sure netdump-server is active at boot time:
   
   chkconfig netdump-server on
3. Modify the /etc/sysconfig/dump configuration file with the following settings:
   DUMPDIR="/var/log/dump"
   DUMP_FLAGS="0x40000000"
   SOURCE_PORT="6688"
   NETDUMP_VERBOSE="yes"
   NOTE: NETDUMP_VERBOSE="yes" is only needed for troubleshooting purposes in the event your netdumps are failing, otherwise set to "no". Verbose logging is recorded in the /var/log/messages file on the netdump server.
4. Create a /var/log/dump directory, writable for the netdump user (which the netdump-server runs under by default):
   
   install -o netdump -g dump -m 777 -d /var/log/dump
   
5. Start the netdump-server service:
   
   /etc/init.d/netdump-server start
   
6. Following a system crash from the core server, create a tarball with the necessary image files. The vmcore will be on the netdump server, and the Kerntypes and System.map files will be on the core server. The kernel image file cannot be read without the Kerntypes andSystem.mapfiles.
   core:/boot/System.map-$(uname -r)
   core:/boot/Kerntypes-$(uname -r)
   netdump:/var/log/dump/ipaddr-date-time/vmcore # where ipaddr-date-time is a specific directory with the values of the core server's IP address and the date and time of the crash

1. Install the lkcdutils RPM.
2. Make sure lkcd-netdump is active at boot time:
   
   chkconfig lkcd-netdump on
3. Modify the /etc/sysconfig/dump configuration file with the following settings -- modify to fit your environment:
   
   DUMP_ACTIVE="1"
   DUMPDEV="eth0"
   DUMPDIR="/var/log/dump"
   DUMP_LEVEL="2"
   DUMP_COMPRESS="2"
   DUMP_FLAGS="0x40000000"
   DUMP_SAVE="1"
   PANIC_TIMEOUT="5"
   BOUNDS_LIMIT="10"
   KEXEC_IMAGE="/boot/vmlinuz"
   KEXEC_CMDLINE="root console=tty0"
   TARGET_HOST="10.0.0.20"
   TARGET_PORT="6688"
   SOURCE_PORT="6688"
   ETH_ADDRESS="aa:bb:cc:dd:ee:ff"
   DUMP_MAX_CONCURRENT=4
4. Enable core dump capturing on the server:
   
   lkcd config
   
   lkcd query
   

   Configured dump device: 0xbffff617
   Configured dump flags: KL_DUMP_FLAGS_NETDUMP
   Configured dump level: KL_DUMP_LEVEL_HEADER|KL_DUMP_LEVEL_KERN
   Configured dump compression method: KL_DUMP_COMPRESS_GZIP
5. Enable Magic Keys (This is useful in the event of a server hang. Only a kernel oops or panic will automatically create a core dump. However, for security reasons, you may want to disable Magic SysRq keys when there is no longer a need.)
   
   • Persistent across reboots:
     yast | Security and Users | Security settings | Miscellaneous Settings (Page 6), select Enable Magic SysRq Keys, select Finish, Quit
   • Temporary change:
     
     echo 1>/proc/sys/kernel/sysrq

1. After the server has rebooted, login as root.
2. Change to the directory that contains the kernel image:
   
   cd /var/log/dump/0
3. Load the image with lcrash
   
   lcrash map.0 dump.0 kerntypes.0
4. For network core images, the files would probably be:
   
   lcrash System.map-* vmcore Kerntypes-*
5. If you are taken to the`>>'lcrash prompt, the kernel image is valid.
6. Press "q" to quit.
7. Tar up all the files and submit them to NTS.

Follow IBM's documentationLinux on System z - Using the Dump Tools
available as http://download.boulder.ibm.com/ibmdl/pub/software/dw/linux390/docu/l26cdt01.pdf
or via http://www.ibm.com/developerworks/linux/linux390/history.html

+ Upgrading to Open Enterprise Server 2 SP1 Linux