Search My Blog

Monday, August 15, 2011

HOWTO: Log a kernel panic.. It can be done! | TOCPCs – The Elite Geeks Blog

HOWTO: Log a kernel panic.. It can be done!

Ever find yourself in a position with a server which you can't watch a kernel panic on, and wonder, how the hell can I find the cause of this issue, with no access to the console, and no way of logging the kernel panic to disk?

It just so happens the situation has happened to me. And I was keen to get to the bottom of these bloody kernel panics. They are annoying.

You see, a server I have has been crashing at random intervals, with no software messages giving reason for the crashes.

After getting tired of pestering the folk who run the rack for reboots, I investigated a way of making the machine reboot by itself.

After a little bit of searching a resolution was found to make it reboot by itself:
edit the boot menu list (the list of kernels it can boot – /boot/grub/menu.lst) to include the line panic=20.

Each time a panic occurs, the server won't sit there waiting for a reboot, it'll simply reboot itself. Much better.

Now, we go back to the original issue, investigating the cause of the panics, when you aren't able to look at the system, and the system panics randomly. Remote logging of panics. Fantastic idea, but how does it work? No idea. Until now.

The feature is called "netconsole", and here is what you do.

Ping the server you want to log the panic: ping -c 1 123.456.789.012

After the ping executes, type arp -a, and find the MAC address (00:12:34:56:78:89) of the server you want logging to go to.

You need two machines in the same network (or you must have the others MAC address).

With that information at hand, we can start logging those bloody panics:
On the logging machine, ensure you have netcat installed (yum install nc). You can after install, execute:
nc -u -l -p 6969

Which starts netcat. Cat is a program for text capture / output. Netcat is the exact same, except it listens on the network.

On the crashing machine, while it is up, enter:


This article ties in to the links below...


How Linux handles hardware problems
How Linux handles hardware problems | ZDNet UK
Memtest86+ - Advanced Memory Diagnostic Tool
HOWTO: Log a kernel panic.. It can be done! | TOCPCs – The Elite Geeks Blog

No comments: