How to Debug Kernel Panics with Crash Dumps on FreeBSD

Kernel panics are critical system failures that occur when the FreeBSD kernel encounters an unrecoverable error. These panics can lead to system crashes and data loss if not properly diagnosed and addressed. Fortunately, FreeBSD provides tools to generate and analyze crash dumps, which contain valuable diagnostic information.

This article walks through the process of debugging kernel panics using crash dumps on FreeBSD, covering setup, collection, and analysis.

Understanding Kernel Panics and Crash Dumps

A kernel panic occurs when the FreeBSD kernel detects an internal error that prevents normal execution. This could be due to hardware failures, driver bugs, memory corruption, or unexpected system states.

To diagnose these panics, FreeBSD can generate crash dumps—snapshots of kernel memory at the moment of failure. These dumps are essential for identifying the root cause.

Setting Up Crash Dumps

Enabling Crash Dumps

Before FreeBSD can generate crash dumps, ensure that:

1. The system has a dedicated swap partition with enough space to store dumps.
2. The dumpon service is configured to enable crash dumps.

To check available swap space:

swapinfo -h

Ensure the swap partition is at least as large as the system’s RAM.

To enable crash dumps, use:

dumpon -v /dev/<swap_partition>

Replace <swap_partition> with your swap device (e.g., /dev/ada0p2). To make this setting persistent, add the following to /etc/rc.conf:

dumpdev="AUTO"

This ensures crash dumps are automatically captured after a panic.

Configuring Dump Collection

After a crash, FreeBSD saves the dump to the swap partition. The savecore utility retrieves and stores the dump upon reboot. Ensure it is enabled:

echo 'dumpdir="/var/crash"' >> /etc/rc.conf
service savecore onestart

This moves dumps from swap to /var/crash for later analysis.

Analyzing Crash Dumps

Locating and Loading the Crash Dump

Once the system reboots, check for dumps:

ls -lh /var/crash

You should see files named vmcore.X and info.X, where X is the dump sequence number.

To analyze the dump, use kgdb, which allows debugging of kernel crash dumps:

kgdb /boot/kernel/kernel /var/crash/vmcore.0

Extracting Key Information

Identifying the Panic Message

Run:

bt

This displays the kernel backtrace, showing function calls leading to the panic.

Checking Kernel Logs

Extract panic messages from logs:

dmesg | less

Or review /var/log/messages for additional context.

Analyzing Stack Trace

The bt command output in kgdb shows function calls. Look for:

• The first function in the list (potential panic trigger)
• Any recently modified drivers or kernel modules

Debugging Kernel Modules

If a module caused the crash, unload it for testing:

kldunload <module>

Reboot and monitor stability.

If necessary, rebuild the module with debugging symbols:

make DEBUG_FLAGS=-g

Then, reanalyze the dump with symbols enabled.

Investigating Memory Corruption

Use show registers in kgdb to inspect register values. If memory corruption is suspected, vmstat -m and show malloc may help track memory allocation issues.

Preventing Future Kernel Panics

1. Keep FreeBSD Updated – Bug fixes and security patches often resolve stability issues.
2. Monitor Logs Regularly – Review /var/log/messages and dmesg for warnings.
3. Use Stable Drivers – Avoid experimental or untested kernel modules.
4. Test Hardware – Run memtest86 for RAM and check smartctl for disk errors.
5. Enable Watchdogs – Use system watchdogs to automatically reboot on failures.

Conclusion

Debugging kernel panics in FreeBSD requires enabling crash dumps, collecting them with savecore, and analyzing them with kgdb. By following a structured approach, you can identify root causes, apply fixes, and improve system stability.

Understanding the stack trace, module involvement, and memory state helps pinpoint the problem. Regular system monitoring and keeping FreeBSD updated can prevent many common panics. By applying these practices, administrators can maintain a more reliable FreeBSD environment.