---
title: "Debugging Tools"
slug: "debug"
description: "This document provides a quick reference to some commands that can be used to examine the internal state of a running program. It is not a comprehensive treatment of debugging, or even the commands involved. The goal is to make it easier to get in situ details in a form that can be easily sent to experts for review."
updated: 2025-07-23T13:03:28Z
published: 2025-07-23T13:03:28Z
canonical: "kb.isc.org/debug"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://kb.isc.org/llms.txt
> Use this file to discover all available pages before exploring further.

# Debugging Tools

## Introduction

This document provides a quick reference to some commands that can be used to examine the internal state of a running program. It is not a comprehensive treatment of debugging, or even the commands involved. The goal is to make it easier to get *in situ* details in a form that can be easily sent to experts for review.

## Preparation

### General

          WARNING

          

Debugging commands often have a performance impact (both CPU and RAM use), and it can sometimes be severe. Capturing the output of such tools can sometimes use a large amount of disk space. In extreme cases, a debugging command can disrupt the operation of a program. Test/lab systems are recommended when practical.

Many of these tools need to be run as root, or at least yield better results when run as root. Thus, root access will likely be needed. Run the commands from a root shell, or prefix with `sudo`.

The software packages containing the tool will need to be installed. The exact packages will vary by distribution and release. Looking for what is needed to install the GNU debugger (GDB) on your distribution may be a good starting point.

### Kernel parameters

On Linux, the following may be required, to tell the kernel to allow users to attach to their own processes for debugging. Some distributions set this to a more restrictive value by default.

```
echo 0 > /proc/sys/kernel/yama/ptrace_scope
```

### Debug symbols

Debugging symbols are information about the names and types in a compiled program or library. When symbols are available, software tools can provide human-friendly names and other details to aid in debugging. Without symbols, only the most basic machine-language information is available. Thus, installing collections of debug symbols is often an essential prerequisite for running any debugging tool.

Distributions often package symbols separately from the main software package, since most people have no use for the symbols.

For Debian and derivatives, these packages are usually named with a trailing `-dbg` or `-dbgsym` (suffixed to the main package name). You may also need to add a separate repository (APT sources list) to find the debug packages.

Red Hat and derivatives use a similar convention with a `-debuginfo` suffix. Also similar, you may need to enable a separate debug repository, using the `dnf --enablerepo=` option.

## Examining processes

In the various commands given below, `$PID` is used as a stand-in for the process identification number (PID) of a running program.

You can use a command like `ps alxw | less` to see all running processes.

One useful utility is `pidof`. You can use a command like `pidof named` or `pidof kea-dhcp4` to find the PID(s) of running programs. You can then examine that process with `ps uw $PID`.

### Fields in ps output

A full treatment of all process fields (columns) is beyond the scope of this document. Some particularly relevant fields are briefly described below. For more information, see the `ps(1)` manual page.

| Field | Meaning |
| --- | --- |
| `VSZ` | Virtual size. Total size of all memory pages potentially mappable. Includes memory-mapped files, hardware address space, and other things that are not RAM. Not generally useful. |
| `RSS` | Resident size. Actual RAM in use by the process. Excludes anything unmapped or paged out. This reflects the current memory consumption. |
| `START` | Time or date the process was started. |
| `TIME` | Amount of CPU time used. If this is increasing, the program is doing something. (Not necessarily something useful.) |
| `STAT` | Process status. See below. |
| `WCHAN` | Wait channel. For a process waiting on the kernel, the syscall that is blocked. |

### Status flags

Again, this list is not comprehensive; see `ps(1)`.

| Field | Meaning |
| --- | --- |
| `R` | Runnable. The process wants CPU time. |
| `S` | Sleeping. The process is waiting for an event or timer. |
| `D` | Device wait. The process is waiting for a system call to return. |

## Examining the stack

### What is a stack?

For our purposes, the *stack* can be thought of as the trail of function calls built-up as a program runs. Looking at the stack tells us the path through the code a program followed to get to the current moment. This information is often extremely useful to a software engineer, especially postmortem.

Each process and thread has its own stack.

### GDB

GDB is the preferred tool here as it is almost always readily available in any distribution. The following shell command instructs GDB to dump the stacks (backtrace) of each thread in the given process, and captures the results to a file.

```
gdb -p $PID -batch -ex 'thread apply all bt' -ex detach > /tmp/backtrace.out
```

### Others

The classic utility for examining the stack was `pstack`, but it is no longer maintained and is usually unavailable. On some current distributions (as of 2025), a shell script is included called `pstack` which provides a wrapper for the GDB command given above.

The direct successor is often considered to be `eu-stack`. It is part of the `elfutils` project. The following command is similar in effect to the GDB command given above.

```
eu-stack -v -p $PID > /tmp/eustack.out
```

## Tracing function calls

A *function* is a named block of program code that can be invoked elsewhere in the program, or by another program. Functions may also referred to as *subroutines* or *procedures*. By tracing these function calls — monitoring which functions are being called — we can know what a program is doing internally.

Commands to trace function calls on Linux are given below. FreeBSD operators might explore the capabilities of the `truss(1)` command instead.

### ltrace

To trace all library and system calls of a running process on Linux:

```
ltrace -p $PID -f -s0 -tt -S > /tmp/trace.out
```

The options above have the following effects:

| Opt | Meaning |
| --- | --- |
| `-f` | Also trace forked children |
| `-s0` | Report entire strings (no length limit) |
| `-tt` | Timestamp with milliseconds |
| `-S` | Also trace system calls (in addition to library calls) |

## See also

- [What to do if your BIND, Kea DHCP, Stork, or ISC DHCP server has crashed](https://kb.isc.org/docs/aa-00340) at the ISC KB (addresses core dumps)
- Distribution guidance on installing debug symbols
  - [Installing debug symbols on Debian](https://wiki.debian.org/DebugPackage)
  - [Installing debug symbols on Red Hat Enterprise Linux](https://access.redhat.com/solutions/9907)
  - [Installing debug symbols on Ubuntu](https://documentation.ubuntu.com/server/explanation/debugging/debug-symbol-packages/)

## Related

- [What to do if your BIND, Kea DHCP, Stork, or ISC DHCP server has crashed](/aa-00340.md)
- [Essential UNIX Commands for BIND, ISC DHCP, and Kea DHCP Administrators](/essential-unix-commands-for-bind-isc-dhcp-and-kea-dhcp-administrators.md)