Debugging Tools

Introduction

This document provides a quick reference to some commands that can be used to examine the internal state of a running program. It is not a comprehensive treatment of debugging, or even the commands involved. The goal is to make it easier to get in situ details in a form that can be easily sent to experts for review.

Preparation

General

WARNING

Debugging commands often have a performance impact (both CPU and RAM use), and it can sometimes be severe. Capturing the output of such tools can sometimes use a large amount of disk space. In extreme cases, a debugging command can disrupt the operation of a program. Test/lab systems are recommended when practical.

Many of these tools need to be run as root, or at least yield better results when run as root. Thus, root access will likely be needed. Run the commands from a root shell, or prefix with sudo.

The software packages containing the tool will need to be installed. The exact packages will vary by distribution and release. Looking for what is needed to install the GNU debugger (GDB) on your distribution may be a good starting point.

Kernel parameters

On Linux, the following may be required, to tell the kernel to allow users to attach to their own processes for debugging. Some distributions set this to a more restrictive value by default.

echo 0 > /proc/sys/kernel/yama/ptrace_scope

Debug symbols

Debugging symbols are information about the names and types in a compiled program or library. When symbols are available, software tools can provide human-friendly names and other details to aid in debugging. Without symbols, only the most basic machine-language information is available. Thus, installing collections of debug symbols is often an essential prerequisite for running any debugging tool.

Distributions often package symbols separately from the main software package, since most people have no use for the symbols.

For Debian and derivatives, these packages are usually named with a trailing -dbg or -dbgsym (suffixed to the main package name). You may also need to add a separate repository (APT sources list) to find the debug packages.

Red Hat and derivatives use a similar convention with a -debuginfo suffix. Also similar, you may need to enable a separate debug repository, using the dnf --enablerepo= option.

Examining processes

In the various commands given below, $PID is used as a stand-in for the process identification number (PID) of a running program.

You can use a command like ps alxw | less to see all running processes.

One useful utility is pidof. You can use a command like pidof named or pidof kea-dhcp4 to find the PID(s) of running programs. You can then examine that process with ps uw $PID.

Fields in ps output

A full treatment of all process fields (columns) is beyond the scope of this document. Some particularly relevant fields are briefly described below. For more information, see the ps(1) manual page.

Field	Meaning
`VSZ`	Virtual size. Total size of all memory pages potentially mappable. Includes memory-mapped files, hardware address space, and other things that are not RAM. Not generally useful.
`RSS`	Resident size. Actual RAM in use by the process. Excludes anything unmapped or paged out. This reflects the current memory consumption.
`START`	Time or date the process was started.
`TIME`	Amount of CPU time used. If this is increasing, the program is doing something. (Not necessarily something useful.)
`STAT`	Process status. See below.
`WCHAN`	Wait channel. For a process waiting on the kernel, the syscall that is blocked.

Status flags

Again, this list is not comprehensive; see ps(1).

Field	Meaning
`R`	Runnable. The process wants CPU time.
`S`	Sleeping. The process is waiting for an event or timer.
`D`	Device wait. The process is waiting for a system call to return.

Examining the stack

What is a stack?

For our purposes, the stack can be thought of as the trail of function calls built-up as a program runs. Looking at the stack tells us the path through the code a program followed to get to the current moment. This information is often extremely useful to a software engineer, especially postmortem.

Each process and thread has its own stack.

GDB

GDB is the preferred tool here as it is almost always readily available in any distribution. The following shell command instructs GDB to dump the stacks (backtrace) of each thread in the given process, and captures the results to a file.

gdb -p $PID -batch -ex 'thread apply all bt' -ex detach > /tmp/backtrace.out

Others

The classic utility for examining the stack was pstack, but it is no longer maintained and is usually unavailable. On some current distributions (as of 2025), a shell script is included called pstack which provides a wrapper for the GDB command given above.

The direct successor is often considered to be eu-stack. It is part of the elfutils project. The following command is similar in effect to the GDB command given above.

eu-stack -v -p $PID > /tmp/eustack.out

Tracing function calls

A function is a named block of program code that can be invoked elsewhere in the program, or by another program. Functions may also referred to as subroutines or procedures. By tracing these function calls — monitoring which functions are being called — we can know what a program is doing internally.

Commands to trace function calls on Linux are given below. FreeBSD operators might explore the capabilities of the truss(1) command instead.

ltrace

To trace all library and system calls of a running process on Linux:

ltrace -p $PID -f -s0 -tt -S > /tmp/trace.out

The options above have the following effects:

Opt	Meaning
`-f`	Also trace forked children
`-s0`	Report entire strings (no length limit)
`-tt`	Timestamp with milliseconds
`-S`	Also trace system calls (in addition to library calls)