How to use strace in Linux to Debug Process Hung issues

The strace utility is a debugging utility for Linux which is used to monitor the system calls used by a program and all the signals it receives. It is similar to the “truss” utility in Solaris.

A simple strace can be:

# strace [program name]

or

# strace -p [PID]

where PID is the process ID of the process in question.

Several useful switches for strace

Time option:

1. pass the -tt option. This tells when each syscall happened – -t prints it to the second, -tt to the microsecond. For system administration problems, this often helps a lot in correlating the trace with other logs, or in seeing where a program is spending too much time.

2. For performance issues, the -T option is useful. It tells how long each individual syscall took from start to finish.

3. -e option: ( -e expr) - A qualifying expression which modifies which events to trace or how to trace them. The format of the expression is:

[qualifier=][!]value1[,value2]...

where : the qualifier is one of trace, abbrev, verbose, raw, signal, read, or write and, value is a qualifier-dependent symbol or number.

The default qualifier is trace. Using an exclamation mark negates the set of values. For example, -eopen means literally -e trace=open which in turn means trace only the open system call. By contrast, -etrace=!open means to trace every system call except open. In addition, the special values all and none have obvious meanings. One can even specify the system call functions to trace using the -e option.

- If you want to see the write() and read() syscalls, use options -e read= and -e write= - If you want to trace only open() and close() function system calls, use -e trace=open,close

# Other important -e switches: -e trace=set - Trace only the specified set of system calls. The -c option is useful for determining which system calls might be useful to trace. For example, trace=open,close,read,write means to only trace those four system calls. Be careful when making inferences about the user/kernel boundary if only a subset of system calls are being monitored. The default is trace=all.

-e trace=file - Trace all system calls which take a file name as an argument. You can think of this as an abbreviation for -e trace=open,stat,chmod,unlink,… which is useful for seeing what files the process is referencing. Furthermore, using the abbreviation will ensure that you don’t accidentally forget to include a call like lstat in the list.

-e trace=process - Trace all system calls which involve process management. This is useful for watching the fork, wait, and exec steps of a process.

-e trace=network - Trace all the network related system calls.

Some examples of strace usage

$ strace [name of the program]
$ strace -o strace_output.txt [program name]
$ strace -p [PID]

where PID is the process ID of the process in question.

To trace only open() and close() function system calls:

$ strace -o strace.out -e trace=open,close [program name]

To see comprehensive output, including all child processes:

$ strace -fe verbose=all -e write=all -e read=all -o /tmp/output.txt /opt/sun/appserver/bin/asadmin start-domain domain1

The following example shows how to run strace while starting the Application server without any child processes:

$ strace -o /tmp/output.txt /opt/sun/appserver/bin/asadmin start-domain domain1

SELinux Caveats

Security-Enhanced Linux (SELinux) is a Linux kernel security module that provides a mechanism for supporting access control security policies. Because of the constraints, it implements on a system, particular problems may be encountered when performing diagnostic work on a system deployed with SELinux.

Specifically, you may encounter problems with running strace, such as no trace data being output. One general option you may consider when doing diagnostic work is to disable SELinux, to eliminate it as an issue in testing. Alternately, you could set it to permissive mode ( permissive mode provides warning messages, rather than enforcing restrictions). You can do this by editing /etc/selinux/config and setting selinux= to either permissive or disabled.

For example: /etc/selinux/config

SELINUX=disabled

Note that this permanently sets the mode SELinux operates in until it is changed. You can also make a temporary change in enforcement, until reboot, by using setenforce. For more details on this, see the documentation on your linux distribution with SELinux or refer to Configuring and Using SELinux.

Also note that strace uses calls to ptrace() for to perform its function. By default, SELinux blocks calls to ptrace(). Disabling SELinux, as in the above example, will resolve this during testing. You may also be able to resolve the problem with strace operation by configuring SELinux to allow ptrace(), using the following SELinux command:

# setsebool allow_ptrace on