Debugging Servers

Servers are multithreaded, so if one hangs, it is most interesting to see what the different threads are doing. Using gdb, you can get a stack trace of all threads (provided, you compiled your server with "--enable-debug").

First, we need to get the pid and the full path of the server (e.g. fileserver):

PID=`ps -eaf | grep fileserver | grep -v grep | awk '{print $2}'`
SRV_PATH=`ps -eaf | grep fileserver | grep -v grep | awk '{print $9}'`

for other servers, this might be different, so double-check

Then we need to attach the gdb and make it display the stack of all threads :

gdb --batch --eval-command="thread apply all where" $SRV_PATH $PID > /tmp/threads.log

After this you might use the attached script to display the intersting threads.

e.g. -g /tmp/threads.log -e rx_GetCall

displays only threads which are not in this boring rx_GetCall (threads are waiting for a new call).

Debugging Clients

unix client tracing with fstrace

Low-level debugging of a client can be done using "fstrace". The analysis of the "fstrace" output requires detailed knowledge of cache manager internals. "fstrace" logs to an internal memory based ring buffer, which can be dumped to a file for analysis.

To capture a trace, set the ring buffer size, enable the tracing to the buffer, run your tests, then dump the buffer. The following example creates a 1 Mb buffer:

fstrace setlog -buffersize 1024
fstrace setset -active
fstrace clear

-- do your tests --

fstrace dump -file /tmp/fstrace.out

To disable the tracing when you are done:

fstrace setset -inactive

intermittent issues

Unfortunately, "fstrace" produces a lot of output, so it is may not be easy to catch an intermittent error-condition.

The attached script (presently for Unix only) gives you the opportunity to continuously run a fstrace,
where the output is stored in rotating log-files.

In case of an external-event (the existence of a predefined file), it saves this log and does not overwrite it again. Thus, all you need to do is to write a script which creates this predefined file, when that event happens.

defined issue

When you exactly know how to reproduce the issue, or you don't want to install python on your client, you can use the attached script ClientTracing.bat (for Windows only). It is setting up the client tracing, starting up a tshark (terminal-version of wireshark) and waits for you to tell it to stop tracing. In order to synchronize the timestamps of the wireshark log and the afsd-trace-log, a "dir \\AFS\" is issued on startup. The logfiles are copied to a predefined directory. The script itself should be self-explanatory.]