Wednesday, June 25, 2008

Remote Attachment

In addition to being able to execute programs on a remote system from within the ZeroBUGS debugger, it is also possible to attach to remote processes.

When the remote proxy plug-in is detected, the UI automatically adds a "Target Parameters" field at the bottom of the "Attach to Process" dialog box.

After you type in the protocol ("remote://") followed by either the name or the IP address of the target system, and press the "Refresh" button the box will be populated with a list of processes running on the remote machine. Just select the desired process and click "Ok".

Sunday, June 22, 2008

May I Have the Remote, Please?

This past weekend I put the final touch on supporting remote debugging in my favorite C/C++ debugger for Linux, ZeroBUGS.

It is under test and will ship in the commercial version within the next five to ten days.

Currently, the remote debugger is not a cross-debugger, that is to say it only works between systems of the same architecture (adding support for cross-debugging is not terribly hard, I just do not want to spend rare Seattle summer days working on a feature, unless users ask for it).

"Well, then what's the point in supporting remote debugging", some may ask. "Can't we just ssh -X into the remote target and run the debugger there?"

Installing a full-fledged debugger, with a UI-module that depends heavily on Gnome may not be a feasible option for small (possibly embedded) systems where resources are scarce.

My solution for remote debugging is to install a thin, lightweight server on the target system, and have the debugger on a Linux workstation do the heavy lifting of building symbol tables, managing breakpoints, and so on.

In order to read debug information, the debugger needs access to the executable and shared objects on the target system. One possible solution is to copy them over to the workstation where the debugger runs, but it is not very practical. The amount of files to be copied can quickly go out of control, by ways of shared library dependencies.

I found that it is simpler to just mount the remote target onto the machine where the debugger runs. SSHFS is ideal for this job. Because the debug info may contain references to absolute path, ZeroBUGS provides the ZERO_REMOTE_PATH environmental variable, which creates an internal mapping between the mount point and the original paths.

Here are the steps for a remote debugging sessions, with examples from my own lab. The debugger runs on a 64 bit system (zulu) running Ubuntu 7.10, and the target executable(s) reside on another 64 bit machine (arnold) running Ubuntu 8.04.

1) Mount the filesystem of the target computer onto the debugger system. Example:

cristiv@zulu:~/workspace/sandbox$ sudo chown cristiv /dev/fuse
[sudo] password for cristiv:
cristiv@zulu:~/workspace/sandbox$ sshfs root@ ~/workspace/remote/
root@'s password:

2) Add mount point to remote map. Example:

export ZERO_REMOTE_MAP=";"

Note that each entry must be ended with a semicolon (even when there is only one entry in the map).

3) Start the ZeroBUGS server on the remote (debug target) system:

cristiv@arnold:~/workspace/zero$ zserver
*** ZeroBUGS Remote Debug Server V. 1.0 ***
*** Copyright (c) 2008 Zero Systems LLC ***

To debug remotely programs running on host
4) Run a remote program and debug it, using the command line:

zero remote://

The UI can also be used to execute remote targets:

As a final note, please remember that the debug server opens a security whole on the target systems, since the client debugger can execute any program that the user who started the server can.

Monday, June 02, 2008

Take the Fork in the Road

A friend of mine (let's call him Andrei) who works on the D Programming Language sent me a bug report the other day. He was having trouble debugging this piece of D code with ZeroBUGS:

import std.stdio, std.process, std.string;
void main(string[] args)
writeln("Started on ", chomp(shell("hostname --short")));

After a brief investigation, I figured out what was going on: the shell call (equivalent to system in C) was triggering a fork(), followd by an exec() call. The debugger automatically attached to the spawned shell. After which point, every time my friend was trying to step through the code, control would jump from one process to the other in an apparent indeterministic fashion (I say apparent because the behavior is actually determined by how the kernel schedules the main, forked, and debugger processes).

I replied to the bug report with an explanation of what was going on, suggesting that the --no-trace-fork switch be passed in the command line (and thus avoid attaching to the forked shell).

My friend argued that this option should be more obvious (i.e. accessible from the graphical user interface) and while you are at it, he said, why not add a "spawn on fork" option, so that the two processes can be debugged in separate windows that do not interfere which each other?

I thought that was a great suggestion, and added (albeit only in the commercially-supported version) two new check-buttons to the Language tab in the Options dialog.

Caveat: Currently, the buttons are grayed out while a program is being debugged (i.e. are enabled only when no target is loaded in the debugger).

(This shortcoming has to do with setting ptrace options down a tree of attached threads, and I hope to address in a future release).

New releases of ZeroBUGS will be available later this week, when I launch the revamped website.