ZeroBUGS Lambda Fun

A friend of mine tried to compile the code from my previous post using GCC, only to get a bunch of error messages. As I suspected, the fix was to specify -std=c++0x on the command line. But before answering my friend's G+ message, I had to verify that the code worked with GCC. And one thing lead to another. After compiling, I was curious to see how my ZeroBUGS debugger copes with lambdas. What else to do on a rainy Sunday afternoon in Seattle, other than playing with some old C++ code while listening to Judas Priest?

ZeroBUGS is a visual debugger for Linux, a project that ate up all of my spare time between 2004 and 2008. I kept making small changes to it since, but very limited in scope. For some reason, since the Big Recession I found myself having lesser and lesser time for on-the-side projects. I tried for a while making ZeroBUGS a commercial enterprise, hoping to leave my day time job(s) and become self employed. I learned the hard way that selling proprietary software to Linux geeks is not very lucrative. Or maybe I should have partnered with a savvy sales guy, the kind that can sell refrigerators to penguins.

In late 09 I put ZeroBUGS on ice and went working on Microsoft Windows for a short bit (just long enough to write a few lines of code that will hopefully make it into the upcoming Windows 8.)

After leaving Microsoft and joining TableauSoftware, I yanked the closed source product off my web site, and re-released ZeroBUGS as open source (and free as in free beer under the Boost License.)

I have not come upon major bugs in the debugger since a few years ago, when I discovered that the "step-over" functionality was broken for recursive functions.

So I was pretty confident the debugger will handle the new C++0X just fine. Except it didn't!

After some debugging, I traced the problem to the unnamed classes that the compiler generates to capture the surrounding variables. My debugger cashes data types by name for performance reasons. Unnamed classes normally occur in some scope, and thus there is no clash. Except that in the case of lambda functions, GCC generates unnamed classes at the outer most scope (i.e. the DWARF entries describing their type is at level 1, immediately nested in the compilation unit scope.) The data structures visualization was completely off, because the debugger used the wrong datatype (the first "unnamed" always won).

A simple hack that appends the file index and the line number to the offending "unnamed" solves the problem for now, as the snapshot above can testify.

While I think of a better solution this one will have to do. I am done with the computer for now, off to enjoy the weather and barbecue in the rain for the rest of the night!

Stepping Over STL Code

When debugging C++ code written using the Standard Template Library (STL) it is not unusual to find yourself stepping through STL code. Most of the time, this is not very useful: The STL implementation typically comes bundled with the C++ compiler, and it has been thoroughly tested by the vendor; it is unlikely that the bug you are after is caused by the STL.

So when a statement such as myVector.push_back(x) is encountered while tracing with the debugger, you normally want to step over it, not into it. Most debuggers offer a "step over" and a "step into" function. So you would chose "step over".

But how about this? You want to debug a routine named my_func(size_t containerSize) and want to step into the body of my_func when this statement is hit: my_func(myVector.size()). If you select "step into", the debugger will first take you into the guts of STL's vector<T>::size() implementation before stepping into my_func.

The ZeroBUGS debugger allows you to avoid such annoyances. Once inside size(), you can right click, and select to "Always step over..." that function, all functions in that file, or all files in the same directory. The debugger will remember your option, and you don't have to see the guts of size(), or any other vector function, or any other STL function, respectively.

The functionality can be used not just with the STL but any code. If you later change your mind, the "Manage" menu allows you to remove functions, files or directories from the step-into blacklist.

Symbol Caches

OK, here I am back from a blogging sabbatical (summer in Seattle is so precious that it is sacrilegious not to spend it outside).

And the past few months have been quite eventful. I thought that everything out here in the blogosphere was going to be overshadowed by Real Life (kind of like Michael Phelps steeling the center stage away from Liam Tancock, a guy with a lot of bronze under his belt). Watching the news was a bit of a distraction I must admit...

But I spent the few rainy summer nights working on a ZeroBUGS new exciting feature: caching symbol tables and other debug information from one session to another.

This allows the debugger to start up faster, and use lesser memory (since the cached info is already indexed on disk, there is no need to build maps and hashes in memory).

I have deprecated an older implementation for caching symbol tables, that was based on memory-mapped files. The new code is built upon SQLite, and ensures data integrity with triggers that enforce foreign key constraints.

For the sake of stability, the commercial release does not feature the SQLite-based caches yet. The plan is to slowly burn in the new code through the beta builds, and have it officially released later this fall.

RTFM

A number of people wrote to complain about ZeroBUGS not working properly, for example not stopping at main().

Guys, have you heard the phrase "the user is always wrong"? RTFM!

This is a known issue: http://www.zero-bugs.com/2.0/known_issues.html

And see:
http://www.granneman.com/blog/2006/06/13/how-virtual-machines-work/
and http://www.softpanorama.org/VM/index.shtml

as well as the wikipedia entry on VMware. You will find that "One must always rewrite; performing a simulation of the current program counter in the original location when necessary and (notably) remapping hardware code breakpoints".

Remote Attachment

In addition to being able to execute programs on a remote system from within the ZeroBUGS debugger, it is also possible to attach to remote processes.

When the remote proxy plug-in is detected, the UI automatically adds a "Target Parameters" field at the bottom of the "Attach to Process" dialog box.

After you type in the protocol ("remote://") followed by either the name or the IP address of the target system, and press the "Refresh" button the box will be populated with a list of processes running on the remote machine. Just select the desired process and click "Ok".

May I Have the Remote, Please?

This past weekend I put the final touch on supporting remote debugging in my favorite C/C++ debugger for Linux, ZeroBUGS.

It is under test and will ship in the commercial version within the next five to ten days.

Currently, the remote debugger is not a cross-debugger, that is to say it only works between systems of the same architecture (adding support for cross-debugging is not terribly hard, I just do not want to spend rare Seattle summer days working on a feature, unless users ask for it).

"Well, then what's the point in supporting remote debugging", some may ask. "Can't we just ssh -X into the remote target and run the debugger there?"

Installing a full-fledged debugger, with a UI-module that depends heavily on Gnome may not be a feasible option for small (possibly embedded) systems where resources are scarce.

My solution for remote debugging is to install a thin, lightweight server on the target system, and have the debugger on a Linux workstation do the heavy lifting of building symbol tables, managing breakpoints, and so on.

In order to read debug information, the debugger needs access to the executable and shared objects on the target system. One possible solution is to copy them over to the workstation where the debugger runs, but it is not very practical. The amount of files to be copied can quickly go out of control, by ways of shared library dependencies.

I found that it is simpler to just mount the remote target onto the machine where the debugger runs. SSHFS is ideal for this job. Because the debug info may contain references to absolute path, ZeroBUGS provides the ZERO_REMOTE_PATH environmental variable, which creates an internal mapping between the mount point and the original paths.

Here are the steps for a remote debugging sessions, with examples from my own lab. The debugger runs on a 64 bit system (zulu) running Ubuntu 7.10, and the target executable(s) reside on another 64 bit machine (arnold) running Ubuntu 8.04.

1) Mount the filesystem of the target computer onto the debugger system. Example:

cristiv@zulu:~/workspace/sandbox$ sudo chown cristiv /dev/fuse
[sudo] password for cristiv:
cristiv@zulu:~/workspace/sandbox$ sshfs root@10.0.1.10:/ ~/workspace/remote/
root@10.0.1.10's password:
cristiv@zulu:~/workspace/sandbox$

2) Add mount point to remote map. Example:

export ZERO_REMOTE_MAP="10.0.1.10:/home/cristiv/workspace/remote;"

Note that each entry must be ended with a semicolon (even when there is only one entry in the map).

3) Start the ZeroBUGS server on the remote (debug target) system:

cristiv@arnold:~/workspace/zero$ zserver
*** ZeroBUGS Remote Debug Server V. 1.0 ***
*** Copyright (c) 2008 Zero Systems LLC ***
cristiv@arnold:~/workspace/zero$

To debug remotely programs running on host 10.0.1.10:
4) Run a remote program and debug it, using the command line:

zero remote://10.0.1.10/home/cristiv/workspace/zero/a.out

The UI can also be used to execute remote targets:

As a final note, please remember that the debug server opens a security whole on the target systems, since the client debugger can execute any program that the user who started the server can.

Take the Fork in the Road

A friend of mine (let's call him Andrei) who works on the D Programming Language sent me a bug report the other day. He was having trouble debugging this piece of D code with ZeroBUGS:

import std.stdio, std.process, std.string;
void main(string[] args)
{
    writeln("Started on ", chomp(shell("hostname --short")));
}

After a brief investigation, I figured out what was going on: the shell call (equivalent to system in C) was triggering a fork(), followd by an exec() call. The debugger automatically attached to the spawned shell. After which point, every time my friend was trying to step through the code, control would jump from one process to the other in an apparent indeterministic fashion (I say apparent because the behavior is actually determined by how the kernel schedules the main, forked, and debugger processes).

I replied to the bug report with an explanation of what was going on, suggesting that the --no-trace-fork switch be passed in the command line (and thus avoid attaching to the forked shell).

My friend argued that this option should be more obvious (i.e. accessible from the graphical user interface) and while you are at it, he said, why not add a "spawn on fork" option, so that the two processes can be debugged in separate windows that do not interfere which each other?

I thought that was a great suggestion, and added (albeit only in the commercially-supported version) two new check-buttons to the Language tab in the Options dialog.

Caveat: Currently, the buttons are grayed out while a program is being debugged (i.e. are enabled only when no target is loaded in the debugger).

(This shortcoming has to do with setting ptrace options down a tree of attached threads, and I hope to address in a future release).

New releases of ZeroBUGS will be available later this week, when I launch the revamped website.

Any Updates?

The zero-bugs.com site is about to be revamped, and the side picture should give you a sense of the new look and feel.

I had tremendous fun indulging my artistic side and created the new design using Inkscape 0.45 and Gimp (except for the product box).

The new site's main goal is to better manage software releases. A new back end component allows users to check for software updates (the debugger UI now has a new menu entry: Help --> Check for Updates).

I have improved my uploading script (which I have been using for pushing the software to the site); meta information is updated on the server side each time a binary is uploaded. The meta information is used by the "check for updates" feature; it is also used by a CGI to dynamically generate the download pages, so that users always download the most up-to-date code.

The debugger program connects to the server every time a user clicks on the "check for update" menu, and consults the meta information. Because the format of the server files containing software meta information is dictated by my uploading script, I did not want to hard-code the way these are read. Instead, I created a new client-side Python program.

I wanted the update mechanism to be as generic as possible, and allow for plug-ins to be updated independently, should such a need arise (for example when plug-ins from third-parties are included in the distribution).

To this end, I added a couple of new interfaces which look something like this:

struct Update : public ZObject
{
 /**
  * URL of the package (DEB, RPM, etc.) that contains an update.
  */
 virtual const char* url() const = 0;

 /**
  * Description of changes in this update.
  */
 virtual const char* description() const = 0;

 virtual void apply() = 0;

 virtual Update* copy() const = 0;
};


struct Updateable : public ZObject
{
 virtual size_t check_for_updates(Enumerator<update*>*) = 0;
};

(The ZObject and the Enumerator are building blocks from my ZDK -- Zero Developer Toolkit -- on which the ZeroBUGS is built. They are not essential for the purpose of this post).

The way the update works is very simple: the engine queries each plugin for the Updateable interface. If the interface is detected, the check_for_updates method is invoked; the Enumerator then populates a container of pointers to Update objects, and the URL of each update is displayed in a UI HTML control.

With this simple, generic mechanism, each pluggable component can implement the details for its own update.

My concrete implementation lives in the Python scripting module which acts as a gateway between the bulk of the debugger code, written in C++, and extension written in Python.

The client-side script is ridiculously simple:

from datetime import datetime
import httplib
import urllib
import zero

server="www.zero-bugs.com"
published_dir="/8001/published/"

def check_for_updates(sysid, date):
    print "checking updates for:", sysid, date
    conn = httplib.HTTPConnection(server)
    url = urllib.quote(published_dir + sysid)
    conn.request("GET", url)
    r = conn.getresponse()
    info = r.read().split("\n")
    conn.close()
    my_build_date = datetime.strptime(date, "%Y-%m-%d")
    build_date = datetime.strptime(info[0], "%Y-%m-%d")

    if my_build_date < build_date:
        url = "http://" + server + info[1]
        info.pop(0) #pop the date
        info.pop(0) #pop the url
        info[0] = info[0] + "<br/>"
        desc = "\n".join(info)
        #
        # functions returns a list of available updates
        #
        return [ zero.Update(url, desc) ]

I plan to launch the new site after Memorial Day. And after that I plan to release a new feature: support for remote debugging. I have written a minimalistic server and a companion plug-in that will allow users to debug machines where resources scarcity does not permit a full-fledged ZeroBUGS installation (you did not think that I took a three week hiatus from my blog just to re-write a website, heh).

Debugging D Unit Tests

I had one slide in my presentation at the D Programming Language Conference last year that demonstrated how to automatically insert breakpoints at all unit test functions in a D program (using, of course, the ZeroBUGS debugger for Linux).

The trick mainly consists in invoking a small Python script at startup:

zero --py-run=dunit.py

where the contents of dunit.py may look something like this:

# dunit.py
import os

import re
import zero

def on_table_done(symTable):
 extension = re.compile(".d$")
 process = symTable.process()
 module = symTable.module()

 for unit in module.translation_units():

     if unit.language() == zero.TranslationUnit.Language.D:
         name = os.path.basename(unit.filename())

         i = 0
         while True:
             symName = "void " + extension.sub('.__unittest%d()' % i, name)

             print symName
             matches = symTable.lookup(symName)
             if len(matches) == 0:
                 break
             for sym in matches:
                 process.set_breakpoint(sym.addr())
             i += 1

(see http://zero-bugs.com/python.html for more information on the scripting support in the debugger).

This contortion is painful but necessary with D, because unit tests have the interesting property of running before anything else in the program.

Without this bit of gymnastics, by the time the debugger (trusted hunting companion) is wagging its tail at the main function, the unit test functions have long completed (or failed).

The script line shown above in bold (which constructs the symbol name of the unit tests) is now broken because of a change in the D 2.0 language. The name is prefixed by the module, as explained here: http://www.digitalmars.com/d/2.0/hijack.html

What happens is that the scripts is looking for functions named
void Blah.__unittest0() , void Blah.__unittest1() , etc whereas they are now called void mymodule.Blah.__unittest0(), void mymodule.Blah.__unittest1(), and so on.

Before, I could synthesize the symbol names to look for, but now this is hardly possible since there's no place to infer mymodule from, unless the compiler produces the (standard) DW_TAG_module DWARF debug info entry.

Unfortunately the DMD compiler does not generate a DW_TAG_module, and so it is almost impossible for the debugger to determine the module name. One possible hack would be to sniff other symbols in the module and look at their prefixes.

Or we could change the script like so:

import os
import re
import zero

def on_table_done(symTable):
    extension = re.compile('.d$')

    process = symTable.process()
    module = symTable.module()

    for unit in module.translation_units():
        if unit.language() == zero.TranslationUnit.Language.D:
            matches = symTable.lookup("_moduleUnitTests")
            if len(matches) == 0:
                break
            for sym in matches:
                process.set_breakpoint(sym.addr()

This stops the program in the debugger a couple of function calls before the unit test functions are executed. However it is not ideal since now we have to step over a bunch of assembly code.

Ideally, the D compiler should produce DW_TAG_module info.

Or the debugger could look up symbol names by regular expressions, but I have to punt on the performance implications of such an approach.on the performance implications of such an approach.on the performance implications of such an approach.

yywrap it up

This weekend I decided to download the beta version of the highly anticipated Ubuntu 8.04.

After installing it on a virtual machine I promptly proceeded to building ZeroBUGS on it.

And I ran into a small roadblock: The debugger has a built-in C/C++ interpreter, for evaluating simple expressions and function calls. This code failed to link, because in this new Hardy Heron version of Ubuntu the yyFlexLexer class (see /usr/include/FlexLexer.h) has a virtual method called yywrap(). In older versions of flex yywrap used to be a standalone, extern "C" function.

To work around this issue in a portable manner I defined a new function in my configure.ac file.

It attempts to compile a small C++ program that assigns a pointer to method. If the compilation fails, I assume that yyFlexLexer::yywrap is not defined.

Here's the code:

AC_DEFUN([AC_YYFLEXLEXER_YYWRAP],
  [AC_CACHE_CHECK([for yyFlexLexer::yywrap],
             [ac_cv_yyFlexLexer_yywrap],
             [ac_cv_yyFlexLexer_yywrap=false
cat > tmp.cpp << ---end---

#include <FlexLexer.h>
int main()
{
  int (yyFlexLexer::*method)() = &yyFlexLexer::yywrap;
  return 0;
}
---end---
  if g++ tmp.cpp 2>/dev/null; then
      ac_cv_yyFlexLexer_yywrap=true
      rm a.out
  fi
  rm tmp.cpp])

  if test "$ac_cv_yyFlexLexer_yywrap" = true; then
      AC_DEFINE([HAVE_YYFLEXLEXER_YYWRAP], [1],
              [Define if yyFlexLexer::yywrap is declared])
  fi
])


AC_YYFLEXLEXER_YYWRAP

And you can test drive a pre-release of ZeroBUGS for Ubuntu 8.04 here: http://zero-bugs.com/8001/releases/zero_1.12-heron-032308_i386.deb

The md5sum is: 5b80e9f0f867b858620c740096e432dd

Mostly For Fun

The past weekend I worked on setting up the e-commerce bits for my ZeroBUGS Linux debugger.

While the commercial build is compiled with optimizations turned on, and has some extra UI goodies, the beta builds are still available for free.

I was doing some marketing research and I came across an interview bit where the main, revolving question was "What's so special about your product that big system vendors do not offer?"

The person being interviewed went on and on trying to explain the great virtues of his company's software, but missed the simple answer: there are classes of products where a small ISV has a much better chance to come up with good software than a large corporation.

Take my example of a debugger for the C++ language. Unlike a browser or an OS, it is designed for a relatively small yet highly specialized niche. It is not a mass consumer product; it will never be a household name. Its users are few, but extremely picky (since they are very sophisticated software developers themselves). This spells: low margin. Not what a large corporation would bet its resources on.

A big company may however assign a team to slapping something together quickly, most likely to bundle it with another product (such as a compiler) so that they can check a box in a feature list; or maybe just give it away "for free" so that they can sell you their pricey consulting services.

It is rather the kind of product a geek would do primarily just for fun, and only secondarily to make money (so that he can eat, buy more hardware, and do more of the same).

And that's why in the long run the geek's code is better.

Watch Expressions

This past weekend I added a couple of new features to the ZeroBUGS debugger.

The user can now monitor arbitrary expressions in the "Watch Variables" window (not just variables, as it was the case up until now). For example you can watch for things such as argv[1], or x * y. (Function calls are supported but I would recommend extreme care in using them; calling arbitrary code in the debugged target may yield some rather interesting effects).

Another new feature is the ability to "map" source and library paths. For example, lets assume that you built your code in /home/joe/cool_project but later moved the source code to /home/joe/cool_source. The debug information inside the binary will still point to /home/joe/cool_project. ZeroBUGS allows you to work around this by setting an environment variable like this:

export ZERO_PATH_MAP="/home/joe/cool_project:/home/joe/cool_source;"

When the debugger encounters source file information prefixed by /home/joe/cool_project it will replace it with /home/joe/cool_source.

You may specify an unlimited number of such pairs, separated by a semicolon. I have uploaded new builds for Ubuntu (32 and 64 bit) and Fedora Core 8 (32bit) this morning, and will update the others over the next week or two.

And the total number of code lines has dropped from 142,992 as reported at the end of December, down to:

Total Physical Source Lines of Code (SLOC)                = 139,824
Development Effort Estimate, Person-Years (Person-Months) = 35.80 (429.61)
(Basic COCOMO model, Person-Months = 2.4 * (KSLOC**1.05))
Schedule Estimate, Years (Months)                         = 2.09 (25.03)
(Basic COCOMO model, Months = 2.5 * (person-months**0.38))
Estimated Average Number of Developers (Effort/Schedule)  = 17.16
Total Estimated Cost to Develop                           = $ 4,836,180
(average salary = $56,286/year, overhead = 2.40).
SLOCCount, Copyright (C) 2001-2004 David A. Wheeler
SLOCCount is Open Source Software/Free Software, licensed under the GNU GPL.
SLOCCount comes with ABSOLUTELY NO WARRANTY, and you are welcome to
redistribute it under certain conditions as specified by the GNU GPL license;
see the documentation for details.
Please credit this data as "generated using David A. Wheeler's 'SLOCCount'."

The decrease is mainly due to code refactoring and to removal of unsuccessful experimental features.

Zero G

Wow. It has been quite a while since my last post. In my defense I can claim that I have been hugely busy, both at my day job and with Zero.

I fixed a few crashes (thanks to all of you who took the time to submit bug reports) and added a new layout for organizing the debugger window. The user could choose between a "classic" and a "tabbed" view, now there is a third option that allows local variables and stack traces to be viewed at the same time.

Another cool, new feature consists of displaying a description of the current errno value in a tool-tip (as shown in the screen shot).


When I do not work on software I spend time with my family. And when I manage to escape their love I read science news magazines (or at least that's what they label themselves) just to see how deep the conspiracy runs. What conspiracy? You are kidding, right?

In the last issue of Seed, Will Self and Spencer Wells agree that

"we'll probably fast forward by changing our own DNA"

The human race will leap forward using technology to pick up where evolution left... Wait. Isn't this what Ray Kurzweil was saying all along? That we will transcend our biology?

The ultimate goal of evolution be as it may for God's Debris to put itself back together, I believe that the purpose of spreading "news" like these is to get more people to watch The Sarah Connor Chronicles.

Okay, maybe I am kidding. Or maybe someone sits up there in the control room as we very speak, and fabricates more news on, say Global Warming (be afraid, vewy vewy afraid), or on Emmanuelle finding the G spot! No seriously, what kind of news is this? (Even the New Scientist covered it at length). Sorry to break it to you guys, Emmanuelle found the G spot in the seventies, this is old news.

Oh, oops. Just in: Doctor Emmanuelle is a man.

The New ZeroBUGS Python Console

This is the latest feature that I have been working on: the Python Console, which allows users to interactively script the debugger.

The scripting part is not new, but I always felt that it was largely underutilized. It seems that no matter how cool, a feature that is not obvious to the user is simply wasted. Hopefully this new UI element will remedy the situation.

Yet Another New Year's Resolution

Because this is a time of the year when people make resolutions about... ahem... loosing weight, and because I recently read in a blog how various programming languages are inherently bloated, I decided I should write down my opinions on fat code.

In this summer post back in June, I was reporting that the ZeroBUGS project had 135,601 lines of code, out of which 120815 (89.10%) were C++. Tonight, after just finishing a big round of changes, the project totals 142,992 lines of code, 126779 (88.66%) being C++.

So one could argue that my project got inflated by roughly 6k lines of (C++) code. And what do I have to show for six months of fattening up?

Bug fixes notwithstanding: I have ported the code to the PowerPC platform, added support for visualizing wide strings and Qt strings, added a feature that allows debug events to be ignored by thread, and (hot from the oven and about to be released) added support for D programming language's associative arrays. In only six thousand lines of code. Not too bad, but that's just my opinion.

What are in general the factors that cause source code to bloat up?

System Refactoring
Back when I was working for Amazon.com, we had to re-write the ordering system (one of the many Amazon software components, the ordering system was in charge of all the magic that happens from the time you click Proceed To Checkout until your items are shipped). At the time when I joined the team that got tasked with the re-write, we had a subsystem consisting mainly of a few tens (or maybe hundreds?) C functions.

This thing was not very flexible, and was already giving up at the seams whenever new functionality was being requested by the business people. It also had statical dependencies to almost anything else in the system. The decision was to replace it with a middle-tier service, with clean APIs.

My boss at that time (an ex Bell Labs guy) had a good plan:

• design a set of object-oriented, abstract interfaces;
• implement them in terms of the legacy C code;
• then rewrite all client code to use this new C++ API;
• and finally, once there is no more coupling to the C implementation, change the implementation, one small piece at a time.
  

And this is what we did, and I think it was a successful project, with a few wrinkles though:

• the migration effort took a couple of years to complete; meanwhile, the old system co-existed and was being actively changed and maintained;
• do not forget the people factor: some of the middle-managers (I hear these days they got promoted, accordingly to the Peter Principle) had personal political agendas that caused the project to take longer than necessary

The overall effect was that in fact we had two parallel systems in existence: a legacy one, and the "new" one.

The problem is that by the time you are ready to throw away the legacy system, the new system is already old enough to be called "legacy" itself.

Work on another system designed to replace the "new legacy" may start before the "old legacy" is completely retired. So the company may end up with three or more systems being maintained in parallel (sure, the plan is to eventually phase out the legacy, but that may not happen as soon as we wish). And here is one place where bloat, and its first cousin needless redundancy thrive: when code bases are being unnaturally kept alive. Two systems in parallel, and old one and a replacement, are fine. Three or more systems that are trying to solve the some problem is not a Good Thing. And just in case you did not catch it: needless redundancy is in itself a needlessly redundant association of words.

Supporting multiple platforms
Another reason for code to grow in size is portability. In order to make your code portable, you need abstractions and indirections. I started ZeroBUGS in late 2003, because I wanted to best GDB.

My first debugger prototype was less than three thousand lines of code. I was inadequately enthused: the code was stable as a rock, but that's pretty much all I can say about it. It did not work with multiple threads; it did not load core dumps. The support for STAB was sketchy, support for DWARF was none. There was no expression interpreter. The code was stable as a rock, and that's pretty much all I can say about it. And GUI? What GUI?

It took another four years or so to add all these features. Maybe a third to a half a of this time was spent making said features portable. And I do not even mean across OS-es or CPU architectures. You see, when I started to write the GUI I went with Gtk-1.2 and the corresponding C++ wrapper, Gtk--. By the time I was done the world had already moved to Gtk-2.x, and the Gtkmm C++ wrapper was a standard package in most distributions. I had to write an adaptation layer, not unlike the ordering system adaptation API back at Amazon. And that bloated my source code. But today I can compile against Gtkmm or Gtk-1.2 (and 95% of the details are transparent to my client GUI code).

But Gtk-1.2 may no longer be relevant, some people may say. And I think they are right. But let's look at Professor Tanenbaum's MINIX for a second, shall we? Not only because it is a lean and robust system (easy when you do not have features, multi-threading anyone?) but because if you read the source code you notice a strange thing: there are a lot of macros dedicated to ensuring compatibility with Kernighan & Ritchie C compilers. In the 2006 3rd edition of the book. What the heck?

I guess that the lesson here is that writing for portability is fine, but keep an eye on things that may become obsolete sooner than you think. The code that deals with one particular OS, compiler, etc, will then turn into dead weight.

So my 2008 resolution is to get ZeroBUGS on the treadmill.
Happy New Year!

ZeroBUGS on PowerPC

Happy Thanksgiving! (Yes, we do celebrate Thanksgiving in Seattle in spite of what the Liberal Turkeys want you to believe).

And what a great long weekend this was. We took advantage of the great weather and hiked in the Lincoln and Discovery Parks and walked around Alki Beach, with our son in the backpack.

I have been looking for a solid block of free time from my daytime job in order to look into porting ZeroBUGS to the PowerPC.

I have an old PowerMac that I bought used from a company in Oregon this past summer, for this specific purpose (and promptly installed Ubuntu on it).

But I did not have the time to look into it seriously until now. The main motivation was to see how far I can push the envelope, with the existing design holding up. In other words, I wanted to see how portable my overall design of ZeroBUGS was. And porting prompts one to revisit, re-test and refactor old code.

Of course, I did not expect the outcome of three or four nights of hacking to be perfect.

But the result, while very rough, is quite usable (at least usable enough for me to bootstrap the debugging of the debugger with itself) and I am quite happy with it.

So what's so hard about porting the debugger to the Power architecture? you may ask. Shouldn't it be quite a straightforward task, given it is written in portable C++?

A debugger interacts with the target program (and the OS) at a low level, and hence parts of it depend on the low-level architecture details. Take stack unwinding for example. The layout of stack frames (or links, in PPC parlance) is different than on the Intel chips. And so is the mapping of DWARF register numbers onto the CPU general purpose registers.

Linux runs on PowerPC in big endian mode (as opposed to Intel chips which are little-endian systems) and thus the binary representation of C and C++ bit fields differs.

The hardware support for debugging (special debug CPU registers) also varies across processors.

Well-thought abstractions should isolate these idiosyncrasies. I found out that I have done a good job designing the stack unwinding mechanism (a "driver", template method that delegates to platform-specific bits) but totally blew the model for hardware debug registers.

Oh well. Back to the drawing board. Luckily it is not a heavily-used feature...

The Pros and Cons of Hitch Hacking... Ubuntu 7.10

After fubar-ing one of my systems last week with Fedora Core 8, I switched to Ubuntu 7.10 Ballsy Baboon as my main development system and I just love it.

I cannot care less about the over-hyped desktop effects, I am interested in software development, and in this context the eye candy is irrelevant.

Ubuntu 7.10 allows you to install all major GCC/G++ versions from 2.95 to 4.2 which is so cool! I can test with older GCC versions without having to boot up an older distro! Excellent.

So that is the Good. Now for The Bad and The Ugly: the PowerPC is not officially supported.

This is hardly any news, 7.4 is not supported either but it worked great on my second-hand G4. Upgrading from 7.4 to 7.10 on the other hand is a terrible mistake. I repeat: if running on PowerPC, do not upgrade from Ubuntu 7.4 to 7.10!

If you do, your machine may not reboot properly. It is very likely that you will find yourself stuck at BusyBox, and then you will have to type:

modprobe ide_core
exit

to bring the system up.

Then you may also see an error box popping up, about HAL not starting. You will have to start it manually:

sudo /etc/init.d/hal start

The network interface may not start either, and there is no sound. But then again, I am only using the G4 for porting ZeroBUGS to PowerPC (which I devote very very little time to, so not a biggie).

As a bonus point: on the AMD64 (my main system) nspluginwrapper and flash are installed auto-magically.

So overall this Manly Monkey distro is great! Oh wait. I got the name wrong again... I think they call it Goofy Gorilla. Or Ghastly Gnome.

Handling Invalid UTF-8 in ZeroBUGS

Based on suggestions from users I have changed the way ZeroBUGS deals with invalid UTF8 code when loading source files. The GtkSourceView widget uses ustrings internally, and when passed single-byte strings will assume UTF-8 encoding and convert them.

In order to prevent accidents such as loading a binary file instead of a source file by the users, I had my UI reject outright any invalid UTF-8 text.

But it turns that sometimes this restriction is too strong, and I have added a dialog that asks before rejecting the file.

Wide Strings Debugger Support

This past weekend I worked on adding support for visualizing wide standard strings and QT strings in the ZeroBUGS debugger. For convenience, they can be displayed as if they where regular C strings.

This feature can be turned off and on from the Edit / Options menu. Under the Language tab, if the checkbox for Qt strings is unchecked, the inner structure of the object is shown. Otherwise Qt strings are displayed as regular strings. std::string and std::wstring instances are handled in a similar fashion.

Before implementing this feature, the values of the variables displayed by the debugger where being represented as C strings.

The solution for wide strings keeps the existing design: I read the Unicode strings from the target process' memory, then call wcsrtombs to UTF-8 encode them.

And Glib::ustring takes it from there:

GTK+ and GNOME chose to implement Unicode using UTF-8, and that's what is wrapped by Glib::ustring. It provides almost exactly the same interface as std::string, along with automatic conversions to and from std::string.

It is still a new feature and it needs more testing and possibly, ahem, some debugging but overall I am pleased with the results.

And if you have any feedback regarding what features should I work on next, please take the poll in the right hand side bar, or leave a comment. Or both. Merci.

AMD's CodeAnalyst

Now that I have stabilized the features of ZeroBUGS, I am looking into improving the overall performance and implicitly the user experience.

This weekend I live searched for profiling tools and I found AMD's CodeAnalyst.

On Linux, Code Analyst uses oprofile.

I downloaded the source code for version 2.6.22 of CodeAnalyst and proceeded to build it on a Fedora 5, 64-bit system equipped with an Athlon processor. It is not clear to me whether the 2.6.22 version is a coincidence. Has the kernel version to be an exact match for best results?

I am running 2.6.20 on that box, and the program ran but collected no samples, as per this message

Oprofile engine has encounter error genrating system view

I grepped for the message and found it here:

void opdata_handler091::read_op_module_data()
{
    string session =  m_session;
    vector <string> non_options;

    reset_data();

    for (size_t i = 0; i < m_op_events.size (); i++) {
        non_options.clear ();
        non_options.insert (non_options.begin (), m_op_events[i]);
        non_options.insert (non_options.begin (), session.data ());

        try {
            generate_apps_summary(non_options, i);
        } catch (op_fatal_error const & e) {
            string msg =
                "Oprofile engine has encounter error genrating system view:\n\n";
            msg += e.what();
            m_ca_display->update_display(msg.c_str());
            reset_data();
            break;
        }
    }
}

(File /src/ca/libs/libopdata/op091/opdata_handler091.cpp). The reason I am showing the code snippet here (broken English not withstanding) is that I noticed a bad coding idiom: the catch block is doing quite a lot of stuff, and I see several things that could possibly throw.

Nit-picking aside, I rebooted the machine in a different configuration (Ubuntu 6.06, 32-bit, running kernel 2.6.15) and this time I got luckier. I even found a few interesting bottlenecks in ZeroBUGS.

I concluded that the oprofiled / 2.6.20 64-bit kernel combination must be the culprit of my initial lack of success.

It definitely looks like CodeAnalyst is a useful tool, once one gets past the initial small road blocks.