Tuesday, February 20, 2007

Debugger Breakpoints

An overview of breakpoints, as implemented in the ZeroBugs debugger for Linux.

Breakpoints are central to the ZeroBugs debugger engine layer. Breakpoints can be set by the user, or by the debugger for internal purposes (such as detecting the creation of new threads).

Physical vs. Logical

Breakpoints can be classified in several ways. One categorization distinguishes between "logical breakpoints" and "physical breakpoints". What this means is that not all the breakpoints that you have inserted in the program are physically there, but the debugger will support the illusion that they are; reality is the realm of physical breakpoints, and logic is derived off perception. So if you perceive a breakpoint as being inserted in the debugged process, it is logically there, even though, physically, the debuggee has not been affected.

Let's consider a couple of examples, to help bring the discussion out of the philosophical realm:

  1. The user inserts a breakpoint at the beginning of a function that is not loaded into memory yet, because it lives in a shared library that has not been mapped into the debugge's memory space (just yet). The debugger nicely shields the user from knowing such details, and may say: "OK. I don't know what the address in memory of function `abc' is; but I know that it is implemented inside the dynamic library libabc.so; I will keep this in mind, so that if I later detect that libabc.so is loded, I will insert the breakpoint. "

  2. Another case may be that the debugger has inserted a breakpoint at the beginning of the pthread_create() function, to internally keep track of newly created threads. The user wants to insert a breakpoint at the same address, and does not need to know that a physical breakpoint is already there. The debugger associates two logical actions with the same physical breakpoint: one that internally updates the list of debugged threads, and another one that initiates an interaction with the user.

The logical breakpoints are implemented as actions associated with physical breakpoints. Each physical breakpoint maintains a list of actions. An action may be temporary (or once-only), which means that it gets discarded after being executed once. Once-only actions are similar to UNIX System V signal handlers. Non-temporary actions are executed each time the physical breakpoint is hit -- similar to BSD signal handlers.

Algorithm for executing breakpoint actions

// Execute actions on given thread
void BreakPoint::execute_actions(Thread* thread)
// The list of actions associated with this
// breakpoint may change during
// the execution of actions, and thus the
// iterators may be invalidated:
// make a copy of the actions and cycle thru
// the copy, to be safe.
ActionList tmp(this->actions_);
ActionList::iterator i = tmp.begin();
for (size_t d = 0; i != tmp.end(); ++d) {
if (is_disabled(*i)) {
++i; continue;
// a temporary action returns false
if ((*i)->execute(thread, this)) {
else {
// remove it from the master list
ActionList::iterator j = actions_.begin();
advance(j, d); actions_.erase(j);
// remove it from tmp as well so that
// destruction is not delayed
i = tmp.erase(i);

Software vs. Hardware

The Intel 386/486/585/686 family of chips offers support for debugging, including breakpoints. The CPU has 6 debug registers: 4 for addresses, one for control, and one for status. Each of the first 4 can hold a memory address that causes a hardware fault when accessed.

In Intel's lingo, a "fault" is a hardware notification, or event, that happens when the CPU is about to access a memory address -- that is, before the access happens. An "exception" is a similar notification, only that it happens after the access has occurred.

Remember: Hardware breakpoints are faults, software breakpoints are exceptions.

The control register holds some flags that specify the type of access (read, read-write, execute) and some other bits; the status register is helpful for determining which breakpoint was hit, when handling a system fault.

Thanks to Operating System magic, the hardware breakpoints are multiplexed, so we can have as many as N times 4 hardware breakpoints per program, where N is the number of threads in the program.Hardware breakpoints have the advantage of being non-intrusive -- the debugged program is not modified. Another advantage is that they can be set to monitor data as well as code. A debugger may use a hardware breakpoint to detect that a memory location is being accessed.

Software breakpoints are implemented as a special opcode (INT 0x3) that is inserted in the code at location to be monitored.

Nicely enough, Intel has a dedicated opcode for breakpoints. Other CPUs (PowerPC, for example) do not have a special opcode; on those platforms software breakpoints are implemented by inserting an invalid code at the desired location.

Software breakpoints have the main drawbacks of being slow and intrusive. The debugged program has to be modified, and the debugger needs to memorize the original opcode at the modified location, so that the debuggee's code is restored when the breakpoint is removed. When a software breakpoint is hit, the instruction pointer needs to be decremented, and the original opcode restored. Then the debugged program has to be stepped out of the breakpoint. After the breakpoint is handled, the breakpoint opcode is reinserted.

On UNIX derivatives (such as Linux), a debugger does not manipulate the debugged program directly; rather, it uses the operating system as a middle man (via the ptrace or /proc API). This implies that every time the debugger reads or writes into the debuggee's memory space, a context switch from user mode to kernel mode happens.

Another disadvantage of soft breakpoints is that they can only monitor code. Software breakpoints cannot be used for watching data accesses.

What makes software breakpoints indispensable is that there's no limit to how many can be inserted. Hardware breakpoints are a very scarce resource (you can run out of the 4 of them quite fast); software breakpoints are intrusive and slower, but can be used abundantly.

The design decision in my debugger is to use software breakpoints for user-specified breakpoints, and prefer hardware breakpoints for internal purposes. Watchpoints (breakpoints that monitor data access) are implemented as hardware breakpoints.

An example of breakpoints maintained by the debugger internally is stepping over function calls. A breakpoint is inserted at the location where the function returns, and control is given to the debuggee to run at full speed until the breakpoint is hit. The breakpoint is removed once it is hit, and the hardware resource can then be reused.

As a rule of thumb, the debugger employs the hardware support for cases where breakpoints are expected to be released after relatively short amounts of time. If no hardware registers are avaialable, the debugger falls back to using a software breakpoint.

Global vs. Per-Thread

Another categorization of breakpoints is by the what threads they affect in a multi-threaded program. A global breakpoint causes the program to stop, regardless of what thread has hit it. Per-thread breakpoints will stop the program only when reached by a given thread. Because all threads share the same code segment, a software breakpoint is also a global breakpoint, since
any thread that reaches the break opcode will stop.

The operating system creates the illusion of each thread running on its own CPU, therefore a hardware breakpoint may be private to a given thread.

A bit in the debug control register of the 386 chip can be used to control the global/per-task behavior of hardware breakpoints.

A thread ID can be added to the data structure or class that represents a software breakpoint. When the breakpoint is hit, the thread ID in the structure may be compared against the ID of the current thread. The behavior of a per-thread breakpoint can be emulated this way.

The debugger uses emulated breakpoints when it needs a hardware breapoint and none of the 4 debug registers is available.

Consider the case where the debugger uses a breakpoint for quickly stepping over function calls. The debugged program must stop only if the breakpoint at the function's return address is hit by the same thread that was current when the user gave the "step over" command.

Sunday, February 11, 2007


I am a big fan of Mac computers and even use the word Mac as synonym for coolness (or lack thereof), as in "There is no mac in emacs".

The other day I made a typo in the attempt of e-mailing a link to this blog to someone: oh my! I used a slash instead of a dot: the-free-meme/blogspot.com

Firefox somehow read through the clutter in my head and took me to the right place (in spite of the error). By contrast Internet Explorer just sat there impotently, summoning up to mind this quote from the Hitchhiker's Guide To The Galaxy: [...] the incredstupid but equally dangerous and ravenous Bug Blatter Beast of Traal. This incredibly stupid monster thinks that if you can't see it, it can't see you!

I dream of a day when humans will be able to talk to computers in fuzzy terms, and the machines pick up the correct message. Think about it: why should we adjust to the demands of the machine, and appease it by addressing it in very precise, unequivocal ways? The human brain is able to understand the meaning of an article in spite of typos; more so, it can detect subtle changes of context, satirical undertones, and so on. Machines? Not so much.

As the song goes, you may think I'm a dreamer, but I am not the only one. I read that the genius who came up with Hungarian notation seems to be harboring similar thoughts. Why do I think this is funny?

Regardless of what camp you are in, both strong-typed languages and dynamic-typed languages strive towards a similar goal: the programmer should not be burdened with the intellectual overhead of remembering the types of the variables. In strong-typed languages the compiler enforces the correct types at... compile-time, and with most dynamic-typed languages the runtime takes care of the needed conversions under the hood.

By contrast, the HN is designed to help the poor programmer remember the types of the data; something you should not care much about in the first place. Instead of going after the root of the problem (i.e. some fuzziness in the C language wrt types), Microsoft went for the quick-and-dirty solution of making developers adjust to the idiosynchrasies of their machines, and scarred them for life: to this day there are some darned souls in Redmond who are still using HN.

And the hidden joke is that HN and Romanian Language don't mix well.

Monday, February 05, 2007

The Show Must Go On

I have never missed someone so much since my late father died in April of 2001.

I do miss Seattle Jazz Icon Floyd Standifer. His gig at The New Orleans kept me sane through many a grim Seattle winter. My wife Juliet and I have a nice Wednesday nights dating ritual since we first started going out together. I used to work for Amazon.com when we first met, and after long, crazy hours of laboring over the ordering system code, I used to call her at home: "Hey Babe, I am walking down Jackson street, see you at the New Orleans in a few?"

I have taken close friends there to share the Floyd Experience, and family, and people from out of town (Andra and Cristian who visited us from Spain last year had a blast). The staff at the New Orleans is friendly, and the band casual. Clarence Acox reminds me of James Earl Jones, only cooler: he can roll. And the Garfield kids that he coaches are amazing.

Jim Wilke, John Gilbreath, Bud Young and many many others gathered tonight at Seattle's First Baptist Church to honor Floyd.

As an immigrant to this Great Country, I cannot but stand by one of the participants' quote: "You cannot talk about America, without talking about Jazz".

And viceversa. I am lucky to be here.

Friday, February 02, 2007

Genuinely Numb

Does it matter if your copy of Windows is genuine? According to Microsoft, yes. If I understand correctly the gist of what they are saying, genuine Windows is less dangerous to its users.

It actually appears that the world is awash in counterfeit products. In Russia alone, 43,000 people die annually as a direct consequence of drinking counterfeit alcohol. Tragically enough, the fake beverages also cause losses estimated to $300 million!

And counterfeit cigarettes were reported to have "75% more tar, 28% more nicotine and about 63% more carbon monoxide" than genuine cigarettes in the UK market.

For Goodness sake, please make sure your smokes are genuine!

Oddly enough, I haven't heard of a genuine Apple computer yet.

And genuine Linux comes with ABSOLUTELY NO WARRANTY.