In an earlier post I compared the performance of ElectricAcclerator and distcc by building samba using each tool in turn on the same cluster. In that test I found that Accelerator bested distcc at suitably high levels of parallelism, but that distcc narrowly beat Accelerator at lower levels of parallelism. At the time I chalked the difference up to slightly higher overhead associated with Accelerator. But you must have known I couldn’t just leave it at that. I had to know where the overhead was coming from, and eliminate it, if possible. The exciting conclusion is after the break.

Samba Reloaded

To recap, I previously found that samba is a very CPU-intensive build, despite being written entirely in C. This fact was demonstrated empirically by examining the performance on one dual-core host using just gmake -j at varying levels of parallelism. Past -j 2, the performance degraded sharply. For the distcc and emake tests, I used a cluster of 12 dual-core hosts. Eleven served as workers, for a total of 22 CPU’s. The remaining host was used as the build host (and cluster manager, for emake tests). Here are the original results:

Until we got to about 11 CPU’s, distcc appeared to have a slight edge on emake. I had lots of theories that could explain the difference: maybe our Electric File System (EFS) was slower than ext3fs, or maybe the Electric Agent was sluggish in supplying metadata to the EFS, or in processing file usage data from it. Maybe lock contention in emake itself was causing the problem.

Of course before I could test any of these theories I had to make sure I could reproduce the original behavoir. I set up a five-node cluster using the same dual-core hosts I used previously, plus one additional node to serve as the build host (unfortunately the other half of the cluster was reserved for other tests — I’m not the only person doing work here at Electric Cloud, after all!). This gave me a total of 10 worker CPU’s. After installing the latest version of Accelerator (4.5.0), I fired off a series of three builds each with Accelerator and distcc, using 10 workers. When those builds completed, I computed the average build time for each tool — and found that Accelerator beat distcc by a small margin.

Deeper Into the Rabbit Hole

This result was wholly unexpected, given the results from the previous tests. The next step was to run a series of builds with varying numbers of workers, from 1 to 10. Here are the results:

Now the results are more in line with my expecation: with low levels of parallelism distcc appears to perform better, but Accelerator catches up and finally surpasses distcc once enough resources are engaged. The breakeven point has moved though, from about 11 CPU’s to about 9 CPU’s. In addition, there was an outlyer in both sets of results: with just one worker, Accelerator was consistently faster than distcc. That didn’t fit well with my theories — if the EFS was slow, for example, then Accelerator would have been slower than distcc with one worker.

A New Theory

As I puzzled over this new data, something clicked that caused me to remember a subtle difference between distcc and Accelerator: they use different strategies to determine how to allocate jobs to workers in the cluster. Accelerator prefers to fully load one host before running jobs on another host; distcc prefers to spread the load across as many hosts as possible before doubling up on any one host. The following images illustrate the result obtained when using these different strategies to assign five parallel jobs to a cluster of ten workers on five hosts:

This realization led to a new theory: perhaps the performance difference observed with low numbers of workers was simply an artifact of this difference in worker allocation strategies. We’ve already seen that this particular build is especially CPU intensive. Two jobs on one host have just two CPU’s that they must share; two jobs on two hosts have four total CPU’s available. This theory would also explain why emake “catches up” with distcc — as more and more jobs are run in parallel, distcc is forced to assign multiple jobs to a single host. Eventually, both systems have fully loaded all the available cluster nodes, so the difference in allocation strategies becomes moot.

Armed with this theory, I altered my benchmark so that emake would use the same allocation strategy as distcc, by explicitly enabling and disabling agents via the cluster manager. For example, for a trial with two agents, I enabled one agent each on two cluster nodes. This technique allowed me to better compare the relative performance of distcc and Accelerator. Here are the results from this test:

With the allocation strategy out of the equation, Accelerator actually has a small, consistent edge over distcc (about 2%) on small clusters. And the previous test showed that Accelerator scales better than distcc, so on large clusters the difference is even more pronounced (about 15%).

Should We Change Accelerator?

An obvious question is whether we would consider changing the allocation strategy in Accelerator. The answer is probably no. The strategy we use, although suboptimal for this particular build, actually works very well across a wide variety of builds. One of the key advantages of this strategy is that it allows Accelerator to minimize network overhead, since agents on a single host can share various kinds of data directly. There are relatively few builds that skew so heavily towards CPU utilization, so changing the strategy to benefit those special cases at the expense of the more common case seems unwise.

A Champion Vindicated

Although we previously declared Accelerator the victor versus distcc when building samba, it was not without some reservations. With the new results shown here, I’m satisfied that we made the correct decision: CPU-for-CPU, Accelerator is more efficient and scales better than distcc, at all cluster sizes.

For this investigation, there are two things that are really interesting about the samba project. First, although it is written entirely in C, it is a highly CPU intense build. This runs counter to my intuition and experience, which is that C compiles tend to be relatively light in terms of CPU usage (at least in comparison with C++ compiles, for example). You’ll see the evidence and impact of this in the results below.

Second, the samba build is structured as a single, non-recursive make invocation. This is markedly different from the other projects I’ve previously built in this series, both of which were built using the traditional recursive make style. The principal advantage of a non-recursive make build is that it enables you to completely, precisely specify all the dependencies across the entire system. In theory, this means that the build should be able to run at very high levels of parallelism, much higher than you might see with a typical recursive make build, due to the manner in which parallelism is implemented in gmake. It also means that in this comparison Accelerator is denied the benefit of one of its key optimizations, which is the ability to coalesce recursive makes into a single logical make. Or, to put it another way, it enables the gmake build to compete with Accelerator on a more even footing.

These tests were performed on a cluster of 12 physical servers, each with Dual 2.4GHz Xeon CPU’s with hyperthreading enabled. One system was designated the build host (and cluster manager, for Accelerator tests); the remaining hosts were used as worker nodes. The build host had 2 GB RAM, and the workers had 1.5 GB each. All systems were connected by a dedicated gigabit Ethernet switch, and they were all running RedHat Desktop 3, update 8. I used GNU make 3.79.1; distcc 3.1; and ElectricAccelerator 4.3.1.25685. I built samba 3.3.0, the latest release available at the time I did the tests.

Samba built cleanly out-of-the-box with both Accelerator and gmake with distcc in “pump” mode. As before I used a trivial driver script to simplify the process of running the builds at various levels of parallelism from 1 to 22; that script simply extracted a clean copy of the sources, ran configure and started the build. I won’t bother reproducing the script here since there’s nothing magic in it, but it is available on request if somebody really wants to see it.

Results

First, let’s look at the complete results for all three build tools: gmake alone; gmake with distcc; and Accelerator (emake):

Although it’s hard to see the differences between distcc and emake in this view, I wanted to include it in order to make a couple points about the performance of gmake alone on this build. As I said previously, this is a build that is very CPU intense. You can see the evidence of that in the performance data for gmake. Serially, the build runs in 30m46s; with -j2, the build time is reduced to 18m11s, about 1.7x faster. But after that point, trying to increase the parallelism actually makes the build run slower! This is because at -j2 we’ve already completed saturated the CPU’s on the build host. If the build were less CPU intensive, we would have seen some (possibly very small) improvement at higher levels, at least until we reached the point of saturation.

Now let’s focus on the distcc and emake results starting with 5 workers, so that we can really get a good look at the difference at high levels of parallelism:

With this graph we can see some really interesting things. First, distcc beats Accelerator at low levels of parallelism, but Accelerator steadily gains on distcc with each additional CPU brought into the build. With about 11 CPU’s in use, performance is tied. Past that, Accelerator takes and holds the lead. These results were a bit of a head scratcher for me. I was pleased to see that Accelerator won out in the end, but I was surprised that distcc demonstrated that early advantage, and that it maintained it for so long. After a few days of puzzling over this, I decided this must be the evidence of two separate factors playing together.

First, Accelerator seems to have slightly higher overhead per job run in the build. This explains why distcc beats Accelerator on this build at low levels of parallelism. But the overhead is pretty small — maybe 10% at most, and in the end, the performance is dominated by the second factor: Accelerator scales better than distcc, so it is able to bring more CPU’s to bear on the task. The fact that Accelerator can run more jobs in parallel more than makes up for the small additional overhead.

Conclusion

Although distcc had an early lead in this round, in the end Accelerator is the clear winner. Here’s the updated scorecard:

Still, it’s clear that there’s room for improvement in spite of this victory. That’s good though — as they say, people seldom improve when they have no other model but themselves to copy after.

UPDATE: I finally got time to examine this build more closely. The results of my investigation are here.

• Dual Xeon 2.4GHz hyperthreaded CPU’s
• 8 systems with 1.5GB RAM; one system with 2GB RAM
• Gigabyte Ethernet connections on a dedicated switch
• RedHat Desktop 3, update 8

I used the following software packages in this test:

• GNU make 3.79.1
• distcc 3.1
• ElectricAccelerator 4.3.1.25685
• MySQL 5.1.31

Process

As previously, I used a driver script to simplify running the tests:

#!/bin/sh

gver="`make --version | head -1 | awk '{ print $4 }' | sed -e s/,//`"
PATH=/opt/ecloud/i686_Linux/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

package=mysql-5.1.31
unset ARCH
unset OUTTOP
unset LD_LIBRARY_PATH
DISTCC_POTENTIAL_HOSTS='blade1 blade2 blade4 blade5 blade6 blade7 blade10 blade11'
export DISTCC_DIR=`/bin/pwd`/.distcc
export DISTCC_POTENTIAL_HOSTS

mkdir "distcc-3.1-pump"
(
  for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
  do
    pfx=../distcc-3.1-pump/distcc$i
    rm -rf "$package"
    tar xzf "$package.tar.gz"
    cd "$package"
    ./configure

    rm -f $DISTCC_DIR/lock/*
    (time pump make -j $i CC=distcc CXX="distcc g++" \
      $targets \
    ) < /dev/null > "$pfx.out" 2>&1
    cd ..
  done

  echo DONE
) < /dev/null > "distcc-3.1-pump/dtest.out" 2>&1

MySQL built cleanly out-of-the-box with both distcc and Accelerator: no makefile modifications were required, and distcc’s “pump” mode worked with no additional configuration needed. You can see for this comparison I ran the build with varying levels of parallelism (by altering the -j parameter for the distcc runs, and by altering the –emake-maxagents parameter for the Accelerator runs). This gave me the data needed to show how distcc and Accelerator scale as you add more and more nodes to your cluster. Note that although the driver script only does one run at each level of parallelism, I ran the driver script in its entirety three times, so that I could get a good average time for each build tool at each level of parallelism. Also note that I ran the Accelerator build once to generate a history file which was used for subsequent runs; the time for that initial build is not included in the results.

Results

Here are the results:

There are a few things I find interesting about this result. First, of course, is the fact that Accelerator beats distcc, by a significant margin at high levels of parallelism. The best time delivered by distcc is about 4m15s, about 4.7x faster than serial time. The best time delivered by Accelerator is about 1m49s, about 11x faster than serial time. According to ElectricInsight, our build visualization and analysis tool, the best possible time for this build is about 1m20s, based on the dependency graph for the build.

It’s also interesting to compare the performance of gmake -j with and without distcc. On its own, gmake -j provides about a 1.9x improvement, which is about what you would expect given that the build system has two physical CPU’s. However, that tells us that of the total 4.7x improvement obtained with gmake -j and distcc, the addition of distcc only accounts for about 2.5x (NB: to get the total improvement from multiple techniques used in conjunction, you can multiply the improvement from each; for example, 2.5x times 1.9x is about 4.7x).

The next interesting thing is that Accelerator scales better on this build. gmake with distcc maxes out roughly when parallelism is about 10 or 11; Accelerator continues to see gains until it hits about 16 agents.

Finally, there is the surprising step result in the Accelerator timings: there is a significant improvement each time the number of agents is pushed passed a multiple of three. I believe this is an artifact of the way I have the cluster configured, with three agents per dual CPU host. The agent allocation algorithm in Accelerator tends to prefer to grab agents on the same host before grabbing agents from a different host, so most likely we’re seeing the effect of completely loading up one host followed by the addition of another host to the in-use pool.

Conclusion

It looks like Accelerator wins this round. Here’s the updated scorecard:

Distcc uses an interesting approach to accelerating builds. It leverages the parallel facilities built into GNU make itself, and adds a distribution mechanism that enables it to take advantage of networked CPU resources. This week I decided to take a look at distcc 3.1, which was released in December 2008. It’s been some time since I last tried it, so I figured it was worth some time to see how the project has evolved and how it compares to Accelerator in its latest incarnation.

Setup

For this experiment, I chose to build the bzImage and modules targets of the Linux 2.6 kernel for the following reasons:

• Including modules, the Linux 2.6 kernel build is fairly substantial — around 20,000 source files
• It’s freely available, should anybody want to replicate my experiments.
• It’s well-known, so if I have made a foolish error in my tests it will be easier for other people to detect and correct.
• The build system was deliberately designed to facilitate parallel builds.

I used the following packages in my tests:

• GNU make 3.79.1
• Distcc 3.1
• Linux 2.6.28.1
• ElectricAccelerator 4.3.1.25685

Finally, my test hardware consisted of 9 systems configured as follows:

• Dual Xeon 2.4GHz with hyperthreading enabled
• 8 systems with 1.5 GB RAM; one system with 2 GB RAM
• Gigabit Ethernet connections on a dedicated switch
• RedHat Desktop 3, update 8

Process

After downloading the kernel sources, I unpacked them and used make menuconfig to generate a .config file with all default settings, which I saved for reuse to ensure that each test run used an identical configuration. I wrote a simple driver script for the tests:

gver="`make --version | head -1 | awk '{ print $4 }' | sed -e s/,//`"
lver="2.6.28.1"
targets="bzImage modules"
mkdir gmake-$gver
(
  rm -rf "linux-$lver"
  tar xjf "linux-$lver.tar.bz2"
  cd "linux-$lver"
  patch -p0 -i ../"linux-$lver.patch"

  for i in 1 2 3 4
  do
    pfx=../gmake-$gver/gmake$i

    make distclean
    cp ../"linux-$lver.config" .config
    make silentoldconfig

    (time make \
      $targets \
    ) < /dev/null > "$pfx.out" 2>&1
  done

  echo DONE
) < /dev/null > "gmake-$gver/gtest.out" 2>&1

Attentive readers will have noticed that I’m applying a patch to the kernel sources before running the build. That patch just removes a couple instances of the order-only prerequisite feature in the kernel makefiles, because neither gmake 3.79.1 nor Accelerator 4.3.1 support that feature.

Serial build

Collecting serial build times was completely uneventful. The build ran successfully, albeit slowly, but then that’s why we’re here, right?

ElectricAccelerator build

After collecting serial build times, I tweaked the driver script to accomodate building with emake. For these tests, I used the system with 2 GB RAM as both cluster manager and emake host, and configured the remaining systems as agent nodes running three agents each. ElectricMake built the 2.6 kernel out-of-the-box, without any special configuration required. I did run one build to generate a history file, then used that history file for each of the subsequent runs, although for this build, the impact of the history file was negligable (that is, there are very few missing dependencies in the build), which is to be expected given the amount of work put into making the build parallel friendly.

Distcc build

Finally, I retooled the script one more time to accomodate building with distcc. For these runs, I used the system with 2 GB RAM as the build host, and used the remaining systems as distcc servers; I invoked distcc in “pump mode” with gmake -j 16. The build ran successfully, but I found errors in the build log indicating that “pump mode”, the banner feature of distcc 3.x, had been disabled, meaning that the build performance was negatively impacted (error formatted for legibility):

ERROR: compile arch/x86/kernel/asm-offsets.c on blade10,cpp,lzo failed
Warning: remote compilation of 'arch/x86/kernel/asm-offsets.c' failed,
    retried locally and got a different result.
Warning: file 'include/linux/autoconf.h', a dependency of
    arch/x86/kernel/asm-offsets.c, changed during the build
Warning: now using plain distcc, possibly due to inconsistent file system
    changes during build

After some investigation I learned that pump mode does not automatically handle header files that are modified during the build. I applied the prescribed workaround, with the addition of the specific header file mentioned in the error message I received, and tried again… with the same result. After several more iterations, covering one and a half days and including some detailed analysis of the build made possible by Accelerator’s file-level annotation, I managed to get distcc working in pump mode. Note that plain distcc worked out-of-the-box; it was only pump mode that gave me trouble (details available on request).

Once I worked out the kinks, I reran the distcc tests with -j 8 and -j 24; and I tried including “localhost” as one of the distcc compile servers. At best these changes had no impact on performance; most of them made the build slightly slower.

Results and Analysis

As you can see, both distcc and Accelerator do a good job of accelerating the build, but for raw speed (and, I think, for ease of implementation) Accelerator takes the crown on this one. Why is that so? I think there are two factors that contribute to our success here:

1. Accelerator distributes all work to the cluster: in addition to compiles, Accelerator distributes code gen, links, packaging, and even makefile parsing to the cluster. This significantly increases the amount of work that can be done in parallel, and reduces the amount of work performed on the build host itself, preventing it from becoming a bottleneck.

2. Accelerator aggressively parallelizes recursive make invocations. In a variety of common situations, gmake does not parallelize recursive makes, even when invoked with -j. For example:

   all:
       $(MAKE) util
       $(MAKE) app1
       $(MAKE) app2
       $(MAKE) app3
   

   In this situation, gmake will not parallelize the work in the recursive makes, and in fact it cannot — to do so runs the risk of breaking the build. But Accelerator can and does parallelize the recursive makes, and it can do so safely because of our conflict detection and resolution technology. This often gives us an edge over other build systems, particularly if there is substantial work in recursive makes. Of course, since distcc uses gmake to handle parallelization, it is subject to the same limitations.

It’s worth mentioning that Accelerator also has an edge in terms of the artifacts left over after the build completes. When the distcc builds finished, I had the build output and the standard build log. When the Accelerator builds finished, I had those same artifacts as well as an Accelerator annotation file, which as you know provides a gold mine of performance and dependency information about the build, which personally I find indispensable.

Conclusion

Although distcc is more capable now than ever before, in the most important measure — raw speed — Accelerator still beats it hands down. I think this experiment also underscores the shortcomings of distcc’s approach to build acceleration — without distributing more work to the cluster, I don’t know that distcc will ever be able to match Accelerator for speed, at least for large builds that produce multiple outputs. For smaller, simpler builds, who knows — but then, that’s not really the target that Accelerator is aiming for.