IT is not taking over coding tasks, but they are certainly taking a much more active role regarding where application development tasks get run.   This is no surprise, as software development has an insatiable appetite for compute resources.  Whether it is vast clusters for testing, dozens of machines to run ALM tools, or the incessant requests for “just one more box,” development teams are always asking for something.  Multiply this by the number of development teams in your company, and it is easy to see why this is a big job for IT.

IT’s response is to increasingly look at centralizing where development tasks are run.  The logic is that with a cloud of development resources, they can get more efficiency in sharing them across groups and offer their customers (development) more resources than they would otherwise get.  However, IT’s goals are largely different than those of the development teams.  IT organizations measure their success by how efficient, smooth and cost effective their compute environment is. They want a large, identically configured, scalable, uninterruptible environment that consistently delivers established levels of performance to the downstream customer.  In other words, the more that they can make things the same, the more efficient they can be.

On the surface, these goals are at odds with development.   Development teams are measured on software output and quality, not on resource efficiency.  Each development group often has unique needs to achieve peek productivity and quality.  The matrix of test environments never shrinks, in conflict with IT’s desire to standardize.  If environment customizations and optimizations make development more effective (which they often do) they want their own resources, even if it means they get fewer of them.

How do you bridge these competing goals?  The wrong answer is compromise: it’s not about finding the midpoint that’s just useless enough to each party’s goals that everyone is unhappy.

The right answer is to define an environment that can deliver against both goals simultaneously.  Allow IT to provision the compute cloud – these are the areas where homogeneity and efficiency shine.  This allows IT to meet development’s needs for peak resource demands by sharing across large pools of compute resources while reducing cost.  Virtualization is an important ingredient in the solution because it meets IT’s need for homogeneity and development’s need for configuration specialization.   However, virtualization is not enough.  What is really needed to bridge the gap is a framework that allows development to maintain control of what processes get run, who runs them, and how and when they end up on the compute cloud.

Is this possible?  Our most successful customers have used ElectricCommander to do just this.

For IT, ElectricCommander enables software development to happen in one large, scalable, reliable development cloud.  For development, they get all of the control that they need, only with a heck of a lot more compute resources.

I wouldn’t presume to call myself a great programmer, but I definitely hate doing unnecessary work. That’s why the concept of build avoidance is so intriguing. If you’ve spent any time on the build speed problem, you’ve probably come across this term. Unfortunately it’s been conflated with the single technique implemented by tools like ccache and ClearCase winkins. I say “unfortunate” for two reasons: first, those tools don’t really work all that well, at least not for individual developers; and second, the technique they employ is not really build avoidance at all, but rather object reuse. But by co-opting the term build avoidance and associating it with such lackluster results, many people have become dismissive of build avoidance.

Subbuilds are a more literal, and more effective, approach to build avoidance: reduce build time by building only the stuff required for your active component. Don’t waste time building the stuff that’s not related to what you’re working on now. It seems so obvious I’m almost embarrassed to be explaining it. But the payoff is anything but embarrassing. On my project, after making changes to one of the prerequisites libraries for the application I’m working on, a regular incremental takes 10 minutes; a subbuild incremental takes just 77 seconds:

Not bad! Read on for more about how subbuilds work and how you can get SparkBuild, a free gmake- and NMAKE-compatible build tool, so you can try subbuilds yourself.

What is a subbuild?

A subbuild is just the smallest part of a full build tree that must be built in order to completely build a single component of the build, including all its prerequisites. For example, my project consists of several applications and the libraries they depend on. Each of these components resides in a separate directory, and we use recursive make invocations to build everything. (Nota bene: if you have a non-recursive make then you probably already enjoy many of the benefits of subbuilds, but you should definitely still check out the other features of SparkBuild!)

The dependency graph for my project looks like this:

You can see that to build the agent component, for example, we only need to build the util, xml, and http libraries, and the agent application code, of course:

This subset defines the agent subbuild.

Subbuilds and developers

What makes subbuilds really interesting for developers is the realization that usually you’re working on just one component at a time. For example, on any given day I might be working on the agent component, or the cm, but rarely both. Most of the edits I make will be on code in the agent directory, with occassional edits to the agent’s prerequisites. As I’m running through the edit-compile-test cycle, I have some choices about how to run the build. The most natural thing for me is to simply run make in the agent directory. After all, most of the changes I make are in that directory, so that will do the right thing most of the time. Of course, if I have made changes to any of the prerequisites, or if I resync with the source depot and pick up somebody else’s changes in one of those prerequisites, I’ll probably get a busted build.

The next most obvious approach is a rebuild from the root of my source tree. This ensures that I always update all the pieces I need for the agent, but at the cost of also building components that are irrelevant to my current focus: if I’m just trying to rebuild to run the agent’s unit tests, there’s no need for me to rebuild the cm application, or the ldap library.

The best choice is the agent subbuild, the minimum set of things that must be built to be sure that the agent component is fully up-to-date. But although it’s possible on a small project like this to execute the subbuild manually, it’s a nuisance, and on a bigger project it may not be practical or even possible. You need a build tool that can automatically determine which parts of the build make up the subbuild for any component, and then automatically execute that subbuild. That tool is SparkBuild emake.

Subbuilds with SparkBuild

Subbuilds with SparkBuild start with a full build, during which emake captures information about which targets are produced by each submake. In subsequent builds, emake references that database anytime it can’t find a rule to build a particular target. If a match is found, emake runs the corresponding submake before proceeding. For example, the rule for the actual agent target looks like this:

$(OUT)/agent/agent: $(OUT)/agent/*.o $(OUT)/xml/xml.a $(OUT)/http/http.a
	g++ -o $@ $^

In a normal build, gmake would see the dependency on $(OUT)/xml/xml.a and use that file if it existed already, regardless of whether it was actually up-to-date; or report “no rule to make” if the file did not exist. With SparkBuild, emake checks the subbuild database for an entry matching $(OUT)/xml/xml.a and sees that it must run make in the xml directory before proceeding. Like magic, each of the agent’s prerequisites is updated without requiring me to take any action other than swapping emake –emake-subbuild-db=my.db for gmake in my build command-line.

Still not convinced that it’s worth a look? Here’s some more concrete results comparing a few different build scenarios from my project. These comparisons assume that I’m actively working on the agent component, and that I ran either a standard incremental, from the root of the source tree, or a subbuild using SparkBuild emake:

Conclusion and Availability

Tools like ccache and ClearCase winkins have co-opted the term build avoidance, but in fact they do object reuse, not build avoidance, and they are not very useful for developer builds. Subbuilds are a simple but highly effective approach to build avoidance that save significant time during developer builds by literally skipping parts of the build tree that are unrelated to your current focus.

If you want to try out subbuilds yourself, you can download SparkBuild from www.sparkbuild.com. It’s completely free (as in beer), so you’ve got nothing to lose… except those long coffee breaks, of course!

Our customers gave us some interesting use cases for using our products in combination with the public cloud. Here are some of their ideas:

We can now do all of this. What can you imagine?

Agile development processes are now the norm rather than the exception. Why not an agile development infrastructure too? Our new integration gives you the ability to use just the build and test machines you need whether they are physical, virtual, or in a public cloud.

Recently I read a good post on Brad Appleton’s Agile Configuration Management Environments (ACME) blog.  It takes the definition of business agility and maps it to the definition of software development agility by including the people factor.  The definition in that article and other tenets of Agile, indicate that success is not only dependent upon team size, but that experience and range of skill set are critical as well.  Consider two of the many principles set forth in the Agile Manifesto.

“Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.”

“The best architectures, requirements, and designs emerge from self-organizing teams.”

These notions indicate that there will be late breaking changes in scope and they rely on the notion that teams can self-evaluate and adjust.  In order to effectively adjust, developers must draw on experience collecting requirements and design to anticipate where the requirements are “squishy”.  Similarly, self-organization requires the ability to look at a situation, evaluate, and adjust to create a better outcome than would have occurred otherwise.  The likelihood of success in this model is significantly improved through experience.  So the best self-organizing teams are those that consist of individuals who have experience working with other teams.  Someone might even maintain that in order for an Agile team to succeed it’s members must have experience with unsuccessful software projects.

One issue is that experience alone does not guarantee that a developer will encounter all the responsibilities in the software development lifecycle.  It is quite common for developers to naturally gain experience in requirements gathering, design, and testing throughout their project history – even if the sole focus of responsibility is supposed to be pure programming.  However there are areas such as release management and automated testing, which the standard developer encounters much less.  These areas are considered to be more specialized, often due to the specific toolsets involved and the detailed understanding of the development infrastructure required.  Subsequently companies end up with armies of developers, but only one or two people who can exercise, troubleshoot and enhance their development infrastructure.  In effect the demography of the corporate software organization is at odds with the needs of the successful Agile team.

Depending on the makeup of the software development organization and complexity of the development infrastructure, it is important that the build-test-deploy processes are encapsulated in a reusable application for the entire team.  ElectricCommander and similar applications traverse the skills gap by capturing the knowledge associated with build-test-deploy and bundling it up behind a BGB (big green button).  Each particular developer doesn’t have to remember the login credentials for each build server, remember which of the automated tests require a special setup script because the test is out of date, etc.  Just as the build scripts or make files themselves are meant to roll up the complexity of the build process into individual targets, applications such as ElectricCommander also roll up the complexity of building your software, running the test harnesses, and deploying to environments into singular processes.

If you’ve ever worked on a significant software development project you’ve probably looked at some part of your process and said ‘we have to change this!’ More often than not the topic is the software production process: that problematic build-test-release cycle. We can all identify places where our process is slow or broken; so why do some initiatives succeed while others fail miserably? If you’re going to make a change which affects six people: stop reading – just go do it! Sixty? You’ll have to get buy-in from colleagues. Six hundred? You’re going to have to talk to your Director of Engineering. Navigating those waters benefits from drawing a map: you should know who your change will impact, how it will benefit them, and where the low-hanging fruit are. Over my four years with Electric Cloud I’ve successfully (and sometimes not so successfully!) sailed the process-improvement trade route; in this article I’m going to give you a heuristic in order to profile your own team and develop a navigation map.

Are you building on a monolithic model, or a modular model?

Consider this the first branch decision in evaluating the layout of your organization: is your product monolithic or modular? There are fundamental differences between these design models, and the challenges of development will be different depending on which model applies to your team.

Monolithic development teams are generally large, with dozens or hundred of developers working on the same final binary kit. Take a look at your build process: do you build the same binary as all of your office-mates? Do they build from the top-level .sln file or type ‘make all’ just like you? Does the build you do on the desktop take two hours? If you trace the execution of code do you see few API calls but a ton of interlaced functions? If so, you’re building a monolithic application (something like Microsoft Office, PTC ProEngineer, or Cisco IOS).

Modular development teams are characterized by large collections of small groups (2-8 people), each providing a separate deliverable which interlocks with other team deliverables. Does your team support your piece of the codebase to the rest of the development staff? Do you have your own design docs which are published out? Does your code compile in five minutes? Do you make a library or depend on a specific version of other internal elements? Do you find out about problems in your code only after release engineering gets through with it? Do you use a build process which is different from release engineering? You’re in a modular shop (think Amazon.com, SonyEricsson, eBay).

Let’s also remember that the most common answer is ‘maybe’: the picture is complicated when you’re a hybrid, but nearly every shop is. (Teams swing back and forth, too) Look for the modular model – small teams, API-as-contract – with a couple ‘core’ libraries or function groups that are huge. This is usually because some part of the product has to be monolithic – backend data processing, the kernel or standard library, a graphics engine or runtime – and everything else is middleware or leaf applications. Microsoft Windows Mobile and Symbian OS are examples of products like this.

Here’s why it matters:
Monolithic developers have more of the same pain as the release team; these developers are more likely to have to link (or even compile) the whole app to test a code change. A developer may not build every language variant or platform (and usually can’t!) but even a single platform ‘full build’ before checkin is expensive, and often means going home for the day. In this case, you’ve got a build speed problem. Release teams in this scenario are generally doing the ‘official’ builds, in blessed sane environments, and it’s only a developer mistake which breaks their build. Speed is still the issue; managing builds is generally making sure every platform and variant is compiled.

Modular developers chuckle at their monolithic colleagues – a component, which is often the extent of a developer’s responsibility before checkin, may compile in sixty seconds – but release teams in modular shops sob. Why? Release engineers are integrating hundreds or thousands of components, each of which was built locally before checkin and generally not tested in the full system, and get to find all the integration problems at once. This leads to tons of ‘try, fix, retry’ loops as they discover problems, notify developers, and wait for new builds, builds which often take hours (due in no small part to the overhead of all those APIs and modular boundaries) instead of seconds. Your developers need a way to asynchronously test their changes in the production environment (a build management problem) and your release team needs help managing component dependencies, versions, and updates (also a build management problem). The release team could also use some speedup, especially when the product is forked and becomes two (or ten) different variants.

So if you want to know where the low-hanging fruit is: look at your product architecture. Monolithic: developer speed. Modular: developer management, release team speed.

Do the boundaries of your product elements line up to the boundaries of your organization?

Now that you know whether the architectural model is monolithic or modular, you know how to identify what changes are worth making. Don’t stop there! You’ll need to scope more than just the benefit of a change: you should know how many people (and in what roles!) the change will impact. One easy way to evaluate this is by looking at the boundaries of code responsibility.

Strong boundaries are characterized by tight grouping of responsibility and clear knowledge limits. Can you identify what part of the product code an engineer works on based on what group she’s in? Is the SCM system aware of which users are approved to work on certain functional areas? The more distinct the divisions between functional areas of code the more likely it is that developers work on narrow areas of responsibility. This is common in modular product shops: groups are independent, and so are their code bases.

Ambiguous boundaries tend to be ad-hoc, more to do with areas of developer preference than intentional roles or assignments. Are there guru or lead programmers in your teams who answer questions for everybody? Do development tasks (and bugs) get fluidly reassigned when someone is on vacation or busy on a special project? Are there lots of developers to ask about any specific technical question? Monolithic projects are more condusive to a large, multipurpose team (but you’ll still find striations along functional units, I guarantee it).

Here’s why it matters:
The reason you should watch this closely has to do with decision-makers: can you get buy-in for your idea from the engineering director? Do you need to? In a team with clear boundaries in its components you might get away with just convincing the component developers. A more amorphous team has to deal with individual developer preferences: if Bob works on component A and B he probably doesn’t want to see A built with ‘make -j’ and B built with MSBuild. Most importantly, keep an eye on standardization: if you want to change a process used by everyone the impact is going to be broad; any change will require lots of support! Know whether you need buy-in from the whole team (and in this case, your champion better be that director, and he’ll want to understand a comprehensive ROI) or just to a specific group (these decisions can often be made in the lunchroom, on technical merits alone).

By scoping the size of the impacted group you’ll know how high in the organization you need to sell your idea, and who might be opposed to your proposed solution.

Do you ship code (and its associated development tools) or do you ship a binary product?

We’ve just looked at scoping impact on development teams within your organization. For some types of product organizations, though, you’re going to have to look at more than just the internal engineering efforts: who uses the product of your labors? This is a key question, and is at the heart of the question of ownership and impact analysis.

Finished products draw a distinct line between the needs of development and the needs of consumers. These types of applications tend to be independent from their development model. Does the user of your product (whether it’s sold retail or is a purely-internal tool) get a fully packaged and self-contained product? Is it used stand-alone? Intuit (for example) builds a binary product – Quickbooks – and ships the executable to its customers. Intuit is thus free to use any build system they want, and can change the infrastructure used to support development as much as they’d like.

Middleware, SDKs, Toolkits and Platforms are more like the components of a sailboat – it’s a finished product, but will be used (and often recompiled or modified) as part of a larger system. Do you have sample applications built on your product? Do you ship source code? Do you make sure your development tools are redistributable? Qualcomm ships source and tools as a board support package with their MSM product; switching from ‘make’ to ‘SCons’ would require all their customers to switch to (and support!) SCons as well.

Here’s why it matters:
If you ship a tool environment outside your walls – or to other groups in your company – you’ll need to look at the impact on your customers before making a change. The visibility and impact of your change may extend outside of development and up to Product Management, Marketing, Support, even Sales. Don’t be deterred; a change which benefits your team will undoubtedly offer benefits for your customers, but the scope of buy-in you’ll need gets much more broad.

Consider asking the vendors of your tools if you can ship their products (some will be happy to have you do so!) If you consume a platform product, look into what the OEM is using (it’s a good bet they use a different process than what they supply to you!) Either way, be sure you know who is affected before proposing a tooling change.

Is your infrastructure centralized and standardized, or do you have autonomy?

Remember we said it’s important to scope how many people and in what roles a proposed change will impact? Another way to evaluate this is to look at the level of common tooling and process is in your organization. This is a tell-tale which can focus your efforts on the guys with the juice to approve (or to quash) a proposed change.

Do you know who administers your SCM system? Bugtracking system? Compiler environment? When a new build machine is delivered, does IT image it for you? In a centralized shop, developers get little control over infrastructure systems, and those systems can sometimes have lots of impact on development practices. Ever heard of a ‘blue’ shop? This is a slang way of referring to development organizations which are consumers of all things IBM; there’s usually a central engineering group which sets policy (you WILL use ClearCase) or provides all the core infrastructure (Here’s your Rose, your Purify, your Team Concert).

Did your lead engineers pick the build tool when the project stated? Does your ‘/usr/local/tools’ directory have Perl, Python, Ruby, and TCL interpreters? Some organizations – JPL is a classic example – let every internal team pick their own tools and generally stick that team with supporting itself. This gives NASA scientists the autonomy to use whatever tools are optimum for their current project, but it also means the project lead is the single most important decision-maker for your change request.

Most shops – just like the monolithic/componentized decision – are a little of both: Electronic Arts provides some central tools, but allows individual studios a significant range of leeway in picking their own tools. This may be the most fertile ground for supporting process change: get a pilot group to adopt a new approach, show some success, then use that success as a case study to gain additional adopters. A snowball effect will lead to an enterprise-wide change once central engineering sees the growth of adoption.

Here’s why it matters:
You’ve got to know who to sell to! Central Engineering (if it exists) has to be engaged in a monolithic model – everyone has to use this stuff! Standardization on Microsoft Team System, for example, pretty much means you’re going to use MSBuild or devenv to run your builds. Want to use SCons? You’re pretty much out of luck. It’s not always easier on the other side, either: component developers are much more flexible – remember you’re just building your little component! – but Release Engineering is going to hate you if you give them the first JAM project in a sea of Visual Studio. Look at who holds the budget, who is impacted, and who has the need for change before mapping out a project plan and you’ll build success in from day one.

Whew! Feeling like Magellan yet? It’s worth it, though; by understanding the shape of the organization you’re in you’ll gain enormous insight to what changes to your software production methodology will help (and make you a star!) and which ones would be crucial mis-steps. Now try it out! Ask your co-workers about the history of the tools you’re using now, and see how your organization maps.