How deep is your build pipeline?

,

[diagram]

The diagram to the right (click for larger version) shows a small part of the build system for the project I’m working on, tracing the data flow from a single source file (“plot descriptions” in the top left) though to the final target (“distribution” in the bottom right) with a handful of other source and intermediate files (with dashed outlines) to give a bit of context. Data from this one source file passes through ten different tools on its way to the target, with the longest path passing through seven tools.

The trouble with long build pipelines like this is that small changes to sources tend to provoke large numbers of recompilations as the effects of change cascades through the system. And since we use “make” (which only knows about modification times on files) these changes are often unnecessary: in this example, a small change to the plot descriptions results in fonts being rebuilt, and so textures re-encoded, packages rebuilt and source code recompiled even though the set of characters in the plot text did not change. As discussed in my previous post. Still, correctness first, optimization later, and it only takes 10 minutes to rebuild the whole project from scratch. Maybe we’ll rewrite the whole thing in SCons some year soon.

This kind of deep build system pipeline is pretty typical in the video games industry; I’m sure that bigger companies could show much deeper examples.

Pity the poor build engineer working on a game containing lengthy pre-recorded video sequences rendered by the game’s engine. To make each build of the game, he has to build a version of the game engine for recording the video sequences, run it overnight on a farm of console development kits, collect the output sequences on an attached PC, encode them (overnight again, using a video encoding farm), then use the encoded sequences as inputs to a second build of the game. The whole procedure is too complex, lengthy, and error-prone to run for each build, so he saves the generated videos from each build and keeps using them in new builds of the game when he doesn’t have time to do a full build or when something goes wrong. When the game comes out to retail, the engineer plays it and thinks to himself, “that cut sequence at the end of level 5 looks like it was rendered using the engine from build 12345, it’s not using the new shader code, how the hell did that happen?”