mgear, a make-based build tool

Status: abandoned; 2007-2009 (supersedes any conflicting remarks left on this page; see the home page for definitions)

Mgear is a general-purpose build tool implemented inside GNU Make >= 3.81.  (Actually, it's under development, but I'll use the present tense anyway.)  It provids several additional features, most of them meant to improve the robustness of the build process.  Mgear is very easy to adopt because it can be included in a software package and users who have GNU Make >= 3.81 can build the software with an ordinary make command.

Note: Mgear was originally named Mage.  I renamed it to mgear on 2007-06-13 to avoid confusion with other meanings of the word "mage".

Motivation and plan[# Top]

When I poked around on the Internet for build tools similar to what I envisioned for mgear, I found many C/C++ build tools with nice features but few general-purpose ones.  Let me explain my meaning with an example.  The C/C++ preprocessor presents an interesting problem for build tools because the result of preprocessing a file depends on the other files it includes.  Several build tools, such as cons, implement their own dependency scanners for C/C++.  Another approach is to use the C preprocessor's own dependency logging capabilities (-M* options); such a build tool could easily be used with any other "dependency-logging command".  While a dependency scanner integrated into the build tool might make some really cool functionality possible, I prefer the second approach because it eliminates the risk of the build tool's and command's ideas of a dependency getting out of sync and I find that it leads to a deeper understanding of the issues involved.

Features I'm considering for mgear include:

• Isolation of inconsequential changes.  If someone adds a comment to foo.c, mgear notices that foo.o didn't actually change and doesn't relink foo.
• Warning replay.  "make all" prints the warnings for all transitive prerequisites of "all", not just the ones rebuilt on that run.  This requires storing the warnings when a file is built so they can be replayed on later runs.  Without warning replay, it's easy for important warnings in a source file to go out of sight and out of mind once the file is compiled successfully.  In fact, as of 2007-06-13, warning markers go out of the Eclipse CDT's mind this way; it needs warning replay so as not to lose markers.
• Override warning.  If you modify foo.o by hand, mgear warns you that you have overridden the rule instead of silently considering foo.o up-to-date.  If you then delete foo.o, mgear rebuilds it according to the rule and the warning goes away.
• Rebuild on command change.
• General support for dependency-logging commands with generated input files.  If mgear compiles foo.c and sees that it unexpectedly included bar.h, mgear updates bar.h and then compiles foo.c again if necessary.  I don't know of any other build tool that supports this.
• Rebuild a target on any change in a prerequisite's mtime, not just if the prerequisite's mtime becomes newer than the target's.  I'm not sure how best to do this.  I requested an enhancement to GNU Make for a $(mtime) function and was told that a Google Summer of Code 2007 project to add support for custom out-of-date criteria is in progress.  I will reconsider this feature when that project is done.

Motivating potential uses for mgear include:

• My Web site.  Its XSLT compilation step is a less commonly used dependency-logging command whose inputs include generated *.dl files.  I also hope to factor out some of the logic related to collecting and manipulating file trees into a reusable library to go along with mgear.
• rsync, as an example of a typical application in C that uses a relatively simple makefile.
• The Linux kernel.  Its "kbuild" system inspired mgear by demonstrating that awesome things can be done on top of GNU Make.

(I want mgear to be capable of building these projects, but obviously I won't try to force any of them to adopt mgear.)

Track development[# Top]

I keep mgear in a git repository mirrored at mgear.git/ relative to this page.  You can follow that link to browse the source code in gitweb.  If you want to try mgear out, I encourage you to clone the repository from that link using the git tools.  Alternatively, you can go into the gitweb interface and download a snapshot of the latest (topmost) revision by clicking one of the formats next to "snapshot".

WARNING: Adopt mgear at your own risk! The basics do appear to work well, but since it is still under development, there may be bugs and/or rough edges that make it unsuitable for day-to-day use by actual developers.  Furthermore, be prepared for frequent incompatible changes.

Current status[# Top]

As of 2007-06-13: The basics (genfiles, isolation of inconsequential changes, warning replay, override warning, rebuild on command change) are working pretty smoothly, but I'm still making occasional improvements.  I wrote a test suite for them.  I decided I didn't like having a separate branch dlc, so I moved it over master.

In order to stay on the same page with myself, I commented mgear.mk heavily.  Read the comments for lots of information about how mgear works.

I successfully built rsync using mgear!  Here is the patch that converts the rsync source tree to use mgear in case you'd like to try this for yourself.

I am working on support for dependency-logging commands.  I've mostly thought it through, but I expect implementing it to take a while because it's incredibly complex and the devil's in the details.  I'm still thinking about how best to handle the differing dependency-logging styles of xsltproc (--load-trace; dependencies written line-by-line to stderr as they are encountered) and cpp (dependencies written as a make rule to a file on successful completion).  To complicate matters, cpp doesn't provide a dependency log when it fails, so I think I'll just do the best I can using all the filenames I've seen in its error/warning output.