Evil Ant Build Properties

The Problem (and intermediate solutions)

I have worked on no less than five projects where the standard, when I arrived, was to have a build.properties file right next to the build.xml file. Both go into version control. What always happens is that somebody changes a property to make the build work on their system. Then they check in their changes and everybody else's build breaks.

The most common first solution is to simply not check in that file. This works kinda-okay until somebody accidentally checks it in. Then at least one person's properties file gets wiped out and .... no backup. And if you are new to the project, you need somebody to feel sorry for you and e-mail you a copy of a properties file to get a start. Sure, there can be documentation - but I have yet to see the documentation be kept up. And I have experienced multiple projects where the solution was to develop to the documentation and when it doesn't work, exchange e-mail with somebody until the documentation works.

The next solution is to put build.properties into your home directory and have ant find it there. This works until you have more than one project. Then you change it to flooberproject.properties or start having properties like flooberproject.db.schemaname. This solves the problem of somebody accidentally checking in the properties, but incoming developers still need to depend on somebody sending them a properties file.

The next improvement is to make a conf directory within the project that is in version control. This directory contains lots of username.properties files. This solves the incoming engineer problem and brings all the pluses that come with version control, but there are a couple of minor problems introduced. One is people that have the same username. Or the same person (with the same username) developing from two different boxes. Another is that sometimes you have data that you are not comfortable with moving into version control (a password perhaps?).

Some Desired Goals

A) Have as few properties defined outside of build.xml as possible.

I have seen projects that have hundreds of build properties defined. I'm sure when they started out they had two or three. And then there were forty. And then it just grew and grew and nobody could come up with a clear reason to not just add a few more now and then. Here's one reason: Anybody new coming to the project needs to understand these before they can configure them correctly. It is far easier to understand eight properties than three hundred. I think that if any project has more than 15 properties, it should be considered a bad smell and something should be done. Maybe the project is too big and should be broken up. Maybe there are a lot of things that can be moved into build.xml - not every developer needs it to be different? Maybe some coding standards could eliminate some properties? Maybe some properties are not being used any more?

The focus of this article is properties that change from developer to developer. Once you have just a few properties to manage, everything else becomes much easier to solve. I have to say that having fewer properties is (IMO) ten times more important than what is being covered in this article. And it would be silly to write an article that simply says "use fewer properties." I only mention it because it is such a serious problem for some shops and there isn't a better time/place to express it.

B) Try to keep as many properties as possible in version control.

If the properties you are currently using are in version control, then when new people come to the project, they can look at the most recent property files that other developers are using and emulate those properties files. Further, if you make changes and those changes turn out to be poor, well you always can look up your older properties in version control!

C) Provide a mechanism for sensitive properties to not be in version control.

Database passwords are an example of something that you might want to keep out of version control. But when we are talking about a development box, and the database is configured to not talk to anybody outside of the box ... well, maybe it isn't such a big deal. As long as some people can have security if they want it and some people can go the version control route if they want that.

D) Provide a mechanism to share build properties between projects.

If you have really good communication between teams and within teams, you will avoid duplicated effort and use shared resources. If any one team will have, say, eight projects they are working on during a year, and some common property needs to change, it just seems wise to facilitate making the change just once and all of the projects still build properly.

E) Properties need to be self documenting.

Be considerate of engineers coming to this project. Also be considerate of the other engineers on a project when you are adding properties.

A doc file, a readme file or a wiki can help, but they can easily become outdated and lead to confusion.

Actual, functioning properties files make for an excellent form of documentation. Using "fail unless" ant tasks can help too!

A Hybrid Solution

This is just one humble solution. I suspect that there are many others that will work great. And I can't help but think that as the years pass, this solution will be enhanced.

I'm a big fan of "do the simplest thing that could possibly work". In this case, all of the solutions mentioned earlier are simpler than what I'm about to propose. But the needs do exceed each of these solutions. So while this hybrid solution does add four lines to the simplest ant script, it does solve several problems. And when combined with making it a standard (as it now is for JavaRanch projects) there is predictability.

The reason why I call this a hybrid is that it attempts to collect properties from any one of three places used in the previous solutions! Each solution has strengths and weaknesses. With all three you have all of the strengths and you have eliminated all of the weaknesses. I think that the optimal use of the hybrid approach is to keep as many properties as possible within version control (a config directory in the project). Keep sensitive properties in one of the user home properties files. Keep properties shared between projects in the common property file.

Drop these ant tasks into the init of your build.xml to pull this off:

        <property file="${user.home}/${project.name}.properties"/>

        <property file="${user.home}/build.properties"/>
        <property name="config.filename" value="${user.name}.properties"/>
        <property file="config/${config.filename}"/>

So when I'm working on the floober project on windoze, I can put my properties in one of these three files:

        C:\Documents and Settings\paul\floober.properties
        C:\Documents and Settings\paul\build.properties
        C:\work\floober\conf\paul.properties

And if I sometimes work from home, where my login is the same, I can define "config.filename" in build.properties in my home dir as, say, "paul_home" and then keep the rest of my properties in paul_home.properties in the conf directory for all projects.

Since ant always uses the first definition of a property and ignores any attempt to change a property, you can declare some default properties in build.properties and override them with project specific properties in your home dir. You might be tempted to read in the config dir properties first, but that will eliminate the ability to define config.filename without adding yet another properties file.

Self Documenting Build Properties

Old, moldy programming pearl: Debug only code. Comments lie.

Any time you leave something to the discipline of a human, you are looking for mistakes to be made. I've worked in shops where the technique to let you know about new properties was to send you an e-mail. So the next time you do a "get latest" from version control, you need to remember to look that e-mail up again and work the new properties in and test to make sure you did it right. Some shops use web pages or wiki pages. Some do both e-mail and wiki. Some put comments in the ant files. The problem with all of these is that things change, people forget to send the info, look up the info, or .... well ... all sorts of common, natural human stuff. And then it takes time to resolve things.

Using the approach where properties go into version control helps a lot. If your build isn't working now and the last time you got the latest code was three hours ago, then somebody checked something in within the last three hours. Take a look at the config directory - if there is a file there that is less than three hours old, you found your culprit. A diff to the last version of the file will show you new or changed properties.

What is proposed next is a pretty significant improvement. It does depend on the developer to remember to add one thing to the ant script, so there is still a hole in this process. But if that developer can remember this one thing, all of the other developers don't need to be as disciplined.

For every property that needs to be defined by a developer, use fail/unless tags. Like so:

        <property name="prop.message" value="must be defined, preferably in config/${config.filename}
                                             (details at javaranch.com/props)"/>
        <fail message="db.schema ${prop.message}" unless="db.schema"/>
        <fail message="db.password ${prop.message}" unless="db.password"/>

Now, every time you have a required property, add one fail/unless tag. The next time the other six people on your team tries to build, each build will fail and a message will spell out exactly what to do.

[Bonus Trick!] Instead of using fail/unless, simply declare a property after reading in the property files. It will act as a default. If somebody wants that value to be different, they can specify what they want in one of their property files!

Beyond Ant

Ant is a fantastic tool. And now I'm ready for whatever is going to replace ant. I tinkered with maven for a while and ended up going back to ant. I had high hopes for gravy (groovy plus ant) and it doesn't seem to have caught on. As a Java programmer, I would like somebody to come up with something that lets me do loops and have a little more control over my build. Something that takes into account multiple developers, code re-use, CruiseControl, IntelliJ, sub projects, common projects, and ... properties!