[caption id="attachment_378" align="alignleft" width="200" caption="Hourglass"][/caption]

One of the most difficult tasks in a software development project is estimating the size of the project. Unfortunatly very often  you have to do it at the very beginning of a project, when you have the least information. The result at the end is very often a large difference between the original estimate and the actual time and money needed.

If the difference is positive as often as it is negative this is kind of OK. But in some teams estimates are too low almost all the time! The obvious strategy often employed is to add a certain percentage to the estimate. But of course that is just fixing the symptoms, because most of the  time nobody knows the reason. So in this article I am going to show a couple of possible reasons for bad estimates, how to identify these and also a possible fix.

These are the kinds of reasons I identified so far:

Super Hero Estimates: Often estimates are done by a very experienced developer (Super Hero). When they estimate a task they imagine themselves doing it. But while the esimate might be correct for the Super Hero in the project more average developers work on the task. And since they aren't Super Heros they need longer.

You can identify this kind of situation by letting different people do the estimates. Including average developers. If the estimate of the Super Hero is constantly below that of the others, the fix becomes obvious: Use the estimate of the other developers. Don't exclude the Super Hero from estimating though. Experienced developers a good at identifying things others tend to forget.

Wrong Team: This one is similar to the Super Hero Estimate, in that the estimate is correct for a team of good (possibly excellent) developers. It differs though, because in this scenario, this is the actual team that should do the project. But for one reason or the other the project gets staffed with a different team. Maybe developers that aren't that good. Or maybe just a team that doesn't no the domain, the framework or the programming language that well.

In order to identify this situation, talk to the people doing the estimates. Let them specifiy the assumption they had in mind when they made the estimate, and compare it to what actually happend in the project.

If this is actually the problem you have two choices: The first is: stop switching teams. This doesn't work most of the time, since you don't switch them just for fun, right? So you are stuck with the second option: Even when you plan to use your top team, make the estimates on the assumption that an average team will do the job. Be honest to yourself about what an average team is.

Guessing in the dark: Often the information available about the project at hand is just not sufficient for a reliable estimate. There is only a vague description available.

In order to identify this as the underlying problem, break the project down into tasks, which you estimate. For each task put down the assumptions you are making: what technology are you using, how complex is the logic to implement. During the project check if your assumptions hold or if there are a lot of changes. This is also most of the solution to the problem. Make the description of your assumption part of your estimate. If the assumptions are wrong, the customer may correct those, resulting in a new more reliable estimate. I he changes his mind later on, you have a clean basis for a change management process, which ensures, that changes are possible, but that the one requesting the change is also paying for it.

Forgetting stuff: This one looks very similiar to the previous. Stuff doesn't get included in the estimate. But this time it's not because the people doing the estimate don't know about it, but because they forget about it.

Using the same strategy as for Guessing in the dark should make this kind of mistake obvious. As a remidy, let multiple people do estimates, and compare the results. You should do this based on the broken down tasks, so missing tasks in the different estimates become obvious.

Ball of Mud Projects: When projects base on a specific code base take too long each and every time, a reason might be that tasks keep getting more complex then anticipated. A typical reason for that would be an overly complex and convoluted code base, where nobody really can predict the effects a given code change may have.

In such a project I'd expect a lot 80% syndrome, few test and many swearing developers. Do a code analysis of the code base, with a concentration on architecture and dependency management. If you find a lot of crap, there is just one solution: Clean it up and pay your technical debt. I.e. invest time and money to improve the architecture. I also recommend implementing tests, that prevent the architecture from starting to rot again.

Multitasking: People are bad at multi tasking. So you can't put your top developer for 20% in five different projects and expect her to work just as efficient as normal. Once you start thinking about it, it should be really obvious if you have the problem or not. And the same should be true for the solution.

Mythical Man Month: Conceptional very similar is the Mythical Man Month effect. Check if your estimates get processed like this: "Frank says he would need 6 months. We need to get done within 1 month, so let's put 6 developers in the project." If this is the case you are missing on all the overhead that a bigger team causes. In order to get a handle on that kind of problem read the book Mythical Man Month. I do think that book is overrated, but the chapter about the actual mythical man month is right to the point.

This problem is almost guaranteed to hit you when you do a project that is a magnitude larger then your normal project. You will underestimate by a large amount, if you do not factor in the exponential behaviour of communication over head.

Lazy Developer: Of course there is always the possibility that the developers just aren't working hard enough. In my experience, most of the time, this isn't a problem. But I guess is it does happen. Symptoms could be lots of open browser tabs, on sites that have no relation to the work at hand, acompanied by hectic wiondow/tab switching, when the boss drops by. If this is a problem with one developer, it might be that the developer has the power to change it. But if many developers spend their time procrastinating it is most probably a management problem. Procrastination is not fun. Getting stuff done is. So if everybody is procrastinating something in the environment is probably highly demotivating. Find out what it is and get it out of the way.