Hourglass

One of the most difficult tasks in a software development project is estimating the size of the project. Unfortunatly very often  you have to do it at the very beginning of a project, when you have the least information. The result at the end is very often a large difference between the original estimate and the actual time and money needed.

If the difference is positive as often as it is negative this is kind of OK. But in some teams estimates are too low almost all the time! The obvious strategy often employed is to add a certain percentage to the estimate. But of course that is just fixing the symptoms, because most of the  time nobody knows the reason. So in this article I am going to show a couple of possible reasons for bad estimates, how to identify these and also a possible fix.

These are the kinds of reasons I identified so far:

Super Hero Estimates: Often estimates are done by a very experienced developer (Super Hero). When they estimate a task they imagine themselves doing it. But while the esimate might be correct for the Super Hero in the project more average developers work on the task. And since they aren’t Super Heros they need longer.

You can identify this kind of situation by letting different people do the estimates. Including average developers. If the estimate of the Super Hero is constantly below that of the others, the fix becomes obvious: Use the estimate of the other developers. Don’t exclude the Super Hero from estimating though. Experienced developers a good at identifying things others tend to forget.

Wrong Team: This one is similar to the Super Hero Estimate, in that the estimate is correct for a team of good (possibly excellent) developers. It differs though, because in this scenario, this is the actual team that should do the project. But for one reason or the other the project gets staffed with a different team. Maybe developers that aren’t that good. Or maybe just a team that doesn’t no the domain, the framework or the programming language that well.

In order to identify this situation, talk to the people doing the estimates. Let them specifiy the assumption they had in mind when they made the estimate, and compare it to what actually happend in the project.

If this is actually the problem you have two choices: The first is: stop switching teams. This doesn’t work most of the time, since you don’t switch them just for fun, right? So you are stuck with the second option: Even when you plan to use your top team, make the estimates on the assumption that an average team will do the job. Be honest to yourself about what an average team is.

Guessing in the dark: Often the information available about the project at hand is just not sufficient for a reliable estimate. There is only a vague description available.

In order to identify this as the underlying problem, break the project down into tasks, which you estimate. For each task put down the assumptions you are making: what technology are you using, how complex is the logic to implement. During the project check if your assumptions hold or if there are a lot of changes. This is also most of the solution to the problem. Make the description of your assumption part of your estimate. If the assumptions are wrong, the customer may correct those, resulting in a new more reliable estimate. I he changes his mind later on, you have a clean basis for a change management process, which ensures, that changes are possible, but that the one requesting the change is also paying for it.

Forgetting stuff: This one looks very similiar to the previous. Stuff doesn’t get included in the estimate. But this time it’s not because the people doing the estimate don’t know about it, but because they forget about it.

Using the same strategy as for Guessing in the dark should make this kind of mistake obvious. As a remidy, let multiple people do estimates, and compare the results. You should do this based on the broken down tasks, so missing tasks in the different estimates become obvious.

Ball of Mud Projects: When projects base on a specific code base take too long each and every time, a reason might be that tasks keep getting more complex then anticipated. A typical reason for that would be an overly complex and convoluted code base, where nobody really can predict the effects a given code change may have.

In such a project I’d expect a lot 80% syndrome, few test and many swearing developers. Do a code analysis of the code base, with a concentration on architecture and dependency management. If you find a lot of crap, there is just one solution: Clean it up and pay your technical debt. I.e. invest time and money to improve the architecture. I also recommend implementing tests, that prevent the architecture from starting to rot again.

Multitasking: People are bad at multi tasking. So you can’t put your top developer for 20% in five different projects and expect her to work just as efficient as normal. Once you start thinking about it, it should be really obvious if you have the problem or not. And the same should be true for the solution.

Mythical Man Month: Conceptional very similar is the Mythical Man Month effect. Check if your estimates get processed like this: “Frank says he would need 6 months. We need to get done within 1 month, so let’s put 6 developers in the project.” If this is the case you are missing on all the overhead that a bigger team causes. In order to get a handle on that kind of problem read the book Mythical Man Month. I do think that book is overrated, but the chapter about the actual mythical man month is right to the point.

This problem is almost guaranteed to hit you when you do a project that is a magnitude larger then your normal project. You will underestimate by a large amount, if you do not factor in the exponential behaviour of communication over head.

Lazy Developer: Of course there is always the possibility that the developers just aren’t working hard enough. In my experience, most of the time, this isn’t a problem. But I guess is it does happen. Symptoms could be lots of open browser tabs, on sites that have no relation to the work at hand, acompanied by hectic wiondow/tab switching, when the boss drops by. If this is a problem with one developer, it might be that the developer has the power to change it. But if many developers spend their time procrastinating it is most probably a management problem. Procrastination is not fun. Getting stuff done is. So if everybody is procrastinating something in the environment is probably highly demotivating. Find out what it is and get it out of the way.

the ease with which a software product can be modified in order to:

• correct defects
• meet new requirements
• make future maintenance easier, or
• cope with a changed environment;

Wow, that’s great. Because it lends itself for an ‘easy’ test for maintainability. Take the completed product, make up some new requirements and measure how long it takes to implement those. If you do this with different products for same task, you can compare the time needed, thus comparing maintainability. Unfortunately most of the time, the requirement is stated before any software is written. And only one piece of software gets written, so there is nothing to compare to. So what to do?

Since one can’t realistically measure maintainability we once more should concentrate on characteristics making software easier to maintain, i.e. easier to change. There are plenty of measures around that are thought to be linked to maintainability. Cyclomatic Complexity being possibly the best known. I personally prefer toxicity, which is based on Cyclomatic Complexity and a couple of other measures, which contribute to the toxicity with the amount they are above a certain threshold:

Metric	Level	Threshold
File Length	file	500
Class Fan-Out Complexity	class	30
Class Data Abstraction Coupling	class	10
Anon Inner Length	inner class	35
Method Length	method	30
Parameter Number	method	6
Cyclomatic Complexity	method	10
Nested If Depth	statement	3
Nested Try Depth	statement	2
Boolean Expression Complexity	statement	3
Missing Switch Default	statement	1

Compared to a single simple measure, this has the benefit of being less prone to optimization for the measurement, although of course this is still possible.

So does a low, even zero toxicity guarantee maintainable code? Hell no. For starters there are a couple of different things to consider:

• Is the code well covered with tests?
• Are the current requirements specified in a suitable manner (e.g. using tests)?
• Is the code written in a consistent style
• Is all the code including the documentation under control of a version control system?
• Is the structure of the application, its architecture defined and documented?
• Is there an automatic process for building new versions of the software (i.e. ant, maven, make scripts or similar)?
• Is the code written in a language that is well known and understood and which has a large user community?

Everything but the last bullet point is pretty much identical with what I consider base practices for any serious software developer.

So I propose: The next time you encounter the vague requirement of maintainability, replace itybe useful and well testable requirements, based on specific practices and metrics. It still won’t guarantee maintainability. But it will increase the chance for it.

Support beams and wires of a bridge

When reading the specification of a piece of software to be written, you are bound to find some non functional requirements. Among these there will be, or at least should be Supportability. But what the heck does that mean? How do you install supportability? Let me present some ideas, what you can do to improve supportability.

Let your application log in a well defined reliable way, to a location that is easily accessible. Flatfiles on a server qualify. If you must log on a client, consider implementing a way to automatically transfer the log file to a support person. If you log into a database, make sure the support person can access it easily.

Make your application easy to shut down and start. This sounds trivial, but it is easy to break this ability. Considere the following check list:

• What happens when you start two different versions of your application against the same database? A nicely supportable application should notice this and react accordingly.
• What happens when you stop your application while it is processing a request?
• If you have batches or batchlike processes, what will happen with those when you try to stop your application while the batch runs? Do you have the 5hours, until the batch finishes? Or will the batch stop and rollback, so you have to wait an hour for the rollback to happen? Or will it stop nicely within a minute, and pick up its work automatically after restart?
• Are you stuffing stuff into a database or a queue? What happens when the queue or the database gets started after the application?
• Do you receive messages from a queue? What happens when the first message arrives, before your application is available? What does happen when you receive a message that you already processed? What does happen when you receive a message for which you already processed a later message?
• How long does it take to shutdown and restart an instance of your application when it is under full load? If this takes more then a few minutes, is it possible to stop and restart only parts of your application?

Make the state of your application visible for support personnel. Most applications just report arbitrary errors when a component of the system is down. It’s up to the supporters to guess if it is the database, a queue, a webservice or the application itself which is causing the problem. Identify the resources your application depend on. Write a check which tests all these resources, and make this check available, for example as a special health check webpage.

Put components that are likely to fail behind some kind of buffer. Your database might be so important for the application that this doesn’t work for it. But if you are posting stuff to a queue (or webservice or …), consider using a local queue as a buffer, so your application can work as usual even when the target queue isn’t available.

Last but not least: Document your application. The agile manifest says that working software is more important then documentation. It doesn’t say you don’t need documentation. And I’d say the documentation for servicing your application might be the most important one. The normal user who uses your application everyday will figure out a way to get along. If not he will call you or your boss. But the poor support person has to support dozens of the applications and since your application just works he’ll encounter it only a few times a year. He will know nothing about it except the stuff documented in the manual. So make sure there are instructions on how to interpret the logs, how to shutdown and restart the application, how to analyze the internal state of the application and what happens when some connected component fails.

Have you noticed something? The vague non functional requirement ‘supportability’ turned into a nice set of very functional requirements. You can attach a price to that, decide what pieces of it you’ll really need and measure if it really works. And I claim this works with all the much hated non functional requirements.

Evolution

Software development has been compared to many things. I’d like to propose another comparison: Evolution.

Why another metaphor?

A metaphor enables you to think about a problem in a different way, thus possibly gaining new insight. It is also useful for explaining something to someone who otherwise wouldn’t understand what you are talking about. Or to use software development wording: It is an abstraction.

Why evolution?

The ideas of evolution are well known and considered valid in the scientific sense. It is also area of abundant scientific work, which actually might be useful in the context of software development. Compare this to other metaphors like gardening and crafting.

And of course I think it fits.

Let’s start in the beginning. A new software development project starts with ideas. Ideas about what the software will be able to do, ideas about all the features it will have and ideas about in which way the software will be implemented. But on their own these ideas are nothing. The moment you stop thinking about them they cease to exist. If you don’t push the ideas forward, nobody else will.

This is similar to the primordial soup. In which many pieces, i.e. molecules needed for life existed. But these molecules couldn’t yet reproduce themselves. And many of these molecules fell apart before finally some where ‘lucky’ enough to form droplets which met the requirements of the definition of life, although they where so extremely primitive, they can hardly be compared with our ideas of life. Consider the description wikipedia gives.

These would combine in ever-more complex fashions until they formed coacervate droplets. These droplets would “grow” by fusion with other droplets, and “reproduce” through fission into daughter droplets, and so have a primitive metabolism

This could only happen because they existed in a suitable environment: They had enough energy (from the sun and vulcanism) but not so hot that everything was boiled immediately. All the needed materials where there suspended in water, so they could interact easily often and fast.

Compare that to your ideas of the next software project. Maybe you don’t have the energy to drive your idea forward. Maybe you don’t have the right partners to work with, maybe you don’t have the necessary tools. All these might be reasons a software project dies, before its first steps. But if you manage to bring your ideas into a suitable environment, you’ll be able to create a first prototype, a sketch, of what your product will be. This sketch is important. If you make it to complicated you won’t get there within this century, just as the dinosaurs weren’t created directly from the primordial soup.

Once you have this first prototype, things get really interesting. Again you need the suitable environment. But if you have that, features will get added to your application, and it will grow, both in size and in complexity. This in itself will create new ideas for features. Sometimes these features will be added to your piece of software, but sometimes they will form into an independent software project. While your project moves a long and gathers features and complexity, it will move with different speed. Sometimes features will get added on a daily bases. Sometimes you will be busy cleaning up stuff, doing small tuning for days in a row, with hardly any new features, and sometimes you (hopefully) will decide, that a cluster of features really does not help the application, and throw it from your code base.

Again the similarities to evolution are abundant. Life sure grew in size and complexity. And many kind of life could only appear because other forms of life where already there: Almost all animals need oxygen, which only exists in the atmosphere because of plants. Carnivores would get extinct pretty fast if there weren’t other animals for food. Similar things are true for parasites. Evolution also doesn’t proceed with a steady pace. There where times, when new species evolved extremely fast, and of course we all know about the extinction of dinosaurs, which is only one case of many cases where nature decided to drop a couple of features.

I hope you enjoyed my comparison so far. But I claimed, that metaphors like this are actually useful. So let’s see, what we can get out of this one:

1. The environment is important. There is no life on Venus, because it is to hot. There is most probably no life on Pluto, because it is to cold, and there is hardly any life if at all on mars, because it is to dry (among other things). I’d say the same is true for software development. You need a good (integrated development) environment, office, coworkers and boss, to develop great software.
2. If you want to create something really great, a detailed plan is only useful if you are a god who can control everything, and event that is disputed.
3. You can’t expect to control a software development project by changing one aspect of the environment. If you have fish, and want evolve them into animals living on land, drying up the oceans will kill the fish, but not create mammals. But making land available for testing, getting plants there for food, and convincing some of the fishes, that they will be safe from sea-born predators on land will eventually convince some of them to develop lungs. This will take time and constant pushing.
4. Simpler is better. Humans seem to think they own the world. The opposite is true. There are many more species, individuals and kilograms of bacteria then humans. Same is true for ants.
5. How ever careful you engineer the environment, sometimes accidents like the human species just happen. In these cases starting over might be your only option.

Tools

When reading blogs you get the impression, that everybody works in high end environments, using the latest greatest distributed version control system. Writing tons of tests, before they even dream about writing actual code and of course the tests a executed by the continuous integration system after every commit, which happens about 30 times per day and developer. But when I look around in the real world, this is not what I see. Instead the way people work on their code like ancient ‘doctors’. Drilling holes in heads in the hope it will reduce the headache of the patient. It probably did. In many cases in a very final way. I urge you: Don’t let that happen to your code (or your career). Practice solid software development. And in order to help you with that I compiled a simple list of things you really really should do. Those are basic practices. If you don’t even adhere to those then I have only two possible explanations: You are not involved in software development at all. Go away, this blog isn’t for you. Or … you are a dabbler.

• Use a Version Control System (VCS). I am not even going to comment on this one.
• Commit your changes at least once a day. This does not mean you should commit what ever is on your filesystem at 5pm, but you should break your tasks in so small pieces, that you finish one or two of them on a normal day.
• Tag or label everything in the VCS you hand out to somebody outside your team (testers, salespersons and of course customers)
• Have a complete and working build script. This means you can build everything you hand of to the customer by getting the source code out of the VCS and start the script. Necessary adjustments are made in a tiny file which contains settings for the local machine. A well commented template for such a file is in the VCS. And NO, hitting the compile button in your IDE is not the same as a build script.
• You must have the complete environment necessary to run your application under your control. That means if your application needs a database, you have a database available. One per developer that is, or at least one schema per developer. If you need a queue or ten queues, you have those, again once for every developer. If you have systems that you interface to, that can’t be installed once per developer, you have mocks, stubs or similar available. In the year 2009 a single developer database for 5 developers is no longer acceptable.
• You have an extensive set of automatic Unit tests. These test should cover at the minimum the main execution paths.
• Let a Continuous Integration system execute your build script, i.e it compiles your code, executes the automatic tests and builds a setup or jar, or what ever you are deploying.
• Have a specification of what a piece of software is supposed to do, before you try to write the software. You don’t need a full specification of the complete application upfront. Maybe you have just the first user story for the first feature, or only a test. But you must have something that tells you where to go. And where not to go. Flying blindfolded is not the same as being agile.
• Adhere to a style guide that describes your naming conventions, indenting and so on. Ideally this gets enforced by the IDE and automatic tests. Fighting about the content of such code conventions is useless. Not having one is dumb. Remember that your tests are code too, so the conventions apply to tests as well.
• Document how your code is structured. It should at least describe the different layers, and the way two layers may depend on each other, and it should not allow for circular dependencies. These rules as well should be enforced by automatic tests. Even if the language you use doesn’t support packages you should use a similar concept, possibly through naming conventions.

These are my points. What did I miss?

The goal of the Agile Process Maturity Model (APMM) is to provide a framework which provides context for the plethora of agile methodologies and practices out there today.

First I was quite impressed by the BWPS ratio (Buzz Word Per Sentence). I count 6 (counting “Agile Process Maturity Model (APMM)” as one). But then I realized that this is what I would highly welcome: Context for Agile Methods. If this means we get an overview of all the process that you need if you do software development and then markers in this process landscape to indicate various Agile Methods to help with this process, that for sure would be helpful. But how would this be a ‘Maturity Model’?

So once again I had no choice but to read the complete article, including the comments. I must say, I was disappointed. What follows is basically a ranking of agile process, methodologies or what ever you want to call these. Such a ranking explains the ‘Maturity Model’ part, but it completely voids the agile part. What value can a ranking of processes have in a context where the common understanding is that processes aren’t that important anyway?

So my clear opinion is: No thanks we don’t need another maturity model, the existing once are causing enough problems. I know some of my coworkers are reading this blog and are by now probably thinking: “Interesting opinion for the quality manager of our company”. So let me explain why I think ISO 9001, SPICE, CMMI & Co are often doing more harm then good.

The problem is the same as with any kind of measurement: If you measure people or they work, they start to optimize for the measurement. For example if you start to measure code coverage and declare high coverage to be good, people will create automated tests for code that gets created automatically, although in most cases these kind of tests are of very limited use. With all the quality norms the auditors look for proofs that a certain aspect of a process gets executed as intended. This is really difficult to do, expect for the cases where documentation is created. But documentation created only for some audit is actually the exact kind of thing you don’t want to have. It costs money, gets outdated fast after the audit and has no benefit for your project, your company or the customer.

How to avoid this pitfall is a topic for a different post.

Aircraft Carrier

Itay Maman tries to answer the question posed in the title:

Why aircraft carriers are not built using an agile methodology?

And my answer is this:
Because the engineers can know up front that it will float

That’s it. That’s the answer. That’s the point I am making.

I think the question is a good one, yet I do not agree on the answer. The mistake Itay makes is to reduce the task of building an aircraft carrier to the question: “Does it float?”

I haven’t been in any kind of military service. I have never been on an aircraft carrier (with the exception of a real cool flightsimulator with a real fighter cockpit). But I’d think the requirements for an aircraft carrier are rather complex. It should float. It should be difficult to hit with rockets or similiar, it should withstand hits with any kind of projectile, it should carry aircrafts and enable them to start and land. And of course it should allow the many women and man needed for the operation to live on the ship. You can’t calculate this from the plans. And I am pretty sure that every single aircraft carrier is full of signs of improvisation where things didn’t work as planned.

On the other hand: proving algorithms in enterprise software is just as doable as calculating if an aircraft carrier floats. The referenced proof that one can’t write a program that predicts for every program if it stops after finite time is next to irrelevant. Because for the typical applications this question is answered rather easily (ignoring bugs). For a desktop application it’s simply: It does stop exactly iff anybody clicks on the close icon.

One of the comments hints in the right direction:

Another reason that agile processes are not used in the construction of aircraft carriers is that people who sign the contract to purchase an aircraft carrier do not change the size and speed requirement after the ship has already been halfway completed. Nor do the bring back a ship that has already been delivered and say, “We need you to make the deck ten feet longer.”

But I’d expect these questions to be rather typical: “Oh we have this new fighter, it needs a longer deck. Can you make it?” And the people building aircraft carriers will try to do it or at least make a proposal on how to fix the problem. Actually this is the same problem Architects have (the real ones with a capital A). Customer request last minute changes all the time in all businesses of the world.

So where is the difference between building Aircraft Carriers and Software?

The first difference is the cost attached to a change! The cost of a change consists of the following parts:

1. Cost of removing what is already there.
2. Cost of adding what is supposed to be there in the new version.
3. Cost of testing, taking care of side effects.
4. Cost of not being able to use the subject of change during the change.

Number 2 and 3 is more or less the same for aircraft carriers and software. But number 1 and 4 differ a lot. If you replace a piece of an aircraft carrier, you have to remove the original piece, it attached to the ship really well I’d guess. With software it is just hitting the delete button a couple of times. Also during the replacement work, your software is fully available (assuming you are not doing changes directly to a production database). Only when everything is done you have to deploy the software which might cause an outage for limited time. This is very different for aircraft carriers. You can’t start on a carrier, when engineers took that catapult out to replace it with a stronger one.

So one real difference between software development and building an aircraft carrier or any other industry producing real physical artifacts is the cost of change.

Once you realize that fact, the next revelation is: The grass is greener on our side! Aircraft Carrier Builder have to plan extremely detailed and carefully, because everybody involved knows: Changes are expensive. They just can’t make mistakes.

Very different for software developers: If we do something wrong the costs are only those attached for doing it the wrong way, which is rather cheap if you realize it immediately.

Therefore agile methods try to detect errors fast, by getting feedback from the customer as often and early as possible.The waterfall approach tries to prevent changes.

The second difference is in the process of automation: The software industry is the only industry that produces it’s own tools. If a developer finds herself in a need of  a certain kind of tool, she has everything available to write it herself. Especially if she finds herself doing things multiple times, it is very likely this job will get automated soon.

A welder working on an aircraft carrier can’t build a new version of welding equipment. She would need skills very different from the ones used in her day to day job.

How does this affect the choice between agile and not so agile processes? Simply put: in software development everything which is done twice gets automated. So most things we do we, do for the first time. Which in turn makes it hard to predict how long it will take. In an aircraft carrier, you can calculate how many km of welding you have to do, compare that to the time needed for a meter of welding and get a pretty good estimate for the full time needed. The effect gets even more extreme when you build two aircraft carriers. The time needed for the second one should be extremely close to the time needed for the first one. Very different in software: Ctrl-C Ctrl-V done.

So software projects are harder to predict, therefore an agile approach makes more sense, because you are doing a detailed planning only for a couple of days or weeks ahead.

The third and final reason for going agile with software projects, but not with aircraft carriers is: We are immature.  Well not the individuals, at least not each, but the industry as a whole. Depending on what exactly you consider a computer, the first one was build somewhere in the middle of the last century. Aircraft Carriers aren’t that much older, but ships, rockets, flying devices and all the pieces (except the computers and software) making up an aircraft carrier are way older. Which again makes estimating much more difficult.

So these im my point of view the differences between classical industries and the software industry. But what are the consequences? What should we do about it?

1. Be happy that we are working in highly interesting, fast moving industry, that can fulfill customer wishes they couldn’t even dream about 20 years ago.
2. Realize that while change is cheap, it isn’t free.
3. While agile is the way to go for many projects it isn’t for every one. If you are building software medical appliances, space ships, airbags or aircraft carriers you should think twice before going down the agile road.

So I should be pretty happy right? Well in the general sense actually I am, but as a developer, I have the feeling that I am operating on the wrong level of abstraction. A lot of the time I am working directly on the code level. Writing getters, setters, exception handling, logging. There are some areas, where the abstraction level moved considerable upwards in the last years. Persistence comes to the mind. Hibernate is pretty reasonable, although it has it’s problems, and it’s enemies. Another one might be rules engines, I haven’t tried those. But other then that …? Of course there has been progress. A lot of progress. We have hundreds of languages, many with very interesting features. We have a large choice of databases with innovative concepts to choose from, and of course a pletorea of java libraries for almost every purpose. We have IDEs so smart, I sometimes wonder if they are really just another piece of code. So what am I complaining about?

I think it is about time for the next paradigm shift. Why are we still coding objects? An example: I want to build the ubiquitous shopping system. I take a jar I found somewhere on the web to get started. Of course I have my own little idea, about how this thing should work, so I want to tweak the shopping cart a little. So what do I do?

public class MyCart extends somebody.elses.Cart {
    public void setRealyCoolFeature() { ...

But most of the time, this doesn’t work. Because the new attribute doesn’t get saved in the database. Or maybe it does, but the repository returns only Cart’s and not MyCart’s. The gui doesn’t show the new attribute, actually the gui of the component is for swing, but I want a web gui. No problem I can write that, but now I have tons of fireProperty noise in the code, that really nobody cares about. Of course it’s better than the other way round, because it is extremely tricky get fireProperty calls into a class without changing the source files.

Hold on a second I know what you are thinking: You fix that with a little AOP here, and this won’t be a problem if the component was cleanly designed in the first place. Agreed. But my point is: I want to use the concept of a shopping system. I don’t want to deal with technical detail like AOP, IoC and similar. Or from the perspective of the component provider: I want to code the concept of a shopping system, and I do not care, how the users want to store it, how they want to show it, and how they want to extend it. That is what I’d call a new paradigm.

Of course I am not the first calling for the next paradigm since OOP. Actually there are many alternatives out there on various levels: AOP, Functional Programming, Meta Programming on the very technical side. Business Process Modeling, Rule Engine, and Components pretend to operate on a higher level, but sometimes I get the feeling they’re just a glorified version of Basic. None of those come even close, to what I imagine. And I don’t even use drugs to get there. Still there might be a solution in sight. Imagine something along these lines:

If you write a component, you use a language that is much more expressive than java, possibly like groovy but two steps further and with a strong type checking (I’m a fan of strong type checking). In this language you specify the structure (the relationships) of you component. Using the equivalent of annotations on stereoids you specify things like transactional behavior, logging behavior, GUI representation, concurrency options, but on a very high abstraction level. For example for concurrency you don’t specify locks, latches and stuff like that, instead you specify data that is related and there for needs protection against concurrent access.

All this is not compiled into a single jar (or whatever we will call it), but actually ends up in separate ajars (Aspect Jars). So as a user, I can take the differnt pieces, and create a extension, specifying the various aspects in the same way, thereby extending all the aspects I need to extend. This is possible, since the aspects are still separated, and not compiled in one big blob. And the original developer separated those, because it was the easiest thing to do.

The guy actually using the component, somebody else created, and extended be  me, uses the components in a high level DSL, that allows him to glue pieces together. For example he could specify which kind of order and invoice component should get created from the cart component. And although the three components come frome completely distinct developers, they would be joined automatically using domain information from the semantic internet. This joining would of course need some tweaking but it should give you a head start. I think this sounds extremely nice, doesn’t it?

Will it happen? Most certainly not. The description is to specific to fit whatever comes, yet way to vague for anybody to actually implement it. BUT I am pretty optimistic something along these lines will actually happen in the next 10 years. DSLs and AOP and new languages that allow Aspects to be written as easily as a for loop will be key players at it.

So if you are a genius with lots of time and a strong developer team at hand, get into contact with me to exchange ideas. If you just a stupid enterprise application developer as I am that needs a job for the next 30-40 years to come, get used to AOP, watch the new languages and don’t worry to much about concurrency.

1. Gather information. Make sure you have reliable information about
   • Which application has a problem
   • What are the steps to reproduce the problem
   • The version of the application
   • Does it happen only to specific users, specific machines or to everyone everywhere
   • When did the problem start to happen

   The important part is the reliable part. Users are known to have a completely different perspective on applications. I had users swearing there was only one message box although there where two (they didn’t understand the content of the second, but it was the information needed by the developer). The last point is a difficult one. In many cases you will hear things like “Oh it just started today. Everything was fine the last two months” Just to find out, nobody had used that feature before. Probably the best source of information is a clean logging trail.

2. Give a first estimate. Yes, that early. Most of the reasonable users are pretty happy with something like: I give you an update in half an hour. They might not be just as happy with: “I think I can look into this next month” but if this is the case it is at least fair to let them know. If the problem is urgent and the resolution time is long a workaround might be needed. Check your estimate every now and then. If you realize that you can’t deliver on your estimate for whatever reason, let the user know.
3. Make sure the bug is a bug and not a butterfly. Although users sometimes have a different oppinion: Software can’t do what they wish. It can only do what they say. So many times the behavior now classified as a bug was considered a feature when it was specified. If this is the case let the user know, give him a reason why it is specified this way, and inform him about the chances to change it.
4. Reproduce the problem. If possible reproduce it on your machine.
5. Simplify and automate. Often the procedure to reproduce a problem is long, but often some steps aren’t necessary. By finding these step you’ll gather information about what influences the problem and what not. If possible you should automate the reproduction of the problem using a test case. With a serious bug you will execute the steps for reproducing the problem a lot. If this takes 2 minutes, spending 2 hours reproducing the problem with a testcase might easily payoff before the bug is fixed. But even if it doesn’t pay off immediatly you will have another test case in your Test Suite.
6. Now we finally start working on the buggy code. Find the place where the state of the application is not the way it is intended to be. Often this will be just before a NullPointerException is thrown, but it might be a line, where an event should get fired but doesn’t, or where the variable Pi is supposed to be equal to 4 but actually holds the value 2. Add debugging output to show the problem.
7. Find a place shortly before the point found in step 6 where everything is ok. Add debugging output to show the absence of the problem.
8. Find a place between the spots identified in step 6 and 7. Check if the problem is present at that spot. Document it with a debugging statement.
9. Repeat from beginning from step 6 until you find the actual faulty line in the application.
10. Fix it.
11. Test it (should be easy with the automated test from step 5).
12. Let the user know when she will receive the new version.

Every process should have a tayloring advice to accompany it. So here it comes: You may not short cut the fixing, testing and communication part with the customer. You may use some short cuts in steps 6 through 9. But be prepared to return to the clean process when the shortcut turns out to be a cul de sac.

You may use your debugger instead of debugging output, but in the complex cases debugging output is easier and faster to use, especially if you have an automated test.

There are special cases that need some consideration, but those have to wait for a future post: Finding suitable starting points in step  6 and 7; Debugging in a distributed environment; Debugging with 3rd party software; reading stack traces.

What process do you use when debugging? What is missing? Let me know.

While I am glad that there are others worrying about the lack of basic software development tools in the database realm, I still think we are really missing to things.

First: all these tools are using small migrations, and manage those. I think this is like directly writing patch files and then using some tools to assemble class files from those. Why not maintain one file for creating the schema or the changes from one release to the next. It is much easier to maintain and the version control system will split it into changes if you need it.

Secondly: We now have dbmigrate, migratedb, liquibase, dbmaintain, dbdeploy but none seems to get real traction. Liquibase seems to be most active but even that doesn’t cause enough search traffic to show up on google trends and it is so horribly ridden with XML that I don’t even want to look at it.

We need a JUnit equivalent for database release management. We need somebody with some traction in the community to pick one of those tools and push it forward real hard. Pramod? Martin? Anybody? Please!