If you ask more experienced developers about the singleton, they will probably tell you its bad and should be avoided.

What is so bad about the singleton that it fell from the heavon of patterns into the hell of anti patterns?

If you check the description of the singleton pattern in the Gang of Four book it talks about two characteristics of the singleton:

• There can be only  a single instance
• There is a central access point to that instance

And there really is nothing wrong about the first point. Sometimes you really can’t have more then one instance, but even if multiple instances don’t hurt, if they don’t give you any benefit (i.e. if they are stateless) you might as well reuse a single instance.

The problem is really with the central access point. This makes the singleton look and behave like a global variable. And how should we obtain our dependencies? Correct through Dependency injection.

So the next time you think: “Looks like a singleton to me”. Go ahead, create a singleton. But also create an interface which gets implemented by the singleton and make it very clear that nobody may use this singleton except via its interface and dependency injection through constructor.

But everything seems to change when it comes to databases. A lot of stuff in and around databases goes untested. Heck some of the scripts don’t even live in version control. If you can’t believe that, because you haven’t seen it, you are a damn lucky bastard.

At least one reason for this state of affairs is obviously: It hurts. Database tests tend to be slow, interdependent and hard to maintain. Granted. But you know what: If it hurts, you should do it more often. It will teach you what exactly the things are that cause pain when testing and eventually you’ll find approaches to make it less painful. Here are some ideas I found helpful when testing database related code:

• Give every developer her own database. This forces you to find a way to set up the database fast, easy and reliable. If your application lives in a single user/schema/namespace it is sufficient for each developer to have his own user/schema/namespace in a single database. For this to work though …
• … the application should be user/schema/namespace agnostic. It makes it much easier to create multiple instances one a single server.
• Let the application live in a single user/schema/namespace. If you have multiple interdependent namespaces (e.g. for modules) you’ll have a hard time making them agnostic of the names.
•  Have separate instances for CI, Demos, QA and so on. Actually ideally it should be trivial to create a fresh instance.
• Stay away from any tool that comes with its own repository. If have seen about a dozen of such tools and although some looked promising in the beginning, they all completely failed to integrate with other tools on the development side of things. Examples of such tools are tools for code generation from UML or ER models and ETL tools.

The previous points where about the setup of your database and infrastructure. Lets have a look at the code:

•  Favor a proper Language (like Java, C, PHP …) over database specific languages like T-SQL or PL/SQL. If you are wondering why, make a comparison between your favorite DB language and your all purpose language. For which do you get the better IDE, Unit testing support, code analysis, code coverage and so on. Reconsider your main language if it doesn’t win in that comparison.
•  Have a small layer that does all interaction with the database. Make sure no SQL or other database dependent code leaks out. Inject that layer as a dependency into anybody in need of it. This will allow you to test almost everything without worrying about the database. Only the tests for that integration layer actually needs a database for testing.
• Favor database independent SQL or a library abstracting away the differences of various databases. Back in the time people claimed they needed that in case they have to switch database vendors, which never happened. Now it does. See below.

The next points will deal with the actual tests:

• Consider an in-memory-database for testing. Many databases can run in an in-memory-mode. They are great for testing, because you can throw them away after the test, and they are way faster then any database writing to disk. This of course is only possible when you work with a database system that can run as a in-memory-database or if your code is database independent. Hence the previous point.
• If you can’t use your database as in memory database, consider putting it on a RAM disk. We got a huge performance gain for our tests with this approach. A solid state disk might be the next best thing although I’m not sure how it would react to the heavy load of continuous database tests.
• Make test failure messages so explicit that you don’t have to look into the database for analyzing test failures.
• Use code for setting up your test data. Make it nice and concise. If you need a row in a table without special requirements for its values you should be able to create that with a single trivial line of code, no matter how many foreign keys the table contains. I.e. you should have a little DSL for your test data. Doing it with plain code will enable all the refactoring power of your IDE for your tests. For load and performance tests other approaches like loading production data or large amounts of generated data might be suitable.
• Make sure your tests clean up after them selves. There are two approaches I found usable in most cases:

1. Recreate the schema for every test. This is slow but really safe.
2. Do a rollback after each test. This only works when there is no commit inside the test. The mean thing is: If a test tries to rollback, but fails because there was a commit inside the test some completely different test will fail. It can be a really frustrating task to find the bad test in such a case.

We covered the testing of  database related code inside your application. But there is another kind of code we need to deal with: Scripts for deploying your application (or upgrading it)

• The scripts that change your database schema from one version to the next are source code just like everything else. Therefore they belong under version control and should get tested continuously.
• The testing process is really simple: Create a database as it looks now. Apply your change scripts and verify it looks as desired.
• For verifying the resulting schema it is useful to have a script that creates your target database (or at least the schema) from scratch, so you compare it with the result of the test.
• For performance reasons you might want to test this with an empty database first.
• But don’t forget to run it on an instance with realistic data as well. Hint: adding a column to huge tables can take loooooong.

Happy testing everybody.

Am I doing the right amount of up-front design?

I never before thought about this question in my software projects. But I also hardly ever had the feeling of “damn we did to much or to little of it back then”. How can this be possible? The reason might be that in some sense my approach to hitting the sweet spot for the amount of design is extremely easy:

Design as much as possible, as early as possible! But not more or earlier!

So some people might imagine me now sitting for weeks in a row working with UML tools and hammering out diagrams and design documents. Nothing could be farther from the truth. The key is in the interpretation of  when design is possible. For me design is all about solving a problem. So in order to do some designing one needs a problem. If you have found a problem do as much design for it as possible. ‘much’ doesn’t get measured in hours or number of pages or diagrams. ‘much’ gets measured in the extend to which the problem gets solved. If you can design a solution that makes a problem go away. Great! If you only find one that makes it a tiny problem: It will have to do until you find a better solution.

But now we are stuck with trying to find a better solution for a not completely solved problem, right? Wrong! In reality we have lots of problems in a project, so we pick the worst one whip up the solution, until the next problem is bigger then the first.

So if you are spending time with a problem that isn’t the most pressing one, you are doing to much up-front design. But everybody would immediately stop doing it once he realizes it, right? If on the other hand you are knowingly ignoring your biggest problem you are doing to little up-front design. But the more common name for this is sabotage.

Of course it does happen that what you thought is a great solution turns out as a bad idea, or that you just can’t find a solution for a especially bad problem, or a problem jumps at you from behind. Congratulations you found the border of your skills. Its an opportunity to learn an to train you problem identification or solving skills. It is not the time to decide, I should do more of it (or less) next time.

It really is a simple process. Yet software design is considered difficult. So where is the difficult part?

There are actually three difficult parts:

• Seeing the problems as early as possible, so you can solve them while they are small
• Finding good solutions fast.
• Having a great team that understands that every software developer is a software designer

Spring (and other DI frameworks) can inject dependencies into private fields. It looks like people like to use this because it combines some compelling properties:

• Very little boilerplate code. All you need is a simple annotation on the field.
• Since the field is private nobody outside the class can change it.

But now when you want to test your class and want to assign e.g. a mock to the field you have to either setup a Spring context in your tests or use reflection in order to access the field (just as Spring does).

Now something is really starting to smell bad here. One of the purposes of Dependency Injection is to decouple code, which in turn should make the code easier to test. Now that didn’t work out to well.

So how can we clean up this mess? We can write a setter for the field. That solves the testing problem. It adds a little boiler plate code, but hey we are talking Java here, so you should be used to that. But now we have a setter which anybody can call at any time. At the very best that doesn’t make any sense at all in the production environment. In the worst case somebody actually uses it and creates some ugly bug.

So what do you do when you don’t want anybody to change a field? Correct you make it final. Great, now you have a compile time error in the setter.You can only set a final field in a constructor. This leads naturally to the solution of all our problems (and possibly to world peace): Make your dependency a constructor argument!

This is the only place where dependencies really belong. If you try to stick to that rule I’d expect you’ll encounter the following two problems.

1. You’ll find classes with lots and lots of constructor arguments. I’ll bet that class does a lot of different things and really needs to get broken down into smaller pieces.
2. You’ll find cases where a class A needs an instance of B and B needs an instance of  A. This is a typical case of a circular dependency and is obviously bad. In my experience the solution is either to make B a part of A when the two are so strongly dependent that they really should be one class. More often though there is at least one more class C hiding in there so that B doesn’t need A but only C.

So really the problems you’ll find when doing constructor based dependency injection are problems that are present in your current code already. So don’t shoot the messenger, but fix the problems in your code and go for constructor arguments for dependency injection.

Lets start with the small part where I do agree.

Since TDD forces you to write test first, it forces you to think about how you want to use an API, because that is the very first thing you write down. This is a good thing and in my case led to more usable APIs (often in the form of Builders or little internal DSLs).

Since it is almost insane to test anything if it isn’t properly decoupled from stuff it uses, TDD also forces you to factor out dependencies in order to be able to mock them. Thats the other point where TDD encourages good design.

But there are bad news: There is more to software design than just nice to use APIs and SOLID principles: Your design actually has to solve a problem! And TDD does almost nothing for you in order to find a good solution for your problem. Yet this is really the hard part in software design.

Can your domain model represent all the cases needed by the business? TDD won’t tell you.

Should represent some edge cases of the domain in a ‘dirty’ way and thereby making the domain model simpler and therefor 99% of the use cases way easier? Or should you go for the complete but awfully complex solution? TDD won’t tell you.

Which of all the design patterns would help you? Or is there some well known algorithm you could use to solve your problem? TDD won’t tell you.

Does the problem become trivial once you approach it in a functional way? TDD won’t tell you.

Is recursion a solution or a dead end? TDD won’t tell you.

You have to understand the problem and know lots of possible approaches to a problem in order to design software well. There is just no way around it.

Once you have the solution in your mind, once you know the basic data structures and algorithm you going to use, TDD will help you to implement it in a clean, well factored way. It will also help you to stay focused on the task at hand (which is making the next test green).

The pragmatic programmer recommends a tracer bullet or spike in this case. This means you implement some kind of prototype in order to clarify how to implement it properly.

And than you throw that away and implement it properly again, which for me means test driven.

Do you really do that? For me this goes against the principle of baby steps and of refactoring instead of rewriting code.

Also I was fearing it wouldn’t help much, because the problem wasn’t so much that I was unclear about the code, but about how to test it.

So last Friday I tried a different approach. I call it ‘Sketch and Test’

I started by implementing the complete thing in one single class. Without tests (scary, I know). But I never actually executed the code. I just wrote it as I thought it should work.

Once this sketch was finished I looked at the code and considered the changes I needed to make in order to test it.

I did the changes and finally created the tests, fixing the bugs I found in the progress.

Only when all the tests where in place and green I actually started the application and tested the new feature.

It worked, i.e. the feature did what I expected. The code qualifies as clean and I had complete code coverage with the only exception of some weird exception handling code, that I found hard to trigger in tests.

What do you think? Is this a viable approach? Or is it just a symptom of my limited TDD skills? Or both?

JUnit: JUnit is pretty much the basis for everything else. Its THE unit testing framework for Java, with great integration in IDEs, build tools and CI Server. I don’t take the term ‘unit’ to seriously though. I use JUnit to execute all other kinds of tests as well.

Mockito: There are many Mocking Frameworks out there but I prefer this one. It has nice DSLish API and I find it nice to use. Only drawback is that in some special cases the standard API doesn’t work and one has to use an alternative syntax.

PowerMock: I actually try not to introduce this into projects. You need PowerMock if you want to mock constructors, static or final methods. If you need this, PowerMock gets the job done, but its better not to need it.

Jenkins: a free continuous integration server. Not exactly a work of beauty, but it works, is easy to setup and does everything I needed so far with the help of a couple of plugins.

CheckStyle: A static code analysis that finds lots of bad practices and can check lots of coding conventions too. Integrates in IDEs and CI server. There are other tools in this area which are worth considering: FindBugs and PMD. You can also use Checkstyle in order to gather simple metrics about your code.

JDepend: does static code analysis of the dependencies of your code. I use it to write tests against cycles between packages in the code and also in order to limit the dependencies to those I’m willing to accept in the code base. Some time ago I found out there are some limitations in JDepend resulting in dependencies that JDepend misses (I think it doesn’t consider classes in annotations or something). Therefore I’m looking at DependencyFinder, which seems to be way more powerful, but is certainly harder to use. I actually build a little tool for visualizing dependencies based on DependencyFinder.

Cobertura/EclEmma: Cobertura and Emma are code coverage tools. I use them both. Emma in the form of EclEmma as an eclipse plugin and Cobertura in Jenkins, because we couldn’t get Emma to work as we wanted on our Jenkins instance.

Sonar: It collects tons of metrics from your code and makes them available as a website. It actually to much numbers for my taste. In a serious project you can spend the whole day looking at numbers. What is really great about sonar is, that it tracks those numbers over time, so you can see if your average method length goes up or down over time. In some projects I configured a graph with the most important numbers plotted over time and added it to the main screen of Jenkins.

Theory: Here kids learn about different patterns (yes there are patterns in chess just as there are in software development). They learn when and how to apply them. Mostly by looking at examples and analyzing them. They also learn rules, like how to properly convert a pawn into a queen if no queen is available (Hint: turning a rook upside down doesn’t work)

Practice: In this kind of training the kids get isolated problems to solve. Puzzles basically. Like Mate in one. They also do games based on chess rules, but not actually chess. Like collecting gummy bear from the chess board with knights.

Play: Starting from a friendly game against family or other club members to tournament.

I think it maps 1:1 to what you should do if you want to improve your software development skills:

Theory: Read books, read blogs, go to conferences. Learn as much as you can about design, algorithms, network protocols and so on. 90% of all developers neglect this. But since you are already reading this blog, you are probably among the other 10%

Practice: Do Katas, Dojos, spikes, tracer bullets, prototypes. Write code. But not in order to have working code in the hand but in order to learn and explore. I’m afraid 90% of those doing the theory part completely neglect this part. You should not. The first couple of times you try to use something which you have read or heard about, you probably will do it wrong. To bad if this is in a real project with time pressure and lots of money at risk.

Work: If you take care of the first two parts work will still bring some surprises. Your agile approach runs into problems because the customer doesn’t like it. Your nicely normalized schema doesn’t happen because you have to use an old existing database, used by lots of other applications, what worked nicely in the kata fails flat on its face when applied to 1 million records … So there is still a lot of stuff you can learn at work, but for this you have to think about what you are doing at work. Chess players eager to improve their skills analyze games they played in order to understand which move was good and which wasn’t. Do you do that? Do you do a personal retrospective in the evening? Thinking what you did well? What you could have done better?

I think you should.

I think this two fold distinction is not sufficient. You write code. Thats fine. But when reading code, there are actually two different kinds of reading. Lets start with an example:

Even if you didn’t follow my articles about ScalaTest you can understand what this line of code is doing. In this sense reading this line of code is extremely easy. I call this ‘skimming’

But what if this line of code is not doing what you expected? Then you need to really understand the code. Then you want to understand what is going on. In the Scala example above you might consider the following options: Is startWith("Hello") a parameter to should? Or is "Hello" an argument passed to the result you get after passing startWith to should? What is should anyway? A method? An object? A function? An instance with apply method?

As you can see what was realy easy to skim now gets quite challenging to ‘understand’.

The same can be the case for libraries. Consider Hibernate. If you have setup Hibernate, CRUD operations on an entity are really easy to write and understand. But once you get one of the dreaded Hibernate exceptions (LazyInitializationException, NonUniqueObjectException, …) you want to really understand what is going on. And at that point many people realize they have no idea what is actually happening when they call session.update or just the getter of an Hibernate Proxy.

Obviously you want all three kinds of handling code to be as easy as possible, and writing might be the least important, because we read more code then we write. But how important is the ability to skim versus understanding?

With anything you use in an application you will do a lot of skimming. Thats why DSLs like the ScalaTest example above are such a great tool.

I suspect if you do complex things with a library or a language, understanding will always be hard. But in some cases it gets extra hard because non standard tricks are used. In Java a typical case of trick is Reflection and Byte Code Manipulation. Many of the more powerfull libraries use it to work around the limitations of ‘normal’ Java. The result is code that is easy to skimm but for many people next to impossible to understand. Examples are Mockito or Hibernate.

So when evaluating a library or writing one you might consider how much magic it is using.

The other way to limit the problem with ‘understanding’ code is to make it unessecary. I think Hibernate is an example where this failed. You have tons of places where you have to understand pretty exactly what Hibernate is doing in order to use it correctly.

Java itself is an example where this works pretty good. In many, many projects nobody has to actually understand how the JVM works. It just works. Only in rare cases you have to dive into the details of the garbage collector or the memory model.

The key here is a clean non leaky abstraction. If you have that it doesn’t matter what kind of magic you use to make it happen. But if your abstraction is leaky, parts of the magic you use inside will leak out with the risc of turning your project into something that acts like it is bewitched.

This metaphor works well to communicate the need for refactorings if at least one person realized the need for it.

But in various cases nobody in the project realizes that there is a problem until the team face a huge debt which seems impossible to pay back.

I see two possible reasons:

Being blind: There is a different level of cleanliness people consider clean enough. In interviews I have repeatedly talked to people considering a 20 line method a good thing and don’t have problem with nested control structures 5 levels deep. Those developers wouldn’t notice anything smelly when looking at code which I would consider CMD (Code of Mass Destruction). This problem is most of the time fairly easy to fix by teaching and coaching. But there is a more insidious possible reason:

Being a frog in boiling water: There is a theory that a frog which sits in cold water doesn’t jump out of the water if you heat it realy slow until it dies. Wikipedia isn’t decisive if this is actually true, but I definetly see this effect in software development. It looks like this: At some point you find something in your code that looks like it has an easy abstraction. So you build a little class encapsulating that behaviour and use that class instead. It works great, so that little class gets used a lot. Sometime later the class gets a new feature to handle a variation of the original case. And it goes on like this, until one day what was a simple abstraction has turned in a complex library, possibly even a framework. And now that framework is a problem on its own. Its getting hard to understand how to use it.

And you realize you are a poor little frog sitting in boiling water with no idea how he got there. Hint: It’s a good idea to jump even when it is a little late.

Why does this happen? Just as the frog has a problem sensing the small change in temperatur and realizing he is getting into trouble the developer doesn’t see he is hurting the quality of the code base until it is to late.

Again: Why? Let’s make the example a little more specific. You have blocks of 10 lines of simple repetitive code in N places in your code base. You replace it with a call to a simple class of 40 lines of code. So you save (9*N – 40) lines. On the call site your code gets significantly simpler, of course the class is a little more complex but thats ok. Now while implementing a new feature you are about to create another of those 10 line blocks. Obviously you want to use the helper class. But it’s not fit for the job. You need to add a feature to it. That’s ok. You also have to add something to the public API to turn that feature on or of. Maybe it’s a new constructor, a new method or an additional parameter. That’s not ok. Until you changed the API of your class the changes where local to the helper class and its usage at the new site. But when you changed the API, you added complexity to all the call sites of your class. Whenever you now call that helper you have to think a little more about the correct API to use. This unfortunatly isn’t easy to see. So it can easily happen that your API turns slowly so complex that using it is more painfull then just writing the 10 lines it replaces down.