Tools

When reading blogs you get the impression, that everybody works in high end environments, using the latest greatest distributed version control system. Writing tons of tests, before they even dream about writing actual code and of course the tests a executed by the continuous integration system after every commit, which happens about 30 times per day and developer. But when I look around in the real world, this is not what I see. Instead the way people work on their code like ancient ‘doctors’. Drilling holes in heads in the hope it will reduce the headache of the patient. It probably did. In many cases in a very final way. I urge you: Don’t let that happen to your code (or your career). Practice solid software development. And in order to help you with that I compiled a simple list of things you really really should do. Those are basic practices. If you don’t even adhere to those then I have only two possible explanations: You are not involved in software development at all. Go away, this blog isn’t for you. Or … you are a dabbler.

• Use a Version Control System (VCS). I am not even going to comment on this one.
• Commit your changes at least once a day. This does not mean you should commit what ever is on your filesystem at 5pm, but you should break your tasks in so small pieces, that you finish one or two of them on a normal day.
• Tag or label everything in the VCS you hand out to somebody outside your team (testers, salespersons and of course customers)
• Have a complete and working build script. This means you can build everything you hand of to the customer by getting the source code out of the VCS and start the script. Necessary adjustments are made in a tiny file which contains settings for the local machine. A well commented template for such a file is in the VCS. And NO, hitting the compile button in your IDE is not the same as a build script.
• You must have the complete environment necessary to run your application under your control. That means if your application needs a database, you have a database available. One per developer that is, or at least one schema per developer. If you need a queue or ten queues, you have those, again once for every developer. If you have systems that you interface to, that can’t be installed once per developer, you have mocks, stubs or similar available. In the year 2009 a single developer database for 5 developers is no longer acceptable.
• You have an extensive set of automatic Unit tests. These test should cover at the minimum the main execution paths.
• Let a Continuous Integration system execute your build script, i.e it compiles your code, executes the automatic tests and builds a setup or jar, or what ever you are deploying.
• Have a specification of what a piece of software is supposed to do, before you try to write the software. You don’t need a full specification of the complete application upfront. Maybe you have just the first user story for the first feature, or only a test. But you must have something that tells you where to go. And where not to go. Flying blindfolded is not the same as being agile.
• Adhere to a style guide that describes your naming conventions, indenting and so on. Ideally this gets enforced by the IDE and automatic tests. Fighting about the content of such code conventions is useless. Not having one is dumb. Remember that your tests are code too, so the conventions apply to tests as well.
• Document how your code is structured. It should at least describe the different layers, and the way two layers may depend on each other, and it should not allow for circular dependencies. These rules as well should be enforced by automatic tests. Even if the language you use doesn’t support packages you should use a similar concept, possibly through naming conventions.

These are my points. What did I miss?

1. Know your audience. If you look at the log files of many applications you quickly realize who the intended audience is: Developers. The log is full with technical detail, but not much of it makes sense to anybody else. The groups you probably should have on your list are: Developers on the search for bugs, performance issues, Administrators trying to install, start or shutdown your application and Support personal trying to detect and analyze problems or trying to verify that a problem did not occur. Once you realize there are different audiences you’ll understand what kind of information is helpfull in the logs.
2. Use a framework. The major options are: Log4j, SLF4J, JUL, Commons Logging or Logback. Each one has it’s pros and cons, but they are all better then System.out.println. They allow you to configure at deploy or runtime, what gets logged where and in what format. If you don’t want to do any research, and don’t have special requirements, use SLF4J.
3. Logging must be rock solid. If a logging causes problems. Throw it out. Don’t do experiments like remote logging, if you can avoid it.
4. Define rules, when to use which Log Level: I propose the following:
   • DEBUG: Not important, except if you are researching a very specific problem.
   • INFO: General information like startup, shutdown, version and configuration of an application. Information about important decisions in the code or changes in the configuration.
   • WARN: Things that are fishy, but don’t represent a problem in themself, but might indicate a problem when appearing very often. E.g. a failed log in attempt is such a candidate. If it happens sometime it is not a problem, but if it happens a lot somebody might want to look into that.
   • ERROR: This should only be used when a problem occurs that somebody needs to look into. Typically but not necessarily relating to an exception in the code. Use this when only a limited part of the application is affected.
   • FATAL: If something goes completely wrong, rendering the application unable to recover this log level is appropriate, very likely accompanied by a shutdown of the application or the session.

   Of course your rules need some tayloring for your context, but absolutely need rules, and every developer must know and understand those.

5. Log everything you log exactly once: When analyzing log files I often see log messages that always come in groups. This happens often when exceptions get caught, logged, and rethrown. Don’t do that. Log at the place where you actually do handle the exception. This is the only way to log an exception exactly once and also being able to decide which log level to use.
6. Include useful information and data: The information should typically include answers to the following questions
   • Why are you logging this, typically the root cause of an exception.
   • What is the affected use case? Did it happen during creation of an order or when importing product data from a remote site?
   • What effect has it on the use case that was executing? Does the user need to repeat the action, or did it succeed? Did the complete batch fail, or just a single step?
   • What instance of the use case was effected? For example an order id.

   In order to make all this information available to your logs MDC (Mapped Diagnostic Context) is tremendously helpful. If you don’t know MDC, follow the link and read about it NOW. MDC exists also in SLF4J and Log4J and should be easy to port for other frameworks as well.

7. Use more then one logging hierachy: The link contains more information about why. The simple idea is, that you can seperate different aspects you want to log by using different prefixes. This becomes especially usefull when adhering to the following points
8. Log the time needed for stuff which might be intersting for tuning: Of course this includes the time used for access to remote resources (e.g. webservices or databases) and execution of complex algorithms, but possibly also the time spend in the various layers in the application. You can use different log level for three categories, a separate logging hierarchy for separating performance logging from other stuff. Performance problems a notorious difficult to hunt down, especially when they appear over time. You’ll love this logs, when performance becomes an issue.
9. Log stuff that worked, not only what failed: Distributed systems become common place more and more. When debugging distributed system one often has the situation that requests just disappear, and it can be extremely challenging to decide which component failed. After all not everybody does such a fine logging as us, right? So I recommend the following log events:
   • RECEIVED: The application received a request through the UI, a Webservice or any kind of interface offered.
   • SEND: The application send a request to some other component.
   • GOT ANSWER:  The application got a reply to a request it send.
   • ANSWERED: The application answered to a request it received.
10. Log how often stuff happens: One possible reason for decreasing performance is a changed usage pattern. So logging how often per day, hour or minute a certain kind of use case is triggered can help tremendously.

While obviously I recommend a lot of logging, it isn’t free of cost. So when doing some logging you should be sure to avoid it in tight loops that get executed millions of times. Also MDC keeps a reference to everything you put into, so make sure to clear it when you are done and think twice before putting complex objects in. And finally think about data privacy. A lot of people might have access to the log files. They shouldn’t find passwords, credit cards or similar information their.

Probably somewhere around point number 8 you might have started to think: “Man this guy is crazy. I’ll have more highly repetitive logging code than anything else in my application!” And if you do everything by hand you are probably right. But if you have cleanly structured code where you can deduct use cases, application layers, access to remote connections and so on from the method signature and the packages involved, most of these logging statement can get injected using AOP. So what is left is to create a framework that handles all this.

Since I couldn’t find an Open Source Solution I started my own: Meta Logga. It is still in an extremely early stage, definitely not ready to be used, but possibly ready to understand where I am heading.

If you agree with me that this is indeed something that would be useful you can help in many extremely easy ways. Comment on this post, tweet it, dzone it, blog about or in any other way make this post more visible to others. All this will help me to keep motivated, thus it will help the project.