• Don’t worry about technology so much. Software development is such a broad field you can’t know everything. I will be able to find some limits of your knowledge. In fact everybody with a couple of years experience will find those limits no matter who the candidate is. If you are any good, you’ll find my limits too. It is not so important what you know, but what you are able and willing to learn. But there are some things you should absolutely know, because they are about yourself:
• Know why you are a software developer and why you want to stay one. What is it that fascinates you about software development? If you don’t know this I’m gonna assume that you are not fascinated by software development at all. That’s OK. I know lots of companies that are looking for exactly this kind of employee. I am not.
• Know what you don’t know. When you are telling me you are an expert in X and I find out you aren’t, this is a big minus. If you tell me you have a working knowledge of Y and I find out this is exactly the case it is a big plus. It is your choice to collect pluses or minuses.
• Know what you learned. As I said I am interested in your ability to learn. So be prepared to tell me what and how you learned during the last project, during the last year and during the last month. What have you read? What did you like, what did you dislike? And of course why?
• Know where you want to go. Do you want to become project managere? Do you want to become a teacher and mentor? Do just want to write code? When you tell me where you want to go, I can tell you if the job offering will match that. If you don’t tell me, I’ll vote for not giving you the chance to find out.

Does this sound harsh and arrogant? Thats becaus it is. One of things I want to reach is a higher quality of software development in my proximity. For that I’m more then willing to learn and to teach, but I’m not willing to waste time with people that don’t even know where they are and where they want to go.

Rusty Car

I just finished ‘Here Comes Everybody’, a must-read for anybody trying to understand what is going on with all this social media stuff. One point Clay Shirky makes, is that  the various web2.0 tools make failing cheap.

If in the 80s you had an idea for a group to form, you had a lot of things to do, many of which cost money: Finding a room to meet, printing fliers or ads to promote your idea and so on. If your idea failed, all that money was wasted.

Compare that to the situation today. Create a webpage, and promote it using facebook, twitter, xing or many of the other tools is free. All you have to invest is your own time. If it fails, nothing is lost. Actually something that fails today might get picked up tomorrow by somebody else and brought to success. Like a wikipedia article which is a stub at first, but then evolves in a well written article, through many mostly small changes an improvements. If one of the changes is bad, again the cost of this is minimal. Somebody will notice it and revert the change. Cost: a couple minutes of online time for bad article and 5 minutes work. Compare that to a typo in a printed encyclopedia. Fixing a typo would cost thousands of euros. So it won’t get fixed until the next revision which might take years to come.

So where is the relationship to the title of this post, namely the ISO 9001? One of the corner stones of modern quality management is to prevent errors to happen. This is a good thing under the assumption that fixing errors is way more expensive then preventing them. For building a car, this is probably still true: “Oh the brakes didn’t work? Probably because we installed any in the first place. Let me fix that right away. What do you mean, you don’t want them anymore? Oh you are dead … I see.” But for building software, for creating documentation for the software, for gathering requirements of the software, this assumption is wrong in many cases.

I am trying to write clean code, tested code, working code all the time and I expect my coworkers to do the same. But that doesn’t prevent me from checking in less then perfect code into the version control system. Because as soon as it available for others to see and to use, they might find bugs, or even fix some. They might provide feed back for improvement or further development of the piece. I would not accept a rule that disallows less then perfect code (or documentation) to be seen by others.

So, is ISO 9001 obsolete? I’d say NO for the following reasons:

• While it is OK to have broken stuff around, it is not OK to leave it that way. An implementation of ISO 9001 may help with making this clear for everybody.
• The agile / social media way of doing things does lead some people to the impression that it is OK to write crappy software. Although it is hard to understand how they come to that conclusion when reading the agile manifesto or any of the well known literature on agile software development, it still happens. Fixed rules inside a company of what must be done before releasing software or any development artifact to a customer.
• While the ISO 9001 is build on the idea of preventing errors it actually isn’t hard coded in a form that requires anything like a Waterfall approach.
• Even in a software developing company there are processes that need attention and that don’t need, and maybe shouldn’t be as flexible as a wiki page (consider the processes of paying the salaries with correct taxes and all)

But there are a couple of things that pop up around ISO 9001: Lengthy specifications and lengthy review processes of that specification. Many companies like to create those, and many people seem to think they are a requirement of the ISO 9001. This is not true. The norm requires that you know what you need, before you build it. A specification of the complete system certainly fits that requirement. But since you don’t create a complete new CRM in an afternoon, you don’t need the complete specification.

If you agree with a customer on a couple of user stories to be implemented in the next two weeks, and write those down on a whiteboard or in a webbased application or on some sheets of paper, any auditor of ISO 9001 conformance will have a problem criticizing that. If you use a whiteboard, you might want to make a photo of that. And if you write it down as a Fitnesse test, the auditor will probably be impressed.

So no, ISO 9001 is not obsolete. What is obsolete are ways of implementing the ISO 9001 that are damaging the reputation of that norm.

The goal of the Agile Process Maturity Model (APMM) is to provide a framework which provides context for the plethora of agile methodologies and practices out there today.

First I was quite impressed by the BWPS ratio (Buzz Word Per Sentence). I count 6 (counting “Agile Process Maturity Model (APMM)” as one). But then I realized that this is what I would highly welcome: Context for Agile Methods. If this means we get an overview of all the process that you need if you do software development and then markers in this process landscape to indicate various Agile Methods to help with this process, that for sure would be helpful. But how would this be a ‘Maturity Model’?

So once again I had no choice but to read the complete article, including the comments. I must say, I was disappointed. What follows is basically a ranking of agile process, methodologies or what ever you want to call these. Such a ranking explains the ‘Maturity Model’ part, but it completely voids the agile part. What value can a ranking of processes have in a context where the common understanding is that processes aren’t that important anyway?

So my clear opinion is: No thanks we don’t need another maturity model, the existing once are causing enough problems. I know some of my coworkers are reading this blog and are by now probably thinking: “Interesting opinion for the quality manager of our company”. So let me explain why I think ISO 9001, SPICE, CMMI & Co are often doing more harm then good.

The problem is the same as with any kind of measurement: If you measure people or they work, they start to optimize for the measurement. For example if you start to measure code coverage and declare high coverage to be good, people will create automated tests for code that gets created automatically, although in most cases these kind of tests are of very limited use. With all the quality norms the auditors look for proofs that a certain aspect of a process gets executed as intended. This is really difficult to do, expect for the cases where documentation is created. But documentation created only for some audit is actually the exact kind of thing you don’t want to have. It costs money, gets outdated fast after the audit and has no benefit for your project, your company or the customer.

How to avoid this pitfall is a topic for a different post.

Every time in my career that I’ve gotten useless feedback it has always been because I hadn’t asked for anything more. Attaching a file to an email and asking for any thoughts the client may have is a sure way to get any thoughts that the client may have.

If you don’t want to start a brain storm about whatever you have created, ask specific questions. When you create mock gui the questions might be:

• Do the colors and layout match your Corporate Identity?
• Are the most important elements easily accessible? How can we improve on that?
• Should we make the fonts larger, smaller or keep them as they are?

When doing a code review questions or tasks might be:

• Look for ill designed methods, that don’t work on a single level of abstraction, or aren’t properly named.
• Is there anything that looks like a bug?
• Is there anything that is difficult to understand and therefor needs refactoring or a comment?
• Is there anything that might result in performance problems?

It is also important to consider if you actually want feedback in order to improve the piece of work, or if you just want a confirmation that everything is Ok and you can consider it done. In the first case ask open questions: “What can we do to improve …?”, “What do you think about …?” If you just want to get an Ok ask closed question, that can be answered with yes or no: “Is … acceptable?”, “Do you agree …?”

Just in case you didn’t followed the link to Des Traynors blog you mist the hilarious video he included in the post, so let me add it here.

• Employees get extra pay for work on billable projects. Undesired result: If an employee is finished with her work she benefits from claiming to still work on the project instead of announcing availability for new tasks.
• Employees get extra pay for the ratio of positive feedback from customers. Undesired result: Negative feedback gets hidden, but the negative feedback is the one you really need to know about.
• Employees get extra pay for the success of their department. Undesired result: Fights between department about who is ‘owner’ of a project; Information hiding between departments.

So should we scrap all key figures? No, I think they are actually quite useful, as long as you never try to tie any kind of automatic action on a key figure that goes beyond some kind alarm, triggering somebody to look into the situation. It also makes the selection of key figures much easier. You don’t have to find a key figure which is fail proof (which you won’t find anyway). Another example: Key figures for Software Development. The aim is to get an indicator when something in the software construction phase is going astray. I propose the following key figures:

• Lines Of Code per man month
• Code Coverage in any of its flavors
• Toxicity

Of course it is a bad idea to base any kind of payment on the lines of code produced. Employees would use cut and paste instead of avoiding duplicate code. The inverse is equally bad. People would probably stop working at all. Yet if you find that one team produces half the lines of code than all the others it is worth a second. Maybe they creating extremely crisp code or they are just lazy, or encounter serious problems. Almost the same applies when code coverage or toxicity deviates from the usual values. If all three numbers evolve as you are used to from other projects, development probably evolves as you are used to. And if something unusual happens it will probably show in the numbers pretty soon.

It really sounds like a trivial thing, but it isn’t. So far I have three times accountered scenarios where tests couldn’t possibly become red. And in all cases this went undetected at least for some time:

• The first time I finally found the problem, when I by excident ran code coverage not only for the code under test but also for the Tests themself. I was quite surprised to find many classes with partial coverage as well with no coverage at all. The partial coverage turned out to be ok. This was due to tests checking for thrown exceptions. Those resulted in segments of the testcode that where never run. But the Tests that never ran at all, well they never ran at all. They were not included in any test suite.
• The next time we were using Teamcity to run our tests on each commit. We were kind of proud of ourself, because the tests where green for quite some time now. We clicked around in the statistic a little just to show to ourself how great we were. But what was that?The number of executed tests wen’t down! Two days ago it went from 396 to 387! Who the heck deleted the tests? Well nobody did. But a change in the code caused an exception during setup of some tests. And this exception was silently ignored by Teamcity. But the associated tests of course didn’t ran, thus reducing the test count. JetBrains is aware of the problem, but it doesn’t seem to be high on the priority list.
• I detected the last on when I tried to gather some code statistics for a project and didn’t get any results. In this case no tests where run at all! The team used the ant junit target with the batchtest nested element. But the pattern they used was wrong. So no tests where picked up. But since no tests failed the build was reported as a success.

All cases where pretty trivial to fix, but also in all cases the not excecuted tests surfaced some problem when they where finally executed. So do yourself a favor and make sure that

• your tests can fail (test first is one way to do that) .
• all your tests do run (using code cover and examination of statistics)

And only than make sure that your tests stay green.

Here is somebody else working on that problem. He is aproaching the problem using FIT or at least based on fit:

That’s the hardest part about testing code that depends on the database: setting up the data the way you want it, and making sure it’s there in a pure and unadulterated form (otherwise, you may get incorrect results). What better way to declare your test data than to put in it tables, just the way it should look in the database? So, we made special set up fixtures that let you declare the name of a table and the columns into which you want to insert your test data. Each row in the HTML or spreadsheet declares a row that will be inserted into the database before the test begins.

Unfortunatly the tool he talks about isn’t open sourced, but I think it is an interesting aproach worth considering when you do database centric software development.

He talks also about code coverage analysis for database code and promises to cover that in a future post in more detail. I do look forward to that post.

One other piece that is typically missing from the database development suite are tools for reporting on test coverage of procedural code by the unit test suite. [...] Our Data Architect, has come to the rescue with an excellent, thorough and flexible solution to this problem. Unfortunately, it’s Oracle-specific, so it will only help you if you work with Oracle. I’ll save the details for another post – stay tuned!

Ein Kommilitone versuchte es mit dieser Antwort:

“Es gibt zwei Möglichkeiten: Entweder es ist ein schon gelöstes Problem, dann hat schon jemand eine Lösung dafür und ich schlage sie nach, oder es ist ein ungelöstes Problem. In diesem Fall ist es auch mit ausgeprägtem Selbstbewusstsein für einen Vordiplomanden unangemessen zu erwarten man könnte es lösen.”

Die Antwort bekommt definitiv einen Bonus für Witz, und für wohlstrukturierte Probleme ist sie auch tatsächlich sehr brauchbar. Wohlstrukturiert heisst alle relevanten Informationen zu dem Problem sind vorhanden und von einer potentiellen Lösung läßt sich verhältnismäßig einfach sagen, ob es sich um eine tatsächliche Lösung handelt. Diese Probleme lassen sich in der Tat am besten lösen, in dem man einen Experten fragt.

Ab viel spannender sind die schlecht strukturierten Problemen: Es gibt unglaublich viele potentielle Lösungen, und ob sie  tatsächliche Lösungen sind, kann man erst sagen, wenn man es ausprobiert hat. Ein Beispiel? Politische Probleme: Sollten wir die Mehrwertsteuer senken um die Wirtschaft zu stärken? Oder Barschecks verteilen? Oder einfach abwarten? Wir haben vermutlich alle eine Meinung. Aber wirklich wissen tut es niemand. Fast alle Probleme die man täglich bei der Arbeit bearbeitet fallen in diese Kategorie: Welches Framework ist für das nächste Projekt das bessere? Struts, Struts2, JSF, Wicked …

Also, wie löst man solche Probleme? Ich kenne zwei Strategien. Die erste nenne ich die Berater Strategie:

1. Es werden ein paar Lösungsalternativen diskutiert
2. Es wird eine Alternative ausgewählt
3. Und diese umgesetzt.

Klingt gut? Ich denke nicht. Warum nicht sollte offensichtlich werden, wenn man sich die etwas ausgefeiltere Methode anschaut, die unter dem Namen S.P.A.L.T.E.N firmiert anschaut:

1. Situationsanalyse
2. Problemeingrenzung
3. Alternativen aufzeigen
4. Lösungsauswahl
5. Tragweite analysieren – Chancen und Risiken abschätzen
6. Einführung und Umsetzung – Maßnahmen und Prozesse
7. Nachbearbeitung und Lernen

Man erkennt die drei Punkte aus der ersten ‘Methode’ wieder. Aber es ist einiges hinzugekommen, was oft (nicht nur von Beratern) vergessen wird. Als erstes wird das Problem erkannt, identifiziert und eingegrenzt, d.h. erst wenn man weiß, dass es ein Problem gibt, und man das Problem kennt, wird versucht eine Lösung für das Problem zu finden.

Und nachdem eine Lösung umgesetzt wurde, wird diese nachbearbeitet, d.h. insbesondere wird kontrolliert, ob das Problem tatsächlich gelöst wurde.

Also: das nächste Mal, wenn euch jemand eine Lösung vorstellt, vor allem wenn er dafür viel Geld haben will, fragt doch einfach mal, was das Problem ist.

First I read Increasing Code Coverage May Be Harmful where Dan Mange describes how adding tests just to increase code coverage might cause problems. Secondly Meera Subbarao asks is code coverage important?

I think there are a lot of problems with the whole code coverage discussion. The phrase with ‘tool’ and ‘fool’ comes to ones mind.

So this is my list of problems I have with 99% of all code coverage discussions

1. Code coverage isn’t properly defined at all. Are you talking about statement, block, path, method coverage? What exact flavor thereof? With out defining it the term code coverage is pretty much useless.
2. Any kind of Code Coverage of 100% percent suggests some kind of completeness of the test. Which is pretty much bullshit. If your tests are wrong, coverage says nothing. If your test is correct (in the sense of checking for the correct result) it doesn’t mean the code will have the correct result in all possible cases. Other things like configurations, annotations and third party libraries, including operating systems and hardware often go completely untested although some metric says we have a code coverage of 100%. So it is important to realize 100% code coverage of your preferred flavor is just some completely arbitrary degree of completness for the test suite.
3. Closely to related to the second point is the assumed ideal of a completely tested code base. Such a complete coverage in the sense of guaranteed surfacing of every possible contained bug is impossible. No matter how trivial you program is. Even for the most trivial program it is easily possible to construct hundreds of testcases.
4. In most projects only automated unit tests contribute to reported code coverage. But normally there is a lot of manual testing to accompany that. So while 100% coverage doesn’t mean there are no bugs, 30% doesn’t mean the software is bug ridden.

So where is the real target for testing? As so often it is a tradeoff. A trade off between effort invested in development and maintanance of test code vs. uncovered bugs in the code base. So one needs to define the limitations in budget and the limitation in bugs to stay undetected in the codebase. Which again are constraint by feasability: Some kind of bugs are difficult to find by tests: Bugs in extremly simple code could hide, when the test code is more complex and therefore errorprone then the code under test. Other Bugs you cannot possibly find by tests: Features that are not required yet implemented (called surprises by Robert V. Binder in his book Testing Object Oriented Systems: Models, Patterns and Tools (Addison-Wesley Object Technology)) as well as not implemented but required features.

When you are working on software that does not control life support systems nor large amounts of money I’d consider it quite naturally that quite a piece of the code isn’t covered by automated tests at all but the tests concentrate on the most complex most error prone pieces of the software.

So what to do? Whenever your actions are controlled by just one number, stop it. Code Coverage is a valuable tool to find areas that need testing.

But before blindly writing tests ask some questions first.

• Do we need to increase the quality (in the meaning of decreasing the number of bugs). Ok most of the time the answer is a big yes to this one.
• How much do we want to increase it.
• How much are we willing to pay for that
• What is the best way to increase the quality? Increase code coverage? Increase thoroughness of the test? Which in turn leads to the question of fault models (what kind of bugs are we looking for) and test strategies (how to hunt bugs effective AND efficient)

And if the team doesn’t want to answer all these questions but still increase the code quality through test, I’d opt for some training possibly by reading the already mentioned book by Robert V. Binder. I have some critism for the book as well, but that must wait for another post. It at least clarifies some facts about what tests can achieve and what they can’t achieve.