Flip up Shades

For about a decade now everybody in IT talks about Agile, but hardly anybody else does. There is a somewhat similar concept of ‘Lean’ in other industries, but hardly anybody in a ‘real’ industry considers things like ‘release early’ a viable strategy. And that’s a good thing. I don’t want a second iteration car. “Ooops sorry, the story about braking is still in the backlog”. I am confident that if those kind of companies would learn agile from the books, they’d need quite some time until they realize the fine difference between ‘ship’ and ‘shippable’.

But maybe these companies won’t learn it from the books, but from their tools. As said before companies will learn about social media, and to some extend it will invade  daily life and work at these companies. But if you start doing your documentation in a wiki, for everybody to see or spurt out your idea on a internal twitter account, then you are presenting stuff as potentially shippable. Not the car you will eventually build. But maybe the concept for a new feature.

Everybody will be able to judge the results, comment, provide feedback. And without realizing it big old companies will become just a little bit agile.

So today instead of mainly complaining, I’going to make a proposal on how to manage you database in a modular fashion.

One of the strengths of databases is the concept of foreign keys, a declarative way of enforcing relationships.
But when it comes to modularity, foreign keys are the source of all evils. They link two tables with each other and there is no standard way to remove the dependency. In object oriented computing languages this would be done with interfaces.

Instead of Class A directly referencing Class B it would reference a nInterface IB and Class B would implement that interface. We now can replace Class B and Instances of Class A won’t notice. How can we achieve this with databases?

Probably the first idea that comes to mind is “views!”. Views are basically select statements with the name. They do hide dependencies to the underlying tables so that is a start, but they do not correspond to interfaces for two reasons.

First: The dependency between a table and a view is just the opposite way than the dependency between a class and an interface. This means we can’t reference views in our code and simply swap the underlying table.

Second: At least Oracle (and I guess this is true for most databases) can’t use a view a foreign key relationship.
So views don’t help that much.

Let’s look once more at classes, interfaces. One thing to realize is that in the example above there is  a dependency between Instances of A and B but not between their classes. The dependency between the instances is created at runtime. How does that help with databases? Typically classes are identified with tables and instances with rows in tables and that works just fine for ORM tools.

But I want to propose a different mapping. How about considering the create script of the table “the class” and the actual table in a database a instance. If you choose this approach, everything falls into place. The interface becomes a create script that just creates the most essential columns of a table, possibly only the primary key. You now can have different implementing tables / scripts, which either add columns for your normal production version of the table or a more simple version with only a couple of fields. Or you create no additional fields at all.

Obviously this makes organizing the scripts more complex in the first step. What are the benefits? Actually this can help in many ways.

You can use it for testing. With database testing a common problem is the setup of test data. Since everything is connected with foreign keys you need to create tons of master data in order to just create a single row that you actually want to use in your test. Know you replace all tables referenced by the table you want to use in the test with dummy implementations, featuring just the primary key, so no further dependencies and the setup can happen easily, possibly automatically.

You can use it for a modular application. Imagine an application that references some kind of ‘Order’. One customer wants an extremely simple order with just one line item with a fixed quantity of 1. For another customer it is the average order with multiple line items, different units of quantities (string is sold by the meter, other stuff is sold by piece …). The third customer has extremely complex orders with rules governing what combinations of line items can go into a single order. You know this kind of order when you ever configured a car online. The forth customer has a different kind of order, but that doesn’t matter because it is a third party system and your application has just to maintain a reference.

Normally you need a table structure that combines all possible implementations, making it more complex than even the most complex implementation that you actually need. With the proposed approach you’ll have four different implementations and can choose the one you need for each deployment.

So how do we actually do this? Well that is the real challenge. Right now I don’t know of any tool supporting this. One could use Hibernate with a dynamically created Configuration. But there are things in a database that you probably don’t want to control via hibernate (e.g. partitions and tablespaces). You could create your own little script assembly tool, store DDL scripts in a directory structure, that has branches for each implementation variant. A little configuration file would define what implementation variant to use for which object. In any case we would need to stop developing directly in a database. We should work on the file system and deploy into a database even in development instead.

I guess there is plenty of room for more ideas and of course tools. What do you think? Is this an approach that could pay off? Is this old stuff and I just reinvented the wheel? Or is all this completely stupid and should be handled differently?

Martin Fowler hat eine Antwort. Er beschreibt ein Projekt, dass auf genau diese Weise als Festpreis vereinbart wurde:

Slowly but steadily the changes ate up the buffer. After about six months or so the changes had used up the buffer completely. We’d been quite open with Nebbiolo during this whole time, so they weren’t surprised when we told them that we couldn’t afford to eat the cost of the changes any more. During that time we’d collaborated closely with Nebbiolo and they’d grown to trust us. We had no problem finding more money, indeed it took another half million to cover the requirements changes that were needed before we delivered.

At the end of all this Nebbiolo agreed that the fixed scope approach was a mirage, and we would do future projects together using a more flexible charging scheme.

Es gibt zwei Dinge die notwendig sind, damit das ganze funktioniert:

1. Vertrauen. Das Team muss es schaffen das Vertrauen des Kunden zu gewinnen, zu verdienen. Und dies schafft ThoughtWorks durch Offenheit und entgegenkommen.
   

   We [.. ] didn’t charge the client for the change

   Aber damit das funktioniert braucht es etwas anderes:

2. Ein hochgradig fähiges Team, denn nur dadurch, konnte ThoughtWorks mit einem Preis antreten, der günstiger war als der, der Konkurenz und dennoch 50% Puffer enthielt:
   

   We charge much higher daily rates for our people, but since we have better and more productive people, we can actually do the job for less.

Also, ihr wollt agile entwickeln, aber eure Kunden wollen Festpreise? Die Lösung ist “einfach”, sucht euch ein Spitzenteam. Macht ein Festpreisprojekt mit großzügigem Puffer und spielt mit offenen Karten, damit der Kunde sieht wie zuverlässig ihr arbeitet, und wie wenig von seinen ursprünglichen Anforderungen wirklich übrigbleibt.

Und wer schon mal versucht hat Spitzenleute anzuheuern weiß, warum einfach oben in Anführungszeichen steht.

Hardware is usually already there or provided by a central services department. And the services department usually charges an outragous amount.

Hinzu kommt in meiner Erfahrung eine Rigide und unangemessene Budgetplanung. Das extremste Beispiel das ich bisher selbst erlebt habe stammt allerdings aus Uni Zeiten und nicht aus einer Firma, aber das scheint mir mehr Zufall als alles andere zu sein:

Während ich an meiner Diplomarbeit arbeitete war das Budget für Büromaterialien derartig knapp, dass alle in der Arbeitsgruppe ihr Büromaterial selbst mitbringen mussten. Das war nicht so tragisch, da wir nicht viel brauchten, außer Papier, Bleistift, Radiergummi und Anspitzer. Absurd wurde es nur, als just zu dieser Zeit eine neue Workstation angeschafft wurde, da im Hardware Budget noch so viel Geld war und wir das Budget aufbrauchen mussten um es auch im nächsten Jahr wieder zu bekommen. Zur Einordnung, zu der Zeit hatte ich mit einem 486DX2 mit 8MB Hauptspeicher einen ordentlichen nicht ganz billigen Rechner, und die Workstation wurde angekündigt mit: “Es gibt eine gute und eine schlechte Nachricht. Die gute: Wir bekommen eine neue Workstation. Die schlechte: Sie hat nur 168MB Hauptspeicher.” Und das war ernst gemeint.

Mir scheint, dies ist genau die Art von Situation die man mit agilen Methoden versucht zu vermeiden. Anstatt an einem Abkommen zu kleben, das längst veraltet ist, setzt man sich mit den Beteiligten zusammen und bestimmt was für die nächste Iteration, d.h. überschaubaren Zeitraum am wichtigsten ist. Und darum kümmert man sich. Bei einem solchen Vorgehen sollte es kein Problem sein ein paar hundert Euro vom Hardware Budget ins Büromaterialbudget zu verschieben.

Also schickt eure Verwaltungs- / Controlling- / Buchhaltungs- Abteilung doch mal zu einer SCRUM Schulung. Aber informiert schon mal den Notarzt, die sollen den Defibrillator bereithalten, falls Konzepte wie Kundenorientierung und Flexibilität auf die Teilnehmer zu bedrohlich wirken.