Schauderhaft

Where is your Mentor?

Jens Schauder — Sun, 06 May 2012 12:44:44 +0000

Back in the days when I had my lack of experience as an excuse to explain my mistakes I was sometimes wondering: “Wouldn’t it be nice to have a mentor?”

What I didn’t realize until a couple months ago was that my mentors were all around me all the time.

My cousin who introduced me to computers
A coworker who pointed out the limitations of VB to me
A coworker in California who tought me lots and lots of things about Oracle databases and how to have fun.
My boss(es) nudging me more or less toward project management
A coworker making it very clear to me that there are alternatives to project management
Another coworker who taught me even more about databases
Dozens, no hundreds of people writing blogs for me to read, questions to be answered or answered questions written by me in various forums.

Its just that I never realized they were my mentors (nor did they in many cases). By not realizing this I missed out on some greate opportunities to learn much much more, much much faster than I actually did.

So let me be your mentor for a second and give you a little task:

Think for a moment: Who in your vicinity is where you want to be in a couple of years? Who can help you getting there? Whom are you learning from? Whom could you learn from?

Now use my advice and just ask her to become your mentor.

Challenges in Requirements Analysis: Finding and Understanding the Correct Terms

Jens Schauder — Sun, 22 Apr 2012 12:18:04 +0000

There are lots of things that make requirements analysis difficult. For this article lets focus on one issue:

What is a Train?

At first sight this looks like a rather simple question and might be answered by having a look at wikipedia:

A railway or railroad train is a connected series of vehicles for rail transport that move along a track (permanent way) to transport cargo or passengers from one place to another place.

This is pretty much what most of us understand when we talk about trains. But when we talk with experts in the various domains that involve trains we might come along these definitions:

A train is a collection of wagons that is assembled during a given time before departure.

A train is the abstract thing that has a unique number and travels on specified weekdays at a specified time from one station to another.

A train is the thing that has a unique number and travels on a specific date from one station to another.

A train is the thing that at a given date and time arrives at a given station and leaves a couple of minutes later.

… and many more

Obviously the various possible definitions of the term train are somewhat related but different. Most of the time experts do have precise words to differentiate between these concepts, but since the different definitions are relevant for different groups of experts each expert calls the concept he is dealing with just ‘train’.

So when you try to gather requirements and designing software you must try to identify these different concepts. How would you do that?

Listen carefully: Often people use additions to a main term in order to differentiate between different concepts. Examples might be: “train in preparation”, “train at a station”

Look out for contradictions: Does somebody describe a train as something repeating every week while another one says something happens at departure? They are probably talking about two different kinds of train.

Collect constraints: What identifies a train uniquely? A number? A date + a number? A date and a track? How does a train relate to a station? 1:1? 1:2? 1:n? Ask the experts for such constraints, ask them if they agree with the constraints identified by others. Don’t stop when you have one answer, but verify it over and over again. It really sucks when 95% of your users use one definition and 5% use a different one. It sucks even more when you find out about it on deployment day.

Look at the documents and systems in use right now: Often header for reports or forms (paper or digital) give cues for constraints. But also intensely used comment field sometimes with obscure DSL like syntax are pointers to a missing concept. The same holds for abused fields. For example when you have a field that contains mostly the number 10, 20, 30, 40, 50 and 60 sometimes the numbers in between and in few cases the value 999 or 998 I am willing to bet that the records are used for at least two different things and boths aren’t properly modeled as a number.

Find and use the correct termsDomain-Driven Design: Tackling Complexity in the Heart of Software

TDD, do we really have to do it?

Jens Schauder — Sun, 15 Apr 2012 07:48:06 +0000

Many really smart people tell us we have to use TDD if we want to create code that is worth a penny. For example take this article about TDD by Uncle Bob (one of the moderate ones). He describes the benefits of TDD in paragraphs like this:

If all your code works every minute, how often will you use a debugger? Answer, not very often. It’s easier to simply hit Ctrl-Z a bunch of times to get the code back to a working state, and then try to write the last minutes worth again. And if you aren’t debugging very much, how much time will you be saving? How much time do you spend debugging now? How much time do you spend fixing bugs once you’ve debugged them? What if you could decrease that time by a significant fraction?

But the one thing he does not provide is anything close to a proof. He offers his own experience, which might be distorted for all kinds of things.

Yet with a practice so simple (note: I didn’t write easy) as TDD and so many people advocating it one would like to see some proof of the benefits of TDD. After writing a little piece about opportunities for more research in software development I was recommended this the book: Making Software: What Really Works, and Why We Believe It. It is highly recommended for anybody interested in what works and what doesn’t work in software development. Just be warned it is a little harder to read than Harry Potter

There actually is a chapter about TDD in it. The result: Not a clear win. I’m not even trying to put the details in here, but the result is basically: Although they tried they couldn’t prove TDD to be as effective as many people claimed. But they also showed it is not as costly as many people claimed. And I think they mist at least one point: TDD can teach you a lot about software development and software design on the micro level.

So what is the answer to the question in the title: Do we have to do TDD?

Nope, don’t think so. There are alternatives, like writing your tests after the code. But I recommend TDD to everybody to try it at least for some time. Its a great chance to learn a lot.

Advanced Database Constraints: Don’t Look for a Second

Jens Schauder — Sun, 08 Apr 2012 07:31:28 +0000

One of the powers of RDBMS as we use them today are constraints. I use unique indexes, not null constraints and foreign keys on a regular basis and if you do any work with RDBMSs you probably do as well.

From time to time one comes at a point where you think “it would be nice to enforce this with some kind of constraint”, but you can’t. Well maybe you just didn’t try hard enough. This is the second part of the mini series about somewhat special constraints for (Oracle) databases.

Let’s assume you want to model a system of superheroes an villain. Each superhero has an arch_villain and each villain has an arch superhero. Thats simple, isn’t it?

CREATE TABLE super_hero (

id number(10),

name varchar2(200) NOT NULL UNIQUE,

arch_villain_id number(10) NOT NULL,

CONSTRAINT super_hero_pk PRIMARY KEY (id)

);

CREATE TABLE villain (

id number(10),

name varchar2(200) NOT NULL UNIQUE,

arch_hero_id number(10) NOT NULL,

CONSTRAINT villain_pk PRIMARY KEY (id)

);

ALTER TABLE super_hero

ADD constraint arch_villain_fk

FOREIGN KEY(arch_villain_id)

REFERENCES villain(id);

ALTER TABLE villain

ADD constraint arch_hero_fk

FOREIGN KEY(arch_hero_id)

REFERENCES super_hero(id);

Not so fast. Try to insert a row into these tables. It doesn’t work. Before you can insert the first super hero you have to insert a first villain to serve as a arch villain for our super hero. But this arch villain needs a super hero first for use as arch super hero. A vicious circle.

Looks like we have to drop at least one of our constraints. Actually no. We can make it a deferred constraint like so:

CREATE TABLE super_hero (

id number(10),

name varchar2(200) NOT NULL UNIQUE,

arch_villain_id number(10) NOT NULL,

CONSTRAINT super_hero_pk PRIMARY KEY (id)

);

CREATE TABLE villain (

id number(10),

name varchar2(200) NOT NULL UNIQUE,

arch_hero_id number(10) NOT NULL,

CONSTRAINT villain_pk PRIMARY KEY (id)

);

ALTER TABLE super_hero

ADD constraint arch_villain_fk

FOREIGN KEY(arch_villain_id)

REFERENCES villain(id)

deferrable initially deferred;

ALTER TABLE villain

ADD constraint arch_hero_fk

FOREIGN KEY(arch_hero_id)

REFERENCES super_hero(id)

deferrable initially deferred;

Now we can enter our data in a pretty natural way:

INSERT INTO super_hero(id, name, arch_villain_id) VALUES (1, 'Spider-Man', 2);

INSERT INTO villain(id, name, arch_hero_id) VALUES(2, 'Green Goblin', 1);

commit;

What happend? Are the constraints still there? Yes they are:

INSERT INTO super_hero(id, name, arch_villain_id) VALUES (1, 'Spider-Man', 2);

INSERT INTO villain(id, name, arch_hero_id) VALUES(3, 'Green Goblin', 1);

commit; – fails with ORA-02091: transaction rolled back ORA-02291: integrity constraint (USER_1DA56.ARCH_VILLAIN_FK) violated – parent key not found : commit

But they are only enforced on commit, not after every DML statement like it is done normaly.

Many Java developer know this feature due to a bug in hibernate that causes illegal intermediate inserts/updates. Most database developers know this feature more as a performance tuning tool. When you insert or update millions of rows Oracle has to executes additional statementes in order to verify constraints. This can cost a considerable amount of performance and often can be done much more efficient when one waits until all the data has its final state.

What Metrics to Use

Jens Schauder — Sun, 01 Apr 2012 18:04:13 +0000

When setting up a project with more code than fits one a piece of paper I like to have some code metrics around.

The purpose of that metrics is two fold:

Identify areas in the code base that need improvement.´

Motivate team members to improve those areas.

But what metrics should one use? I tried to use Sonar but found it to be of very limited use. The problem is that it produces just to many metrics. You can spent hours browsing, slicing and dicing these metrics without really learning anything from it. Once again: Less is more.

I also used toxicity and I still like it. But I also learned from the book Making Software: What Really Works, and Why We Believe It that all these fancy metrics are really pretty much equivalent to lines of code when it comes to predicting faults in your software.

So here is my first code metric I recommend: Size. I don’t think it matters to much what exact measure you use, but you should have a measure of size for the various artifacts in your project. For a typical java project you might end up with:

Number of classes per package

Lines of Code per class

Lines of Code per method

The second metric I’d like to see in my projects is Test Coverage. I don’t expect test coverage to be a useful predictor of faults in the software, but writing unit test teaches you a lot about modular design and if you know about modular design and have to write tests for your code I expect this combination to result in the application of modular design. That’s why I have an eye on test coverage. Oh and good tests might actually help in maintenance. There are tons of different kinds of test coverage metrics. I’d choose the one that is most difficult to get to 100% and is provided by your tooling. I’d expect that to be either path or branch coverage.

In the book Making Software: What Really Works, and Why We Believe It there is a chapter in which metrics are analysed for their ability to predict faults in software. One of the metrics that actually works is the amount of change: If you change a file a lot it is more likely to contain bugs. Therefore I’d like to have The Number Of Commits During the last N days as a code metric. Unfortunately I have no easy way to make that available to a project team, but I assume for now this is just because I haven’t looked yet.

I think with these three metrics you should be all set for the first purpose I mentioned above.

But what about the motivational part. Is a test coverage of 85% motivating? I don’t think so. What I found motivating in the past is the visualization of metrics over time. Measure the three metrics mentioned above for every day and make a graph out of it. Display it on the start page of your CI build. I find this part especially nice for systems that aren’t in a very healthy state as far as code quality is concerned. It is very frustrating to look at the toxicity of your project and thinK: Oh man we have to get rid of 5739 points of toxicity. But when you can change that in: “Cool while adding this feature I refactored this class and now the toxicity is ten points lower then a week ago it becomes actually motivating. At least for number geeks like me.

Advanced Database Constraints: There Can Be Only One

Jens Schauder — Sun, 25 Mar 2012 16:17:56 +0000

One of the powers of RDBMS as we use them today are constraints. I use unique indexes, not null constraints and foreign keys on a regular basis and if you do any work with RDBMSs you probably do as well.

From time to time one comes at a point where you think “it would be nice to enforce this with some kind of constraint”, but you can’t. Well maybe you just didn’t try hard enough. This is the first in a little series of articles pointing out some tricks you can do with database constraints.

I’ll talk Oracle here since that is what I know best, but many of these things are possible in other systems as well.

Lets assume we have a person table and a hobby table which holds the hobbies of a person:

CREATE TABLE person (

id number PRIMARY KEY NOT NULL,

name varchar2(200) NOT NULL

);

CREATE TABLE hobby (

id number PRIMARY KEY NOT NULL,

person_id number REFERENCES person NOT NULL,

name varchar2(200) NOT NULL

)

Just ignore that the hobby table probably needs some normalization. Now assume you want to flag the favorite hobby of a person and of course there can be only one favorite hobby per person.

You could create an additional table holding that information, linking person entries and hobby entries, with a unique constraint on the person_id. From a theoretical point of view this is the correct normalized way to do it. But it makes you add a complete table when you just need a flag. There is a somewhat denormalized way to do it, which is in cases like this one simpler to use.

ALTER TABLE hobby

ADD is_favorite varchar2(1) UNIQUE;

ALTER TABLE hobby ADD constraint check_is_favorite

CHECK (is_favorite = 'Y');

The idea is to add a is_favorite column and make it unique, but also only allow a single value different from null using a check constraint. Since null values don’t get indexed you can have as many null values in a unique column as you want. This does look probmising, but there is a problem. The is_favorite column should only be unique per person, so we have to include the person_id. But now all the null values get indexed, since a unique key only ignores null values when all indexed columns are null. So we have to replace the null values with something unique. We can do that by creating a function based column which is identical to is_favorite for not null values and replaces the null values with the primary key (which of course is unique by definition).

ALTER TABLE hobby

ADD is_favorite_unique AS (nvl(is_favorite, id));

ALTER TABLE hobby ADD CONSTRAINT unique_favorite_hobby

UNIQUE(person_id, is_favorite_unique);

If you remove the unique constraint from the is_favorite column you finally have the desired behavior:

CREATE TABLE person (

id number PRIMARY KEY NOT NULL,

name varchar2(200) NOT NULL

);

CREATE TABLE hobby (

id number PRIMARY KEY NOT NULL,

person_id number REFERENCES person NOT NULL,

name varchar2(200) NOT NULL

);

ALTER TABLE hobby

ADD is_favorite varchar2(1);

ALTER TABLE hobby ADD constraint check_is_favorite

CHECK (is_favorite = 'Y');

ALTER TABLE hobby

ADD is_favorite_unique AS (nvl(is_favorite, id));

ALTER TABLE hobby ADD CONSTRAINT unique_favorite_hobby

UNIQUE(person_id, is_favorite_unique);

INSERT INTO person VALUES (1, 'Jens');

INSERT INTO person VALUES (2, 'Alfred');

INSERT INTO hobby (id, person_id, name, is_favorite) VALUES (10,1, 'coding', 'Y');

INSERT INTO hobby (id, person_id, name, is_favorite) VALUES (20,2, 'soccer', NULL);

INSERT INTO hobby (id, person_id, name, is_favorite) VALUES (21,2, 'reading', NULL);

INSERT INTO hobby (id, person_id, name, is_favorite) VALUES (22,2, 'swimming', 'Y');

– this fails with a unique exception

– insert into hobby (id, person_id, name, is_favorite) values (23,2, 'watching highlander', 'Y');

SIP-18. Why?

Jens Schauder — Sun, 18 Mar 2012 17:18:11 +0000

SIP-18 is a Scala Improvement Proposal. Its intent is … well … to be honest. I don’t get it. But the plan is to introduce a feature in Scala which would disable some of Scalas features. In order to turn these features back on you would have to import them with statement like these:

import language.macros

import language.{structuralTypes, existentials}

So this feature is rather simple and kind of obvious to understand. But what is its intent? The proposal itself says:

The hope is that this will provide a good balance between the wish to provide the most powerful abstraction facilities possible and the wish to control of these features by making their usage more explicit.

Smart people say when looking for the real reason for anything one shouldn’t stop at the first answer to the question “why?” but instead ask again: Why does anybody whish to make the usage of these features more explicit?

Knowledgeable people seem to agree that you can use these features without actually wanting to with quite unfortunate side effects:

It’s quite easy for structural types to creep into a public API unintentionally [..] To give you an idea of the magnitude of the problem, the textual representation of the inferred type was 225kb. Yes, kilobytes.

So there is actually good reason for a way disable some features for those that don’t want punch holes in their feet. We have a problem and we have a solution so everything should be fine, right? I don’t think so.

Lets look at possible alternative solutions first.

code reviews and proper education of developers … mmh, ok that won’t work in most cases.

tools separate from the the language. I think Scala has a perfect setup for this. It allows compiler plugin and naive as I am I would think it should be perfectly feasible to write a tool or a compiler plugin that checks for usage of certain features. It would easily enforceable by a continuous integration setup which by now everybody should use who is working on some serious code.

Lets look at the consequences of the proposed solution second.

It is another language feature, so it makes the beast that is so difficult to tame for many even more difficult. And it is a bad kind of feature in itself. It uses an existing Feature (Imports) in order to do something completely different than the normal usage does (enabling of features vs. managing of namespaces)

It is zero protection for those not understanding the risk or willing to take it or both, even when they are only the minority of the team (note this is not possible with a compiler plugin or a separate tool used in the CI build)

It makes usage of the existing features even more difficult, because now all the little hints and tips found in the internet fail to mention that from version X on you have to make an import to use a certain feature. I’m already waiting for the stackoverflow questions “I want to do Y and do just as described HERE and it doesn’t work”

There is a never ending discussion going on if Scala is complex or complicated or difficult. I think this SIP acknowledges this to be true. Which is a good thing (the acknowledgment). After all you don’t make guns saver be telling everybody these things are harmless.

Martin Odersky, Scala to stay a lean language:

In the future my ambition is to make Scala an smaller language, not a larger one.

Why would he want to introduce such a weird feature of dubious value?

As said before: I don’t get it. Anybody wants to explain it to me in the comments? Serious, it would be welcome.

About Estimating

Jens Schauder — Sun, 04 Mar 2012 08:46:42 +0000

One of the most difficult tasks in software development is estimating also known as guessing. Here are a couple of thoughts about the topic.

Estimates aren’t Commitments. At least they shouldn’t. Lets assume you estimate you need 1 hour from your home to the airport, through all formalities and into the plane. Do you leave your house 1 hour before departure of the plane? Probably not. You’ll add all kinds of buffers and leave your house a lot earlier. Same should apply to any kind of project. If you think it takes 6 months it is a bad idea to promise a delivery date in 6 months. Promise 7, 9 or 12 months, depending how bad it is when you show up late.

Estimates aren’t measurments. If you want to build a shelf, you can measure the space you have available, and if you measure properly you can cut the boards according to that measurement and they will fit. If they don’t it is probably your fault because you didn’t measure properly. With the tools in your house you might be able to measure size up to a couple meters with an error of about 1/1000 and if you want to measure more precise you just have to invest in better tools and measure more carefully.Not so with software. If you estimate a task to take a day you might be done after a day. Or maybe after 6 hours. Or you give up after 3 days. There is no way to be sure. You can’t just put more effort in estimating to make it more precise. Tools won’t help you much. The only thing reducing the margin of error is actually implementing it.

Some basics in statistics. If you roll a dice once you will score approximately 3.5 points. With a margin of error of about 100%. If you roll the dice 1 million times the average value will be 3.5 with pretty good precision. This is no accident, but pretty strong math. The basic idea is that you can add up indipendent random events and that the relative uncertainty is smaller for the added up events than for each single event. This is good news since if you estimate a project you typically break it down into tasks and estimate each task. Then you add up everything and get your result. Which has a smaller range of error.

The rule above applies does not apply. It holds only when the events (guesses/estimates) follow a normal distribution and are independent of each other. Neither is true for software estimates. Go figure.

We learn through feedback
It’s almost impossible to learn anything without getting feedback on how you are doing. Imagine learning chess without anybody ever telling you when you make a bad or even illegal move. You just have a book with the rules and play games against yourself. How likely do you think it is to ever become a respectable chess player? Nil? I think so too. But with estimating we are often in such a situation. When you do waterfall projects you often do estimates for 100s of tasks in one go and only a year later you get the feedback: The sum of your estimates was 20% of. It’s really hard to learn this way. The situation improves a lot when you follow agile practices, where you typically estimate smaller batches, closer to the time when they are actually performed. This enables you to actually learn how long tasks take.

The First Not Carbon Based Sentient Being

Jens Schauder — Sun, 26 Feb 2012 08:34:00 +0000

With my last blog post I answered to a post of Ted Neward. In that post he also mentioned AI being an interesting topic back then:

Artificial Intelligence was one such thing: the search to try and bring computers to a state of human-like sentience drove a lot of interesting ideas and concepts forward, but over the last decade or two, AI seems to have lost almost all of its luster and momentum

I disagree! Maybe AI as a research topic is not as popular any more, but I think it pretty much became omnipresent and accessible for anybody with a little advanced computer skills. It just happened to happen partially in a very different way than anticipated. Lets start with the obvious:

Siri, Google and Wolfram|Alpha are pretty close to passing the turing test and a computer even won a game show for humans. So AI achieved a lot although we still don’t have humanoid robots in our homes and we probably never will.

But consider this: Imagine a blob of green jelly which sits embedded in the top of an anthill. It produces a smelly goo which the ants love and when an ant nibbles of the goo the green jelly absorbs some of the body fluids of the ant.

One day a neighbouring ant population approaches the ant hill threatening the ants. The ants in distress consume a lot of the smelly goo in order to sooth their stress. Suddenly the green jelly starts to spray a weak acid at the foreign ants until those retreat.

Question: Based on this observation, would you consider the green jelly a living being? If so tell me what is the difference between the green jelly, the friendly ants and the foreign ants on one hand and the Internet, the social media community and the politicians in favor of SOPA and PIPA?

Many scientists searching for extra terrestrial life seem to be quite sure that when they see foreign life forms they’ll recognize them as such. I think they missed one right under their nose. Just because it is so huge it spans a complete planet and because the role of bacteria inside the body of mammals is played by mammals inside the Internet.

Just because its Not New doesn’t mean its Old

Jens Schauder — Sun, 19 Feb 2012 15:18:18 +0000

Ted Neward wrote a piece in which he complains about not seeing anything new and exciting in the IT industry and ends with a question: If you’ve been programming for twenty years, what about the industry today gets your blood moving and your mind sharpened?

So if you count my first experiments with a clunky programmable calculator and my dabbling with an Apple II I do qualify. So here is my answer.

Lets start with a general note: the complaint that their is nothing new is old in it self. I remember the complains when Java started of: “Oh we already have C++/SmallTalk there is nothing new in it” or my boss back than when he finally got the basics of html and http straight: “Oh, its just a fancy terminal!”

But just because a topic is not new doesn’t mean its Old.

For me one such topic is functional programming. I learned abut functional programming about 15 years ago (and it was old back than). I found it to be cool, hardly grasped the very basics and than found out that while the way of programming was cool, the performance was abysmal (which might have been due to my lack of understanding). And I thought: “Now it would be cool if some time I could actually use this kind of stuff”. Today we have Scala (or F# if you prefer) which allows you to use these concepts in nice languages and integrate it with your more conservative main stream languages like Java or C#. Make no mistake: these languages won’t save the industry, but they allow me to explore concepts that might be older than the first computer, but are new and interesting to me.

I have a long list of books and paper a want to read and undstand. Its new stuff to learn and to put to good use. It would only get better if someone would be paying me for this.

The other thing is the ubiquity of computing power and internet connectivity. As it has been all the time the interesting challenge is not so much to write an application, but to come up with interesting ideas for applications. With an interesting idea you can easily reach hundreds or even thousands of potential users. This is awesome. Yeah it’s true there is nothing that is really new. But the scale of availability changed and continues to change.