Ever found more broken things in a project than you possible could fix in one go? I'm exactly in that situation. We have a test which checks for certain dependencies in our project. The problem: We discovered that the test was broken and didn't report all disallowed dependencies. We do know how to fix it, but if we do it will fail. And since there are many violations it will take quite some time to fix all of them. But of course we want to prevent even more violations of our dependency rules.

The solution could be a ratchet on the tests. A contraption which accepts broken tests, but which doesn't allow tests to fail once they succeeded.

For ScalaTest you can create appropriate tests using this simple Ratchet trait

package de.schauderhaft.ratchet

import org.scalatest.AbstractSuite

import org.scalatest.Suite

import org.scalatest.TestFailedException

trait Ratchet extends AbstractSuite {

    self : Suite =>
    private var tests = Set[String]()
    def ratchet(ratchetedTests : Set[String]) {

        tests = ratchetedTests

    }
    override abstract def withFixture(theTest : NoArgTest) {

        if (tests.contains(theTest.name)) {

            var failedToFail = false

            try {

                super.withFixture(theTest)

                failedToFail = true

            } catch {

                case ex : TestFailedException =>

            }

            if (failedToFail)

                fail("Remove '%s' from the ratchet it doesn't fail anymore".format(theTest.name))

        } else

            super.withFixture(theTest)

    }

}

It adds a ratchet method to your suite. You pass it a Set of test names. These are the tests that you expect to fail. If they do fail, the Ratchet will convert that failure to a success. If a test which you expect to fail succeeds, the Ratchet will make sure it does fail with a message saying you should remove it from the tests expected to fail. Tests not registered with the ratchet method behave just as normal tests do. This is how the contraption looks in action with an example test suite: (Note tests which start witch 'expected:' do fail

class RatchetTestDemo extends FunSuite with ShouldMatchers with Ratchet {

    ratchet(Set(

        "a failing test with ratchet does not fail",

        "expected: a succeeding test with ratchet fails"))

    test("a failing test with ratchet does not fail") {

        fail

    }
    test("expected: a failing test without ratchet fails") {

        fail

    }
    test("a succeeding test without ratchet succeeds") {

    }
    test("expected: a succeeding test with ratchet fails") {

        println("hallo")

    }

}