Ratchet Tests with ScalaTest
Ever found more broken things in a project than you possible could fix in one go? I’m exactly in that situation. We have a test which checks for certain dependencies in our project. The problem: We discovered that the test was broken and didn’t report all disallowed dependencies. We do know how to fix it, but if we do it will fail. And since there are many violations it will take quite some time to fix all of them. But of course we want to prevent even more violations of our dependency rules.
The solution could be a ratchet on the tests. A contraption which accepts broken tests, but which doesn’t allow tests to fail once they succeeded.
For ScalaTest you can create appropriate tests using this simple Ratchet trait
-
package de.schauderhaft.ratchet
-
import org.scalatest.AbstractSuite
-
import org.scalatest.Suite
-
import org.scalatest.TestFailedException
-
-
trait Ratchet extends AbstractSuite {
-
self : Suite =>
-
-
private var tests = Set[String]()
-
-
def ratchet(ratchetedTests : Set[String]) {
-
tests = ratchetedTests
-
}
-
-
override abstract def withFixture(theTest : NoArgTest) {
-
if (tests.contains(theTest.name)) {
-
var failedToFail = false
-
try {
-
super.withFixture(theTest)
-
failedToFail = true
-
} catch {
-
case ex : TestFailedException =>
-
}
-
if (failedToFail)
-
fail("Remove '%s' from the ratchet it doesn't fail anymore".format(theTest.name))
-
} else
-
super.withFixture(theTest)
-
}
-
}
It adds a ratchet method to your suite. You pass it a Set of test names. These are the tests that you expect to fail. If they do fail, the Ratchet will convert that failure to a success. If a test which you expect to fail succeeds, the Ratchet will make sure it does fail with a message saying you should remove it from the tests expected to fail. Tests not registered with the ratchet method behave just as normal tests do. This is how the contraption looks in action with an example test suite: (Note tests which start witch ‘expected:’ do fail
-
class RatchetTestDemo extends FunSuite with ShouldMatchers with Ratchet {
-
ratchet(Set(
-
"a failing test with ratchet does not fail",
-
"expected: a succeeding test with ratchet fails"))
-
-
test("a failing test with ratchet does not fail") {
-
fail
-
}
-
-
test("expected: a failing test without ratchet fails") {
-
fail
-
}
-
-
test("a succeeding test without ratchet succeeds") {
-
}
-
-
test("expected: a succeeding test with ratchet fails") {
-
println("hallo")
-
}
-
}
Interesting, that looks like the pendingUntilFixed method: http://www.scalatest.org/scaladoc-1.5.1/org/scalatest/Suite.html but more like ‘successUntilFixed’ in that case.
@Eric
I agree it is similar.
An important difference is the fact that the information if a test is expected to fail is out side the test.This allows it to work with Suites where the number of tests is determined dynamically.
Imagine a case where a list of packages is created through checkstyle or something similar and then a test is created for each package:
val list = createListOfPackages()
list.foreach{p =>
test("do a a test with " + p ){ p should be not to big}
}
With pending until fixed I would have to code the logic into the test.
You can still write something like:
val pendingTests = Seq(“p1″, “p45″)
def testOrPending[T](name: String, id: String)(expectations: =>T) = test(name + ” ” +id) {
if (pendingTests contains p) pendingUntilFixed(expectations)
else expectations
}
val list = createListOfPackages()
list.foreach{p =>
testOrPending(“do a a test with “, p) {
p should be not to big
}
}
@Eric you are right.