Unit test exception messages using JUnit ExpectedException Rule

Traditionally JUnit supported expected exceptions using the @Test annotation like this

@Test(expected = ParsingException.class)
public void parseGame_when_fileIsEmpty_then_throwsParsingException() throws Exception {
   // Arrange
   BufferedReader bufferedReaderMock = mock(BufferedReader.class);
   when(bufferedReaderMock.readLine()).thenReturn("");

   // Act
   gameJsonParser.parseGame(bufferedReaderMock);
}

This has two inconveniences. The expectation should be defined outside the test body and it does not support asserting the exception message.

A workaround commonly used is handling the exception with a try/catch block and assert it manually.

@Test
public void parseGame_when_fileIsEmpty_then_throwsParsingException() throws Exception {
   // Arrange
   BufferedReader bufferedReaderMock = mock(BufferedReader.class);
   when(bufferedReaderMock.readLine()).thenReturn("");

   // Act
   try {
     gameJsonParser.parseGame(bufferedReaderMock);
     fail("Should have thrown ParsingException but did not!");
   }
   catch(ParsingException e) {
     String message = "Parsing failed. Empty game file.";
     assertEquals(message, e.getMessage());
   }
}

As you can see that brings lot of boilerplate code to the test.

Since JUnit 4.7 we can use the ExpectedException core rule to assert exceptions in a more natural flow and also to assert their messages.

ExpectedException: Allows a test to specify expected exception types and messages in the test itself.

@Rule
public ExpectedException exception = ExpectedException.none();

@Test
public void parseGame_when_fileIsEmpty_then_throwsParsingException() throws Exception {
   // Arrange
   BufferedReader bufferedReaderMock = mock(BufferedReader.class);
   when(bufferedReaderMock.readLine()).thenReturn("");

   exception.expect(ParsingException.class);
   exception.expectMessage("Parsing failed. Empty game file.");

   // Act
   gameJsonParser.parseGame(bufferedReaderMock);
}

References:
https://paucls.wordpress.com/2013/11/12/using-junit-rules
https://www.infoq.com/news/2009/07/junit-4.7-rules

Using JUnit Categories to group tests

JUnit allows to organize tests in groups using Categories. A well-organised​ suit of tests is really important, annotating​ our tests with categories could help to differentiate fast from slow tests or stable from unstable/maturing tests.

Defining Categories
We define our categories creating interfaces

public interface CriticalTests {}
public interface SanityTests {}
public interface FastTests {}
public interface SlowTests {}
public interface StableTests {}
public interface UnstableTests {}

And annotating our test class or individual tests with the Category annotation

public class MyFeatureTest {
	@Test @Category(FastTests.class)
	public void testFast() {
		System.out.println("fast");
	}
	@Test @Category(SlowTests.class)
	public void testSlow() {
		System.out.println("slow");
	}
	@Test @Category({SanityTests.class, SlowTests.class})
	public void testSanity() {
		System.out.println("sanity");
	}
}

Notice, that we can annotate a test with multiple categories.

@Test @Category({SanityTests.class, SlowTests.class})

Run Tests by Category
Configuring the POM
We can use categories from maven configuring properly the version of maven-surefire-plugin and selecting the junit47 provider.

			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-surefire-plugin</artifactId>
				<version>2.17</version>
				<dependencies>
	                <dependency>
	                    <groupId>org.apache.maven.surefire</groupId>
	                    <artifactId>surefire-junit47</artifactId>
	                    <version>2.17</version>
	                </dependency>
	            </dependencies>
			</plugin>

Run tests specifying categories
We can run all the test under a category using the -Dgroups maven attribute. For example run all the fast and stable tests:

mvn test -Dgroups="com.test.groups.FastTests, com.test.groups.StableTests"

That argument could be used in combination with -Dtest to run only a especific test in a test class.

mvn test -Dtest=AppTest.java -Dgroups="com.test.groups.SanityTests"

Run tests with a category profile
The second option is to define maven profiles per category or group of categories. This allows to run maven using a profile

mvn test -PsanityTests

Another advantage of profiles is that allows to combine group selection with file inclusion and exclusion.
Example of profile configurations for multiple categories:

...
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-surefire-plugin</artifactId>
				<version>2.11</version>
				<dependencies>
	                <dependency>
	                    <groupId>org.apache.maven.surefire</groupId>
	                    <artifactId>surefire-junit47</artifactId>
	                    <version>2.12.2</version>
	                </dependency>
	            </dependencies>
				<configuration>
					<groups>${testcase.groups}</groups>
					<excludes>
                    	<exclude>${exclude.tests}</exclude>
	                </excludes>
    	            <includes>
						<include>${include.tests}</include>
                    </includes>
				</configuration>
			</plugin>
		</plugins>
	</build>
 
	<profiles>
	    <profile>
	        <id>sanityTests</id>
	        <properties>
	            <testcase.groups>com.emc.gs.atlas.ui.categories.CriticalTests</testcase.groups>
				<exclude.tests>**/x/**/*.java</exclude.tests>
	            <include.tests>**/y/**/*.java</include.tests>
	        </properties>
	    </profile>
	    <profile>
	        <id>fastTests</id>
	        <properties>
	            <testcase.groups>com.test.groups.SanityTests</testcase.groups>
	        </properties>
	    </profile>
	    <profile>
	        <id>fastTests</id>
	        <properties>
	            <testcase.groups>com.test.groups.FastTests</testcase.groups>
	        </properties>
	    </profile>
	    <profile>
	        <id>slowTests</id>
	        <properties>
	            <testcase.groups>com.test.groups.SlowTests</testcase.groups>
	        </properties>
	    </profile>
	    <profile>
	        <id>stableTests</id>
	        <properties>
	            <testcase.groups>com.test.groups.StableTests</testcase.groups>
	        </properties>
	    </profile>
	    <profile>
	        <id>unstableTests</id>
	        <properties>
	            <testcase.groups>com.test.groups.UnstableTests</testcase.groups>
	        </properties>
	    </profile>
	</profiles>

Documentation:
Oficial documentation:
https://github.com/junit-team/junit/wiki/Categories
https://maven.apache.org/surefire/maven-surefire-plugin/examples/junit.html
Other links:
http://java.dzone.com/articles/unit-and-integration-tests
http://www.togsblom.com/2010/04/categorizing-junit-tests.html
https://weblogs.java.net/blog/johnsmart/archive/2010/04/25/grouping-tests-using-junit-categories-0
http://technicaltesting.wordpress.com/2012/09/24/using-junit-category-and-maven-profiles-to-get-stable-test-suites/
http://stackoverflow.com/questions/15648557/maven-unit-tests
http://stackoverflow.com/questions/3100924/how-to-run-junit-tests-by-category-in-maven

Using JUnit Rules

JUnit rules allows the definition of work to be done before and after a test case execution. Than means before either the test execution and also all the @before annotated methods. This facility allows creation of more powerful and cleaner tests. It is a concept similar to JUnitRunners but with the advantage that we can combine multiple rules.

There are a couple of useful rules already implemented in JUnit. For example the TemporaryFolderRule allows creation of files and folders that are guaranteed to be deleted when the test case finishes. A complete rule list is available in https://github.com/junit-team/junit/wiki/Rules.

If we want to create our custom rules we can simply do:

public class MyTestRule implements TestRule {
	public Statement apply(final Statement base, Description description) {
		return new Statement() {
			@Override
			public void evaluate() throws Throwable {
				System.out.println("before");
				try {
					base.evaluate();
				} finally {
					System.out.println("after");
				}
			}
		};
	}
}

And we will use it in our tests via the @Rule annotation.

public class MyTest {
	@Rule
	public MyTestRule myTestRule = new MyTestRule();
 
	@Test
	public void testSanity() throws Exception {
		System.out.println("test execution");
	}
}

Running this test case will print:

before
test execution
after

that proves that the rule interceptor is working as expected.

A real example could be a rule to take snapshoots when a selenium ui tests fails, for example:

public class ScreenshotRule implements TestRule {
	public Statement apply(final Statement base, final Description description) {
		return new Statement() {
			@Override
			public void evaluate() throws Throwable {
				try {
					base.evaluate();
				} catch (Throwable t) {
					takeScreenshot();
					throw t; // Re-throw to allow the failure to be reported to JUnit
				}
			}
			private void takeScreenshot(){
                                WebDriver driver = new FirefoxDriver();
				File imageFile = ((TakesScreenshot) driver).getScreenshotAs(OutputType.FILE);
				String failureImageFileName = "testfailureimage.png";
				File failureImageFile = new File(failureImageFileName);
				FileUtils.moveFile(imageFile, failureImageFile);
			}
		};
	}
}
public class MySeleniumTest {
	@Rule
	public ScreenshotRule screenshotRule = new ScreenshotRule();
 
	@Test
	public void testThatFails() {
		WebDriver driver = new FirefoxDriver();
		driver.get("http://www.google.com");
		driver.findElement(By.id("non_existing_element"));
	}
}

Using rules we can reduce the boilerplate code in our tests, it’s especially useful for Integration Tests or Automation Tests that usually share a complex context or environment set up previous the test execution.

Other interesting use cases of rules are to use them as a Runners replacement, when we want to combine more than one Runner. You can see an example for SpringJUnit4ClassRunner or MockitoJUnitRunner in http://www.alexecollins.com/content/tutorial-junit-rule/

An other interesting point is to combine Rules with custom annotations.

Ordering Rules
We must be careful using multiple rules because the order of excution is not assured by default. To illustrate that problem we can use the next custom rule:

public class LoggingRule implements TestRule {
	private String name;
	public LoggingRule(String name) {
		this.name = name;
	}
	public Statement apply(final Statement base, Description description) {
		return new Statement() {
			@Override
			public void evaluate() throws Throwable {
				try {
					System.out.println("starting " + name);
					base.evaluate();
				} finally {
					System.out.println("finished " + name);
				}
			}
		};
	}
}

And create a little test with three rules.

public class RulesOrderTest {
	
	@Rule public LoggingRule rule1 = new LoggingRule("rule 1");
	@Rule public LoggingRule rule2 = new LoggingRule("rule 2");
	@Rule public LoggingRule rule3 = new LoggingRule("rule 3");

	@Test
	public void test() {
	}
}

Running this test case will print:

starting rule 3
starting rule 2
starting rule 1
finished rule 1
finished rule 2
finished rule 3

But that order could be different if we execute it in other java VM.
If we want to assure the ordering of the rules execution we should use the RuleChain test rule. The order in the chain will be the order of the rules execution.

public class RulesOrderTest {
		
	@Rule 
	public TestRule chain = RuleChain
                           .outerRule(new LoggingRule("outer rule"))
                           .around(new LoggingRule("middle rule"))
                           .around(new LoggingRule("inner rule"));
	@Test
	public void test() {
	}
}

Now it will print:

starting outer rule
starting middle rule
starting inner rule
finished inner rule
finished middle rule
finished outer rule

Documentation:
https://github.com/junit-team/junit/wiki/Rules
http://www.alexecollins.com/content/tutorial-junit-rule/
http://www.codeaffine.com/2012/09/24/junit-rules/
http://junit.org/javadoc/latest/org/junit/rules/RuleChain.html