Cucumber ATDD ImageAcceptance Test Driven Development (ATDD) is a process where every member of a project team may develop acceptance criteria for a particular feature. By using ATDD, a team can produce a feature from inception to implementation with minimal surprises regarding the final product. A developer following ATDD will use acceptance tests to iteratively drive the coding process until the feature is fully implemented. The practice of ATDD provides a self-documenting process that results in an automated test library describing the behavior of the system in a language that programmers and non-technical members alike can understand. Cucumber is the Behavior Driven Development (BDD) testing tool that will be used as examples to describe ATDD in this post.

Acceptance Criteria

ATDD gives the power of traditional Test Driven Development (TDD) to the entire project team. The first step a team takes in the ATDD process is to collaborate and define the acceptance criteria for the feature to be developed. The acceptance criteria is written in the BDD Given When Then test scenario format that the Cucumber testing tool understands. Once the test can be executed against the implemented feature, your entire team can advertise with confidence that the feature is functioning as everyone on the team expects.

Team Collaboration

Similar to a developer writing a unit test while practicing TDD, the first thing a team should do when developing an acceptance test is to determine the desired behavior that the test will verify. This process is an opportunity for the product owner to ensure that the stakeholders for your project will receive a feature that behaves exactly as they need it to. It is also the engineers' opportunity to clearly define the scope of the feature. The team will have a much better understanding of what implementing the feature will entail by crafting the acceptance criteria.

It is entirely possible that everyone subscribing to the team's idea of what the feature is supposed to be will drastically change during the acceptance criteria crafting process. Determining acceptance criteria should be part of the story writing process in a story grooming session. If a team is having difficulty crafting the acceptance criteria for a feature, it could be a good indication there may be a problem with the feature. Perhaps the feature scope is too broad. Maybe the concept simply will not work in the real world. The beauty of having this discussion now is that problems are discovered before any lines of code have been written.

Given When Then

The final result of acceptance criteria determination is a script that will serve as the actual acceptance test for a feature. Acceptance tests are written in the BDD Given When Then format. For example, a test scenario for the CLC IPS Product Offering would be:

Given a Centurylink customer has a server
When IPS is installed on that server
Then the server is protected with IPS

The Given When Then format follows the typical approach taken for writing unit tests:

  1. Setup test conditions.
  2. Execute code under test.
  3. Verify code functionality.

Tests Instill Confidence

A comprehensive library of acceptance tests that the entire team has collaborated on gives each member a warm, fuzzy feeling about the product they are producing. It also helps marketing to espouse the capabilities of the system without fear of making unfounded promises. The implemented acceptance tests coupled with a Continuous Integration (CI) system such as Jenkins gives the developers confidence that the product is still functioning as advertised.

High-Level Iterative Development

We can begin implementing our test now that we have a fully-vetted acceptance test scenario. Cucumber makes test implementation a breeze.

  1. Put your test scenario into a feature file: Note: Source for the examples is provided in GitHub.

    ~/workspace/atdd-cucumber/src/test/java/com/ctl/example/cucumber/feature/ips.feature

    @CLC_IPS
    Feature:  CLC_IPS - Install
    
    @CLC_IPS_INSTALL
    Scenario: IPS for Linux
    Given a Centurylink customer has a server
    When IPS is installed on that server
    Then the server is protected with IPS
    
  2. Execute the test runner:

    ~/workspace/atdd-cucumber/src/test/java/com/ctl/example/cucumber/runner/RunCukesTest.java

    @RunWith(Cucumber.class)
    @CucumberOptions(monochrome = true,
    tags = {"@CLC_IPS"}, // You must explicitly state which tests to run
    features = {"src/test/java/com/ctl/example/cucumber/feature"},
    glue = {"com/ctl/example/cucumber/step"}
    )
    public class RunCukesTest {
    
    }
    

    The source code for the step definitions is printed to the console:

    Generated Steps

  3. Add steps to a Step Definition Class:

    ~/workspace/atdd-cucumber/src/test/java/com/ctl/example/cucumber/step/IpsSteps.java

    public class IpsSteps {
    
    @Given("^a Centurylink customer has a server$")
    public void a_Centurylink_customer_has_a_server() throws Throwable {
        // Write code here that turns the phrase above into concrete actions
        throw new PendingException();
    }
    
    @When("^IPS is installed on that server$")
    public void ips_is_installed_on_that_server() throws Throwable {
        // Write code here that turns the phrase above into concrete actions
        throw new PendingException();
    }
    
    @Then("^the server is protected with IPS$")
    public void the_server_is_protected_with_IPS() throws Throwable {
        // Write code here that turns the phrase above into concrete actions
        throw new PendingException();
    }
    
    }
    

Now, we can start writing our test.

  1. Start by adding the action that we are testing in the When clause, so that we learn what needs to be set up for the test. We surmise that we need an IPS install client to interact with the IPS API.

    @When("^IPS is installed on that server$")
    public void ips_is_installed_on_that_server() throws Throwable {
        ipsInstallClient.install();
    }
    

    This is where our acceptance test begins to drive the code to be developed.

    Note: Before the next step, you should develop the ipsInstallClient.install() using standard TDD.

  2. Next, we'll put the new version of ipsInstallClient.install() into our When step:

    @When("^IPS is installed on that server$")
    public void ips_is_installed_on_that_server() throws Throwable {
        ipsInstallClient.install(ipsInstallConfiguration, bearerToken);
    }
    

    After developing the ipsInstallClient.install() method we see that we need an ipsInstallConfiguration and bearerToken to execute our test.

  3. So, we make ipsInstallConfiguration and bearerToken members of our step class so that we can initialize them in our Given step.

    private IpsInstallConfiguration ipsInstallConfiguration;
    private String bearerToken;
    
    @Given("^a Centurylink customer has a server$")
    public void a_Centurylink_customer_has_a_server() throws Throwable {
        ipsInstallConfiguration = null;
        bearerToken = null;
    }
    

    Now that we're setting up our test and executing the action that we want to test, we can verify that the install is functioning in the Then step. Here, we have to figure out how we will verify that the action in the When step is functioning.

  4. Let's execute the test to see if that will give us an idea for the verification. Running the test yields the following error (exception):

    org.springframework.web.client.HttpServerErrorException: 500 Internal Server Error
    at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:94)
    at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:641)
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:597)
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:557)
    at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:475)
    at com.ctl.security.ips.client.IpsInstallClient.install(IpsInstallClient.java:38)
    at com.ctl.example.cucumber.step.IpsSteps.ips_is_installed_on_that_server(IpsSteps.java:34)
    at ✽.When IPS is installed on that server(ips.feature:7)
    
  5. When our feature is completed and working we should no longer receive the error. Let's gracefully catch the exception and set a member to that exception.

    private HttpServerErrorException httpServerErrorException;
    
    @When("^IPS is installed on that server$")
    public void ips_is_installed_on_that_server() throws Throwable {
        try{
            ipsInstallClient.install(ipsInstallConfiguration, bearerToken);
        }
        catch(HttpServerErrorException e){
            httpServerErrorException = e;
        }
    }
    

    Our Then step verifies success by asserting that the exception is null.

    @Then("^the server is protected with IPS$")
    public void the_server_is_protected_with_IPS() throws Throwable {
        assertNull(httpServerErrorException);
    }
    
  6. Now when we run our test we get a nice expected failure:

    Expected Failure

    We can use this failing test to drive our next steps of development. Is there a problem with the installation client? Are we calling the client properly? The above testing and developing cycle provides an example of the iterative development process that an acceptance test can provide.

  7. We want the final state of the Then step to be something like this:

    @Then("^the server is protected with IPS$")
    public void the_server_is_protected_with_IPS() throws Throwable {
        assertNull(httpServerErrorException);
    
        boolean ipsIsInstalled = ipsInstallClient.verifyInstallation(ipsInstallConfiguration, bearerToken);
    
        assertTrue(ipsIsInstalled);
    }
    

    Now that we have this high level acceptance test in place, we can change any part of the underlying implementation with confidence that the overall functionality remains.

Self-Documenting Automated Tests For Everyone

As can be seen from the above feature tests, we now have a document that anyone can read to understand how a particular feature works. A Given When Then test scenario can serve as the centerpiece for a demonstration of the current state of a feature. A library of feature tests gives a developer very clear feedback on the state of the entire system. Coupled with a continuous integration server such as Jenkins, a team can know very quickly if any part of their system is failing. Here is a sample of the type of report that can be automatically generated using the Jenkins Cucumber Report plug-in:

Cucumber Report 1.jpg Cucumber Report 2.jpg

Cucumber is one of many test tools that allows teams to create acceptance tests. No matter what tool is used, the most important aspect of ATDD is the collaboration it fosters between all team members.

The image at the top of this post is a combination of two images with creative commons licenses, located here and here.