I can sum this up in a couple of bullet points:

  • When it should be a unit test
  • When nobody outside the delivery team cares about it

When it should be a unit test

The value of automated scenarios are at their highest while the features are still being delivered. Particularly in areas where you need to have a lot of discussions with your stakeholders.

Once a feature has been completed, the scenarios still have value as regression tests, but it is not really necessary to have all the behaviour defined in Cucumber.

After delivery of each feature, there should be a comprehensive suite of class level specifications. Any business rules ought to be covered by these and therefore shouldn’t need to be tested again by Cucumber.

While it’s useful to keep a few scenarios around as integration tests, most over the behaviour can and should be specified at a lower level.

If you think about an integration test, it’s testing the integration, it’s making sure the glue between the parts is OK, it shouldn’t need to test the parts themselves. If we take as an example (very simplified to make the point) a basic CRUD page for a widget app. You might want to have 2 scenarios for each operation:

  1. Successful - make sure everything is hooked up and on success, the right thing happens from point of view of the user or consumer.
  2. Failure - make sure it doesn’t crash and some sort of error message (specified elsewhere) makes it into the response.

There might be a few special cases, for instance an unusual response or redirect under a particular set of circumstances but it’s unlikely there will be too many, and most of these could easily be specified at a lower level.

There are, however likely to be a lot of business rules. As long as we can prove the UI displays (or the service returns) what it should on the success/failure of one of these rules, the various rules can be specified at the class or service level.

When nobody outside the delivery team cares about it

Scenarios are useful for thrashing out the details of a feature using examples, collaboratively, before it gets implemented. The cheapest time to make a change is before you start coding.

Only the more interesting scenarios, the ones that aren’t quite ironed out yet and still need some discussion retain their value.

If you have lots of cucumber scenarios acting as regression or integration tests that are slowing down your build and the underlying functionality is unlikely to change. You don’t need them as a tool for communication so delete the scenarios and test using unit or lower level integration tests.