Creating Maintainable Automated Acceptance Tests
51 likes21,508 views
There's a recording of this talk at Agile 2012 here: http://www.youtube.com/watch?v=v-L_2y6g5DI Creating automated end-to-end functional acceptance tests is hard. Maintaining them over time is harder. Some agilistas even claim that the cost outweighs the benefit. In this lecture we present five principles for creating valuable, maintainable acceptance test suites. We discuss practices such as layering acceptance tests to reduce coupling between the test harness, and talk about how teams should be organized in order to efficiently manage acceptance test driven development. The core of the talk discusses how to manage the evolution of acceptance tests by organizing them as scenarios rather than as suites of story tests. Finally we show how to manage data for acceptance tests.
![var AjaxTracker = { https://gist.github.com/3315690
PENDING_REQUESTS: $A([]),
onCreate: function(request){
if (!request.url.match(/gadgets/js//)) {
this.PENDING_REQUESTS.push(request.url);
}
},
onComplete: function(request){
this.PENDING_REQUESTS = this.PENDING_REQUESTS.without(request.url);
},
onException: function(request, exception){
try {
this.onComplete(request);
}catch(e){
if (Prototype.isFireBugsEnabled) {
console.log("Got Exception on request: " + request.url);
console.log(e);
throw(e);
}
}
},
allAjaxComplete: function(includeCardSummary){
var requests = this.PENDING_REQUESTS.reject(function(url) {
return url.match(/cards/card_summary/) || url.match(/also_viewing/);
});
return requests.size() == 0;
}
};
Ajax.Responders.register(AjaxTracker);
Thursday, August 16, 12](https://image.slidesharecdn.com/automatedtestsworkshop-120816124836-phpapp02/85/Creating-Maintainable-Automated-Acceptance-Tests-24-320.jpg)
More Related Content
What's hot (20)
Viewers also liked (18)
Similar to Creating Maintainable Automated Acceptance Tests (20)
Recently uploaded (20)
Creating Maintainable Automated Acceptance Tests
- 1. Maintainable Acceptance Tests Badrinath Janakiraman Jez Humble Agile 2012 Dallas @badrij | badri@thoughtworks.com @jezhumble | jez@thoughtworks.com http://thoughtworks-studios.com/ Thursday, August 16, 12
- 2. what to expect • Creating high quality acceptance tests • How to structure a maintainable acceptance test suite • Patterns for effective teamwork • Managing test data Thursday, August 16, 12
- 3. what to take away • Quality is everybody’s responsibility • High quality test suites are continuously curated - by testers and developers working together • Test code needs to receive the same care as production code • Exhaustive story-level testing is not a good basis for maintainable acceptance suites Thursday, August 16, 12
- 4. different kinds of tests Business facing AUTOMATED MANUAL Showcases Support programming Functional acceptance Usability testing tests Critique project Exploratory testing Unit tests Non-functional Integration tests acceptance tests System tests (performance, scaling, ...) AUTOMATED MANUAL / AUTOMATED Technology facing Diagram invented by Brian Marick Thursday, August 16, 12
- 5. UI Mike Cohn: Succeeding with Agile Service Unit Thursday, August 16, 12
- 6. End to End Business Facing Localized Technology Facing Thursday, August 16, 12
- 7. principle 0 Writing good acceptance tests is hard. (good: when the tests are green, we know the software works) Thursday, August 16, 12
- 8. mingle 2006 2012 20 tests 3000 tests 500 LOC 50k LOC 2 minutes 12 hours • actual running time: 55 minutes • 7-10 times a day • for 6 years, across 4 offices now Thursday, August 16, 12
- 9. why do test suites decay? for the same reasons code does we don’t pay enough attention to expressing intent only testers care about maintaining tests Thursday, August 16, 12
- 11. principle 1 Tests are first-class citizens of your project Thursday, August 16, 12
- 12. preventing decay in test code treat test code as production c0de refactor relentlessly don’t repeat yourself don’t repeat yourself use record-playback tools to build your suite Thursday, August 16, 12
- 13. preventing decay of intention given-when-then is insufficient separate intention from mechanics express the test as steps of a user's journey Thursday, August 16, 12
- 14. a solution use natural language to express intentions use a general purpose programming language to express test mechanics use a tool that allows you to operate in either domain transparently Thursday, August 16, 12
- 15. Thursday, August 16, 12
- 16. page object https://gist.github.com/3345556 public class LoginPage { private final SeleniumSession browser; public LoginPage(Selenium browser){ this.browser = browser; } public HomePage loginAs(String user, String password){ browser.type('#login', login); browser.type('#password', password); browser.submit('#login-form'); return new HomePage(this.browser); } public HomePage loginExpectingError(String user, String password){ browser.type('#login', login); browser.type('#password', password); browser.submit('#login-form'); return new LoginPage(this.browser); } } Thursday, August 16, 12
- 17. Acceptance Criteria Customer Tester Test implementation Developer Tester Thursday, August 16, 12
- 18. tester / quality analyst ...is a role, not a person ...is not a failed developer ...advocates for the user and makes the quality of the system transparent ...should not be primarily working on manual regression testing ...should be focussed on exploratory testing & maintaining automated acceptance tests Thursday, August 16, 12
- 19. remember passing acceptance tests are necessary (but insufficient) for “done” encapsulate! the acceptance tests are owned by - and the responsibility of - the team Thursday, August 16, 12
- 20. principle 2 always interact with the system under test the same way a user would Thursday, August 16, 12
- 21. browser based tests are unreliable "the test fails in CI, but when I run the app, everything seems fine" usually an indication that test mechanics and user interaction patterns differ ajax based tests? JS heavy applications, which need non-zero processing time to modify the UI Thursday, August 16, 12
- 22. some solutions Test your application the way a user might use it. Understand when behavior is asynchronous and account for it explicitly Don’t use bare sleeps: poll If it’s hard to write the test, you need to have a conversation with the team Thursday, August 16, 12
- 23. some solutions wait-utils (https://github.com/itspanzi/WaitUtils) for ajax tests, if your JS framework provides a pre- and post-call hook, intercept those to count the number of active calls before proceeding Thursday, August 16, 12
- 24. var AjaxTracker = { https://gist.github.com/3315690 PENDING_REQUESTS: $A([]), onCreate: function(request){ if (!request.url.match(/gadgets/js//)) { this.PENDING_REQUESTS.push(request.url); } }, onComplete: function(request){ this.PENDING_REQUESTS = this.PENDING_REQUESTS.without(request.url); }, onException: function(request, exception){ try { this.onComplete(request); }catch(e){ if (Prototype.isFireBugsEnabled) { console.log("Got Exception on request: " + request.url); console.log(e); throw(e); } } }, allAjaxComplete: function(includeCardSummary){ var requests = this.PENDING_REQUESTS.reject(function(url) { return url.match(/cards/card_summary/) || url.match(/also_viewing/); }); return requests.size() == 0; } }; Ajax.Responders.register(AjaxTracker); Thursday, August 16, 12
- 25. remember make time to go back and refactor your tests use layers and encapsulation: separate high level intent and low level mechanics use page object to interact with SUT; run against service layer where possible Thursday, August 16, 12
- 26. principle 3 continuously curate the structure of your test suites Thursday, August 16, 12
- 27. #1312 #1301 As a... I want... So that... As a... I want... So that... #1310 As a... I want... So that... Given... Given... Given... Given... #1306 When... When... When... When... As a... I want... So that... Then... Then... Given... Given... Then... Then... When... #1304 When... #1315 Given... Given... AsThen... a... I want... So that... Then... When... When... As a... I want... So that... Then... Then... Given... Given... Given... Given... #1308 When... When... When... When... As a... I want... So that... Then... Then... #1307 Then... Then... #1313 As a... I want... So that... Given... Given... As a... I want... So that... #1317 When... When... Then...#1311 As a... I want... Then... So that... Given... Given... #1303 Given... Given... When... When... As a... Given... So that... As a... I want... So When... that... When... I want... Given... Then... Then... When... When... Then... Then... Given... Given... Given... Then... Given... Then... When... When... When... When... Then... Then... Then... Then... #1305 #1309 As a... I want... So that... #1302 As a... I want... So that... As a... I want... So that... #1318 Given... Given... I want... So that... Given... #1316 Given... When... As a... When... Given... #1314Given... When... As a... I want... So that... When... When... As a... I want... So that... When... Then... Then... Given... Given... Then... Then... Then... Then... Given... Given... When... When... Given... Given... When... When... Then... Then... When... When... Then... Then... Then... Then... Thursday, August 16, 12
- 28. journey tests #1612 As a customer I want a gift wrapping option So that I don’t have to wrap Buy Product them and post them myself Buy Product Search product catalogue Search product catalogue Add product to cart Add product to cart Check out Check out Create new account Create new account Provide address details Provide address details Provide credit card details Provide credit card details Complete order Select gift wrapping option Verify order created Complete order Verify credit card debited Verify order created Verify email sent Verify gift wrapping option Verify credit card debited Verify email sent Thursday, August 16, 12
- 29. some solutions identify user journeys ( journey: the path a persona takes through the application to achieve an end goal) most applications have very few distinct personas most stories in iterative development are enhancements to existing journeys Thursday, August 16, 12
- 30. features Basic shopping cart functionality Searching for a product - searching for a word should bring up all products which have that word in their name - searching for a phrase should bring up all products which have any of the words in their name - searching for a quoted phrase should bring up all products which have all words in the the quoted phrase in their name Paginating search results - return 25 results per page by default - if there are fewer than 25 results, do not show pagination links - provide a "previous" link on every page other than the first page of results - provide a "next" link on every page other than the last page of results - if user supplies a page number which is less than 1 in the URL, stay on the first page - if the user supplies a page number greater than the last page of results, stay on the last page Gift-wrap option Thursday, August 16, 12
- 31. story tests Story tests for search - test that searching for "friends" brings back 782 results -- results should include how to win friends and influence people - test that searching for dead friends brings back 8900 results -- results should include <how to win friends and influence people> -- results should include <The Zombie Survival Guide: Complete Protection from the Living Dead> - test that searching for "dead friends" brings back 57 results -- results should include <all my friends are dead> Story tests for pagination - with a catalog of 3 products, I should see no pagination links - with a catalog of 25 products, I should see no pagination links - with a catalog of 26 products, I should see 1 link to page two, along with a next link but no previous link - with a catalog of 26 products, on page 2, I should see one product, with a link to page one, a previous link but no next link Story tests for gift wrapping Thursday, August 16, 12
- 32. journey tests Journey of user buying a book - Login as user "bob" - Search for <"my friends" dead> - Make sure that 3 pages of results show - Verify that "All My Friends Are Dead" by "Avery Monson" is on the first page - Add two copies of the book to the shopping cart - Gift wrap one of them - Proceed to checkout Thursday, August 16, 12
- 33. more solutions extract journeys from your acceptance tests make them fast and run them first do test the most likely path that the team, business and UX folk agree upon do not test every possible path through the system extract negative tests and edge cases into a regression suite which runs after your journey tests Thursday, August 16, 12
- 34. build quality in “Cease dependence on mass inspection to achieve quality. Improve the process and build quality into the product in the first place” W. Edwards Deming Thursday, August 16, 12
- 35. why cross-functional teams? output of testing is not just bug reports feedback from testing to product design feedback from test framework to system architecture developers and testers share knowledge and skills Thursday, August 16, 12
- 36. principle 4 everyone owns acceptance tests Thursday, August 16, 12
- 37. when acceptance tests break Triage to find root cause 1. There was an environmental problem 2. There is a bug with the test 3. An assumption changed 4. The test actually caught a bug Fix the problem Add a guard to prevent it happening again Optimise your test suite: detect failures fast Optimise your process for time to fix tests Thursday, August 16, 12
- 38. intermittent failures flaky tests are worse than useless quarantine flaky tests - but not forever http://martinfowler.com/articles/ nonDeterminism.html Thursday, August 16, 12
- 39. external systems not all tests should call the external system parameterize connections to external systems Run integration smoke tests before full acceptance suite Thursday, August 16, 12
- 40. impersonator pattern create a proxy from SUT to external system cache results from integration smoke tests run integration smoke tests before acceptance suite periodically expire cache only run acceptance suite if integration smoke tests pass! Thursday, August 16, 12
- 41. principle 5 acceptance tests are responsible for managing their own test data Thursday, August 16, 12
- 42. test data Test-specific data Test reference data Application reference data Ensure tests are independent Don’t use production data dumps (except for performance testing and staging) Thursday, August 16, 12
- 43. recap 1. treat acceptance tests like production code 2. always interact with the SUT like a user would 3. continuously curate your user journeys 4. collective ownership of acceptance tests 5. acceptance tests own their data Thursday, August 16, 12
- 44. take-aways • quality is everybody’s responsibility • high quality test suites are continuously curated - by testers and developers working together • test code needs to receive the same care as production code • exhaustive story-level testing is not a good basis for maintainable acceptance suites Thursday, August 16, 12
- 45. questions @badrij | badri@thoughtworks.com @jezhumble | jez@thoughtworks.com http://continuousdelivery.com/ © 2011 ThoughtWorks, Inc. http://thoughtworks-studios.com/ Thursday, August 16, 12