Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Mozmill Crowd
Similar to Mozmill Crowd (20)
Recently uploaded
Recently uploaded (20)
Mozmill Crowd
- 1. quality.mozilla.org qa automation Mozmill Crowd A change in how we test Firefox Henrik Skupin <hskupin@mozilla.com> Lead of QA Test Automation
- 3. Overview • Release schedule for Firefox • Testing Firefox releases • How the Automation helps QA • Who contributes to QA? • Crowd-sourced Testing (Mozmill Crowd) • Demo
- 5. FIREFOX RELEASES Firefox 4.0 Beta • Lately approx. 2 weeks beta cycles • Various bug fixes and improvements Firefox 3.5 and 3.6 • Security releases each 4 to 6 weeks • Both releases happen simultaneously • About 50 - 80 fixed bugs per release
- 7. FIREFOX RELEASES Testing Firefox Releases • Exploratory and scripted testing of affected areas • Execution of functional tests, including: • approx. 40 Smoketests • approx. 330 BFT tests (Basic Functional Tests) • Time to ship to beta audience - only a few days • Time to ship the release - about 1 week
- 9. FIREFOX RELEASES Testing Firefox Releases • Software Update • Tests from previous versions (partial / complete) • Testing ‘betatest’ and ‘beta’ channels before beta • Testing ‘releasetest’ and ‘release’ channels before release
- 10. FIREFOX RELEASES
- 11. FIREFOX RELEASES How Automation Helps QA • Usage of Mozmill to run automated functional tests • approx. 110 (out of 330) functional tests automated • Execution of localization specific tests for all locales • (Nearly) fully automated update tests for all locales • Simultaneously execution on all supported platforms
- 12. CONTRIBUTING
- 13. CONTRIBUTING Who contributes to QA? • Small QA team (~30) and contractors for Releases • Community members execute manual tests on Litmus • 80.000 nightly users for Firefox 4.0 builds • 15.000 nightly users for older Firefox branches
- 14. CONTRIBUTING Who contributes to QA? • Small QA team (~30) and contractors for Releases • Community members execute manual tests on Litmus • 80.000 nightly users for Firefox 4.0 builds • 15.000 nightly users for older Firefox branches How can we outreach testing?
- 15. CONTRIBUTING How can we outreach testing?
- 16. CONTRIBUTING Who contributes to QA? • Small QA team (~30) and contractors for Releases • Community members execute manual tests on Litmus • 80.000 nightly users for Firefox 4.0 builds • 15.000 nightly users for older Firefox branches How can we outreach testing?
- 17. CROWD SOURCED TESTING Crowdsourcing relies on people, and thus works when participants are treated as people
- 18. CROWD SOURCED TESTING Crowd-sourced Testing Crowdsourcing relies on people, and thus works when participants are treated as people
- 20. CROWD SOURCED TESTING What do we expect? • Improved quality of localized builds • Knowledge how Firefox behaves in different locations • Results from machines with unknown configurations • Performance related information and memory usage • ...
- 22. CROWD SOURCED TESTING What’s necessary? • No global installation of additional software • Automatic setup of the Mozmill-CLI environment • Wide range of different test scenarios (BFT, L10n, ...) • Reporting and analysis of collected test results • Promoting tests even outside of our community
- 24. CROWD SOURCED TESTING How it works • Mozmill-Crowd add-on for Firefox 4.0 and 3.6 • Automatic setup of the prepared test environment • Simplified UI for the configuration of the test-run • Automatic download of the latest tests • Test execution and reporting to our database
- 25. CROWD SOURCED TESTING User Profile Testing Profile 3. 1. Repositories 2. (Tests and Scripts) MozMill CLI 4. Database Virtual Test Environment
- 26. CROWD SOURCED TESTING How it works User Profile Testing Profile 3. 1. Repositories 2. (Tests and Scripts) MozMill CLI 4. Database Virtual Test Environment
- 28. CROWD SOURCED TESTING Analyzing test results L10n test results • Multiple access keys • Cropped elements • Screenshots
- 30. CROWD SOURCED TESTING What’s left • Proof reliability of our Mozmill tests • Fix focus problems with Mozmill (native events) • Offer more options in the extension for contribution • Add more test-runs (i.e. add-ons and endurance tests) • We need your feedback!
- 31. REFERENCES Demo
- 32. REFERENCES Mozmill Crowd: https://addons.mozilla.org/firefox/addon/mozmill-crowd QA Test Automation team page: http://quality.mozilla.org/teams/automation/ Mozmill Code: https://github.com/mozautomation/mozmill Mozmill Tests: https://developer.mozilla.org/en/Mozmill_Tests
- 33. QUESTIONS & ANSWERS Any Questions?
- 34. Thanks
Editor's Notes
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n