SlideShare a Scribd company logo
1 of 34
Download to read offline
quality.mozilla.org   qa automation




     Mozmill Crowd
A change in how we test Firefox

   Henrik Skupin <hskupin@mozilla.com>
       Lead of QA Test Automation
Mozmill Crowd
Overview

• Release   schedule for Firefox

• Testing   Firefox releases

• How   the Automation helps QA

• Who   contributes to QA?

• Crowd-sourced Testing        (Mozmill Crowd)

• Demo
FIREFOX RELEASES
FIREFOX RELEASES



                       Firefox 4.0 Beta

   • Lately   approx. 2 weeks beta cycles

   • Various    bug fixes and improvements

                      Firefox 3.5 and 3.6

   • Security   releases each 4 to 6 weeks

   • Both   releases happen simultaneously

   • About    50 - 80 fixed bugs per release
FIREFOX RELEASES
FIREFOX RELEASES



                   Testing Firefox Releases

   • Exploratory    and scripted testing of affected areas

   • Execution   of functional tests, including:

     • approx. 40   Smoketests

     • approx. 330    BFT tests (Basic Functional Tests)

   • Time   to ship to beta audience - only a few days

   • Time   to ship the release - about 1 week
FIREFOX RELEASES
FIREFOX RELEASES



                   Testing Firefox Releases

   • Software    Update

     • Tests   from previous versions (partial / complete)

     • Testing ‘betatest’ and ‘beta’ channels   before beta

     • Testing ‘releasetest’ and ‘release’ channels   before
       release
FIREFOX RELEASES
FIREFOX RELEASES



                How Automation Helps QA

   • Usage   of Mozmill to run automated functional tests

   • approx. 110   (out of 330) functional tests automated

   • Execution   of localization specific tests for all locales

   • (Nearly)   fully automated update tests for all locales

   • Simultaneously   execution on all supported platforms
CONTRIBUTING
CONTRIBUTING



                 Who contributes to QA?

   • Small   QA team (~30) and contractors for Releases

   • Community     members execute manual tests on Litmus

   • 80.000   nightly users for Firefox 4.0 builds

   • 15.000   nightly users for older Firefox branches
CONTRIBUTING



                 Who contributes to QA?

   • Small   QA team (~30) and contractors for Releases

   • Community     members execute manual tests on Litmus

   • 80.000   nightly users for Firefox 4.0 builds

   • 15.000   nightly users for older Firefox branches


                  How can we outreach testing?
CONTRIBUTING




               How can we outreach testing?
CONTRIBUTING



                 Who contributes to QA?

   • Small   QA team (~30) and contractors for Releases

   • Community     members execute manual tests on Litmus

   • 80.000   nightly users for Firefox 4.0 builds

   • 15.000   nightly users for older Firefox branches


                  How can we outreach testing?
CROWD SOURCED TESTING




    Crowdsourcing relies on people, and thus works when
            participants are treated as people
CROWD SOURCED TESTING



               Crowd-sourced Testing




    Crowdsourcing relies on people, and thus works when
            participants are treated as people
CROWD SOURCED TESTING
CROWD SOURCED TESTING



                    What do we expect?

   • Improved    quality of localized builds

   • Knowledge    how Firefox behaves in different locations

   • Results   from machines with unknown configurations

   • Performance    related information and memory usage

   • ...
CROWD SOURCED TESTING
CROWD SOURCED TESTING



                     What’s necessary?

   • No   global installation of additional software

   • Automatic   setup of the Mozmill-CLI environment

   • Wide   range of different test scenarios (BFT, L10n, ...)

   • Reporting   and analysis of collected test results

   • Promoting   tests even outside of our community
CROWD SOURCED TESTING
CROWD SOURCED TESTING



                        How it works

   • Mozmill-Crowd     add-on for Firefox 4.0 and 3.6

   • Automatic    setup of the prepared test environment

   • Simplified   UI for the configuration of the test-run

   • Automatic    download of the latest tests

   • Test   execution and reporting to our database
CROWD SOURCED TESTING




                      User Profile        Testing Profile



                                    3.
                         1.
                                               Repositories
                                    2.      (Tests and Scripts)
                   MozMill
                    CLI             4.
                                                Database
        Virtual Test Environment
CROWD SOURCED TESTING



                              How it works


                      User Profile        Testing Profile



                                    3.
                         1.
                                               Repositories
                                    2.      (Tests and Scripts)
                   MozMill
                    CLI             4.
                                                Database
        Virtual Test Environment
CROWD SOURCED TESTING
CROWD SOURCED TESTING



                  Analyzing test results

   L10n test results



    • Multiple   access keys

    • Cropped     elements

    • Screenshots
CROWD SOURCED TESTING
CROWD SOURCED TESTING



                           What’s left

   • Proof   reliability of our Mozmill tests

   • Fix   focus problems with Mozmill (native events)

   • Offer   more options in the extension for contribution

   • Add    more test-runs (i.e. add-ons and endurance tests)

   • We    need your feedback!
REFERENCES




              Demo
REFERENCES


  Mozmill Crowd:
  https://addons.mozilla.org/firefox/addon/mozmill-crowd

  QA Test Automation team page:
  http://quality.mozilla.org/teams/automation/

  Mozmill Code:
  https://github.com/mozautomation/mozmill

  Mozmill Tests:
  https://developer.mozilla.org/en/Mozmill_Tests
QUESTIONS & ANSWERS




                      Any Questions?
Thanks

More Related Content

What's hot

Network Protocol Testing Using Robot Framework
Network Protocol Testing Using Robot FrameworkNetwork Protocol Testing Using Robot Framework
Network Protocol Testing Using Robot FrameworkPayal Jain
 
Robot Framework Introduction & Sauce Labs Integration
Robot Framework Introduction & Sauce Labs IntegrationRobot Framework Introduction & Sauce Labs Integration
Robot Framework Introduction & Sauce Labs IntegrationSauce Labs
 
Apereo OAE development and release process
Apereo OAE development and release processApereo OAE development and release process
Apereo OAE development and release processBert Pareyn
 
Integration Group - Robot Framework
Integration Group - Robot Framework Integration Group - Robot Framework
Integration Group - Robot Framework OpenDaylight
 
Robot framework Gowthami Goli
Robot framework Gowthami GoliRobot framework Gowthami Goli
Robot framework Gowthami GoliGowthami Buddi
 
Robotframework
RobotframeworkRobotframework
RobotframeworkElla Sun
 
Introduction to Robot Framework
Introduction to Robot FrameworkIntroduction to Robot Framework
Introduction to Robot FrameworkCarl Su
 
Acceptance testing plone sites and add ons with robot framework and selenium
Acceptance testing plone sites and add ons with robot framework and seleniumAcceptance testing plone sites and add ons with robot framework and selenium
Acceptance testing plone sites and add ons with robot framework and seleniumAsko Soukka
 
Robot Framework for beginners and what is new at 2019
Robot Framework for beginners and what is new at 2019Robot Framework for beginners and what is new at 2019
Robot Framework for beginners and what is new at 2019Laura Ojala
 
Introduce fuego
Introduce fuegoIntroduce fuego
Introduce fuegos60030
 
Introduction to Robot Framework (external)
Introduction to Robot Framework (external)Introduction to Robot Framework (external)
Introduction to Robot Framework (external)Zhe Li
 
.NET Standard - Under the Hood
.NET Standard - Under the Hood.NET Standard - Under the Hood
.NET Standard - Under the HoodImmo Landwerth
 
BePUG - PowerShell and REST - Expanding your Horizon
BePUG - PowerShell and REST - Expanding your HorizonBePUG - PowerShell and REST - Expanding your Horizon
BePUG - PowerShell and REST - Expanding your HorizonEurocontrol
 
.NET Standard - Introduction
.NET Standard - Introduction.NET Standard - Introduction
.NET Standard - IntroductionImmo Landwerth
 
Introduction to Robot Framework – Exove
Introduction to Robot Framework – ExoveIntroduction to Robot Framework – Exove
Introduction to Robot Framework – ExoveExove
 
Robot framework
Robot frameworkRobot framework
Robot frameworkboriau
 

What's hot (20)

Network Protocol Testing Using Robot Framework
Network Protocol Testing Using Robot FrameworkNetwork Protocol Testing Using Robot Framework
Network Protocol Testing Using Robot Framework
 
Robot framework
Robot frameworkRobot framework
Robot framework
 
Robot Framework
Robot FrameworkRobot Framework
Robot Framework
 
Robot framework
Robot frameworkRobot framework
Robot framework
 
Robot Framework Introduction & Sauce Labs Integration
Robot Framework Introduction & Sauce Labs IntegrationRobot Framework Introduction & Sauce Labs Integration
Robot Framework Introduction & Sauce Labs Integration
 
Apereo OAE development and release process
Apereo OAE development and release processApereo OAE development and release process
Apereo OAE development and release process
 
Integration Group - Robot Framework
Integration Group - Robot Framework Integration Group - Robot Framework
Integration Group - Robot Framework
 
Robot framework Gowthami Goli
Robot framework Gowthami GoliRobot framework Gowthami Goli
Robot framework Gowthami Goli
 
Robotframework
RobotframeworkRobotframework
Robotframework
 
Introduction to Robot Framework
Introduction to Robot FrameworkIntroduction to Robot Framework
Introduction to Robot Framework
 
Acceptance testing plone sites and add ons with robot framework and selenium
Acceptance testing plone sites and add ons with robot framework and seleniumAcceptance testing plone sites and add ons with robot framework and selenium
Acceptance testing plone sites and add ons with robot framework and selenium
 
Robot Framework for beginners and what is new at 2019
Robot Framework for beginners and what is new at 2019Robot Framework for beginners and what is new at 2019
Robot Framework for beginners and what is new at 2019
 
Introduce fuego
Introduce fuegoIntroduce fuego
Introduce fuego
 
Introduction to Robot Framework (external)
Introduction to Robot Framework (external)Introduction to Robot Framework (external)
Introduction to Robot Framework (external)
 
.NET Standard - Under the Hood
.NET Standard - Under the Hood.NET Standard - Under the Hood
.NET Standard - Under the Hood
 
BePUG - PowerShell and REST - Expanding your Horizon
BePUG - PowerShell and REST - Expanding your HorizonBePUG - PowerShell and REST - Expanding your Horizon
BePUG - PowerShell and REST - Expanding your Horizon
 
.NET Standard - Introduction
.NET Standard - Introduction.NET Standard - Introduction
.NET Standard - Introduction
 
JavaCro'14 - Test Automation using RobotFramework Libraries – Stojan Peshov
JavaCro'14 - Test Automation using RobotFramework Libraries – Stojan PeshovJavaCro'14 - Test Automation using RobotFramework Libraries – Stojan Peshov
JavaCro'14 - Test Automation using RobotFramework Libraries – Stojan Peshov
 
Introduction to Robot Framework – Exove
Introduction to Robot Framework – ExoveIntroduction to Robot Framework – Exove
Introduction to Robot Framework – Exove
 
Robot framework
Robot frameworkRobot framework
Robot framework
 

Similar to Mozmill Crowd

Test parallelization using Jenkins
Test parallelization using JenkinsTest parallelization using Jenkins
Test parallelization using JenkinsRogue Wave Software
 
Testing API's: Tools & Tips & Tricks (Oh My!)
Testing API's: Tools & Tips & Tricks (Oh My!)Testing API's: Tools & Tips & Tricks (Oh My!)
Testing API's: Tools & Tips & Tricks (Oh My!)Ford Prior
 
How to stuff a 900 pound gorilla into a smartphone
How to stuff a 900 pound gorilla into a smartphoneHow to stuff a 900 pound gorilla into a smartphone
How to stuff a 900 pound gorilla into a smartphoneJoel Maher
 
Real-Time Communication Testing Evolution with WebRTC
Real-Time Communication Testing Evolution with WebRTCReal-Time Communication Testing Evolution with WebRTC
Real-Time Communication Testing Evolution with WebRTCAlexandre Gouaillard
 
Badoo: Cross platform Mobile Test Automation and Continuos Delivery
Badoo: Cross platform Mobile Test Automation and Continuos DeliveryBadoo: Cross platform Mobile Test Automation and Continuos Delivery
Badoo: Cross platform Mobile Test Automation and Continuos DeliveryCodeFest
 
VA Smalltalk Update
VA Smalltalk UpdateVA Smalltalk Update
VA Smalltalk UpdateESUG
 
Automated Testing Environment by Bugzilla, Testopia and Jenkins
Automated Testing Environment by Bugzilla, Testopia and JenkinsAutomated Testing Environment by Bugzilla, Testopia and Jenkins
Automated Testing Environment by Bugzilla, Testopia and Jenkinswalkerchang
 
Unit Testing in JavaScript
Unit Testing in JavaScriptUnit Testing in JavaScript
Unit Testing in JavaScriptRob Scaduto
 
20060712 automated model based testing of community-driven open-source gui ap...
20060712 automated model based testing of community-driven open-source gui ap...20060712 automated model based testing of community-driven open-source gui ap...
20060712 automated model based testing of community-driven open-source gui ap...Will Shen
 
Version Control and Continuous Integration
Version Control and Continuous IntegrationVersion Control and Continuous Integration
Version Control and Continuous IntegrationGeff Henderson Chang
 
Automated Testing with Docker on Steroids - nlOUG TechExperience 2018 (Amersf...
Automated Testing with Docker on Steroids - nlOUG TechExperience 2018 (Amersf...Automated Testing with Docker on Steroids - nlOUG TechExperience 2018 (Amersf...
Automated Testing with Docker on Steroids - nlOUG TechExperience 2018 (Amersf...Lucas Jellema
 
ONOS System Test - ONS2016
ONOS System Test - ONS2016ONOS System Test - ONS2016
ONOS System Test - ONS2016Suibin Zhang
 
Infrastructure testing with Molecule and TestInfra
Infrastructure testing with Molecule and TestInfraInfrastructure testing with Molecule and TestInfra
Infrastructure testing with Molecule and TestInfraTomislav Plavcic
 
Testing sync engine
Testing sync engineTesting sync engine
Testing sync engineIlya Puchka
 
Growing Trends of Open Source UI Frameworks
Growing Trends of Open Source UI FrameworksGrowing Trends of Open Source UI Frameworks
Growing Trends of Open Source UI FrameworksSmartBear
 
What I Learned From Writing a Test Framework (And Why I May Never Write One A...
What I Learned From Writing a Test Framework (And Why I May Never Write One A...What I Learned From Writing a Test Framework (And Why I May Never Write One A...
What I Learned From Writing a Test Framework (And Why I May Never Write One A...Daryl Walleck
 

Similar to Mozmill Crowd (20)

Test parallelization using Jenkins
Test parallelization using JenkinsTest parallelization using Jenkins
Test parallelization using Jenkins
 
Testing API's: Tools & Tips & Tricks (Oh My!)
Testing API's: Tools & Tips & Tricks (Oh My!)Testing API's: Tools & Tips & Tricks (Oh My!)
Testing API's: Tools & Tips & Tricks (Oh My!)
 
How to stuff a 900 pound gorilla into a smartphone
How to stuff a 900 pound gorilla into a smartphoneHow to stuff a 900 pound gorilla into a smartphone
How to stuff a 900 pound gorilla into a smartphone
 
Real-Time Communication Testing Evolution with WebRTC
Real-Time Communication Testing Evolution with WebRTCReal-Time Communication Testing Evolution with WebRTC
Real-Time Communication Testing Evolution with WebRTC
 
Bug zillatestopiajenkins
Bug zillatestopiajenkinsBug zillatestopiajenkins
Bug zillatestopiajenkins
 
How We Test Linux
How We Test LinuxHow We Test Linux
How We Test Linux
 
Badoo: Cross platform Mobile Test Automation and Continuos Delivery
Badoo: Cross platform Mobile Test Automation and Continuos DeliveryBadoo: Cross platform Mobile Test Automation and Continuos Delivery
Badoo: Cross platform Mobile Test Automation and Continuos Delivery
 
Introduction to DevOps
Introduction to DevOpsIntroduction to DevOps
Introduction to DevOps
 
VA Smalltalk Update
VA Smalltalk UpdateVA Smalltalk Update
VA Smalltalk Update
 
Automated Testing Environment by Bugzilla, Testopia and Jenkins
Automated Testing Environment by Bugzilla, Testopia and JenkinsAutomated Testing Environment by Bugzilla, Testopia and Jenkins
Automated Testing Environment by Bugzilla, Testopia and Jenkins
 
Unit Testing in JavaScript
Unit Testing in JavaScriptUnit Testing in JavaScript
Unit Testing in JavaScript
 
20060712 automated model based testing of community-driven open-source gui ap...
20060712 automated model based testing of community-driven open-source gui ap...20060712 automated model based testing of community-driven open-source gui ap...
20060712 automated model based testing of community-driven open-source gui ap...
 
Functional Tests. PHP Unconf 2016
Functional Tests. PHP Unconf 2016Functional Tests. PHP Unconf 2016
Functional Tests. PHP Unconf 2016
 
Version Control and Continuous Integration
Version Control and Continuous IntegrationVersion Control and Continuous Integration
Version Control and Continuous Integration
 
Automated Testing with Docker on Steroids - nlOUG TechExperience 2018 (Amersf...
Automated Testing with Docker on Steroids - nlOUG TechExperience 2018 (Amersf...Automated Testing with Docker on Steroids - nlOUG TechExperience 2018 (Amersf...
Automated Testing with Docker on Steroids - nlOUG TechExperience 2018 (Amersf...
 
ONOS System Test - ONS2016
ONOS System Test - ONS2016ONOS System Test - ONS2016
ONOS System Test - ONS2016
 
Infrastructure testing with Molecule and TestInfra
Infrastructure testing with Molecule and TestInfraInfrastructure testing with Molecule and TestInfra
Infrastructure testing with Molecule and TestInfra
 
Testing sync engine
Testing sync engineTesting sync engine
Testing sync engine
 
Growing Trends of Open Source UI Frameworks
Growing Trends of Open Source UI FrameworksGrowing Trends of Open Source UI Frameworks
Growing Trends of Open Source UI Frameworks
 
What I Learned From Writing a Test Framework (And Why I May Never Write One A...
What I Learned From Writing a Test Framework (And Why I May Never Write One A...What I Learned From Writing a Test Framework (And Why I May Never Write One A...
What I Learned From Writing a Test Framework (And Why I May Never Write One A...
 

Recently uploaded

Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 

Recently uploaded (20)

20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 

Mozmill Crowd

  • 1. quality.mozilla.org qa automation Mozmill Crowd A change in how we test Firefox Henrik Skupin <hskupin@mozilla.com> Lead of QA Test Automation
  • 3. Overview • Release schedule for Firefox • Testing Firefox releases • How the Automation helps QA • Who contributes to QA? • Crowd-sourced Testing (Mozmill Crowd) • Demo
  • 5. FIREFOX RELEASES Firefox 4.0 Beta • Lately approx. 2 weeks beta cycles • Various bug fixes and improvements Firefox 3.5 and 3.6 • Security releases each 4 to 6 weeks • Both releases happen simultaneously • About 50 - 80 fixed bugs per release
  • 7. FIREFOX RELEASES Testing Firefox Releases • Exploratory and scripted testing of affected areas • Execution of functional tests, including: • approx. 40 Smoketests • approx. 330 BFT tests (Basic Functional Tests) • Time to ship to beta audience - only a few days • Time to ship the release - about 1 week
  • 9. FIREFOX RELEASES Testing Firefox Releases • Software Update • Tests from previous versions (partial / complete) • Testing ‘betatest’ and ‘beta’ channels before beta • Testing ‘releasetest’ and ‘release’ channels before release
  • 11. FIREFOX RELEASES How Automation Helps QA • Usage of Mozmill to run automated functional tests • approx. 110 (out of 330) functional tests automated • Execution of localization specific tests for all locales • (Nearly) fully automated update tests for all locales • Simultaneously execution on all supported platforms
  • 13. CONTRIBUTING Who contributes to QA? • Small QA team (~30) and contractors for Releases • Community members execute manual tests on Litmus • 80.000 nightly users for Firefox 4.0 builds • 15.000 nightly users for older Firefox branches
  • 14. CONTRIBUTING Who contributes to QA? • Small QA team (~30) and contractors for Releases • Community members execute manual tests on Litmus • 80.000 nightly users for Firefox 4.0 builds • 15.000 nightly users for older Firefox branches How can we outreach testing?
  • 15. CONTRIBUTING How can we outreach testing?
  • 16. CONTRIBUTING Who contributes to QA? • Small QA team (~30) and contractors for Releases • Community members execute manual tests on Litmus • 80.000 nightly users for Firefox 4.0 builds • 15.000 nightly users for older Firefox branches How can we outreach testing?
  • 17. CROWD SOURCED TESTING Crowdsourcing relies on people, and thus works when participants are treated as people
  • 18. CROWD SOURCED TESTING Crowd-sourced Testing Crowdsourcing relies on people, and thus works when participants are treated as people
  • 20. CROWD SOURCED TESTING What do we expect? • Improved quality of localized builds • Knowledge how Firefox behaves in different locations • Results from machines with unknown configurations • Performance related information and memory usage • ...
  • 22. CROWD SOURCED TESTING What’s necessary? • No global installation of additional software • Automatic setup of the Mozmill-CLI environment • Wide range of different test scenarios (BFT, L10n, ...) • Reporting and analysis of collected test results • Promoting tests even outside of our community
  • 24. CROWD SOURCED TESTING How it works • Mozmill-Crowd add-on for Firefox 4.0 and 3.6 • Automatic setup of the prepared test environment • Simplified UI for the configuration of the test-run • Automatic download of the latest tests • Test execution and reporting to our database
  • 25. CROWD SOURCED TESTING User Profile Testing Profile 3. 1. Repositories 2. (Tests and Scripts) MozMill CLI 4. Database Virtual Test Environment
  • 26. CROWD SOURCED TESTING How it works User Profile Testing Profile 3. 1. Repositories 2. (Tests and Scripts) MozMill CLI 4. Database Virtual Test Environment
  • 28. CROWD SOURCED TESTING Analyzing test results L10n test results • Multiple access keys • Cropped elements • Screenshots
  • 30. CROWD SOURCED TESTING What’s left • Proof reliability of our Mozmill tests • Fix focus problems with Mozmill (native events) • Offer more options in the extension for contribution • Add more test-runs (i.e. add-ons and endurance tests) • We need your feedback!
  • 31. REFERENCES Demo
  • 32. REFERENCES Mozmill Crowd: https://addons.mozilla.org/firefox/addon/mozmill-crowd QA Test Automation team page: http://quality.mozilla.org/teams/automation/ Mozmill Code: https://github.com/mozautomation/mozmill Mozmill Tests: https://developer.mozilla.org/en/Mozmill_Tests
  • 33. QUESTIONS & ANSWERS Any Questions?

Editor's Notes

  1. \n
  2. \n
  3. \n
  4. \n
  5. \n
  6. \n
  7. \n
  8. \n
  9. \n
  10. \n
  11. \n
  12. \n
  13. \n
  14. \n
  15. \n
  16. \n
  17. \n
  18. \n
  19. \n
  20. \n