SlideShare a Scribd company logo
1 of 28
Download to read offline
Building trust within the
organization, first steps towards
            DevOps
         Guido Serra, txtr
What’s the role of a DevOp(s)?
•   Deliver
•   Be bridge of trust between DEVs and SysOPs
•   Stop the “throw the ball over the fence” game
•   Mediate
•   Drive non-functional requirements

           … DevOp or DevOps, talking of one? 
Introduce a DevOp(s)
• In ‘txtr, starting as a QA Manager, specialised
  on backend systems, seems to have worked
• Other organizations tends to call it Site
  Reliability Engineer / Site Reliability Operation
• But… QA != Testing, not strictly at least
  – Testing should be only a subset of QA, but that is
    not how it is normally perceived
  – Non-functional requirements did not seem to fit in
Non-functional requirements?
• Functional requirements == features
• Non-functional requirements == everything that
  OPS would need to run the service, or even
  things that Product Owners would want but has
  not thought of at the design time
  – Logging
     • Which kind of informations?
     • How?
  – Health checks / Load Balancer required URL
  – Live sales report / Dashboard / Charting
Steps that worked so far
• Listen …to OPS, to PMs, to QA, to R&D
• See how the people have solved their specific
  needs trying to gather informations
• Match all the tools that have been built
• Try to gather the essence of those tools, and
  come up with non-functional requirements
• Discuss those with the R&D organization and
  push them at Product level to be prioritized
  over features
TRUST
Means…
• Not having to duplicate work
  – wrongly testing the backend to see if it is
    answering
  – or testing to measure the response times
  – or creating tests again, when there are plenty of
    them that are simply not shared and/or broadly
    understood
The answer is 42?
…no, the answer is DATA!
• Creating a single point of data collection and
  graphing, people are gaining trust in the
  backend
• Logs need to be shared too
• Tests needs to be commonly understood
Logging

SHARE LOGS WITH EVERYONE
Tools
• Logging
  – Slf4j > Log4j / JUL > GELF > GrayLog2
     • Logging to syslog from a Java based backend, is pretty
       bad. The stacktrace become very hard to be fetched
       and reported in a ticket. Instead, one link and a
       screenshot, or a cut&paste of a complete stacktrace
       from a web interface is much more easy to be digested
     • GELF is a notification format, encapsulating the full
       stacktrace as a message
     • GrayLog2 is a ruby/MongoDB FIFO queue with a nice
       web interface, and an alerting email system
Why?
• Slf4j
   – It is an abstraction layer on logging facilities
          • I’ll not explain why an “abstraction layer” is good
• Log4j or JUL, at your choice
   – They are the most commonly used
          • Means: their code is maintained
• GELF
   – It keeps a full stacktrace in a single message. There is no need of
     reconstructing it from syslog, spread on multiple lines and with
     additional garbage/timestamps
• GrayLog2
   – We have an in-house developer, and it is working pretty well
   – Has threshold based alerting per streams of events (regexp)
Building trust within the organization, first steps towards DevOps
Building trust within the organization, first steps towards DevOps
Results seen so far
• 1st level support team is gaining trust in the
  application.
  – Logs are getting more and more readable
  – Events can be correlated much more easily
• 2nd level support (OPS) can set thresholds of
  alerts and react promptly, having alerts tight
  to real traffic data and not “one time probes”
• I have a better feeling of the trend of issues in
  production, and I don’t have to dig for logs
Instrumented metrics

PRODUCTION PERFORMANCE
Tools
• Instrumented metrics
  – JMX > Jolokia > JSON > Graphite
     • MIN / MAX / AVG response time of each API
     • Worst response times with related API parameters
     • Success / failure counters
     • All the above aggregated over the last 5 / 15 minutes, 1
       hour, 24 hours
     • Plus all the standard exposed JConsole / JMX infos
Why?
• JMX
  – It is built in in Java, and it is non-invasive
     • R&D loves it, cause it does not need an invasive agent as
       many profiling agents that are normally used in such cases.
       Standard profiling agents tend to interfere with the
       application and decrease the overall performance.
  – It is a standard, so there are many tools that plug into
    it natively
• Jolokia
  – It is a standard tool that plugs into JMX and expose it
    as JSON encoded format
Why?
• Graphite
  – It can correlate data from many sources
  – Gives me the freedom of structuring graphs as I
    want, directly from the web interface
     • This is a definitive WIN over Munin or Cacti
  – It lets me select specific timeframes
     • In case of outage investigation. Thing which is not
       possible with Munin
  – Can create dashboards
Data are in transactions “per 5 minutes” in this graph
…you can see this specific service is currently being used
100 transactions per second
uhmm… at 7a.m., ok 11a.m. in India
someone is testing…
Results seen so far
• No need of load and performance testing
  – Apart of specific cases, to try to reproduce the
    issue to let DEVs work on it.
  – Producing a proper load test is problematic, and
    can bring to false assumptions about the product.
    Having the possibility to watch what the business
    logic is doing in production is the best load test.
• DEVs are proactively watching and fixing
  performance issues on their own. The overall
  product gets better and better.
Testing

SHARE TESTS AND RESULTS
Tools
• Testing
  – BDD / Cucumber-Nagios executed by Jenkins
     • Cover all the fast HTTP action via Watir
     • API calls via JsonRPC or Soap4Rr
     • Javascript based UI via Selenium / Capybara
• These tests are actually very valuable at
  deployment time, since there is no need of
  manual testing. All is in the hand of whom
  follows the deployment.
Why?
• BDD
  – Not everyone wants to read your code
  – Not everyone is a coder
  – You don’t want to have to explain your test again and
    again and again, and you hate documenting
• Cucumber-Nagios / Ruby
  – It is off-the-shelf, it works.
  – It generates standard JUnit XML report
     • Means: it directly integrates with Jenkins ( ex Hudson )
  – It generates an awesome HTML report
  – It can be extended pretty easily
Why?
• Watir
  – It is the default HTTP client in Cucumber-Nagios
     • BUT: it has tons of bugs… I have a long backlog to fix
  – It is fast
• Soap4r
  – Pretty easy SOAP ruby gem/library
• JsonRPC
  – Very simple and basic JSON RPC gem/library
     • BUT: it does not support proxy settings
Why?
• Selenium
  – Cause it is the only one?
  – It supports Javascript
  – It supports clustering of testing nodes
  – It is supposed to be easy to integrate with
    Cucumber (it is NOT …I’m working on it)
Building trust within the organization, first steps towards DevOps
Upcoming…
• Health checks (normally used for load
  balancing purposes) are based on business
  logic historical data from within the
  instrumented metrics
• Continuous integration
  – Configuration management
• Data mining
guido.serra@txtr.com

QUESTIONS?

                       http://slidesha.re/rVzd8F

More Related Content

What's hot

So you-want-to-go-faster
So you-want-to-go-fasterSo you-want-to-go-faster
So you-want-to-go-fasterOoblioob
 
Joxean Koret - Interactive Static Analysis Tools for Vulnerability Discovery ...
Joxean Koret - Interactive Static Analysis Tools for Vulnerability Discovery ...Joxean Koret - Interactive Static Analysis Tools for Vulnerability Discovery ...
Joxean Koret - Interactive Static Analysis Tools for Vulnerability Discovery ...RootedCON
 
Requirements Engineering - a tale from the trenches
Requirements Engineering - a tale from the trenchesRequirements Engineering - a tale from the trenches
Requirements Engineering - a tale from the trenchesEric D. Schabell
 
jBPM Migration - generating your process future
jBPM Migration - generating your process futurejBPM Migration - generating your process future
jBPM Migration - generating your process futureEric D. Schabell
 
Static Analysis Techniques For Testing Application Security - Houston Tech Fest
Static Analysis Techniques For Testing Application Security - Houston Tech FestStatic Analysis Techniques For Testing Application Security - Houston Tech Fest
Static Analysis Techniques For Testing Application Security - Houston Tech FestDenim Group
 
Bringing CD to the DoD
Bringing CD to the DoDBringing CD to the DoD
Bringing CD to the DoDGene Gotimer
 
The challenges and pitfalls of database deployment automation
The challenges and pitfalls of database deployment automationThe challenges and pitfalls of database deployment automation
The challenges and pitfalls of database deployment automationDBmaestro - Database DevOps
 
Automating development-operations-v1
Automating development-operations-v1Automating development-operations-v1
Automating development-operations-v1Sumanth Vepa
 
Security Implications for a DevOps Transformation
Security Implications for a DevOps TransformationSecurity Implications for a DevOps Transformation
Security Implications for a DevOps TransformationDevOps.com
 
Enhanced Verification Flow with Nextop's Assertion Synthesis Technology
Enhanced Verification Flow with Nextop's Assertion Synthesis TechnologyEnhanced Verification Flow with Nextop's Assertion Synthesis Technology
Enhanced Verification Flow with Nextop's Assertion Synthesis TechnologyDVClub
 
Generalization in Auto-Testing. How we put what we had into new Technological...
Generalization in Auto-Testing. How we put what we had into new Technological...Generalization in Auto-Testing. How we put what we had into new Technological...
Generalization in Auto-Testing. How we put what we had into new Technological...SQALab
 
The Continuous delivery value - Funaro
The Continuous delivery value - FunaroThe Continuous delivery value - Funaro
The Continuous delivery value - FunaroCodemotion
 
Code refactoring
Code refactoringCode refactoring
Code refactoringLalit Kale
 
Rob Sabourin: On Testing
Rob Sabourin: On TestingRob Sabourin: On Testing
Rob Sabourin: On TestingTechWell
 
Software Defects and SW Reliability Assessment
Software Defects and SW Reliability AssessmentSoftware Defects and SW Reliability Assessment
Software Defects and SW Reliability AssessmentKristine Hejna
 
“Sensu and Sensibility” - The Story of a Journey From #monitoringsucks to #mo...
“Sensu and Sensibility” - The Story of a Journey From #monitoringsucks to #mo...“Sensu and Sensibility” - The Story of a Journey From #monitoringsucks to #mo...
“Sensu and Sensibility” - The Story of a Journey From #monitoringsucks to #mo...Puppet
 
Ginsbourg.com presentation of open source performance validation
Ginsbourg.com presentation of open source performance validationGinsbourg.com presentation of open source performance validation
Ginsbourg.com presentation of open source performance validationPerfecto Mobile
 

What's hot (20)

So you-want-to-go-faster
So you-want-to-go-fasterSo you-want-to-go-faster
So you-want-to-go-faster
 
Joxean Koret - Interactive Static Analysis Tools for Vulnerability Discovery ...
Joxean Koret - Interactive Static Analysis Tools for Vulnerability Discovery ...Joxean Koret - Interactive Static Analysis Tools for Vulnerability Discovery ...
Joxean Koret - Interactive Static Analysis Tools for Vulnerability Discovery ...
 
Requirements Engineering - a tale from the trenches
Requirements Engineering - a tale from the trenchesRequirements Engineering - a tale from the trenches
Requirements Engineering - a tale from the trenches
 
jBPM Migration - generating your process future
jBPM Migration - generating your process futurejBPM Migration - generating your process future
jBPM Migration - generating your process future
 
Static Analysis Techniques For Testing Application Security - Houston Tech Fest
Static Analysis Techniques For Testing Application Security - Houston Tech FestStatic Analysis Techniques For Testing Application Security - Houston Tech Fest
Static Analysis Techniques For Testing Application Security - Houston Tech Fest
 
Bringing CD to the DoD
Bringing CD to the DoDBringing CD to the DoD
Bringing CD to the DoD
 
Benefits from AATs
Benefits from AATsBenefits from AATs
Benefits from AATs
 
The challenges and pitfalls of database deployment automation
The challenges and pitfalls of database deployment automationThe challenges and pitfalls of database deployment automation
The challenges and pitfalls of database deployment automation
 
Automating development-operations-v1
Automating development-operations-v1Automating development-operations-v1
Automating development-operations-v1
 
Static code analysis
Static code analysisStatic code analysis
Static code analysis
 
Security Implications for a DevOps Transformation
Security Implications for a DevOps TransformationSecurity Implications for a DevOps Transformation
Security Implications for a DevOps Transformation
 
Enhanced Verification Flow with Nextop's Assertion Synthesis Technology
Enhanced Verification Flow with Nextop's Assertion Synthesis TechnologyEnhanced Verification Flow with Nextop's Assertion Synthesis Technology
Enhanced Verification Flow with Nextop's Assertion Synthesis Technology
 
Generalization in Auto-Testing. How we put what we had into new Technological...
Generalization in Auto-Testing. How we put what we had into new Technological...Generalization in Auto-Testing. How we put what we had into new Technological...
Generalization in Auto-Testing. How we put what we had into new Technological...
 
The Continuous delivery value - Funaro
The Continuous delivery value - FunaroThe Continuous delivery value - Funaro
The Continuous delivery value - Funaro
 
Code refactoring
Code refactoringCode refactoring
Code refactoring
 
Extreme Makeover OnBase Edition
Extreme Makeover OnBase EditionExtreme Makeover OnBase Edition
Extreme Makeover OnBase Edition
 
Rob Sabourin: On Testing
Rob Sabourin: On TestingRob Sabourin: On Testing
Rob Sabourin: On Testing
 
Software Defects and SW Reliability Assessment
Software Defects and SW Reliability AssessmentSoftware Defects and SW Reliability Assessment
Software Defects and SW Reliability Assessment
 
“Sensu and Sensibility” - The Story of a Journey From #monitoringsucks to #mo...
“Sensu and Sensibility” - The Story of a Journey From #monitoringsucks to #mo...“Sensu and Sensibility” - The Story of a Journey From #monitoringsucks to #mo...
“Sensu and Sensibility” - The Story of a Journey From #monitoringsucks to #mo...
 
Ginsbourg.com presentation of open source performance validation
Ginsbourg.com presentation of open source performance validationGinsbourg.com presentation of open source performance validation
Ginsbourg.com presentation of open source performance validation
 

Viewers also liked

The end of traditional enterprise IT - ING's journey to the next generation I...
The end of traditional enterprise IT - ING's journey to the next generation I...The end of traditional enterprise IT - ING's journey to the next generation I...
The end of traditional enterprise IT - ING's journey to the next generation I...NLJUG
 
Deploying 30 times a day, and making sure everything stays 200 OK by Eric Sigler
Deploying 30 times a day, and making sure everything stays 200 OK by Eric SiglerDeploying 30 times a day, and making sure everything stays 200 OK by Eric Sigler
Deploying 30 times a day, and making sure everything stays 200 OK by Eric SiglerDevOpsDays Baltimore
 
Minding your P's and Q's: Enrich-ing Enlighten
Minding your P's and Q's: Enrich-ing EnlightenMinding your P's and Q's: Enrich-ing Enlighten
Minding your P's and Q's: Enrich-ing Enlightenenlightenrepository
 
Cevora ICT Symposium - Graph Databases
Cevora ICT Symposium - Graph DatabasesCevora ICT Symposium - Graph Databases
Cevora ICT Symposium - Graph DatabasesRik Van Bruggen
 
Northbridge capital organised retail pharmacy india 2011
Northbridge capital organised retail pharmacy india 2011Northbridge capital organised retail pharmacy india 2011
Northbridge capital organised retail pharmacy india 2011rohit_shankar
 
Gestion de la Innovacion en Cofas
Gestion de la Innovacion en CofasGestion de la Innovacion en Cofas
Gestion de la Innovacion en CofasFarmaceuticos Mundi
 
Channel Partner data sheet
Channel Partner data sheetChannel Partner data sheet
Channel Partner data sheetIntelePeer
 
Raport z monitoringu budżetu obywatelskiego w Sochaczewie
Raport z monitoringu budżetu obywatelskiego w SochaczewieRaport z monitoringu budżetu obywatelskiego w Sochaczewie
Raport z monitoringu budżetu obywatelskiego w SochaczewieMarcin Germanek
 
Palladium Magazine (Special Summer 2014)
Palladium Magazine (Special Summer 2014)Palladium Magazine (Special Summer 2014)
Palladium Magazine (Special Summer 2014)Palladium Hotel Group
 
Act. iv ethernet y planificación y cableado de redes
Act. iv ethernet y planificación y cableado de redesAct. iv ethernet y planificación y cableado de redes
Act. iv ethernet y planificación y cableado de redesAry Roque
 
U.S. Travel Trailer And Camper Market. Analysis And Forecast to 2020
U.S. Travel Trailer And Camper Market. Analysis And Forecast to 2020U.S. Travel Trailer And Camper Market. Analysis And Forecast to 2020
U.S. Travel Trailer And Camper Market. Analysis And Forecast to 2020IndexBox Marketing
 
How has the internet affected the way we
How has the internet affected the way weHow has the internet affected the way we
How has the internet affected the way weaimeechh
 
~~Putting~~ Convincing the Ops in DevOps by Jamie Jones
~~Putting~~ Convincing the Ops in DevOps by Jamie Jones~~Putting~~ Convincing the Ops in DevOps by Jamie Jones
~~Putting~~ Convincing the Ops in DevOps by Jamie JonesDevOpsDays Baltimore
 
PresentacióN Movilflota 2011
PresentacióN Movilflota   2011PresentacióN Movilflota   2011
PresentacióN Movilflota 2011jfcuellogarcia
 
Influencias escolha de repertorio
Influencias escolha de repertorioInfluencias escolha de repertorio
Influencias escolha de repertorioMárcia Lyra
 
Democracia y Derecho Constitucional - David Mercado Pérez
Democracia y Derecho Constitucional - David Mercado PérezDemocracia y Derecho Constitucional - David Mercado Pérez
Democracia y Derecho Constitucional - David Mercado PérezTaEthekai
 
Calidad Del Aire En Torrelavega Red
Calidad Del Aire En Torrelavega RedCalidad Del Aire En Torrelavega Red
Calidad Del Aire En Torrelavega Redplataformabesaya
 
Presentacion historia del facebook y su impacto en la sociedad
Presentacion historia del facebook y su impacto en la sociedadPresentacion historia del facebook y su impacto en la sociedad
Presentacion historia del facebook y su impacto en la sociedadANGELICA YADIRA OYAGUE VELEZ
 

Viewers also liked (20)

The end of traditional enterprise IT - ING's journey to the next generation I...
The end of traditional enterprise IT - ING's journey to the next generation I...The end of traditional enterprise IT - ING's journey to the next generation I...
The end of traditional enterprise IT - ING's journey to the next generation I...
 
Deploying 30 times a day, and making sure everything stays 200 OK by Eric Sigler
Deploying 30 times a day, and making sure everything stays 200 OK by Eric SiglerDeploying 30 times a day, and making sure everything stays 200 OK by Eric Sigler
Deploying 30 times a day, and making sure everything stays 200 OK by Eric Sigler
 
Minding your P's and Q's: Enrich-ing Enlighten
Minding your P's and Q's: Enrich-ing EnlightenMinding your P's and Q's: Enrich-ing Enlighten
Minding your P's and Q's: Enrich-ing Enlighten
 
Interpolacion POLINOMICA DE NEWTON
Interpolacion POLINOMICA DE NEWTONInterpolacion POLINOMICA DE NEWTON
Interpolacion POLINOMICA DE NEWTON
 
Cevora ICT Symposium - Graph Databases
Cevora ICT Symposium - Graph DatabasesCevora ICT Symposium - Graph Databases
Cevora ICT Symposium - Graph Databases
 
Northbridge capital organised retail pharmacy india 2011
Northbridge capital organised retail pharmacy india 2011Northbridge capital organised retail pharmacy india 2011
Northbridge capital organised retail pharmacy india 2011
 
Gestion de la Innovacion en Cofas
Gestion de la Innovacion en CofasGestion de la Innovacion en Cofas
Gestion de la Innovacion en Cofas
 
Channel Partner data sheet
Channel Partner data sheetChannel Partner data sheet
Channel Partner data sheet
 
Raport z monitoringu budżetu obywatelskiego w Sochaczewie
Raport z monitoringu budżetu obywatelskiego w SochaczewieRaport z monitoringu budżetu obywatelskiego w Sochaczewie
Raport z monitoringu budżetu obywatelskiego w Sochaczewie
 
Palladium Magazine (Special Summer 2014)
Palladium Magazine (Special Summer 2014)Palladium Magazine (Special Summer 2014)
Palladium Magazine (Special Summer 2014)
 
Act. iv ethernet y planificación y cableado de redes
Act. iv ethernet y planificación y cableado de redesAct. iv ethernet y planificación y cableado de redes
Act. iv ethernet y planificación y cableado de redes
 
U.S. Travel Trailer And Camper Market. Analysis And Forecast to 2020
U.S. Travel Trailer And Camper Market. Analysis And Forecast to 2020U.S. Travel Trailer And Camper Market. Analysis And Forecast to 2020
U.S. Travel Trailer And Camper Market. Analysis And Forecast to 2020
 
How has the internet affected the way we
How has the internet affected the way weHow has the internet affected the way we
How has the internet affected the way we
 
Texto 3
Texto 3Texto 3
Texto 3
 
~~Putting~~ Convincing the Ops in DevOps by Jamie Jones
~~Putting~~ Convincing the Ops in DevOps by Jamie Jones~~Putting~~ Convincing the Ops in DevOps by Jamie Jones
~~Putting~~ Convincing the Ops in DevOps by Jamie Jones
 
PresentacióN Movilflota 2011
PresentacióN Movilflota   2011PresentacióN Movilflota   2011
PresentacióN Movilflota 2011
 
Influencias escolha de repertorio
Influencias escolha de repertorioInfluencias escolha de repertorio
Influencias escolha de repertorio
 
Democracia y Derecho Constitucional - David Mercado Pérez
Democracia y Derecho Constitucional - David Mercado PérezDemocracia y Derecho Constitucional - David Mercado Pérez
Democracia y Derecho Constitucional - David Mercado Pérez
 
Calidad Del Aire En Torrelavega Red
Calidad Del Aire En Torrelavega RedCalidad Del Aire En Torrelavega Red
Calidad Del Aire En Torrelavega Red
 
Presentacion historia del facebook y su impacto en la sociedad
Presentacion historia del facebook y su impacto en la sociedadPresentacion historia del facebook y su impacto en la sociedad
Presentacion historia del facebook y su impacto en la sociedad
 

Similar to Building trust within the organization, first steps towards DevOps

Profiling and Tuning a Web Application - The Dirty Details
Profiling and Tuning a Web Application - The Dirty DetailsProfiling and Tuning a Web Application - The Dirty Details
Profiling and Tuning a Web Application - The Dirty DetailsAchievers Tech
 
Performance tuning Grails applications
Performance tuning Grails applicationsPerformance tuning Grails applications
Performance tuning Grails applicationsLari Hotari
 
Behavior-Driven Development (BDD) Testing with Apache Spark with Aaron Colcor...
Behavior-Driven Development (BDD) Testing with Apache Spark with Aaron Colcor...Behavior-Driven Development (BDD) Testing with Apache Spark with Aaron Colcor...
Behavior-Driven Development (BDD) Testing with Apache Spark with Aaron Colcor...Databricks
 
Road to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.comRoad to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.comAviran Mordo
 
Presentation application server diagnostics
Presentation   application server diagnosticsPresentation   application server diagnostics
Presentation application server diagnosticsxKinAnx
 
Performance tuning Grails Applications GR8Conf US 2014
Performance tuning Grails Applications GR8Conf US 2014Performance tuning Grails Applications GR8Conf US 2014
Performance tuning Grails Applications GR8Conf US 2014Lari Hotari
 
" Performance testing for Automation QA - why and how " by Andrey Kovalenko f...
" Performance testing for Automation QA - why and how " by Andrey Kovalenko f..." Performance testing for Automation QA - why and how " by Andrey Kovalenko f...
" Performance testing for Automation QA - why and how " by Andrey Kovalenko f...Lohika_Odessa_TechTalks
 
Webinar: Keep Calm and Scale Out - A proactive guide to Monitoring MongoDB
Webinar: Keep Calm and Scale Out - A proactive guide to Monitoring MongoDBWebinar: Keep Calm and Scale Out - A proactive guide to Monitoring MongoDB
Webinar: Keep Calm and Scale Out - A proactive guide to Monitoring MongoDBMongoDB
 
5 Steps to Jump Start Your Test Automation
5 Steps to Jump Start Your Test Automation5 Steps to Jump Start Your Test Automation
5 Steps to Jump Start Your Test AutomationSauce Labs
 
Topic production code
Topic production codeTopic production code
Topic production codeKavi Kumar
 
Cerberus : Framework for Manual and Automated Testing (Web Application)
Cerberus : Framework for Manual and Automated Testing (Web Application)Cerberus : Framework for Manual and Automated Testing (Web Application)
Cerberus : Framework for Manual and Automated Testing (Web Application)CIVEL Benoit
 
Cerberus_Presentation1
Cerberus_Presentation1Cerberus_Presentation1
Cerberus_Presentation1CIVEL Benoit
 
I pushed in production :). Have a nice weekend
I pushed in production :). Have a nice weekendI pushed in production :). Have a nice weekend
I pushed in production :). Have a nice weekendNicolas Carlier
 
Infrastructure as Code for Network
Infrastructure as Code for NetworkInfrastructure as Code for Network
Infrastructure as Code for NetworkDamien Garros
 
Change Management in Hybrid landscapes 2017
Change Management in Hybrid landscapes 2017Change Management in Hybrid landscapes 2017
Change Management in Hybrid landscapes 2017Chris Kernaghan
 
[DPE Summit] How Improving the Testing Experience Goes Beyond Quality: A Deve...
[DPE Summit] How Improving the Testing Experience Goes Beyond Quality: A Deve...[DPE Summit] How Improving the Testing Experience Goes Beyond Quality: A Deve...
[DPE Summit] How Improving the Testing Experience Goes Beyond Quality: A Deve...Roberto Pérez Alcolea
 

Similar to Building trust within the organization, first steps towards DevOps (20)

Profiling and Tuning a Web Application - The Dirty Details
Profiling and Tuning a Web Application - The Dirty DetailsProfiling and Tuning a Web Application - The Dirty Details
Profiling and Tuning a Web Application - The Dirty Details
 
Performance tuning Grails applications
Performance tuning Grails applicationsPerformance tuning Grails applications
Performance tuning Grails applications
 
Behavior-Driven Development (BDD) Testing with Apache Spark with Aaron Colcor...
Behavior-Driven Development (BDD) Testing with Apache Spark with Aaron Colcor...Behavior-Driven Development (BDD) Testing with Apache Spark with Aaron Colcor...
Behavior-Driven Development (BDD) Testing with Apache Spark with Aaron Colcor...
 
Road to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.comRoad to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.com
 
Presentation application server diagnostics
Presentation   application server diagnosticsPresentation   application server diagnostics
Presentation application server diagnostics
 
Redundant devops
Redundant devopsRedundant devops
Redundant devops
 
Performance tuning Grails Applications GR8Conf US 2014
Performance tuning Grails Applications GR8Conf US 2014Performance tuning Grails Applications GR8Conf US 2014
Performance tuning Grails Applications GR8Conf US 2014
 
ADF Performance Monitor
ADF Performance MonitorADF Performance Monitor
ADF Performance Monitor
 
" Performance testing for Automation QA - why and how " by Andrey Kovalenko f...
" Performance testing for Automation QA - why and how " by Andrey Kovalenko f..." Performance testing for Automation QA - why and how " by Andrey Kovalenko f...
" Performance testing for Automation QA - why and how " by Andrey Kovalenko f...
 
NYC MeetUp 10.9
NYC MeetUp 10.9NYC MeetUp 10.9
NYC MeetUp 10.9
 
Kku2011
Kku2011Kku2011
Kku2011
 
Webinar: Keep Calm and Scale Out - A proactive guide to Monitoring MongoDB
Webinar: Keep Calm and Scale Out - A proactive guide to Monitoring MongoDBWebinar: Keep Calm and Scale Out - A proactive guide to Monitoring MongoDB
Webinar: Keep Calm and Scale Out - A proactive guide to Monitoring MongoDB
 
5 Steps to Jump Start Your Test Automation
5 Steps to Jump Start Your Test Automation5 Steps to Jump Start Your Test Automation
5 Steps to Jump Start Your Test Automation
 
Topic production code
Topic production codeTopic production code
Topic production code
 
Cerberus : Framework for Manual and Automated Testing (Web Application)
Cerberus : Framework for Manual and Automated Testing (Web Application)Cerberus : Framework for Manual and Automated Testing (Web Application)
Cerberus : Framework for Manual and Automated Testing (Web Application)
 
Cerberus_Presentation1
Cerberus_Presentation1Cerberus_Presentation1
Cerberus_Presentation1
 
I pushed in production :). Have a nice weekend
I pushed in production :). Have a nice weekendI pushed in production :). Have a nice weekend
I pushed in production :). Have a nice weekend
 
Infrastructure as Code for Network
Infrastructure as Code for NetworkInfrastructure as Code for Network
Infrastructure as Code for Network
 
Change Management in Hybrid landscapes 2017
Change Management in Hybrid landscapes 2017Change Management in Hybrid landscapes 2017
Change Management in Hybrid landscapes 2017
 
[DPE Summit] How Improving the Testing Experience Goes Beyond Quality: A Deve...
[DPE Summit] How Improving the Testing Experience Goes Beyond Quality: A Deve...[DPE Summit] How Improving the Testing Experience Goes Beyond Quality: A Deve...
[DPE Summit] How Improving the Testing Experience Goes Beyond Quality: A Deve...
 

Recently uploaded

20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 

Recently uploaded (20)

20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 

Building trust within the organization, first steps towards DevOps

  • 1. Building trust within the organization, first steps towards DevOps Guido Serra, txtr
  • 2. What’s the role of a DevOp(s)? • Deliver • Be bridge of trust between DEVs and SysOPs • Stop the “throw the ball over the fence” game • Mediate • Drive non-functional requirements … DevOp or DevOps, talking of one? 
  • 3. Introduce a DevOp(s) • In ‘txtr, starting as a QA Manager, specialised on backend systems, seems to have worked • Other organizations tends to call it Site Reliability Engineer / Site Reliability Operation • But… QA != Testing, not strictly at least – Testing should be only a subset of QA, but that is not how it is normally perceived – Non-functional requirements did not seem to fit in
  • 4. Non-functional requirements? • Functional requirements == features • Non-functional requirements == everything that OPS would need to run the service, or even things that Product Owners would want but has not thought of at the design time – Logging • Which kind of informations? • How? – Health checks / Load Balancer required URL – Live sales report / Dashboard / Charting
  • 5. Steps that worked so far • Listen …to OPS, to PMs, to QA, to R&D • See how the people have solved their specific needs trying to gather informations • Match all the tools that have been built • Try to gather the essence of those tools, and come up with non-functional requirements • Discuss those with the R&D organization and push them at Product level to be prioritized over features
  • 6. TRUST Means… • Not having to duplicate work – wrongly testing the backend to see if it is answering – or testing to measure the response times – or creating tests again, when there are plenty of them that are simply not shared and/or broadly understood
  • 7. The answer is 42? …no, the answer is DATA! • Creating a single point of data collection and graphing, people are gaining trust in the backend • Logs need to be shared too • Tests needs to be commonly understood
  • 9. Tools • Logging – Slf4j > Log4j / JUL > GELF > GrayLog2 • Logging to syslog from a Java based backend, is pretty bad. The stacktrace become very hard to be fetched and reported in a ticket. Instead, one link and a screenshot, or a cut&paste of a complete stacktrace from a web interface is much more easy to be digested • GELF is a notification format, encapsulating the full stacktrace as a message • GrayLog2 is a ruby/MongoDB FIFO queue with a nice web interface, and an alerting email system
  • 10. Why? • Slf4j – It is an abstraction layer on logging facilities • I’ll not explain why an “abstraction layer” is good • Log4j or JUL, at your choice – They are the most commonly used • Means: their code is maintained • GELF – It keeps a full stacktrace in a single message. There is no need of reconstructing it from syslog, spread on multiple lines and with additional garbage/timestamps • GrayLog2 – We have an in-house developer, and it is working pretty well – Has threshold based alerting per streams of events (regexp)
  • 13. Results seen so far • 1st level support team is gaining trust in the application. – Logs are getting more and more readable – Events can be correlated much more easily • 2nd level support (OPS) can set thresholds of alerts and react promptly, having alerts tight to real traffic data and not “one time probes” • I have a better feeling of the trend of issues in production, and I don’t have to dig for logs
  • 15. Tools • Instrumented metrics – JMX > Jolokia > JSON > Graphite • MIN / MAX / AVG response time of each API • Worst response times with related API parameters • Success / failure counters • All the above aggregated over the last 5 / 15 minutes, 1 hour, 24 hours • Plus all the standard exposed JConsole / JMX infos
  • 16. Why? • JMX – It is built in in Java, and it is non-invasive • R&D loves it, cause it does not need an invasive agent as many profiling agents that are normally used in such cases. Standard profiling agents tend to interfere with the application and decrease the overall performance. – It is a standard, so there are many tools that plug into it natively • Jolokia – It is a standard tool that plugs into JMX and expose it as JSON encoded format
  • 17. Why? • Graphite – It can correlate data from many sources – Gives me the freedom of structuring graphs as I want, directly from the web interface • This is a definitive WIN over Munin or Cacti – It lets me select specific timeframes • In case of outage investigation. Thing which is not possible with Munin – Can create dashboards
  • 18. Data are in transactions “per 5 minutes” in this graph …you can see this specific service is currently being used
  • 19. 100 transactions per second uhmm… at 7a.m., ok 11a.m. in India someone is testing…
  • 20. Results seen so far • No need of load and performance testing – Apart of specific cases, to try to reproduce the issue to let DEVs work on it. – Producing a proper load test is problematic, and can bring to false assumptions about the product. Having the possibility to watch what the business logic is doing in production is the best load test. • DEVs are proactively watching and fixing performance issues on their own. The overall product gets better and better.
  • 22. Tools • Testing – BDD / Cucumber-Nagios executed by Jenkins • Cover all the fast HTTP action via Watir • API calls via JsonRPC or Soap4Rr • Javascript based UI via Selenium / Capybara • These tests are actually very valuable at deployment time, since there is no need of manual testing. All is in the hand of whom follows the deployment.
  • 23. Why? • BDD – Not everyone wants to read your code – Not everyone is a coder – You don’t want to have to explain your test again and again and again, and you hate documenting • Cucumber-Nagios / Ruby – It is off-the-shelf, it works. – It generates standard JUnit XML report • Means: it directly integrates with Jenkins ( ex Hudson ) – It generates an awesome HTML report – It can be extended pretty easily
  • 24. Why? • Watir – It is the default HTTP client in Cucumber-Nagios • BUT: it has tons of bugs… I have a long backlog to fix – It is fast • Soap4r – Pretty easy SOAP ruby gem/library • JsonRPC – Very simple and basic JSON RPC gem/library • BUT: it does not support proxy settings
  • 25. Why? • Selenium – Cause it is the only one? – It supports Javascript – It supports clustering of testing nodes – It is supposed to be easy to integrate with Cucumber (it is NOT …I’m working on it)
  • 27. Upcoming… • Health checks (normally used for load balancing purposes) are based on business logic historical data from within the instrumented metrics • Continuous integration – Configuration management • Data mining
  • 28. guido.serra@txtr.com QUESTIONS? http://slidesha.re/rVzd8F