The Devops Wonder
•Download as PPTX, PDF•
17 likes•8,631 views
In today's digital economy, it is not the big that eat the small, but the fast that eat the slow. Traditional IT project management has led to spectacular failures in terms of delays and risk management, on the other hand web leaders like Amazon, Netflix or Google have reached incredible velocity through disruptive implementation of agile principles. Among the different agile movements, Devops gathers development and operation experts on how agility can be implemented all the way from the technical conception to the production environment to reach maximum velocity. Learn from our experience on many large-scale Symfony2 projects, with up to 15 developers. I will present what Devops means for us and how it helps us deliver much faster and much more reliably. We will see the many cultural, management and technical challenges we faced and the solutions we tried, based on a very powerful technical stack: Symfony2, OpenStack, Puppet, Vagrant, Capifony, Jenkins, Behat and many others.
Recommended
Recommended
More Related Content
More from Fabrice Bernhard
More from Fabrice Bernhard (7)
Recently uploaded
Recently uploaded (20)
The Devops Wonder
- 1. The Devops Wonder How to accelerate your IT flow 8/11/2013 1 THEODO
- 2. The Devops Wonder How to accelerate your IT flow A preach in the Symfony Live Berlin church on finding the path to fast IT 8/11/2013 2 THEODO
- 3. The need for speed in IT The Devops wonder… and facts that prove it How to start practising Devops, step-by-step 8/11/2013 3 THEODO
- 4. Business needs velocity The need for agility 8/11/2013 4 THEODO
- 5. V-Cycle is slow and risky The problems of the alternative ? Source: http://www.ucl.ac.uk/syseng/images/vDiag 8/11/2013 5 THEODO
- 6. Much slower and riskier than what can be estimated The natural mistake A study by two Oxford professors on 1 471 large IT projects shows : • 1 project out of 6 has cost on average 3x more than expected! • large-scale computer spendings were 20 times more likely to spiral out of control than expected! Large-scale example: • FoxMeyer Drugs’ bankruptcy after switching brutally to SAP Oxford study: http://users.ox.ac.uk/~mast2876/WP_2011_08_15.pdf Fox-Meyer bankruptcy: http://fr.slideshare.net/jmramireza/the-foxmeyer-drugs-bankruptcy-was-it-a-failureof-erp-2332065 8/11/2013 6 THEODO
- 7. Our brain sees the world through Bell curves Black swan blindness Average scenario Worst case scenario Good case scenario Probability density of possible outcomes 8/11/2013 7 THEODO
- 8. Complex systems do not act like Bell curves Black swan blindness Average scenario Worst case scenario Good case scenario Probability density of possible outcomes 8/11/2013 8 THEODO
- 9. This is why small iterations are so important! Lean IT Machiavel: Divide et impera! (Divide and conquer) 8/11/2013 9 THEODO
- 10. Agile development is about small FAKE iterations Agile development Source: http://www.mypmhome.com/scrum-methodology/ 8/11/2013 10 THEODO
- 11. Because features will just pile up in front of IT operations The dev VS ops silos 8/11/2013 11 THEODO
- 12. The Scrum prophets showed the light, and the first steps to it Scrum is not the end 8/11/2013 12 THEODO
- 13. The need for speed in IT The Devops wonder… and facts that prove it How to start practising Devops, step-by-step 8/11/2013 13 THEODO
- 14. Devops is about solving the silos problem The definition of Devops Devops is • making the whole IT organisation work as a unique team • with a common goal: business-oriented IT performance 8/11/2013 14 THEODO
- 15. The Devops trinity: tools, process, people The three layers of Devops PROCES S TOOL S DEVOP S PEOPLE Devops is • Tools: how to do it • Process: when to do it • People: why to do it 8/11/2013 15 THEODO
- 16. The people part is important: devs and ops are different! Devops is a lot about culture DHH, the epitom of the hipster developer Stallman, the epitom of the paranoid GNU/Linux guru 8/11/2013 16 THEODO
- 17. Devs want speed and think short-term Devops is a lot about culture I just coded this really cool feature using CouchDB/PouchDB/Node.js/S 3 Let me deploy it now, business wants it online today! The typical dev The typical ops 8/11/2013 17 THEODO
- 18. Ops want stability and think long-term Devops is a lot about culture CouchDB/PouchD B/Node.js/S3??? The typical dev The typical ops 8/11/2013 18 THEODO
- 19. In the worst case, you get neither short nor long-term results Devops is a lot about culture What about Backups? Documentation? Monitoring? Business is not going to be happy… Scalability? Performances? Security? The typical dev The typical ops 8/11/2013 19 THEODO
- 20. Devops efficiency is highly documented on the web Devops studies • • • IT Ops & DevOps productivity report 2013 by RebelLabs surveyed 620 engineers 2013 State of DevOps Report by PuppetLabs & IT Revolution Press surveyed 4000 IT ops and devs And all the testimonials from web leaders like Amazon, Netflix, Etsy, Flickr, etc. (at Velocity Conference, DevopsDays, etc.) 8/11/2013 20 THEODO
- 21. Devops ship code 30x more frequently! 2013 State of Devops report by PuppetLabs and IT Revolution Press Mean time between deploys 8/11/2013 21 THEODO
- 22. Devops ship code 8000x faster! 2013 State of Devops report by PuppetLabs and IT Revolution Press Mean time spent deploying 8/11/2013 22 THEODO
- 23. Devops have 50% less failures! 2013 State of Devops report by PuppetLabs and IT Revolution Press Failed deployments rate 8/11/2013 23 THEODO
- 24. Devops restore service 12x faster! 2013 State of Devops report by PuppetLabs and IT Revolution Press Mean time to resolution 8/11/2013 24 THEODO
- 25. Amazon deploys in average more than 300 times an hour! Devops success stories • • • • Founded in 1994, 61 billion$ revenue in 2012 Mean time between deployments : 11.6s (310 times per hour) Max number of deployments in an hour : 1,079 Mean number of hosts receiving a deployment : 10,000 http://assets.en.oreilly.com/1/event/60/Velocity%20Culture%20Presentation.pdf http://en.wikipedia.org/wiki/Amazon.com 8/11/2013 25 THEODO
- 26. At Etsy every engineer deploys, and starts on the first day! Devops success stories • • • • Founded in 2005, 1 billion$ transactions in 2013 250+ committers, everyone deploys 30+ deploys a day Every engineer at Etsy deploys on the first day http://codeascraft.com/2012/03/13/making-it-virtually-easy-to-deploy-on-day-one/ https://speakerdeck.com/astanway/bring-the-noise-continuously-deploying-under-a-hailstorm-of-metrics http://gigaom.com/2013/08/23/meet-the-man-behind-new-yorks-other-billion-dollar-internet-companythis-one-makes-money/ 8/11/2013 26 THEODO
- 27. The English government did a radical adoption of Devops in less than a year! Devops success stories What is gov.uk? “The websites of all government departments and many other agencies and public bodies” • Alpha launched in 2011 with 4 devs. Site was live in october 2012 • 15-20 deploys a day • Main selling point: faster than other suppliers. • • “this is what we’d like. Can you work out if it’s possible? let’s meet again in a month” -> built, shipped in 3 days http://vimeo.com/album/2384821/video/66622266 https://github.com/philandstuff/devopsdaysparis#kushal-pisavadia-kushalp-how-we-ship-software-at-govuk https://www.gov.uk/service-manual 8/11/2013 27 THEODO
- 28. The need for speed in IT The Devops wonder… and facts that prove it How to start practising Devops, step-by-step 8/11/2013 28 THEODO
- 29. Creating a culture of teamwork between devs and ops The three (or four) maturity levels of Devops adoption 1. Bring devs closer to the constraints of operations 2. Bring ops to empower devs 3. Further embed devs and ops in each other’s teams 8/11/2013 29 THEODO
- 30. Let us start by teaching junior devs basic ops concepts The common developer background The typical junior developer • learnt Java at University • uses Windows… because of the games • has installed Linux because he is curious • maintained a website for one of his clubs at University… • … « deployed » changes using FileZilla Source: http://www.essentialbaby.com.au/ 8/11/2013 30 THEODO
- 31. Step 0: make the baby dev a respectable dev Basic dev skills • • • • • Linux development environment Git versioning Correct git branching strategy Scrum methodology Unit and functional tests Source: http://www.aceshowbiz.com/ 8/11/2013 31 THEODO
- 32. Step 1: give the devs a server to play with One dedicated server per developer «I improved a lot when I started renting my own server» • • • Give a tiny instance to everyone Free to do whatever you want, whenever you want …as long as it is legal… 8/11/2013 32 THEODO
- 33. Step 2: make them set up a simple server… at least twice Basic system administration skills • • Amazing how many junior devs have never set up a server… Twice: to get the taste for automated provisioning What we do: • First a simple setup with Apache + mod_php • Then a more subtle setup with Nginx + fpm 8/11/2013 33 THEODO
- 34. Step 3: bring devs to deploy… and understand how it works Automated deployment • • • Shell scripting: forget it. Capistrano: too magic? Fabric: good compromise? Our experience: every project has a deploy.py script in a «devops» folder (or a Capfile…): • Everybody is now able to deploy automatically • Juniors seem to check if there is an experienced guy around before deploying… 8/11/2013 34 THEODO
- 35. Fabric makes deployment automatic and not too abstract Fabric sample @roles('prod') def deploy(): tag = "%s/%s" % (_getrole(), strftime("%Y/%m-%d-%H-%M-%S")) local('git tag -a %s -m "%s"' % (tag, _getrole())) local('git push --tags') run('git fetch’) run('git fetch origin --tags') run('git checkout ' + tag) 8/11/2013 35 THEODO
- 36. Step 4: make monitoring an obvious step Monitoring • • • Monitoring has become very easy: think NewRelic and AppDynamics By making it an early step, you positively change the dev culture Easy access to: performance issues and hidden 500 errors 8/11/2013 36 THEODO
- 37. Step 5: make continuous integration a dev responsibility Continuous integration • • • Setting up Jenkins is easy: all devs should be expert at doing it To avoid conflicts between jobs: use a separate slave node for each projet Why not Travis-ci? Good solution, but Jenkins proficiency is necessary https://wiki.jenkins-ci.org/display/JENKINS/Distributed+builds 8/11/2013 37 THEODO
- 38. Step 6: normalised environment with Vagrant Development VM • • • • • Macbook, Ubuntu, 32 bits, memory_limit=2048M, ulimit… Development environments are different from production in subtle ways Vagrant: wrap a VirtualBox VM that launches the same OS flavour Bonus: « vagrant package » and you can share your box with a new developer, makes him operational in a few minutes Important performance tips: http://www.whitewashing.de/2013/08/19/speedup_symfony2_on_vagrant_boxes.html 8/11/2013 38 THEODO
- 39. It is time for ops to collaborate and empower devs Production feedback into development 8/11/2013 39 THEODO
- 40. Step 7: use Vagrant with Puppet/Chef Automated provisioning ? • • • Puppet (or Chef) is painful for a dev. But by now he should find manual provisioning painful too! Make an experienced ops create a Puppet/Chef template Make an experienced dev challenge everything and simplify things! 8/11/2013 40 THEODO
- 41. Step 8: enable self-service Infrastructure as a Service • • Public cloud or private cloud OpenStack is now mature (enough) 8/11/2013 41 THEODO
- 42. Step 9: make backuping easy Backups • • S3 / Swift to push backups to Or solutions like Idera ServerBackup 8/11/2013 42 THEODO
- 43. Step 10: create a self-service load testing platform Performance • • • Create an easy to access (VNC) server with a Jmeter and large bandwidth Let the ops teach the devs what a good lead testing scenario is Test the load regularly (if necessary) with devs+ops together 8/11/2013 43 THEODO
- 44. Step 11: enforce pair-devopsing Pair devopsing • • • The dev team has an issue with Puppet/Chef? The load tests indicate a subtle performance issue? Ops see cryptic errors in the logs? Pair devopsing! • • Scrum compatible Works with curious people hire curious people Source: http://www.turnbacktogod.com 8/11/2013 44 THEODO
- 45. Step 12: use visual management as a common incentive Visual management techniques Example: how do you make devs focus on performance? First make it part of the DONE definition Include performance solutions in the standard provisioning AND do visible performance graphs 8/11/2013 45 THEODO
- 46. How do you bring devs to feel responsible for production? Sharing responsibility Why do ops feel that responsible, and not devs? Dev: «It is not my job» … it is up to the business to decide if it is worth putting devs on call. 8/11/2013 46 THEODO
- 47. Next steps: learn from the web leaders Sources • • • • Velocity conferences • http://lanyrd.com/2013/velocity/coverage/ • http://lanyrd.com/2012/velocity/coverage/ • Etc. Gene Kim’s “Top 11 things you need to know” • http://www.thinkhdi.com/~/media/HDICorp/Files/WhitePapers/whtppr-1112-devops-kim.pdf Etsy’s blog: http://codeascraft.com/ Netflix’s blog: http://techblog.netflix.com/ 8/11/2013 47 THEODO
- 48. Iterate to improve continuously Continuous improvement Scrum, Devops, Lean Startup: it is all about continuous improvement! • • • Measure everything Identify problems Try countermeasures And iterate 8/11/2013 48 THEODO
- 49. And I hope that you now believe in the Devops wonder The miraculous performances of Amazon Werner Vogels, CTO at Amazon, 300+ deploys per hour 8/11/2013 49 THEODO