Ops Meta-Metrics: The Currency You Pay For Change

Editor's Notes

1. This is about metrics about YOU! Metrics *about* the metrics-makers!
2. They are basically taken from both Flickr and Etsy.
3. HOW MANY: write public-facing app code? maintain the release tools? release process? respond to incidents? have had an outage or notable degradation this month? that was change-related?
4. Too fast? Too loose? Too many issues? Too many upset and stressed out humans?
5. Everyone is used to bug tracking, it’s something worthwhile....
9. If this is a feeling you have often, please read on.
11. All you need is to see this happen once, and it’s hard to get out of your memory. No wonder why some people can start to think “code deploy = outage”.
12. Mild version of “Critical Incident Stress Management”? Change = risk, and sometimes risk = outage. And outages are stressful.
13. Not supposed to feel like this.
15. Details about the change play a huge role in your ability to respond to change-related incidents.
16. Details about the change play a huge role in your ability to respond to change-related incidents.
17. Details about the change play a huge role in your ability to respond to change-related incidents.
19. We do this by tracking our responses to outages and incidents.
20. We can do this by tracking our change, and learning from the results.
21. We need to raise confidence that we’re moving as fast as we can while still being safe enough to do so. And we can adjust the change to meet our requirements...
22. Why should change and results of changes be any different?
23. Type = code, schema, infrastructure, etc. Frequency/Size = how often each type is changed, implies risk Results = how often each change results in an incident/degradation
24. Lots of different types here. Might be different for everyone. Not all types of change bring the same amount of risk.
25. This info should be considered mandatory. This should also be done for db schema changes, network changes, changes in any part of the stack, really.
26. The header of our metrics tools has these statistics, too.
27. The tricky part: getting all prod changes written down without too much hassle.
28. Here’s one type of change....
29. Here’s another type of change....
30. Here’s yet another type of change...
31. Size does turn out to be important. Size = lines of code, level of SPOF risk, etc.
32. This seems like something you should do. Also: “incidents” = outages or degradations.
33. Just an example. This looks like it’s going well! Getting better!
34. Maybe I can’t say that it’s getting better, actually....
36. Some folks have Techcrunch as their incident log keeper. You could just use a spreadsheet.
37. An example!
38. You *are* doing postmortems on incidents that happen, right? Doing them comes at a certain point in your evolution.
42. Without the statistics, even a rare but severe outage can make the impression that change == outage.
43. Just examples. It’s important to categorize these things so you can count the ones that matter for the user’s experience. #4 Loss of redundancy
44. Just examples. It’s important to categorize these things so you can count the ones that matter for the user’s experience. #4 Loss of redundancy
45. Just examples. It’s important to categorize these things so you can count the ones that matter for the user’s experience. #4 Loss of redundancy
46. Just examples. It’s important to categorize these things so you can count the ones that matter for the user’s experience. #4 Loss of redundancy
47. Just examples. It’s important to categorize these things so you can count the ones that matter for the user’s experience. #4 Loss of redundancy
48. Just examples. It’s important to categorize these things so you can count the ones that matter for the user’s experience. #4 Loss of redundancy
49. Just examples. It’s important to categorize these things so you can count the ones that matter for the user’s experience. #4 Loss of redundancy
50. Just examples. This normally comes from a postmortem meeting. A good pointer on Root Cause Analysis is Eric Ries’ material on Five Whys, and the wikipedia page for RCA.
51. http://www.flickr.com/photos/mattblaze/2695044170/
52. What happens in our response to a change-related incident is just as important as the occurrence of the incident.
53. What happens in our response to a change-related incident is just as important as the occurrence of the incident.
54. What happens in our response to a change-related incident is just as important as the occurrence of the incident.
55. What happens in our response to a change-related incident is just as important as the occurrence of the incident.
56. What happens in our response to a change-related incident is just as important as the occurrence of the incident.
57. Th
58. Th
59. This might also be known as a ‘diagnose’ point.
60. This might also be known as a ‘diagnose’ point.
61. These events usually spawn other events.
62. These events usually spawn other events.
63. This should be standard operating procedure at this point,
64. These events usually spawn other events.
65. Some folks might notice a “Time To Diagnose” missing here. ALSO: it’s usually more complex than this, but this is the gist of it.
68. Do incidents increase with size of change? With frequency? With frequency/size of different types?
71. If you don’t track: Change, Incidents, and Responses, you’ll never have answers for these questions.
74. Reasonable questions.
75. *YOU* get to decide what is “small” and “frequent”.
78. THIS is what can help give you confidence. Or not.
80. The longer an outage lasts, the bigger of a bummer it is for all those who are working on fixing it.
81. The longer an outage lasts, the more mistakes people make. (and, as the night gets longer) Red herrings...
82. put two points on this graph
83. put two points on this graph
84. put two points on this graph
85. put two points on this graph
86. It should, because it is.
87. How we feel about change and how it can (or not) cause outages is important. Some of the nastiest relationships emerge between dev and ops because of these things.
92. “Normal” = lots of change done at regular intervals, change = big, time = long.
93. 2 weeks? 5000 lines?
94. Scary Monster of Change! Each incident-causing deploy has only one recourse: roll it all back. Even code that was ok and unrelated to the incident. Boo!
95. Silly Monster of Nothing to Be Afraid Of Because His Teeth Are Small.
96. Problem? Roll that little piece back. Or better yet, roll it forward!
97. This looks like an adorable monster. Like a Maurice Sendak monster.