Measurement and analytics for social apps 18 may - speakernotes

Editor's Notes

1. I like to make sense of stuff. I’m an analyst. Pretty much everything I’ve done has an ‘analyst’ flavour - data analysis, discourse analysis, user behaviour analysis, technology analysis, market analysis, business model analyst…. (not yet ‘psychoanalyst’). What I’ve been doing recently is trying to make sense of the options available for measuring and analysing social apps. I’m here to share a bit of my enjoyment with you today. The reason I’ve been researching this is that I think it’s the most interesting thing going on at the moment. Specifically, I think relationship between [characteristics of the social graph] and [application design and content optimisation] is the most interesting thing going on. So I want to understand it better. What I’m going to be talking about is work in progress. An alpha version. A demo with no code. Your role in this is to throw tomatoes. Ideally, not ‘til the end of the talk. But afterwards – yes, please. That’s how I make stuff better.
2. I promised you a poem about elephants. It’s a 19thC poem by John Godfrey Saxe about an ancient Hindu, and later Buddhist, and Muslim legend. Everyone keeps borrowing the story. Why? Because it’s very good. Here are the first few verses: IT was six men of Indostan  To learning much inclined, Who went to see the Elephant  (Though all of them were blind), That each by observation  Might satisfy his mind. ii. The First approached the Elephant,  And happening to fall Against his broad and sturdy side,  At once began to bawl: "God bless me!—but the Elephant  Is very like a wall!" iii. The Second , feeling of the tusk,  Cried:"Ho!—what have we here So very round and smooth and sharp?  To me 't is mighty clear This wonder of an Elephant  Is very like a spear!" And so it goes.
3. The situation with social apps analytics strikes me as being quite similar. There are a lot of different aspects to it, and only grabbing on to one is misleading.
4. Of course what kind of elephant you want might be a matter of what you want it to DO. Lift logs, eat popcorn, make more elephants.
5. I’d like to start the serious bit by putting up a fence up between measurement and analytics. What I’m talking about is the difference between WHAT and SO WHAT. Measurement is about the WHAT. How many elephants, how big are they? Analytics I see as being about the SO WHAT. Do we have enough elephants? Are they different from the ones we had last week? What will they do next? So the domain of analytics is about thresholds, trends, associations, inference, and prediction. Analytics is more of a process than a product, though for sure there are products can save you from having to add it all up on your toes.
6. At its simplest, we are talking about actions performed by people, using applications, in a universe which encodes their relationships to objects, and to other people. Notice I haven’t used the F-word. This isn’t just about Facebook. But Facebook is particularly fun because of the way it makes relationships explicit and discoverable, and weaves those relationships into their own content provision. One thing that’s absolutely fundamental for insight is the ability to track users through sessions, and identify when they recur. This isn’t always possible, but it’s always worth striving for. If you don’t do this you end up relying on measures like the stickiness ratio (DAU/MAU). This is a bit like used chewing gum. Perhaps better than nothing. Cheap and cheerful. It tells you sweet-all about retention. You need to be able to tell whether you are pouring marketing money into a leaking ship or not.
7. "If you wish to bake an apple pie from scratch, you must first invent the universe” Carl Sagan Another distinction to get a grip on is the difference between applications and media. I view media as being a specific type of application – one that is preoccupied with perfoming actions on content. In fact media – which is to say Content applications are getting smarter – as content is customised for presentation using rules that take advantage of what the apps knows about the user, and the user’s social graph. Facebook is the 800 pound gorrilla here, but they aren’t the only one in the jungle. One startup worth watching is Cognitive Match, who do AI-based real-time content optimisation for ads. Another is Playnomics, which is so stealthy it’s a bit hard to tell exactly what they do, except that it’s very on-trend.
8. Here are some dimensions you can use to tell what bit of the elephant you’ve got hold of. I think these dimensions are fairly straightforward. There’s some magic involved in the more complicated measures, when you look at the detail. Where it gets interesting is in how they relate to each other . I’d say that’s where analytics plays best. So, for instance, you can track how messages are transformed as they propagate through networks, and how this transformation varies depending on what kind of network neighbourhoods they go through. (Kind of like a ‘regional accent’….;-) Or you can look at how changing some aspects of an app affects the extent to which people invite other people to share it, and the degree of success these invites have. Or on how ‘connected’ an individual is affects how influential they are. The answer isn’t always obvious. There’s a fair amount of published research on this, which is very interesting, but the best generalisation I can make about it is that it is worth measuring for yourself, for your own app and your own ecosystem. Usually if I stare at something like this for long enough it will resolve into a set of dimensions I can put on a graph. I’m still waiting on this one…. any ideas?
9. “ a terminological jungle, in which any newcomer may plant a tree” J. Barnes There’s a fascinating sub-world out there which is associated with measures for looking at social network structure. [QUOTE] Whatever you want to measure there’s at least one way to do it, and you don’t need a special license to make stuff up. One quick point: it can can pay to look at more than just local structure. Let’s think about ‘A’ in the graph. A is not highly connected but she links different groups. In my own social graph, I’ve got a few of those. You can see how they could be even more valuable than people with lots of connections. Another quick point: how interconnected a group is can affect how easy it is to address. Media6 Degrees has come up with something they call ‘Tribal Brand Index’ – I don’t know the algorithm details but it measures how interconnected and similar customers are. This predicts how well that community responds to some types of social media-based campaigns. More homogenous and interconnected communities are potentially easier to reach. Regarding dynamic processes, traditional models of diffusion of innovation, or ‘virality’ assume a ‘fully mixed’ population model. However it is clear that network structure affects the dynamics of how signals can be propagated. As a stochastic process it isn’t so analytically tractable as the old-fashioned models. There’s lots of research work going on on this topic which at the moment is fairly brute force empirical – based on observation and simulation.
10. I’d say that in terms of where we’re at, at the moment, the real hot spots are in doing analysis on social action dynamics, social network structure, and media/content evaluation. It’s by combining those with the relatively well understood disciplnes of application design testing and customer lifecyle tracking that companies are going to get the most traction.
11. Here’s the really scary bit. The demo. Here’s a test-drive of how to use the framework I’ve established in order to tell what kind of elephant is in the room with you. For sure, it’s wrong in some details. But that’s easy to fix. I’d be interested in hearing whether you find it a useful starting point for your own thinking.