Social media measurement – are we staring at the stones?
Social media measurement and ROI is a hot subject. It was one of the issues debated recently at Social Media Influence 09 and listening to that debate and some of the frustrations and difficulties that people were expressing got me thinking. I developed the suspicion that we might be going about this the wrong way. Perhaps our approach to measuring social media is conditioned by the approaches and tools we used in the highly measurable web1.0 environment. Perhaps we are falling for the classic mistake, which is the failure to recognise the fundamental difference between the landscape of social media and the previous on-line, digital or traditional media environment. Perhaps we are trying to micro measure processes rather than understand the system as a whole.
This line of thinking took me back to a story about measurement that I heard at university more than 20 years ago. I was studying geomorphology – and this was the story. Bear with me, like any decent story it takes a few minutes to tell, you might want to get a cup of coffee.
By way of background, for much of its life geomorphology (the study of landscape, rivers, erosion etc) had been an observational and descriptive science. Essentially it was about the history of landscape and how, over the course of millennia, mountains and valleys had been shaped by the power of wind, water and ice. It wasn’t big into measurement. Then, in the 1960s it was decided to try and make the subject more scientific, the definition of more scientific being more measurable. This was driven partly by a desire to find new insights but also because new tools were emerging, principally of course the computer, which allowed large amounts of data to be crunched and therefore spawned a rush to develop data collection tools so that geomorphologists could have the raw material needed to play with these new wonder machines.
The great hope was that these new tools would help them crack what was thought to be a big problem. This problem was that while the old techniques of observation had proved that processes of erosion had shaped the landscape over the course of millions of years, no-one had been able to actually see erosion as it took place or measure it in real time. This was assumed to be because these processes must be very small and gradual and their effect was a cumulative one over a very long period of time. A pretty logical conclusion on the face of it.
Geomorphologists started to develop an obsession with slopes. The hillside suddenly became a laboratory – a place to stake out and measure in order to reveal the hidden secrets of erosion. Everyone assumed that some major new discoveries were going to be made, perhaps something as mundane as earthworms would be found as the key variable in the erosion equation, perhaps it would all be about rainfall – x amount of rain on any given day would be shown to equate to a y amount of increase in erosion.
It is against this background that our particular story takes place. A group of post-grad students decided that they were going to use every technique in the book on a particular hillside to once and for all measure erosion. So they began to stake out the hillside with the latest, most advanced and sensitive measuring devices available. Not a single stone could shift a millimetre without them knowing about it and this information being fed into a computer to be crunched. Once they had completed wiring everything up they switched it all on. Nothing happened, a total flat-line, nothing was moving. They checked the equipment and everything was fine, so they assumed today was just a slow erosion day and tomorrow they would get something. But the next day was just the same, and the day after that. A week went past and then another and still nothing.
Two months passed and it began to rain and it rained some more, and it kept raining almost continually for a week – but even all this rain wasn’t causing any detectable signs of activity. Then, thousands of miles away in China there was an earthquake. The tremors traveled around the globe until the faintest wobble reached the hillside, with its now saturated soil. And this triggered a massive landslide which swept all the sensitive monitoring equipment, along with thousands of tons of earth and rock, away down the hill and off on its way to the sea. It was though the hillside had looked at the students and said “So, you thought you could measure me with all your fancy equipment? Well measure this then.”
What did the students do wrong? They were measuring the wrong thing, based on a wrong assumption. They should have lifted their heads away from studying and analysing individual actions and looked to understand the system as a whole. Then they would have realised that actually they should have been measuring simple and easy stuff like rainfall, soil moisture content, soil and rock structure, slope angle, seismic activity and see how these interacted. If they had done this they would not just be in a position to analyse activity as it was happening or measure it after the event – but actually predict future events.
Could it be that in social media we are making the same mistake as the students? We are just staring at the stones and waiting to see them move. We have taken all the sensitive measuring stuff that worked in the web1.0 world and the mentality of micro-measurement and tried to wire it into the social media landscape. Perhaps we have made the classic mistake of assuming that because we have a tool to measure something, that thing is therefore the right thing to measure.
It seems to me we need to better observe and understand the structure of social media systems and networks. This will then tell us what to measure.
Incidentally, the story of the students is almost certainly not true in the way that is has come to be told, but it did the rounds (perhaps is still doing the rounds) because it neatly illustrated a paradigm shift in geomorphology. This was the realisation that much of the physical features in our world were caused by big events, followed by long periods of seemingly no movement, albeit these periods of still contained plenty of activity that could and should be measured in order to work out when the big movements were going to occur. It lead to a huge re-appraisal of the whole discipline and a recognition that the evidence of this activity had always been there for all to see, but had been overlooked because of a desire to measure what people wanted to see, and match evidence to pre-conceived assumptions or conclusions. Some enterprising chap called Gladwell also latched onto something similar, although about 15 years later and wrote a book about it which became quite successful.
An example of some of the new learning that emerged is found on the east coast of Australia. Here, geomorphologists had noted gigantic scour marks on the cliffs high above the sea. These looked as though they were caused by a huge wave, but this idea had been dismissed because it was assumed that waves of such size could never exist and alternative micro-erosion theories were developed to explain these features. However, the original observation was right. These were indeed caused by the impact of massive waves generated by the catastophic collapse of volcanic islands in Hawaii. Which in turn caused scientists to realise they had better keep a close eye on cracks and fissures on Hawaiian volcanoes if they wanted to think about protecting Australia. The cracks are there by the way. At some point in the not too distant future (geomorphologically speaking) there is some mighty big surf on the way.
P.S. – have written an update to this post, more specific to social media.