Social media measurement – are we staring at the stones?

Social media measurement and ROI is a hot subject.  It was one of the issues debated recently at Social Media Influence 09 and listening to that debate and some of the frustrations and difficulties that people were expressing got me thinking.  I developed the suspicion that we might be going about this the wrong way.  Perhaps our approach to measuring social media is conditioned by the approaches and tools we used in the highly measurable web1.0 environment.  Perhaps we are falling for the classic mistake, which is the failure to recognise the fundamental difference between the landscape of social media and the previous on-line, digital or traditional media environment.  Perhaps we are trying to micro measure processes rather than understand the system as a whole.

This line of thinking took me back to a story about measurement that I heard at university more than 20 years ago.  I was studying geomorphology – and this was the story.  Bear with me, like any decent story it takes a few minutes to tell, you might want to get a cup of coffee.

By way of background, for much of its life geomorphology (the study of landscape, rivers, erosion etc) had been an observational and descriptive science.  Essentially it was about the history of landscape and how, over the course of millennia, mountains and valleys had been shaped by the power of wind, water and ice.  It wasn’t big into measurement.  Then, in the 1960s it was decided to try and make the subject more scientific, the definition of more scientific being more measurable.  This was driven partly by a desire to find new insights but also because new tools were emerging, principally of course the computer, which allowed large amounts of data to be crunched and therefore spawned a rush to develop data collection tools so that geomorphologists could have the raw material needed to play with these new wonder machines.

The great hope was that these new tools would help them crack what was thought to be a big problem.  This problem was that while the old techniques of observation had proved that processes of erosion had shaped the landscape over the course of millions of years, no-one had been able to actually see erosion as it took place or measure it in real time.  This was assumed to be because these processes must be very small and gradual and their effect was a cumulative one over a very long period of time.  A pretty logical conclusion on the face of it.

Geomorphologists started to develop an obsession with slopes.  The hillside suddenly became a laboratory – a place to stake out and measure in order to reveal the hidden secrets of erosion.  Everyone assumed that some major new discoveries were going to be made, perhaps something as mundane as earthworms would be found as the key variable in the erosion equation, perhaps it would all be about rainfall – x amount of rain on any given day would be shown to equate to a y amount of increase in erosion.

It is against this background that our particular story takes place.  A group of post-grad students decided that they were going to use every technique in the book on a particular hillside to once and for all measure erosion.  So they began to stake out the hillside with the latest, most advanced and sensitive measuring devices available.  Not a single stone could shift a millimetre without them knowing about it and this information being fed into a computer to be crunched.  Once they had completed wiring everything up they switched it all on.  Nothing happened, a total flat-line, nothing was moving.  They checked the equipment and everything was fine, so they assumed today was just a slow erosion day and tomorrow they would get something.  But the next day was just the same, and the day after that.  A week went past and then another and still nothing.

Two months passed and it began to rain and it rained some more, and it kept raining almost continually for a week – but even all this rain wasn’t causing any detectable signs of activity.  Then, thousands of miles away in China there was an earthquake.  The tremors traveled around the globe until the faintest wobble reached the hillside, with its now saturated soil.  And this triggered a massive landslide which swept all the sensitive monitoring equipment, along with thousands of tons of earth and rock, away down the hill and off on its way to the sea.  It was though the hillside had looked at the students and said “So, you thought you could measure me with all your fancy equipment? Well measure this then.”

What did the students do wrong?  They were measuring the wrong thing, based on a wrong assumption.  They should have lifted their heads away from studying and analysing individual actions and looked to understand the system as a whole.  Then they would have realised that actually they should have been measuring simple and easy stuff like rainfall, soil moisture content, soil and rock structure, slope angle, seismic activity and see how these interacted.  If they had done this they would not just be in a position to analyse activity as it was happening or measure it after the event – but actually predict future events.

Could it be that in social media we are making the same mistake as the students?  We are just staring at the stones and waiting to see them move.  We have taken all the sensitive measuring stuff that worked in the web1.0 world and the mentality of micro-measurement and tried to wire it into the social media landscape.  Perhaps we have made the classic mistake of assuming that because we have a tool to measure something, that thing is therefore the right thing to measure.

It seems to me we need to better observe and understand the structure of social media systems and networks.  This will then tell us what to measure.

Incidentally, the story of the students is almost certainly not true in the way that is has come to be told, but it did the rounds (perhaps is still doing the rounds) because it neatly illustrated a paradigm shift in geomorphology.  This was the realisation that much of the physical features in our world were caused by big events, followed by long periods of seemingly no movement, albeit these periods of still contained plenty of activity that could and should be measured in order to work out when the big movements were going to occur.  It lead to a huge re-appraisal of the whole discipline and a recognition that the evidence of this activity had always been there for all to see, but had been overlooked because of  a desire to measure what people wanted to see, and match evidence to pre-conceived assumptions or conclusions.  Some enterprising chap called Gladwell also latched onto something similar, although about 15 years later and wrote a book about it which became quite successful.

An example of some of the new learning that emerged is found on the east coast of Australia.  Here, geomorphologists had noted gigantic scour marks on the cliffs high above the sea.  These looked as though they were caused by a huge wave, but this idea had been dismissed because it was assumed that waves of such size could never exist and alternative micro-erosion theories were developed to explain these features.  However, the original observation was right.  These were indeed caused by the impact of massive waves generated by the catastophic collapse of volcanic islands in Hawaii.  Which in turn caused scientists to realise they had better keep a close eye on cracks and fissures on Hawaiian volcanoes if they wanted to think about protecting Australia.  The cracks are there by the way.  At some point in the not too distant future (geomorphologically speaking) there is some mighty big surf on the way.

P.S. – have written an update to this post, more specific to social media.

7 comments

  1. Social Media

    There seems to be much focus recently on a host of “measurement tools” for social media, which in a way, to me, seems a little like a way to create more work for the marketing staff, which leads to more meetings to discuss their new reports, which leads to questions and revisions, and so forth.

    How bout measuring results based on how many new customers we gained? Or how many new email addresses we captured…ads we sold, etc.

    Are our sales growing, are they flat, or are they declining? To me, that is the ultimate metric of any marketing program. The rest is just chairs on the Titanic’s deck.

    Jim

  2. richardstacy

    Jim – you are right, I think inventing measurement tools for measurement tool’s sake is a problem. You are also right in that ultimately sales, customer acquisition etc. is the key metric. However, the problem is establishing a causal link between social media activity and such mtrics. This has been PR’s perpetual problem – you can assume it has effect, but you can’t chart the link between a given input and a measurable output.

    Hence – the only way forward, I think, is to demonstrate the generic link between a given social media “state of health” and a given level of customer engagement (possibly Net Promoter Score would be a good metric here) – and then actually develop metrics that measure an organisation’s state of social media health.

  3. Pingback: Social media measurement - think carrots « Richard Stacy @ Stacy Consulting
  4. Pingback: PR ultimate measurement « Word of Moss
  5. Richard Randolph

    Great post, Richard. Two thoughts to toss out there…

    First, I enjoyed your story a lot — great meaning. Instead of characterizing the early researchers as “wrong” or “foolish”, I think this is an excellent illustration of fundamental learning in action. We start with what we know (or think we know…) then test. When things go wrong, we adjust our thinking, move up one level, then repeat. There’s nothing wrong (in my opinion) with a good-faith trial that fails.

    Second, I’m just learning to “trust the tribe” through social media. From my point of view, as well as that of my Clients, the issue is one of control. In traditional “push/interruption” marketing, we were in total control of the output (or at least, that’s what we thought…). We controlled the message and the media. We controlled the timing, frequency and intensity. Then we measured our results and adjusted the mix to try to improve the outcomes. Then we SOLD, SOLD, SOLD! Features, Advantages, BENEFITS! Again, we controlled the message and delivery.

    How did today’s managers rise to the top? Through exercising control over people, resources and processes to achieve results. Control has been learned and consistently reinforced. It’s internalized and “known.” We all love to “be in control.” In fact, don’t we demean someone who’s “out of control”?

    Now, it seems to me, the “new” social/community architecture reverses the locus of control. The implications are clear: You (your brand, your offer) has to be relevant and meaningful to somebody. You have to matter in some way to someone. If you don’t, you’ll be ignored. If you try to push your message through, you’ll be ignored.

    In other words, you have to give up control to the tribe (thanks to Seth Godin for the metaphor). Today it’s necessary to be proactively reactive.

    To be relevant and meaningful, it’s critical to understand in detail and depth your intended user, their life, and the “job” they’re trying to do (i.e., be customer-centric). There’s an excellent (if somewhat dated now) article “Get Inside the Lives of Your Customers” by Patricia Seybold at Harvard Business Review (I found it on Amazon) that helps describe this process.

    It’s a new (marketing) world today — one where the community has control and the supplier has to work very hard to tune in and satisfy Customer needs, wants and demands. It’s marketing’s version of the old Trust exercise: “fall back into your partner’s arms — they’ll catch you…” No more “push” — hello “pull.”

    //Richard Randolph

  6. richardstacy

    Thanks Richard.

    You are right, the control model is becoming redundant. Power is shifting from brand owners to brand users. Almost everything that worked in the old world, doesn’t work in the social media world. That’s why it is a bit tricky

  7. Pingback: Three lessons from #LRNY « Richard Stacy @ Stacy Consulting

Post a comment

You may use the following HTML:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>