Category: Big Data

Algorithms and the growth of sensorship

Here is a quick thought.  As I have previously said, I think we are moving from the age of the printing press into the age of the algorithm.  Printing led to the growth of censorship whereas algorithms are going to lead to the growth of sensorship.

I was prompted to write this today because of the announcement that one of the UK’s largest electronic goods retailers is linking up with one of the UK’s largest mobile phone retailers.  Electronic goods are basically forms of sensor that monitor human behaviour via how they are used (note: there are now even cameras in Barbie dolls).  Mobile phone retailers basically sell connection to the internet and also provide mobile handsets, which are the most comprehensive form of personal sensor currently out there.  I heard the CEO of the new company on Radio 4’s Today programme make no bones about the fact that the underlying logic behind the deal was the growth in The Internet of Things (with electronic things being the most obvious and easy of such things to connect to the internet).

We are just at a start of a form of data detection landgrab – the Scramble for Data if you like.  Continue reading

The sword, the printing press and the algorithm. Three technologies that changed the world

It is always a good game to identify the game-changers: to reduce the complexities of history (and perhaps even the future) into simple cause and effect relationships.  No more is this so than with technology, given that we like to think we are living in a technological age and thus there is a certain vested interest in either talking-up, warning of, or dismissing the impact of technology on the course of our lives and our societies.

I am not a real fan of technological determinism.  Technology is (or should be seen as) a tool that helps us achieve certain objectives.  Focus on, or worship of, the tools can lead us into dangerous territory.  Nonetheless, I do think there have been certain technological breakthroughs which have played a fundamental role in shaping the way our world has evolved.  Interestingly, these technologies have been so fundamental, they have become invisible – insofar as we focus on the effect these technologies introduced often without fully appreciating the connection between a technological shift and subsequent events.  They are a bit like foundations – you see the building that sits on top (the effect) but the connection between a building and its foundation remains invisible.

The three technological shifts I would single out are the sword (specifically the iron sword), the printing press and the algorithm.  The interesting thing about these three is that they have all superseded each other to a large extent.  We have moved from the age of the sword, into the world of the printing press and are about to enter the age of the algorithm.  Here is what I mean.

The age of the sword

If I had to go back in time and live my life again, I think I would head-on back to the middle bronze age.  Life was pretty cool around 1500 BC (at least it was in the area now known as Great Britain).  A lot of the complications and hassles associated with the tricky business of agriculture had been sorted out by the geeks of the time, resources were in abundance, the weather was pretty good, religion was seen as a shared set of practices, beliefs and endeavours (such as dragging large stones around the country), rather than an instrument of power and everything was generally sweet.  But then some clever geek went and invented iron, and what did the powers that be then go and do with this?  They created swords.  Now swords had been around for some time, but they were more ceremonial than anything else.  You could cause a bit of damage by thrusting one of them into something, but in a full on clash of bronze against bronze they very soon lost their edge.  Iron swords, on the other hand (especially if the hand that held them was a fiery-tempered Celt), could give you serious power and influence.  Result: the quiet and gentle societies of the bronze age faded away into misty-eyed myth and the world became an altogether more brutal place.  I oversimplify, but I think the fundamental truth remains.

It wasn’t so much that possession of iron weaponry made us more violent, it just gave violence a greater competitive advantage.  For millennia groups of men had been clobbering each other using little more than sticks and stones.  Now sticks and stones can break your bones, but they don’t scale very easily.  If you had vast armies facing each other intent on annihilation, armed only with sticks and stones, they would have to go at it for quite a long time before they started to make serious inroads into the business of killing.  Battles would last longer than cricket matches and also have to have tea breaks.  In fact cricket is pretty much a sticks and stones sort of game, a relic perhaps of our stone age ancestry.  Sticks and stones were therefore used to solve relatively small scale, local disputes.  Or to look at it another way – larger scale disputes were simply not feasible.  You could not project power and influence over a large area using a sticks and stones army.  You could not build an empire based on sticks and stones.

Iron swords, however, gave violence a scalable benefit.  Land ceased being something that had only localised value, with a value cap limited by your personal capability to exploit it.  With a group of men armed with swords, you could extract value from land at some distance because you didn’t have to exploit it yourself, you could force the people exploiting it to pass some of that value onto you.  Thus both empires and taxation were born at the same time.  Some bloke sat in Rome could expect another bloke in the north of Britain to hand over a portion of his cash because he knew that if he didn’t there was a system in place which would deliver a posse of blokes with swords to his doorstep in pretty short order.

Armies became a finite and precious resource and thus, like all finite and precious things they ended up in the hands of a small, elite group who then were able to call themselves kings and emperors.  Or rather, if you aspired to become a king or emperor, you first had to get yourself an army.

And so the age of empires and armies (facilitated by swords) continued.  I guess you could say that after a while guns took over from swords – but I don’t count these as a fundamental technological shift, because this didn’t really change the order of things.  Swords gave violence a scalable benefit and guns just simply extended this.  They didn’t change the rules of the game, just conferred upon those that had them the ability to play the game more effectively.

The age of the printing press

A printing press is somewhat different from a sword or an army.  Not that we should necessarily be surprised by this.  Revolutionary shifts are usually defined by the fact the new thing doesn’t look like the old thing it is replacing.

What the printing press did was shift the battle away from a clash of iron to become a clash of ideas.  Ideas ended up becoming more powerful than armies, albeit armies were sometimes employed in the service of ideas.  Ideas allowed you to control the actions of people on the other side of the world without having to put a gun to their head or a sword to their throat.  It is down to the question of scalable benefit again.  If Galileo hadn’t had access to a printing press, his ideas would have lived and died within Italy – largely because the dominant institution of the time (the Catholic Church) would have supressed them, by suppressing him, in order to ensure that it retained its monopoly on ideas.  Printing allowed Galileo’s ideas to escape beyond the reach of the church.  The church could suppress the man, as it did, but it couldn’t imprison his ideas.  Printing gives ideas a scalable benefit.  It allows them to become something that can challenge the established order without having to raise an army.

Printing, or rather the ability to give scale to the distribution of information, does a whole lot of other things as well.  It allows you to give scale to trust and reputation.  Money lenders can become banks because banks can build a reputation that encourages strangers to trust them with their money, even if those strangers have had no previous personal experience of transacting with them.  Pretty much every institution associated with the modern world, from science to modern democracy – can trace its lineage back to printing and the ability to give information a means of mass distribution.  In fact you could say that democracy represents the ultimate triumph of the idea over the sword in that it has allowed large numbers of nations to organise their internal and external affairs without resolving things on a battlefield.

But just as armies were finite and precious resource and thus the monopoly of kings and emperors, the ability to distribute information (publication) was likewise finite and expensive.  This meant that its power could only be wielded by institutions, or rather institutions evolved in order to wield its’ power – first amongst them, of course being the institution that we call the media.  This is why Rupert Murdoch is more powerful than prime ministers and also why Procter & Gamble is the world’s largest advertiser.

But then something happened which changed the rules of the game.  The ability to control the mass distribution of information was no longer limited to institutions.  This thing called social media gave this power to individuals.  The social media revolution is all about the separation of information from its means of distribution and the associated shift of trust and power from institutions into transparent processes.  I used to think that this shift was the next big game-changer: the end of the Gutenberg age and the dawn of something new.  But now I am not so sure, because something else has emerged that confers a new form of institutional (and thus elite) advantage on those who can have access to it – and this thing is the algorithm.

The age of the algorithm

Algorithms are nothing new, but what has changed is that the thing that they feed on has exploded.  This thing is data.  In the world of small, or restricted, data – algorithms had to remain likewise constrained.  Even in the area where algorithms have perhaps carved out the most important role, which is financial asset trading, they have still remained constrained by the limited availability of financial data and haven’t broken out into the world beyond the markets.

Again it is a question of scalable benefit.  Until recently there wasn’t really a scalable benefit available for algorithms outside of what we might call data rock-pools.  But now the tide of data is coming in allowing these to become connected and for the algorithm to become the master of the ocean rather than the rock-pool.

Once algorithms can be fed with large, multi-layered and multi-dimensional data sets, they acquire an almost magical ability.  They can predict the world and at the same time have the power to make the world conform to their predictions.  They can predict the behaviour of consumers, or citizens and thus shape the response of the brand or the government.  In relatively short order, algorithms will define the identities of almost every person on the planet.  You will not be able to walk into a shop without an algorithm determining your desirability as a potential consumer and devising a pricing structure accordingly.  It won’t be long before goods will not have price labels, algorithms will estimate your desire for a product and your ability to pay and pitch you an appropriate price.  Goods may even be discounted according to their ability to harvest data from you – and thus ‘improve’ the ability of algorithms manage your relationship with the supplier of the product you have just bought.  Indeed – in the future we won’t own products anymore, because their primary allegiance will always be to their data masters.  But buying and selling goods will just be the start – algorithms will determine access to all resources, both those of the state and those of the market.   They will determine the insurance premiums you pay, the interest rates you are charged, your ability to benefit from access to healthcare and thus the healthcare you receive.

It is almost impossible to conceive of an aspect of life which algorithms cannot control, for wherever there is data, so will there be algorithms.  Forget quaint notions like artificial intelligence.  Algorithms are not in the business of allowing machines to become as smart as humans, or act in a human way – they are about predicting the actions of humans so that they (we?) can do things that transcend human capability or even comprehension.  Algorithms tell you what the world is like, or will be, without the essentially human need to understand why it is like that.

And here is the thing.  Algorithms are tricky things to make.  Anyone can write a blog post, or write a review, but not anyone can write an algorithm.  Like swords and printing presses algorithms confer an advantage upon an elite.

And that is why I think algorithms will be the third great technological game-changer.  We will have moved from a world of Alexander the Great, to Rupert the Great to … what?  Who can really wield the algorithm or will we have reached that dystopian point at where we become the tools and technology becomes the master?

Yet more evidence for why 2014 will be the year of the Internet of Things

Check out this post from Digital MR.  Yet more evidence for why the Internet of Things will be the next big thing on the internet and why the privacy debate will extend to our cars, electrical appliances and clothing – not just our identity on Facebook.

No data is inconsequential anymore and the algorithm is the most powerful instrument for social control invented since the sword

You had either better learn how to use data and algorithms, (or find you some people that do).

 

It is official: the internet of things will be the next big internet thing (for a few months anyway)

I think we can now officially declare that the internet of things will be the next big thing (sort of Big Data now gets even bigger).  I note it is now officially capitalised and acronymised – as in Internet of Things (IoT).

See this from Business Insider.

Big Barbie

FireShot Screen Capture #267 - 'There's a Camera in This Barbie Doll [VIDEO]' - mashable_com_2013_10_08_barbie-doll-cameraHere is a cool thing I have just seen on Mashable – a Barbie doll with a camera in it.

Here is not such a cool thing – a Barbie doll with a camera and an embedded SIM or RFID, and then maybe a microphone thrown in for good measure.  Barbie can now apply for a job at the NSA.  Welcome to the Internet of Things.

Thongs on the internet: the next big internet thing

FireShot Screen Capture #259 - 'Buy Internet Thongs & G-Strings I Personalised I - CafePress UK' - www_cafepress_co_uk_+internet+underwear-panties_cat=100115Around this time last year I was asked by Savas Onemli, the editor of Digital Age in Turkey, what I thought the big thing in 2013 was going to be.  I said Big Data.  Phew!  So recently I was asking myself the question, what might be the big thing of 2014.  I think Big Data is still going to be pretty big, but I think the new thing we will be talking about (which is closely linked to Big Data) is The Internet of Things.

I have just got back from Brussels where I was taking part in a panel discussion on Big Data at the annual get together of the European Association of Communications Agencies (EACA).  One of the issues I raised was that the amount of data out there was about to explode on account of the fact that things, not just people, were now becoming connected to it.  In the Big Data context I called this ‘the kettles that spy on your life’ – for this, essentially, is what happens when we connect things to the internet: they become spies, either on your  life specifically or on life in general.

I was first introduced to this idea about 18 months ago when I saw a presentation by Andy Hobsbawn at the Social Media Influence conference.  Andy was an enthusiastic supporter of the idea and one of the possibilities he suggested was what might happen when jeans, or other items of clothing, acquire an internet identity.  As soon as he said this the thought that flashed into my mind, however, was not ‘how intriguing’, it was ‘oh, my God’.  The whole thing seemed to be a terrifying prospect, not just the ability of these things to become spies but also the multiplicity of issues that might emerge when we start to give things independent identities and personalities that interact with our own.  Does this mean we will have to start giving things rights for example?  We are having enough difficulties dealing with issues like our own rights to privacy and data protection in relation to Big Data as it is (one of the issues also discussed at the EACA event), let alone dealing with our items of clothing – underwear, data and privacy: welcome to the Internet of Thongs.

Joking aside, the fact that things will join us on the internet as producers and quite possibly consumers of data clearly re-inforces what I already believe, which is that we are not going to solve this privacy and data protection thing by looking at initial sources of the data, because otherwise we will end up giving rights to our underwear and asking its consent.  The answer has to lie in controlling how data is used, not by controlling the way in which it is sourced.

However, the reason I now know this will be the Big Thing of 2014 is that I have just heard a piece on it on the BBC’s World at One programme.  It was approached in the same way the BBC and Radio 4 always approach these things which is to damn it with frivolity.  “Goodness me, a rubbish bin connected to the internet, whatever will these (silly) people think of next” was the general tone of the piece.  “Goodness me, a computer that everyone can own, whatever will these (silly) people think of next”.  Whenever the BBC (in fact traditional mainstream media in general) doles out this sort of treatment – you can be sure they are talking about the next big thing.

 

 

Big data: turning hay into needles

Here is a quick riff on an analogy.  Small data analysis is all about looking for needles in haystacks.  Big data analysis is all about turning hay into needles (or rather turning hay into something that achieves what it is we used needles to do).

Being more specific.  Small data analysis (i.e. the only form of data analysis we have had to date) was a reductive process – like everything else in the world where the data and information channels were likewise restrictive, largely as a result of their cost of deployment.  Traditional marketing, for example, is the art of the reduction – squeezing whole brand stories into 30 second segments in order to utilise the expensive distribution channel of TV.  Academic analysis likewise – squeezing knowledge through the limited distribution vessel that is either an academic or a peer-reviewed publication.

As a result the process of data analysis was all about discarding data that was not seen to be either relevant or accurate enough, or reducing the amount of data analysed via sampling and statistical analysis.  The conventional wisdom was that if you put poor quality data into a (small) data analysis box – you got poor quality results out at the other end.  Sourcing small amounts of highly accurate and relevant data was the name of the game.  All of scientific investigation has been based on this approach.

Not so now with big data.  We are just starting to realise that a funny thing happens to data when you can get enough of it and can push it through analytical black boxes designed to handle quantity (algorithms).  At a certain point, the volume of the data transcends the accuracy of the individual component parts in terms of producing a reliable result.  It is a bit like a compass bearing (to shift analogies for a moment).  A single bearing will produce a fix on something along one dimension.  Take another bearing and you can get a fix in two dimensions, take a third and you can get a fix in all three dimensions.  However, any small inaccuracy in your measurement can produce a big inaccuracy in your ability to get a precise fix.  However, suppose you have 10,000 bearings.  Or rather can produce a grid of 10,000 bearings, or a succession of overlapping grids, each comprised of millions of bearings.  In this situation it is the density of the grid, the volume of the data and, interestingly, often the variance (or inaccuracies) within the data that is the prime determinant of your ability to get an accurate fix.

To return to haystacks, it is the hay itself which becomes important – and rather than looking for needles within it it is a bit like looking into a haystack and finding an already stitched together suit of clothes.This is why big data is such an important thing – and also why a big data approach is fundamentally different to what we can now call small data analysis.  It is also why there is now no such thing as inconsequential information (i.e. hay) – every bit of it now has a use provided you can capture it and run it through an appropriate tailoring algorithm.

Article on Big Data in Sunday Telegraph’s Business Technology supplement

FireShot Screen Capture #242 - 'Richard Stacy_ The algorithm is the most powerful tool of social control since the sword - Business Technology' - biztechreport_co_uk_2013_07_richard-stacy-the-algorithm-is-most-powerfHere is a small article on Big Data I wrote as the opening shot in the Business Technology supplement published yesterday in the Sunday Telegraph.

Big Data is certainly a big buzzword, but there are those out there who say Big Data is nothing really new.  As a rule I find these people have careers based on what we can now call small data (or perhaps that should be Small Data).  Big Data certainly is something new, and there are two reasons why it is aptly named.

First, Big Data is really big.  It is not just a bit larger than the data we had before, nor is it just lots more of small data.  Big Data is defined by the fact that it is so large, it cannot be handled by the tools or techniques conventionally associated with data analysis (one of the reasons its rubs small data people up the wrong way) and this also means we can use it to do things which were not possible when all we had was small data. Continue reading

Big Data: gold mine or fool’s gold?

(This was published in the print edition of Digital Age in Turkey earlier this month.  It also appeared as few days later as a Digital Age blog post – if you want to read it in Turkish!)

There is a lot of buzz about the concept of Big Data.  But it is really the potential gold mine that some are suggesting?

Back in July I was at the Marketing Week Live show in London participating in an event organised by IBM.  We were looking at data and consumer relationships within fashion retailing, using high-end women’s shoes as the example.  The big issue fashion retailers face is that everyone walking into a store is a stranger.  The sales assistants know nothing about them, other than what they can deduce from their appearance and any conversation they can then strike-up.  We therefore asked ourselves the question: how might it be possible to use data from the digital environment so that potential customers were no longer strangers?  How might we be able to create a digital relationship so that when a potential consumer walks through the door the sales assistant would be able call-up this relationship history and pull this on-line contact into an off-line sales conversation?  One of the IBM analysts put it thus, “we need to be able to identify the exact moment a potential consumer starts to think about buying a new pair of shoes, identified from conversations they have with their friends in social networks and be able to then join those conversations”.

Welcome to the world of Big Data.  In the world of Big Data it is theoretically possible to know as much about your consumers as they know about themselves: to be able to anticipate their every thought and desire and be there with an appropriate product or response.  It is a world of ultimate targeting and profiling Continue reading