This is the
golden age of analytics when practical implementation of interesting data
driven ideas is happening at a fast clip. The only concern raised as of now is
in relation to privacy- Big Data shows signs of enabling the creation of
Orwellian Big Brother. But there is one more area of concern which I want to
draw attention, that of the possibility of Analytics actually slowing down the
pace of innovation. If true, society will pay a heavy price.
We are the
products of evolution with strange cognitive quirks. Cognitive science is just
about beginning to grapple with the fundamental question – what makes us tick?
Daniel Kahneman, in his seminal book “Thinking, Fast and Slow”, says that one
of our cognitive flaws – Delusional Optimism- might well be the engine of
Capitalism.
To quote
Kahneman: “Most of us view the world as more benign than it
really is, our own attributes as more favorable than they truly are, and the
goals we adopt as more achievable than they are likely to be”. He goes on
to say “When action is needed, optimism, even of the mildly delusional
variety, may be a good thing. The evidence suggests that an optimistic bias
plays a role- sometimes the dominant role- whether individuals or institutions
voluntarily take on significant risks”. And finally, “The
optimistic risk taking of entrepreneurs surely contributes to the economic
dynamism of a capitalistic society, even if most risk takers end up
disappointed”
Instances of such delusion are well known and documented. To
give a few examples (from Wikipedia entry under Illusory Superiority)
- In a survey of faculty at the University of Nebraska, 68% rated themselves in the top 25% for teaching ability.
- In a similar survey, 87% of MBA students at Stanford University rated their academic performance as above the median.
- In a survey of participants to evaluate their position on eight different dimensions relating to driving skill almost 80% of participants had evaluated themselves as being above the average driver.
Kahneman gives a similar example of entrepreneurial
delusion. Though the chances that a small business will survive for five years
in the US are about 35%, fully 81% of the entrepreneurs put their personal odds
of success at 7 out of 10 or higher, and 33% said their chance of failing was
zero.
Anything which reduces the rate of risk taking in society
will negatively impact the rate of innovation and consequently growth. One way
this can happen is when enough people stop being delusional, and see the world
as it really is.
What has Analytics got to do with this? There is a limit to
which we are capable of self delusion. Given enough data, most of us would be
forced to accept the truth of our limitations, and the true odds that we
face. For instance, if driving capability becomes an objective measure for
which data is widely available- something which is already beginning to happen
with on-board monitors and insurance premium linked to our driving performance,
it will be difficult to overestimate our driving capability. And this is what Analytics is likely to do-make more data readily available in all spheres of
our life along with personalised predictions of the odds for success making
self-delusional optimism increasingly difficult to sustain.
And when that happens, our risk taking capability will go
down along with the rate of innovation and growth – the engine of capitalism
may start sputtering.
This problem is compounded when you factor in the key
insight of Nassim Taleb in his book Black Swan, that “A small number of Black
Swans (or extreme rare events) explains almost everything in our world, from
the success of ideas and religions, to the dynamics of historical events, to
elements of our own personal lives”
Large impact innovations are Black Swan events, which are not
a result of systematic planning but are primarily a matter of chance fuelled by
risk taking. With fewer people taking risks and following their deluded dreams,
even the Black Swan may go the way of the Dodo.
Image (black swan: http://commons.wikimedia.org/wiki/File:Cygnus_atratus_Running.jpg)
Would you please explain the black swan analogy?
ReplyDeleteBlack Swan events are high impact events, and which by definition are not predictable though explained away post event.
DeleteTaleb says, and I agree based on my own life experience, that the most important events in your life (and the world) are Black Swans - events which took place not because of systematic planning on your part but because of chance, possibly fuelled by risk taking.
As we move to a data based analytics driven world focusing more on optimisation rather than unbridled risk taking, the possibility of encountering Black Swans go down. We need some amount of delusion to believe that we are infallible and to go ahead and take risks, objective data would only show that we are no different from others, its only chance that makes the outcome substantially different.