Tableau and the Politics of Political Maps - Visualising change in the population profile of India

Forty years (1971 to 2011) has seen the population profile of India change significantly. We increased in numbers from 566 Million to 1.21 Billion, but this growth was not uniform across the States – this can be seen in the simple filled map chart below where the colour encodes the percentage change in the relative population of the States.

This issue is getting media attention now as the 15^th Finance Commission tasked with recommending the share of each State from the divisible pool of taxes allocated to the States has been asked to use the population data of 2011 while making its recommendations (instead of the 1971 population as had been done by the earlier few Finance Commissions). This has far-reaching implications on the Finances of the States, and I am sure would be addressed appropriately by the Commission and the Government.

I write to highlight another issue – representation of India in the map charts made on Tableau. The default map shows India with the (painful) double lobotomy – the map of Jammu and Kashmir is not in keeping with the official map of India that we in India are so used to seeing.

When I first saw a map chart of India on Tableau about five years ago and pointed out this glaring error in the representation of India to the Tableau representative- he smilingly said – just change the Workbook Locale Setting to India, and voila – we see the official map now. This can be seen in the two figures below – and this is what I have been doing all these years. In a way, I was willing to acknowledge the existence of post-truth era we live in – Truth, like beauty, lies in the eye of the beholder.

But with the recent release of Tableau version 10.5, changing the workbook local to India no longer corrects the error in the default representation of the map of India.

This is unacceptable – Tableau is well advised to ensure that the map representation of India is in keeping with the official maps as released by the Survey of India, or risk losing one big segment of its user base in India.

I for one would not be using Tableau for any of the map-based visualisations – not with most of my visualisations being made for use by Civil Servants and Officers of Government of India.

Maybe PowerBI..

(PS – Changing the workbook locale to Pakistan does not change the Map of India any further, but changing the locale to China does another surgery in the North East – Arunachal Pradesh visibly shrinks)

Civil Services Examination - GoT / Game of Chance Update

A year ago, I had likened the Civil Services Examination, one of the most selective examinations in the world (success rate of a measly 0.23%), as a "Game of Chance". This was in my blog post, Civil Services Examination - Game of Thrones or Game of Chance?

I had come to this conclusion based on the following insights gleaned from an analysis of the CSE Exam results for the three years, 2013, 2014 and 2015, 

1) The interview score of the candidates has an effective weight of 40 % in determining the final rank of the candidate (and not 14 % as may appear prima facie by a simple computation of 275/2025). This is on account of the higher variability in the Interview (Personality Test) scores vis-a-vis the written scores. and 

2) The UPSC Interview had low "Reliability",  as there was little or no relation between the score obtained by the same candidate in two different years. The R-Squared value was in the region of  0.10, ie the interview score of a given year had less than 10% predictive power in explaining the interview score in the next year.

3) There is little or no relation between the Written and Interview marks scored by any candidate. Whatever ability (IAS-ness?) was being assessed by UPSC through these two proxies, it surely did not show up in the correlation between the proxies. 

The scatter plot below (taken from my last year's analysis) shows this weak (lack of?) relation between the interview scores of the same candidate in two successive years.

I had thereafter concluded that the UPSC interview is statistically seen to be of low Reliability, and by implication, of low Validity, as there can be no Validity without Reliability.

The only viable strategy available to the Civil Services aspirant was to acknowledge that this is a game of chance, and therefore to maximise the probability of success by taking as many attempts as possible, till such time that he gets the service of his/her choice (or exhausts his/her attempts).

This was a painful recommendation to make - it is bad enough that the youth of India spend (waste?) many years of their life trying to clear this exam, and here I was exhorting them to reappear, and spend another year or more till they get the service of their choice. 

I get to meet a fresh batch of young Officer Trainees of various Accounts and Finance Service (ICAS, IDAS, IP&TAFS, IRAS, IA&AS), every year at NIFM (where I am on deputation as Professor), where they undergo a part of their probationary training. Most Officer Trainees with attempts left reappear for the Civil Services Exam, but true to the game of chance analogy, only a few end up significantly improving their rank, they are as likely of failing to clear any of the three hurdles - Prelims, Mains and Interview.

Without any shadow or doubt, we can say that the Civil Services Examination as a Civil Services Capability/Aptitude assessment tool has very low Reliability (hence very low validity).

I conducted a similar analysis for the Civil Services Exam 2016 results, just to make sure that the earlier years were not an aberration. Here is what I found - once again:

1) A very weak relation between marks obtained in Personality Test (Interview) for the same candidate across two successive years ie CSE 2015 and CSE 2016. (A total of 188 candidates were identified who managed to clear the exam on both occasions, and it is their marks that are shown on the scatter plot)

2) No relation between the marks obtained by any candidate in Written Test and Personality Test. (The scatter plot is shown for the top 25 % of the candidates by marks secured in the Written Test)

So, nothing has changed in a year.

Let me make this clear - each and every person who clears the Civil Services Exam, any stage, is highly meritorious. Clearing the prelims itself puts you in the top 3 % of the applicants - that sure is something. My point is, from this stage on, it is luck (for want of a better word) that may be the key factor. So don't lose heart if you don't get the service of your choice, nor take pride in your ability for having "aced" this exam with a top-50 or top-100 rank.

I hope that UPSC will one day review the way it conducts its interviews, and find a way to increase its Reliability. Maybe they can learn from Google, which changed its interview process a few years ago, transitioning to "Structured Interview", or the US Office of Personnel Management, which also encourages government agencies to use structured interviews in hiring.  

Impact of GST on Small e-comm vendors - Why what you are reading in the papers is probably wrong

The Economic Times ran this story on its front page today with the title - "Small e-comm vendors will have to pay GST upfront".
There are factual inaccuracies in this article, which makes me realise that GST awareness is way below what it should be, given the fact that the GST rollout is hardly two months away.
So here is a quick lowdown on GST - and the error of understanding in the impact of GST on small e-Commerce vendors.

About GST: GST, or Goods and Services Tax, is undoubtedly the most ambitious and remarkable tax reform in the independent history of India. It is an Indirect Tax - a destination based consumption tax. For the first time, Goods and Services are being taxed at the same time, and by two different levels of the Government, the Centre and States, simultaneously. It is in the form of Dual Tax - both Centre and States would be levying GST on every transaction dealing with the Supply of Goods or Services. The Constitution had to be amended to allow both Centre and States to have joint powers in taxing Goods and Services. For the first time, we will see, One Tax One Nation principle in operation for most Goods and Services, with roughly the same rates of tax across India.
GST and small e-Commerce vendors: Among the various GST provisions is that of Tax Collection at Source (TCS) by electronic commerce operator (Section 52 of the CGST Act). As per this section, vendors selling goods on the various-Commerce sites would have upto 2 % of the sale proceeds collected by the e-Commerce operator for depositing with the Government,  1% on behalf of the Central Govt. (Central GST) and 1 % on behalf of the State Government (State GST). In case the sale is classified as an Inter-state sale, the tax collected at source would be upto 2 % under the Integrated GST Act (IGST). As per news report, the Revenue Secretary Hasmukh Adhia has informed that the GST council in its recent meeting in Srinagar has decided a 1 per cent Tax Collected at Source (TCS)).

Image Courtesy - Economic Times (22/05/2017). Image edited with X superimposed on image

The first error in interpretation of the GST Acts is that deduction of 1 % will impact all vendors registered on e-Commerce sites. As per the CGST Act, only those e-Commerce operators will have to make the deduction where the consideration with respect to such supplies is to be collected by the operator.  So, if you are a small e-Commerce operator, and you are only using the e-Commerce operator to showcase your product, you make your own shipment and collect the proceeds of sale directly from the customer, then the operator need not collect tax at source - the  money does not flow through him to collect it at source.
The second error in interpretation is that if your sale through an e-Commerce operator is liable for Tax Collection at Source of 1 %, and such tax amount gets deducted, then you can claim refund if your aggregate turnover in a year is below Rs 20 Lakh. This too is wrong - and for a very fundamental error in understanding regarding Indirect Tax, and the exemption limit. GST is a tax on consumption - not on supply.Though the seller or provider of Goods and Services is identified by the GST Act as a "Taxable Person", he is in effect a tax collection agent for the Government as on payment of tax on goods or services supplied by him "he is deemed to have passed on the full incidence of such tax to the recipient". It is the consumption which is taxed - not the supply. The exemptions are provided not on consumption, but on certain category of "Taxable Persons" who need to collect and pay tax, primarily to ease the compliance cost for small suppliers of goods and services. The definition of a Taxable person is given in the GST Act as: “taxable person” means a person who is registered or liable to be registered under section 22 or section 24". In case your aggregate turnover in a year is less than Rs 20 lakh, then you do not fit the definition of Taxable Person under Section 22, but you are a Taxable Person under Section 24 of the Act (24(ix)), which states that "persons who supply goods or services or both, ..through such electronic commerce operator who is required to collect tax at source under section 52". So, if you supply through an e-Commerce operator who is required to Collect tax at source under Section-52, then you are a Taxable Person, and you are liable to pay tax.
It is however to be noted that in case the vendor is making inter-state supplies, then again he is a Taxable person, irrespective of the turnover, and nature of e-Commerce operator.

***

NIFM - (National Institute of Financial Management - www.nifm.ac.in) , is an autonomous body under Ministry of Finance, is an accredited GST Training Partner of NACEN. We conduct regular GST Awareness program of 3 days duration for officer of GoI, and for trade and industry.

Government e Marketplace - A GeM of an Idea with seeds of becoming the ultimate disruptor

On 9th August 2016, the Government of India launched the Government e Marketplace (GeM) without much fanfare.
The platform, which allows online purchase of common office use Goods and Services by Government buyers,  was inaugurated by the Minister of Commerce and Industry, Smt Nirmala Sitharaman. As the blurb on the GeM site. says, "GeM represents our Government's firm commitment to bring greater transparency and efficiency in Public Procurement"

That it does, but the platform can and will do more, and has the potential of disrupting the "disruptors" of the information age (Think any platform - Amazon/Flipkart, Uber/Ola, TaskRabbit/UrbanClap). Here's why:

Why I expect GeM to Take-Off

1. GeM platform is inherently well placed to solve the chicken-or-egg problem faced by conventional platforms.
It makes procurement extremely easy for the Government buyers providing them the required incentive to procure through GeM. The Government rules were amended in May 2016 (GFR 141-A) to allow procurement through this portal. Orders can now directly be placed on any Supplier/Vendor on GeM  provided that the order amount is below Rs 50,000/- or if the Supplier/Vendor happens to be offering the lowest price among the available suppliers on GeM. There is no upper limit on the amount of order that can be placed in the latter case.
It (GeM Platform) welcomes Supplier/Vendor with an open arm with a Trust First Verify Later approach.  Suppliers can self register themselves - no waiting outside a Government procuring agency to get empaneled as a Government Supplier. The supplier provides relevant details about his/her firm which is validated online against Government databases  (MCA-21 Company Database, Udyog Aadhar number, PAN Database, TIN Number etc.). These would be subsequently verified through an offline process.
2. There is a strong support from the leadership team - The responsibility for running this platform has been placed on DGS&D, the Central Procuring Authority of India, whose leadership is fully committed to its success. Large number of trainings are being organised, both for Government buyers and sellers to make them understand the benefits of GeM (We at NIFM have conducted training for over 500 buyers and sellers in the last three months). The DGS&D, which is the central procuring agency, has decided to do away with the Rate Contracts (empanelment of suppliers) for all items that are made available for procurement on GeM. In a way, this is a commitment that there is no turning-back - the bridges are being burnt.

Why I think (and hope) that GeM can be the ultimate "Disruptor of Disruptors"

At the heart of GeM user authentication lies the Aadhar ID, the 12-digit unique identification number issued by the Indian government to every individual resident of India. Not only is Aadhar number unique, it is mapped to the biometric details of the citizen (fingerprint, iris scan).Few nations have their citizen's trust their government with this level of personal details. Every transaction on GeM, be it by the buyer or the seller, is e-Signed using the Aadhar number of the person initiating the transaction. With almost all interactions of the citizens/residents with the Government now becoming Aadhar enabled, and with the Aadhar enrolment in India at over 90 % , it has now reached the stage wherein it is ready to serve as the Trust Platform of the Government. The Government can embark on creating a trust score associated with each Aadhar ID for different types of transactions. The various Information age platforms, be it Uber/Ola or UrbanClap/TaskRabbit, or even Facebook ultimately derive value from the trust that they are able to provide to the transacting parties. If a trust platform is created by the Government using Aadhar, and this is made available as a Service to  a Government owned or Not-for-profit platforms, then the platform aggregators will stand disrupted. The sharing economy(a misnomer for the Information age disruptors) will truly live upto its name when it is the interacting and transacting parties which share all of the value generated by the exchange, and not the scraps doled out by the aggregator after taking its cut. We all know that Uber or Airbnb can charge 10 to 30 % as their commission for the matching service, and this when their algorithms are not open to public scrutiny!

As a first step, GeM can be expanded to allow residents and citizens, and not just Government buyers, to procure Goods and Services. The algorithms which match the buyer and the seller can have societal benefit as its primary goal, and not profit maximisation.  A government owned or backed platform can make its algorithms, both for trust scoring and buyer-seller matching available for audit- say by the Comptroller and Auditor General of India.

This will be a revolutionary development whose time has come. Information has to be recognised as a Public Good, and the Government has to step in to ensure that the digital exhaust of the citizens does not become the property of data aggregators be it Google or Facebook or Amazon, At the same time, any algorithm that works on this data has to be subject to public scrutiny lest the delicate balance between efficiency and fairness is lost for good.

Brexit - Analysis indicates Referendum may be the Wrong Way for Contentious issues

The world is still trying to come to grips with Brexit - Much has been said and much will be said on this emotive issue over the next few years. This is an event whose impact will slowly but inexorably play out impacting the lives of millions.
Here is an analysis of the referendum results - with some findings which merit serious discussion by all democracies who may think that Referendum is the way to go for all BIG decisions.
My conclusion - Referendum without mandatory voting by All is wrong. Had voting been mandatory - the Brexit referendum results could as easily have been different. A 4 % gap is no gap at all when 28% of the population chose to watch the show from the sidelines.
1. Results were close: Remain: 48.11 % Leave: 51.89 % - the gap between the two: 3.78 %.

The results are available for the 382 different local areas. In terms of sheer count of the local areas voting Leave - the gap appears wider.

2.Voter turnout was high- but anything below 100 % won't do for deciding on such a contentious issue 

While the overall voter turnout at 72.1 % was quite high as voting in democracies go (The last general elections in India had the highest ever voting % for India at 66.38%), it was far short of 100%. Let us understand why even a 72% voting may not be good enough to truly measure the will of the people. The plot below shows the voter turnout versus the absolute gap between the Leave and Remain vote %.

The graphic clearly shows that the Voter Turnout % going down as the absolute Gap between Leave and Remain increases. Thus the more uniform the views of a local area (with a clear majority in favour of Leave or Remain), the lesser is the voter turnout. Thus, the voting turnout is highest when the Leave and Remain voters were almost equally divided (the left hand side of the graphic), and lower where there was a clear gap in either direction. My explanation for this pattern of behaviour is the thinking embodied in all Democracies with "First Past the Post" system. Once a voter knows that her area is predominantly going to vote in a given direction, she may feel that her vote does not count and hence would not go out to vote. This kind of thinking does not work in Referendum when every single vote counts, and local area wins count for nothing individually - there is no MP or Congressman getting elected in the referendum process.
One may argue that a 72% turnout should be able to measure the overall sentiment accurately. This is backed by the Law of Large Numbers thinking - where a truly random sample, though small, can fairly accurately assess the % in favour or against a particular issue. This is why the poll results are reasonably accurate,
For this thinking to work, the 72 % who voted should be no different from the 28% who did not vote. But such was probably not the case in this Brexit referendum, Refer the graphic again, and you can see that the voter turnout % shows a steeper decline in case of Remain than in the case of Leave. These are two different groups, with different levels of commitment to the cause as measured by the voter turnout.

3, Voter turnout % differs for the Leave and Remain groups with consequent impact
The relationship between the voter turnout and the absolute gap (between Leave and Return %) is clear. But the data also shows that the Leave group was more committed and the local areas where Leave had a majority shows a higher voter turnout. This can be seen in the tabulation and graphic below. I have categorised Local Areas with an absolute gap of > 20 % as COMPLACENT, and those with gap of less than 20 % as ENGAGED.

As can be seen above, the Voter Turnout % is overall higher when the voters know that there is a tough fight ahead (Engaged Category - Gap of less than 20 %) than when the voter is complacent knowing that the results in her area is clearly going one way. This behaviour of increased voter turnout is seen in both the Local Areas which voted to Remain and those which voted to Leave. However, the Local Areas where the majority voted to Leave shows a higher voter turnout in each category.This is the key to the final conclusion. The 72 % voters may not be truly representative (in statistical terms) of the 100 % population, and the results could have been different if there was 100 % turnout.
It is standard thinking based on the work done by behavioural economists that losses loom twice as large as gains. In a Zero-Sum game with the population equally divided between those who perceive the outcome as gains and those who perceive it as a loss, it is the group which perceives the outcome as a loss which would be more motivated in putting up a fight. In such a case, the people who actually come out to vote would be skewed in favour of those who perceived the outcome as a loss, and hence not be representative of the population.This pattern would not be so easily discerned in a pre voting poll, but the self-selection bias would manifest itself on the day of the actual voting.
The Graphic and tabulation above shows that the voters in favour of "Leave" ensured a higher turnout. More than the Status-Quo bias, it was the asymmetry of Gains and Losses which finally decided the issue.
If the Leave and Remain groups were equally divided, and the Leave group was 5 % more motivated than the Remain group to go out and vote,(ie, if Remain Group has 70% Voter turnout rate, then Leave group at 5% higher rate would be 73.5% turnout)  then any voter turnout less than 96% of the total electorate would lead to the Leave group winning by 51.2 % against 48.8 %. Only a 100% voter turnout could reveal that the two groups were equal.
This is the problem with Referendum, where the two opposing sides, though having clear views on which way they stand on the issue may have different levels of motivation to go out and vote. Even a 5 % difference in the motivation level and consequent voter turnout would lead to results different from what the population as a whole wants.
Referendums should be used only when voting is made mandatory and near 100 % turnout is assured. I hope that we do not see a spate of referendums in the EU with an outcome which does not truly represent the views of a fairly equally divided population.


Election results data available at the UK Electoral Commission website and can also be downloaded here.

Civil Services Examination - Game of Thrones or Game of Chance?

The Civil Services Examination of India is one of the most selective examinations in the world with probably the lowest success rate for any competitive examination. In 2015, a total of 465,882 candidates appeared for the first stage of the Exam – Prelims, of which 15,008 (3.2%) were selected to appear for the next stage of written examinations – Mains. Around 20 % of these candidates, specifically, 2797 were called for third and final stage – the Personalty Test or Interview of which a total of 1078 candidates were finally selected and recommended for appointment to one of the many Services of the Government of India, notably, the IAS, IFS, IPS, IA&AS, IRS, ICAS, IDAS, IRAS, IP&TAFS etc. The success rate is thus a measly 0.23%. 

I took the Civil Services Examination in 1994  and again in 1995, cleared on both occasions, and  based on my 1995 exam rank of 136, joined the Indian Audit & Accounts Service (IA&AS) in 1996, and have been happily working with the Audit Department since then.

In those days, the marks secured by the successful candidates were not available in the public domain. Recently, when  I came to know that this data is available on the UPSC website, I decided to analyse the same. Here is my analysis, based on the presentation that I made to the Civil Services Officer Trainees (Probationers) of the various Accounts and Finance Services  undergoing training at NIFM (where I am presently on deputation as Professor).

The dataset in MS Excel and pdf format can be downloaded here. (MS Excel / PDF)

The above visualisation shows the stratification of final Rank (bin size of 85 ranks) with relative share of the four different categories in different rank bin. This is fairly stable till about Rank 600, which has been taken as the cut-off rank for subsequent analysis.

The above visualisation (Histogram) shows the very narrow band in which the written scores of most candidates lie - between 40 to 45 % in both Civil Services Exam 2015 and 2014. (Each bin of width 1 %)
The Personality Test (Interview) however shows a very different behaviour - the spread here is far wider. A very interesting feature of the interview scores distribution is the "lumpiness" of the data, with prominent spikes at round number scores of 55 %, 60 %, 65 % and 70%. Based on this, one can safely infer that a) The UPSC Interview board is assessing the candidates on a percentage scale and then converting it into marks (out of 275) and b) Some of the Interview Boards are not attempting to be very precise in their assessment, and are willing to grade candidates in round number scores (say 65% instead of 64 % or 66 %).

What could be seen visually in the histogram is now clear in the above tabulation of the Standard Deviations of the Written and Personality Test (PT - Interview) Scores.

The importance of each Mark in determining the final rank of the candidates can be seen in the visualisation above, which shows the number of candidates with the same Marks - right from the highest score of 1063 with Rank 1 at the left to a score of around 877 with Rank of 600 at the right . While there is sufficient gap between the candidate's marks in the top 10 or so ranks, we have 4-5 candidates per mark around the 50th Rank and 9-10 candidates per Mark around the 100th Rank, going as high as 40 candidates with the same score at around Rank 600 !!
Each mark matters - and with Interview Marks showing a round number bias and using a % scale (with each % equal to 2.75 marks) added to the low reliability, an element of randomness/chance is introduced in the selection process of the candidates.

The final rank of the selected candidates is based on the total of the Written and Personality Test (Interview) scores. If UPSC was looking for a certain attribute, lets call it "IAS-ness" for want of any better name, which was present in both Written Exam and Personality Test, then these two scores would have shown some relation (correlation) with each other. For example,  the oft-quoted spurious correlation seen between ice cream sales and shark attacks occurs because both are related to the common variable - Temperature. We however do not see any correlation between the Written and Interview Scores - actually , a negative correlation is seen. This happens because the successful candidate with a poor Interview score necessarily has to have a higher Written Score and vice-versa; else she would have not made the cut-off.

The above definitions and examples of Reliability and Validity are taken from the excellent book by Richard Nisbett - Mindware-Tools for Smart Thinking. (This book, along with Daniel Kahneman's Thinking Fast & Slow are mandatory reading for all).
Which brings us to a very important Question:

A serendipitous natural experiment dataset was available within the Civil Services results for the year 2013,2014,2015. As a fairly large number of candidates clear the exam in successive years, their Interview scores in the two years could be examined to see the Reliability of UPSC Interview.
The above visualisation- scatter plot, which shows the correlation between the Interview scores of the same set of candidates in two successive years was the most surprising (and disturbing) finding. One can very clearly see that there is little if any relation between how a candidate may fare in the Personality Test in two successive years. With an R-Squared of around 0.1, the Interview process can be said to have limited Reliability(and as a consequence also of limited Validity.).
Whatever the Personality Test tries to assess (the "IAS-ness") lack of reliability leads to a doubt on the validity of the measurement.
Daniel Kahneman speaks about the "Illusion of Validity" as he recounts his experience of evaluating candidates for officer training as part of a group of evaluators. In his words "Our impression of each candidate's character was as direct and compelling as the color of the sky.... A single score usually came to mind and we rarely experienced doubts or formed conflicted impressions." However, the assessment of the candidate was in variance with the actual performance of the candidates in the officer-training school. As Kahneman goes on to say-"our ability to predict performance at the school was negligible. Our forecasts were better than blind guesses, but not by much". 
Richard Nisbett calls it the "Interview Illusion", and says that "predictions based on the half-hour interview have been shown to correlate less than 0.10 with performance ratings of undergraduate and graduate students, as well as with performance ratings for army officers, businesspeople, medical students, Peace Corps volunteers, and every other category of people that has ever been examined".

While the Personality Test (Interview) is seen to show limited Reliability, the Written scores of the same candidates across successive years shows a slightly higher consistency (Greater R-squared value), but not as high as a typical measure of pure ability.

The final rank of the candidate is based on the total of her Written and Interview Scores. The higher the score, the better the rank. Since it is not the absolute but the relative score that determines your rank, a regression model was built to predict the rank in percentile terms based on the candidate's percentile on her Written score and the percentile on her  Interview score in relation to the other candidates. The model was developed for the top 600 ranks for Civil Services Exam 2014 and 2015, and is shown below:

The model shows a good fit with the data as measured by the high R-Squared value, and based on the relative weights to the two parameters, it is seen that the Personality test carries a 40% weight in determining the final Rank!

Interview or the Personality Test has a high 40% impact weight, but at the same time it shows low reliability (year to year score consistency). The Written scores too can vary across years.
So what should the Civil Services aspirant do?

While the above is a strange (and funny) trend with lower roll number more likely to lead to success in the exam (Which way does the Causal arrow point?), the candidates need to rethink their Civil Services strategy.

So, any Civil Services aspirant should be prepared for the long haul.
The median number of attempts of the candidates who finally join the service is likely to continue increasing.
And what a waste of precious years of the talented  young Indian boys and girls slogging away to clear a highly unpredictable exam.
I hope that the expert committee constituted by the Government under the chairmanship of Sh BS Baswan to examine the various issues regarding the Civil Services Examination is able to fix some of the problems in the current system, and more importantly, prevent the colossal waste of time and effort of the youth of India.

Bounties for the Well-Off?

In one stroke of his pen, Arvind Subramanian – Chief Economic Adviser to the Government of India, has made me and most of you become the new Indian economic elite. We have been kicked hard - Upstairs - and have vaulted from our cosy conception of being part of the Indian "middle class" to the Piketty reviled elites forming the 1 %. (ET ran this article on 27th Feb 2016)

Be warned, for the Occupy Dalal Street movement that you were probably contemplating of leading or joining, is now a protest against you- the economic elite of India.

NO, the economic adviser did not promise any Bounties to make us all well-off, he  did the math and said that anybody earning more than 2 lakh rupees in India is Well Off (thats you and me, and anybody you know). And the meager tax breaks that he/she gets, are the bounties that needs to be clawed back by the Government of India.

His simplistic argument can be simplified further.

Anybody in the top 2 % of the population is a part of the economic elite, top 5-6% are definitely Well-Off and not a part of the vaunted middle class of India.

Based on the Income Tax data, any person in India having an income of more 2 lakh is in the top 6 % of the population, earning more than 5 lakh puts her in the top 1. 5% and more than 10 lakh places her in the envious top 0.5 % of the population.
A key assumption that is made here, which only the very naive would accept, is that this actually represents the true income distribution in India. The number of tax payers has remained near stagnant at around 3 crores for at least a decade now; and not because the number of people with taxable income has not increased.

And this cake shaped graphic takes the cake:

The cake above conceals more than it reveals. Lets try to understand it better.

The graphic is drawn to show the percentiles. (the axis on the left). But the markings on the right, which shows the income slabs are not drawn to scale.
The gap between 5 and 10 lakh appears to be less than a third of the gap between 2 and 5 lakh. And the upper slab of taxable income, which spans Rs 10 lakh to Rs 160 Crore (Yes, 1600 times more than the entry income of 10 lakh) is shown as a narrow sliver. Had this been drawn to the scale of income, the "cake" visualisation would have looked more like burj Khalifa, probably taller.
In this narrow looking band of 0.5 % of the population is where the real income variation is seen, and the true "Rich" and "Well-Off" reside.
To say that the meagre investment made by people in PPF or GPF Accounts, where the maximum investment (with tax saving incentive) is capped at Rs 1.5 lakh per annum, leads to a subsidy for the "well-off" adds insult to injury. This is no "bounty" to the well-off - for most, this is the only investment that they make from their sub-10 lakh salary.
The rich, who inhabit the broad earning range from 10 lakh to 160 Crores, are not investing their surplus in PPF accounts. Their investments lie in financial assets where capital gains are taxed far more favourably.
Instead of targeting the small savings (Yes EA, these are small savings) made by people with income between 4 lakh and 10 lakh, the Economic Advisor could have raised tougher questions - like why capital gains should not be taxed at a higher rate, and why the peak income tax slab starts at Rs 10 lakh and stops at 30 %, when we have people in India with income a thousand times greater than this? Even the US, whose free market ideology we seem to be aping, has higher income tax rates of 33%, 35% and 39.6 % for the top income slabs.
Hope the government did not take the Economic Adviser's advice seriously, and ignored chapter 6 of the Economic Survey (Bounties for the Well-Off) while finalising the Budget proposal.

Reponse - TRAI Consultation Paper on Differential Pricing for Data Services

The Telecom Regulatory Authority of India has recently issued a Consultation Paper on the Differential pricing for Data Services. This paper, and the final decision of TRAI on the same is likely to have a big impact on India, and on the world too, as the action taken by India would set the tone for similar action by developing countries world wide.
Much has been said about the issue of Net Neutrality. The FCC ruling on "Protecting and Promoting the Open Internet" of 12 March 2015 addresses the core issue in a comprehensive manner. The FCC identified the following Clear, Bright-Line Rules:
1. No Blocking: A person engaged in the provision of broadband Internet access service, insofar as such person is so engaged, shall not block lawful content, applications, services, or non-harmful
devices, subject to reasonable network management.
2. No Throttling: A person engaged in the provision of broadband Internet access service, insofar as such person is so engaged, shall not impair or degrade lawful Internet traffic on the basis of Internet content, application, or service, or use of a non-harmful device, subject to reasonable network
management.
3. No Paid Prioritization: A person engaged in the provision of broadband Internet access service, insofar as such person is so engaged, shall not engage in paid prioritization. “Paid prioritization” refers to the management of a broadband provider’s network to directly or indirectly favor some traffic over other traffic, including through use of techniques such as traffic shaping, prioritization, resource reservation, or other forms of preferential traffic management, either (a) in exchange for consideration (monetary or otherwise) from a third party, or (b) to benefit an affiliated entity.

These are sound principles that the TRAI must consider adopting.

However, the current raging issue of "Free-Basics" is not in direct violation of the FCC ruling. Free basics is a form of paid prioritization through zero-rating. FCC notes in para 153 of its report "Given the unresolved debate concerning the benefits and drawbacks of data allowances and usage-based pricing plans, we decline to make blanket findings about these"
The FCC has also stated on the issue of Paid prioritization that: "The Commission may waive the ban on paid prioritization only if the petitioner demonstrates that the practice would provide some significant public interest benefit and would not harm the open nature of the Internet"

The TRAI needs to go further than the FCC in favouring net neutrality.
A clear and firm no to free-basics would be a step in that direction.

Here's my response to the issues raised in the TRAI consultation paper:

Question 1: Should the TSPs be allowed to have differential pricing for data usage for accessing different websites, applications or platforms?

Response Q1: No. TSPs should not be allowed to have differential pricing for data usage for accessing different websites, applications or platforms.

Question 2: If differential pricing for data usage is permitted, what measures should be adopted to ensure that the principles of non-discrimination, transparency, affordable internet access, competition and market entry and innovation are addressed?

Response Q2: A firm stand should be taken to prohibit any form of differential pricing in data usage which directly impacts access to specific parts of Internet. The justification for this stance is the “thin edge of the wedge” argument - once differential pricing (including Zero-rating for some content) for data is permitted, the TSPs can exercise the power that they technically have to become active Gatekeepers of the Internet. They can hold both the consumers of Internet, and the content providers to ransom – and in the long run lead to the shrinkage of Internet access, and simultaneous increase in cost of access. The content providers will be willing partners to this “ransom”. In a networked age where Power law distribution with increasing concentration is becoming the norm, even a small advantage can make all the difference between success and failure. Thus content providers would be willing to pay a substantial amount to the TSPs to be given preferential treatment in access to their site by consumers, say by getting it Zero-rated. If this option of getting zero rated is kept open to all content providers without any discrimination- then the advantage to any individual content providers is lost,  but the cost of content distribution over the Internet would go up. This would adversely affect the dynamism and innovation potential of Internet.

Question 3. Are there alternative methods/technologies/business models, other than differentiated tariff plans, available to achieve the objective of providing free internet access to the consumers? If yes, please suggest/describe these methods/technologies/business models. Also, describe the potential benefits and disadvantages associated with such methods/technologies/business models?

Response Q3: Internet is all of the Web as we understand it. Any subset of the Internet – say a few websites, and apps cannot be called Internet. The zero-rated plans which allow access to a subset of Internet cannot be construed as a model for providing universal access to Internet. They are best viewed as sponsored content – and hence not at all meeting the “objective” of providing free Internet access.

Enabling a certain basic access of Internet to all is a laudable objective. Digital divide is understood to be a form of social and economic inequality which needs to be tackled by society. However, since provisioning of Internet access consumes economic resources, we need to accept that somebody has to pay for it even if it is not the end consumer. The key decision lies in identifying the party which would pay for Internet access to meet the societal goal of bridging the digital divide.

One possible model of funding universal Internet access to meet the challenge of Digital Divide is through the CSR spending which has been made mandatory by the Companies Act 2013.  The Act which applies to large companies meeting certain financial criteria[1]  mandates such companies to spend a minimum of 2 per cent of the average net profit made during the three immediately preceding financial years. Government may make funding universal internet access as one of the permissible activities for CSR spending, and include it in Schedule VII of the Companies Act 2013.

All TSPs may then voluntarily contribute their CSR spend towards providing this universal Internet Access.

The universal Internet access may be based on a bare bones data plan to be specified by TRAI, comprising a fixed amount of data access (say 200 MB per month to begin with – this can be periodically revised upwards). This plan can then be made available by all TSPs to their subscribers for free – to the extent of availability of funds. The plan may come with the rider that any person who would like to consume Internet access more than the minimum provided in the plan has to pay for additional access at a rate which also covers the cost of the “free” component.

While it is understood that CSR contribution by the TSPs by themselves would not be able to provide the basic Internet access to all their subscribers, this can be supplemented by the CSR contributions from other corporates. A suitable message can be displayed to the Internet consumer informing him/her that the Internet access has been sponsored by the particular corporate entity/TSP. TSPs may be encouraged to report on the number of subscribers being provided free Internet access through such CSR funding.

Question-4: Is there any other issue that should be considered in the present consultation on differential pricing for data services?

Response Q4: TRAI may consider porting of Broadband internet access service provider over mobile networks in the same way as porting of mobile numbers ie unbundle Internet access from voice communication service.

---

[1] Net profit of Rs. 5 crore or more or net worth of Rs. 500 crore or more or a turnover of Rs. 1000 crore or more in any financial year [Section 135(1)]

TRAI picks up the call – but why did the market drop the message?

In a very consumer friendly move which far exceeded expectation, the TRAI issued a notification on 16th October 2015 - Telecom Consumers Protection (Ninth Amendment) Regulations, 2015 - by which relief was provided to the consumers facing call drops. From 1st January 2016, every consumer who faces a call drop will be credited by one rupee by the originating cellular mobile telephony service provider.
I was one of the many individuals who had provided feedback on the Consultation paper on "Compensation to the Consumers in the Event of Dropped Calls" floated by TRAI. (You can read it here).
I had then observed - "The consumer derives value from the call only when the sum and substance of the intended conversation stands completed" and that "--compensation, while correcting the wrong of the inconvenience to the consumer, would also serve as a financial disincentive to the service provider"
I am gratified that TRAI, in its Explanatory memorandum to its 16th October 2015 order has invoked both these arguments(Verbatim, actually).
But the compensation of Rs 1 for every dropped call exceeded my wildest imagination. My best case scenario was a compensation of about 30 seconds for consumers on a per second billing rate, and that too for calls dropped within the first 30 seconds. The present TRAI order provides for Rs 1 compensation as credit, or roughly 100 seconds of talk time at the current billing rate, for EVERY call dropped, irrespective of the duration.
If this order is implemented, the estimated loss of revenue in the form of compensation to subscribers would be in the region of 2 % of the billed amount. This is substantial - and may account for over 10 % of their profit before Tax.
So should not the market have responded to this news by hammering the stock prices of these mobile telephony service providers?
Here is how the stock prices (of three TSPs - Bharti Airtel, IDEA, and RCOM) looked in the period 16-21 October 2015 - notice the sharp dip in prices on the date of the TRAI order (16th October), and the almost immediate recovery. So what does the market know that this order does not convey? Is it optimism or a frank assessment that the big money Telecom guys will be able to ensure that the order in its current shape would not get implemented.
My money, sadly, is on the order getting diluted substantially. The current order appears to have become too consumer friendly to go uncontested. Did the consumer lose in the process?

Junk the Bell curve – but in the right places and for the right reason

Maybe there were loud cheers in the Infosys flock with the announcement that the company is doing away with the bell curve based performance assessment. And if Economic Times which ran this story is to be believed, the “decision is already showing results” with a substantial reduction in attrition. Looks like one of those Efficient Market Hypothesis scenario, where the employee attrition rate at Infosys has already factored in this “junk the bell curve” decision of the management to be implemented from this quarter. Or maybe it’s a post hoc fallacy, but that is not the issue for discussion here.

I am all for the end of Bell Curve way of thinking, but only for its universal applicability to domains where it is simply not valid leading to errors in judgment.

The normal distribution, popularly known as the bell curve, has a strong hold on our way of thinking. We almost believe it to be a law of nature (It is not). But fundamentally, it is a distribution of errors. You can see how the bell curve equation can be derived in this 5 minute YouTube video.

Frankly, Bell curve leads to a fairly benign performance appraisal system.A Bell curve is symmetric, it’s mean and median are the same - the average is truly sitting at the centre, with 50% of the values above and below it.

Where else in the world of human interactions to do you see such an egalitarian (Yes – egalitarian) distribution?

Whenever people interact, minor advantages conferred by our fickle preferences or chance snowballs to huge differences over time. This is the classic Matthew effect

Daniel Kahneman succinctly put this as his “favourite equation” –

success = talent + luck

great success = a little more talent + a lot of luck

Any appraisal system that replaces a bell curve based system, and which also wants to rank the employees for rewarding/penalising the top/bottom performers is likely to be far more discriminatory than the poor, much maligned and misapplied Bell. Employee performance, if measured by pure output, is going to show a Power Law distribution. Thomas Piketty, in his book Capital in the 21^st Century, has shown the prevalence of such skewed distribution in income and wealth of individuals.

Are organisations and performance appraisal systems willing to accept the role of luck in success, and the perception of success which leads to further opportunities for the winner in turn leading to the creation of superstars?

The graphic below shows the relative share of various segments of population for Income and Wealth in USA (for 2010, based on data from Piketty’s book), and the share under normal distribution (say of Income, with mean of USD 45,000 and standard deviation of USD 15,000).

The bell curve leads to a far lower level of inequality. 

Now if the idea behind junking the bell curve is to do away with the forced identification of the “non-performers” – then I am all for it. 

I would live in a bell curve world any day - and wish the same for the world – but fear that the Power Law is here to stay.

Call Drops in India - Does the TRAI Consultation Paper address the core issues?

Dropped calls have become a part of the telecom scene in India. We don't even get angry any more when we face a call drop- we simply accept it and call again, or if we are indoors, try calling again using the land line.
Recently, when the issue of call drops started hitting the front page with the Prime Minister of India also commenting upon the issue, the Telecom Regulatory Authority of India came out with a consultation paper on 4th September 2015 titled - "Compensation to the Consumers in the Event of Dropped Calls" . The paper sought comments from all stakeholders by 21st September 2015.
Like a good citizen and a willing stake holder (sic)  I reviewed the consultation paper, and dutifully forwarded my comments to TRAI.
Let me say upfront, I was not very satisfied with the drift of the questions - it appeared to suffer from the logical fallacy of "petitio principii"- begging the question. When you are given a finite set of options to choose from, not only is your choice constrained, but even the overall scope of options that you can imagine gets constrained. The TRAI paper is riddled with such assumptions.
Here are my responses to the questions raised by TRAI in its consultation paper.

Response to TRAI Consultation Paper No. 4/2015

Compensation to the Consumers in the Event of Dropped Calls

Q1: Do you agree that calling consumers should not be charged for a call that got dropped within five seconds? In addition, if the call gets dropped any time after five seconds, the last pulse of the call (minute/second) which got dropped, should not be charged. Please support your viewpoint with reasons along with the methodologies for implementation.

Response Q1: The calling consumer should not be charged for a call that gets dropped within a finite duration of the beginning of the call. However, the cut-off duration of 5 seconds for deciding when not to charge for the call, as suggested in the Consultation paper, is too low, and appears biased in favour of the telecom service provider. The consumer derives value from the call only when the sum and substance of the intended conversation stands completed.  What is the most appropriate cut-off duration value can be debated, but in my assessment, it should be the median call duration. An analysis of the calls made by me over last 8 months shows that less than 3 % of the calls are of less than 5 seconds duration, with the median call duration being 30 seconds. It is proposed that calls that get dropped within 30 seconds of the call initiation should not be charged.

The distribution of call duration (over last 8 months) is shown below. The X-Axis shows the call duration binned in intervals of 5 seconds, the Y-Axis shows the % of calls in each bin (as bar), and the cumulative count of calls (as line)

Q2: Do you agree that calling consumer should also be compensated for call drops by the access service providers? If yes, which of the following methods would be appropriate for compensating the consumers upon call drop:

(i) Credit of talk-time in minutes/ seconds

(ii) Credit of talk-time in monetary terms

(iii) Any other method you may like to suggest Please support your viewpoint with reasons along with the methodologies for implementation.

Response Q2: Yes, I agree with the TRAI Consultation paper that calling consumers should be compensated for the call drops by the access service provider.

While the response to Q1 addressed the issue of the dropped calls which should not be billed by the service provider, this by itself does not compensate the consumer for the poor quality of service. An additional compensation, while correcting the wrong of the inconvenience to the consumer, would also serve as a financial disincentive to the service provider. This compensation may be in the form of Credit of talk-time in minutes/ seconds. For consumers who are on a per-minute pulse, the credit could be of 1 minute, while those with per-second pulse, the credit could be of 30 seconds (being the assessed median call duration).

Q3: If the answer to the Q2 is in the affirmative, suggest conditions/limits, if any, which should be imposed upon the provision of crediting talk-time upon call drop and usage thereof.

Response Q3: As mentioned in the response to Q2, the compensation may be in the form of Credit of talk-time in minutes/ seconds. For consumers who are on a per-minute pulse, the credit could be of 1 minute, while those with per-second pulse, the credit could be of 30 seconds (being the assessed median call duration).

The only condition/limit on the talk-time credited on account of call-drop could be that the credit be consumed on the same day that it is credited.

Q4: Is there any other relevant issue which should be considered in the present consultation on the issue of call drops?

Response Q4: The following additional relevant issues should be considered in the consultation paper:

1) Quantum of penalty / financial disincentive on the TSP: Presently, a penalty of “Not exceeding Rs 50,000” can be imposed on the TSP by TRAI for not meeting the Quality of Service parameters, including call drop. This is too low to serve as a disincentive. This may be suitably increased to the point where it actually starts functioning as a disincentive.

2) Mode of computation of the Quality of Service(QoS) parameter of Call Drop: Presently, call drop is measured at an aggregate level by averaging all the call drops over the period of 1 month. The consultation paper recognises the inherent limitation of this aspect and states: “sometimes averages may not give the full picture”.  The current target of average call-drop of less than 2 % thus does not adequately ensure this quality of service for the individual subscriber, but only for subscribers on average. This parameter may be reinterpreted as quality of service for individual subscriber. TSPs may be asked to provide data on the percentage of subscribers who faced call drops in excess of 2 %, in addition to the average call drop rate. TRAI may set the Quality of service target of this parameter as: “Ensure that not more than 5 % of the subscribers face call drops in excess of 2%”. A basic statistical computation leads to a translation of this parameter as a requirement that the average call drop rate should be less than 0.9%, which in turn would lead to not more than 5 % of subscribers having call drop rate in excess of 2% (Based on an average of 200 calls per subscriber per month). This is illustrated in the normal curve shown below: