Civil Services Examination - GoT / Game of Chance Update

A year ago, I had likened the Civil Services Examination, one of the most selective examinations in the world (success rate of a measly 0.23%), as a "Game of Chance". This was in my blog post, Civil Services Examination - Game of Thrones or Game of Chance?

I had come to this conclusion based on the following insights gleaned from an analysis of the CSE Exam results for the three years, 2013, 2014 and 2015, 

1) The interview score of the candidates has an effective weight of 40 % in determining the final rank of the candidate (and not 14 % as may appear prima facie by a simple computation of 275/2025). This is on account of the higher variability in the Interview (Personality Test) scores vis-a-vis the written scores. and 

2) The UPSC Interview had low "Reliability",  as there was little or no relation between the score obtained by the same candidate in two different years. The R-Squared value was in the region of  0.10, ie the interview score of a given year had less than 10% predictive power in explaining the interview score in the next year.

3) There is little or no relation between the Written and Interview marks scored by any candidate. Whatever ability (IAS-ness?) was being assessed by UPSC through these two proxies, it surely did not show up in the correlation between the proxies. 

The scatter plot below (taken from my last year's analysis) shows this weak (lack of?) relation between the interview scores of the same candidate in two successive years.

I had thereafter concluded that the UPSC interview is statistically seen to be of low Reliability, and by implication, of low Validity, as there can be no Validity without Reliability.

The only viable strategy available to the Civil Services aspirant was to acknowledge that this is a game of chance, and therefore to maximise the probability of success by taking as many attempts as possible, till such time that he gets the service of his/her choice (or exhausts his/her attempts).

This was a painful recommendation to make - it is bad enough that the youth of India spend (waste?) many years of their life trying to clear this exam, and here I was exhorting them to reappear, and spend another year or more till they get the service of their choice. 

I get to meet a fresh batch of young Officer Trainees of various Accounts and Finance Service (ICAS, IDAS, IP&TAFS, IRAS, IA&AS), every year at NIFM (where I am on deputation as Professor), where they undergo a part of their probationary training. Most Officer Trainees with attempts left reappear for the Civil Services Exam, but true to the game of chance analogy, only a few end up significantly improving their rank, they are as likely of failing to clear any of the three hurdles - Prelims, Mains and Interview.

Without any shadow or doubt, we can say that the Civil Services Examination as a Civil Services Capability/Aptitude assessment tool has very low Reliability (hence very low validity).

I conducted a similar analysis for the Civil Services Exam 2016 results, just to make sure that the earlier years were not an aberration. Here is what I found - once again:

1) A very weak relation between marks obtained in Personality Test (Interview) for the same candidate across two successive years ie CSE 2015 and CSE 2016. (A total of 188 candidates were identified who managed to clear the exam on both occasions, and it is their marks that are shown on the scatter plot)

2) No relation between the marks obtained by any candidate in Written Test and Personality Test. (The scatter plot is shown for the top 25 % of the candidates by marks secured in the Written Test)

So, nothing has changed in a year.

Let me make this clear - each and every person who clears the Civil Services Exam, any stage, is highly meritorious. Clearing the prelims itself puts you in the top 3 % of the applicants - that sure is something. My point is, from this stage on, it is luck (for want of a better word) that may be the key factor. So don't lose heart if you don't get the service of your choice, nor take pride in your ability for having "aced" this exam with a top-50 or top-100 rank.

I hope that UPSC will one day review the way it conducts its interviews, and find a way to increase its Reliability. Maybe they can learn from Google, which changed its interview process a few years ago, transitioning to "Structured Interview", or the US Office of Personnel Management, which also encourages government agencies to use structured interviews in hiring.