Do Coasian Solutions Work in Real Life?

This is a crosspost from an earlier post on Pragathi Express.

Anupam Manur in his post “Dealing With Construction in Your Neighborhood” sets up an interesting problem – construction activity in his neighbourhood undertaken by his neighbour is disturbing the peace and tranquility of his life. Anupam then rightly concludes that his neighbour’s actions are imposing a negative externality on him (actually the entire neighbourhood) and then invokes the Coase Theorem to solve for it. The solution would involve the offending party (his neighbour) paying Anupam a certain amount. Anupam would then promptly buy a new pair of heavy duty earbuds, soundproofs his house and perhaps gets some extra cleaning help.

Sounds simple enough. But why did that not happen in this case. I would surmise that getting his neighbour to understand the economics of externalities and Coase theorem is still not going to help matters here. Why is it that then perfectly rational actors fail to trade even when there are gains to be had by doing so. The answer to this comes from another equally important but not as famous theorem called the Myerson-Satterthwaite theorem.

In presence of private information about the value of certain good (Anupam’s peace and tranquility) to the buyer and the seller, Myerson-Satterthwaite theorem says that no mechanism exists that guarantees that a trade will always happen. The problem here is that of information-asymmetry. In a Coasian world, all information about the externality is public and bargaining will ensure that an efficient outcome for both parties is reached.

In Myerson-Satterthwaite’s world, people’s valuation of the good are private i.e. the buyer only has a vague sense of the what the seller is willing to accept and similarly the seller only has a notion of what the buyer is willing to pay (for the math inclined: buyers view of sellers cost is uniformly distributed on the interval [0,1] and vice versa). This private information induces sellers to act as if the costs of their goods are higher than they actually are and similarly for the buyer to act as if good is of lesser value than their private valuation of it. The theorem then shows that the gains of trade are not sufficient enough for either of the players to honestly reveal their costs and values. This implies that no fool proof mechanism can be designed that guarantees a trade will happen even when there is a price that would be agreeable to both parties.

Turns out that worst case scenario is the case of a single buyer and a single seller. As the number of buyers and sellers increase, these informational problems disappear and markets become ‘efficient’.

So Anupam might then just be better off buying those heavy-duty earbuds himself; Coase is not going to help him much in this case.

Note: While I highlight only the applicability of M-S theorem in case bargaining does occur, there are many other reasons why bargaining might not even occur. Social norms, inability to assign the responsibility of the externality, hold out problems and other issues often have large transaction costs dissuading the participants from even bargaining in the first place.

Jagadhari #1 and the Third law of demand

I graduated from NIT Kurukshetra (RECK back then), Haryana in 2000. That is, I started college in the same month and year the state went dry. It was a bit of shock to realize that not only was alcohol off limits but non-vegetarian food was also banned in the holy city of Kurukshetra. Talk about a double whammy.

A little background – Haryana Vikas Party – BJP alliance campaigned on the slogan of prohibition and won the assembly elections in 1996. Banning alcohol was their first order of business. In spite of warnings of loss of revenues, the emergence of liquor mafia and the practical difficulty of enforcing such a ban in a landlocked state surrounded by Punjab, Delhi, and Rajasthan (none of which had such prohibitions) the government decided to go ahead with the ban. This prohibition, like most others such experiments, ended up a disaster and was lifted in April 1998. The misadventure cost the taxpayers Rs.1200 crore in lost revenues alone. Countless people were imprisoned, and many ended up with a criminal record for something that was no longer a crime.

jagadhari.jpeg

And now to Jagadhari Number 1. The ban was brutal for those of us who came to RECK seeking a carefree life for four years filled with some if not regular sessions of inebriation. The only way to procure alcohol then was to seek out your local thele /rickshaw walla for the stuff available on the black market. And the only spirit that fit into our meager budget was always Jagadhri #1. Jagadhri #1 was(is?) a highly potent locally brewed(desi) whiskey and apparently quite popular among the locals. Jagadhri got the job done for us back then, and after the ban was lifted in 1998, I don’t think any of us ever went back to it.

What is interesting, thinking back about it now, was why Jagadhri #1 was the only spirit available in the market during that period. And the answer I think lies in the Alchian-Allen effect. The Alchian-Allen Theorem also called the third law of demand, says that when a fixed cost is added to substitutable goods, the more expensive one becomes relatively inexpensive.

Let me illustrate with an example. Say the best  variety (A) of Indian Mangoes costs £10 per box, and the standard variety (B) costs £5 for the same box in India. But say shipping to the UK is a flat £5 per box. Now the cost of the same box in the UK is £15 for the A and £10 for B. That is the relative price of A in India is 2 boxes of B whereas in the UK it is only 1.5 boxes of B. In other words variety (A) is relatively cheaper in the US compared to what it is in India and hence consumers should prefer to buy A to B.  The best stuff gets shipped out ! This is perhaps a reason why Indian groceries/produce you find in the US/UK are sometimes better than what you get back home.

How does this logic work for Jagadhri #1 then? The prohibition  imposed a set of fixed costs on the buyers and sellers. The risk of getting caught, potential for fine and incarceration raises the fixed costs of consuming alcohol (similar to the transportation costs in above example) making a potent Jagadhri more attractive compared to the alternatives. It is just not worth the trouble to smuggle a beer when you could get a bottle of Jagadhri which gives you a bigger ‘kick’ when the risk that you bear in smuggling either of them is the same.

Sad, but an important thing to keep in mind every time you read about tainted alcohol killing people in dry states like Bihar. Perhaps someone was trying to maximize his alcohol to risk equation and got the balance wrong.

Omphaloskepsis (Silly and Self Indulgent)

Everyone engages in Ego searching i.e. searching for your own name on Google and so do I. While on such a trip I came across all my search history that google had preserved for me since late 2013.What I present you here is a little bit of exploratory data analysis of my google search history.

What I realized was simply combing through the timestamps of the searches and breaking them down to days/hour/weekday level said a lot about of my internet/online habits. Adding the search text into the mix pretty much illustrated the key events of my life over the last two years.

Getting Your Google Search History

If you have allowed Google to save your search history and are always logged into google account when using using google even for search (which you should since Google is then able to personalize the results) then you can download your search history from the Google History page. Once you are on the page , navigate to the the three dots the top right corner of the page. From there you can download your search history by simply clicking the Download Searches link. Google will email you your search results as zip file containing the json files.
Screen Shot 2016-02-13 at 2.50.56 AM.pngOnce you have the zip file (and if you want to follow along) unzip the file and place the json files along with this notebook.

For some reason I could not get Pandas to parse the json files straightaway into dataframe, instead had to write a small script to convert the json files into a csv format. Here is the gist if you want to try it out.

Though I have used Python/Pandas & Plotly fro this kind of analysis before, I use Tableau this time around. Here is the simple graph that illustrates the number of searches per day over the last two years. CompleteTimeSeries   The graph shows an overall increasing trend which is not surprising but not terribly useful either. I am more interested in understanding my online behavior i.e. what time of the day am I active the most, which days and months have I searched more.

Activity By Month

MonthlyGrouping   Clearly August and September (followed by December) perhaps are my most active while May and March are the least active ones.

Activity By Day

WeekDaySearchCounts This visualization was bit of a surprise as I always thought I was much more active online on the weekends than I was during the day…turns out its not the case. One big surprise of course is Friday. I knew I slowdown by Friday but did not expect it to be by that much.

By the Hour of the day

DailyWeekendBrowsingBehavior

Looks straightforward and corresponds to my intuition though I am a little surprised that my weekend pattern is so similar to my weekday patterns. I was expecting it to be lot more divergent i.e. much more subdued online activity throughout the day with peaks only the evenings or night…turns out not to be the case.

Hourly – Over the Years

HourlyByYear1   My data for 2013 (when I started collecting my search results) and 2016 is incomplete. Once we disregard the relatively sparse data from those 2 years, clearly my search activity has increased over the years but what is surprising is how consistent my activity patterns have been over the course of the day. My activity around midnight is still close to my average during the day which is a bit worrisome when I should expect it to be much lesser. Time to get real about getting to bed a little early and getting more sleep.

What was I searching for ?

Since google search history also contains the actual search text, with a little bit of text mining (sklearn and nltk) its easy to generate a snapshot of what you were looking for at that point in life. I chose to do a simple wordcloud using the python wordclound package. Generating wordclouds using that package is fairly straightforward.

Here are a few snapshots from my life curated from my google search words.

Sept 2013

 

Nov 2013

Jan 2014

Sept 2014

Jan 2015 (frantic job search time)

Feb 2015 (move to the US)

Oct 2015

Padma Shri’s over the Years

Padma Shri (or Padmashree) is the fourth highest civilian award given by the Indian government after the Bharat Ratna, the Padma Vibhushan, and the Padma Bhushan.

The awards are given for excellence in all fields of activities – disciplines such as art, literature and education, sports, medicine, social work, science and engineering, public affairs, civil service and trade and industry.

While there is no reason to believe that all fields should be equally represented, some fields have seen their share awards represented rather disproportionately than what you would expect.Here is a quick graphic of all PadmaShri’s grouped by their categories.

Number of Padma Shri Awards

Clearly Arts has claimed more of these awards than any other category. But that wasn’t the case always, though. Arts only became the dominant category after 1984. It was rather a mixed bag until then.

PadmaShri Awards over the years by category

But after 1984, Arts has been the dominant category in every year after that.

PadmaShri Awards over the years(for selected categories)

Of course, the  story here is the secular rise in the overall number of awards over the years. And here is why those sharp peaks are intriguing

Number of Padmashri awardees over the years

Make it what you will of the pattern – almost all peaks occur in the election years.

Here is what I think is happening- purely speculative of course.

Recommendations for these awards are solicited(from the states) around August and the winners declared before/on the republic day which is in January and federal elections happen in May/June(of the election year). So given the timing, I suppose it’s the ruling party in center acquiescing to the demands of the ruling parties in the states  either for their support in the upcoming elections or simply repaying for any favors they might have done in the past before cycle end of their rule.

Either way, looking at the states from which these awardees came from and the specific categories (arts-cinema vs arts-dance etc.) for which they were awarded might provide more clues to what is happening.An exercise for some other time.

GIYF

A nice little trick to scrape some data from a list or a table on website is by using the ‘ImportHTML’ function in Google Sheets/Docs. The ‘IMPORTHTML’ function takes 3 parameters –

  • url – the url of the page you want to scrape
  • query – ‘list’ or ‘table’ html element that contains the data
  • index – the index, starting at 1, of the list or table on the page.

Here is a simple example

And the result of the import.

Useful trick to have in your back pocket to quickly scrape data off without having to resort to scrapy or beautiful soup or other tools.

The Mobile Numbers in India

The story of the mobile phones in India is a veritable success story. Its common knowledge that India is among the fastest growing mobile phone markets in the world adding more subscribers every month than any other country. India still adds around 5-7 million new subscribers each month and there is still quite a distance to go before this surge plateaus, at least in the rural market.

Teledensity - Rural India 2001-2014

The urban market, on the other hand, is showing signs of saturation where the emphasis of the providers is moving from away from SIM card sales to more value-added purchases.

Teledensity - Urban India 2001-2014

Any way, thats not really the point of this post. What has been surprising from the data is the leap  Indian consumer made from no-connectivity to a mobile phone, skipping the landline stage completely. The rise is the tele-density in India simply reflects the rise of mobile phones.

Teledensity - India 2001-2014
Teledensity - Wired India 2001-2014
Teledensity - Wireless India 2001-2014

And what is even more remarkable is how far left behind are the public providers in this race. This should put to rest to any arguments that private sector cannot provide for services in the rural India where the margins potentially are not as high and hence  the role of Government (or its agencies like BSNL) in the market place.

Teledensity - Private & Public India 2001-2014

The biggest loser in of all of this growth :

Posts and Letters Traffic 2000-2011

Admittedly, the traffic crashed between 2000-2002 before the surge in Mobile Phones began, but whatever little uptick that happened around 2003-04 could barely the withstand the onslaught of cellphone revolution. An interesting contrast. Data on the mail is here

Summer of Crime in Bihar

Is there an impact of weather/season and the amount of crime happens  –  The Jury is out on that (see here,here and here). Clearly the issue has been studied quite a bit (this from a paper from 1952) and this one more recently with no conclusive result just yet.

The most commonly seen result (so far) is that precipitation does not have a significant correlation with crime but temperature apparently does. Here is a nice summary of the research about this topic. I doubt if such a correlation would hold in an Indian Context where the amount of economic activity is significantly dependent on how the amount of rainfall a particular region gets in the season.

Anyway, while researching for this earlier post , I landed on a crime stats dataset for Bihar that had data split by different types of crime per month for the last years.Once I started I digging into the data initially for murders (since that was the topic of the previous post)a curious observation jumped out from the graphs. For some reason the number of murders seemed to be always higher in months between May and July for every year.

Murders in Bihar over the last 5 years

Intrigued as I was, I checked this trend for other violent crime categories listed by the Bihar Police. Here is what I find.

Cognizable Crimes in Bihar over the last 5 years
Rapes  in Bihar over the last 5 years
Robbery in Bihar over the last 5 years
Theft in Bihar over the last 5 years
Road_Robbery in Bihar over the last 5 years
Riots in Bihar over the last 5 years

Notice those huge bumps in the middle of the year.Only one category of crime (as listed in the data set) did not show this behavior.

Kidnapping in Bihar over the last 5 years

I am not sure if these trends mean anything in a behavioral sense i.e. people don’t get extra violent in summer or grow more docile in the winter. If any thing, these trends might just reflect some underlying economic patterns/hardships which perhaps manifested in increased crime.

Lying with statistics – # 1

Here is a recent article  about the  purported rise in murders in Bihar (which of course  was promptly and endlessly  recirculated on Social Media).

The report does not mention when the data is from but dog-whistles that new government had some thing to do with it. Thats kinda funny  because the new govt. was only sworn in the mid of November. So its definitely not 2 months yet.

So, scouring for the data  I found this from Bihar Police website. I assume this is what they were looking at because the last two months for which the data is available (Sept and Oct of 2015) do add up to 578 – the number of murders they mention in the article.

Any way, when stated without context the number of murders do look significant and the article does make it seem as if there has been a significant rise in murders (or “Jungle Raaj” as the article puts it) after the new/old government took over (even though that is clearly not true).

So, I thought I’d put it in context. (Data from here )

Picture

See the big jump in murders in September and October …neither do I.

But Bihar is a violent state, every one knows that. These numbers are probably higher than those for other states(an exercise for another time) and hence sound scary. But I don’t see how these are any thing different from what they were in previous months.

Just for comparison,  look at the what they were in 2014.

Picture1

The numbers look much better in 2015  than they were in 2014. But that doesn’t make a great headline does it ?

Digging in further, I found an interesting pattern in the number of murders (gruesome as it sounds) , and  the specific the time of the year when they seem to rise. More about it in the next post.

Curious Case of Failing Prediction Markets in India (or Not)

Here is an interesting article that summarizes the failure of election predictions in India, at both national the state level.

This report by IDFC Institute says not a single survey since 1996 has been accurate (when defined at 5% confidence interval level). And the worrying trend is rather this :

“Accuracy levels of these surveys show no signs of an improving trend since 1996 though sample sizes and number of surveys have increased substantially.”

Now if more data is not improving your accuracies then one needs to question the sampling procedures these surveys employed.

On a related note, some of the smartest enterpreneurs are already on the task to solve this prediction crisis….truly in an Indian style ofcourse.

Stars Lining up for Indian Astrology startups

All those graduates of the Astrology programs have to be employed some where.