The Skeptics Guide to Institutions – Part 1

NOTE: The Growth Economics Blog has moved sites. Click here to find this post at the new site.

I’m starting a run of several lectures on institutions in my growth and development course. By revealed preference, so to speak, I take the institutions literature seriously. But there are some issues with it, and so I’m going to teach this literature from a particularly skeptical viewpoint and see what survives. These posts are going to sound very antagonistic as I do this, which isn’t completely fair, but makes it more fun to write.

This first post has to do with the cross-country literature on institutions. The 1st-generation of this research (Mauro, 1995; Knack and Keefer, 1995; Hall and Jones, 1999; Easterly and Levine, 2003; Rodrik et al, 2004; Acemoglu and Johnson, 2005) regressed either growth rates or the level of income per capita on an index of institutional quality along with other controls. In general, this literature found that institutions “matter”. That is, the indices were statistically significant in the regressions, and the size of the coefficients indicated big effects of institutions on growth or income per capita.

These results are the prima facie evidence that institutions are a fundamental driver of differences in development levels. The significance combined with the large absolute values of the estimate effects indicated that even small changes in institutions had a big impact on GDP per capita. We’ll get to talking about questions of whether in fact these are well-identified regressions in a future post. For now, let’s just take these regressions as they are.

The first big issue with this literature is that all the indices of institutions used are inherently arbitrary, and yet are used as if they have a strict numerical interpretation. (see Hoyland, et al, 2012; Donchev and Ujhelyi, 2014) This is easiest to talk about by using an example.

Let’s take the 7 point index for “constraint on the executive” used by Acemoglu and Johnson in their 2005 paper. 1 is “not so many constraints” and 7 is “lots and lots of constraints”. There are more official definitions of these categories. They comes from the Polity IV database, and I will concede that it is coded up by smart, reasonable people. I have no argument with how each individual country is coded. Minor quibbles about how we rank constraints on executives are not going to overturn the results of the regressions using this to measure institutions.

But does Australia (7) have seven times as many constraints at Cuba (1)? Does the one-point gap between Luxembourg (7) and South Korea (6) have a similar meaning to the one-point gap between Liberia (2) and Cuba (1)? Using this as a continuous variable presumes that the index values have some actual meaning, when all they are is a means of categorizing countries.

So what happens if you use the constraint on executive scores simply as categorical (i.e. dummy) variables rather than as a continuous measure? You’ll find that all of the action comes from the category for the 7’s (Western developed countries) relative to the 1’s (Cuba, North Korea, Sudan, and others). That is, the dummy variable on the 7’s indicates that their income per capita is statistically significantly higher than income per capita for the 1’s. Country’s with 2’s, 3’s, 4’s, and 5’s are not significantly richer than 1’s (2’s, 3’s, and 4’s are actually estimated to be *poorer* than 1’s). Country’s with 6’s have marginally significant higher income than 1’s. The finding is that having Western-style social-democracy constraints on executives is what is good for income per capita, but gradations in constraints below that are essentially meaningless.

But there is a more fundamental empirical problem once we use constraints on executive to categorize countries. Regressions are dumb, and don’t care that we have a particular interpretation for our categories. They just load *any* differences in income per capita onto those categorical variables. The dummy variable for category 7 countries captures the average income per capita difference between those countries and the category 1 countries. There might be – and certainly are – a number of things that distinguish North Korea from the U.S. beyond constraints on the executive, and the dummy is picking all those up as well. Even if I control for additional factors (geographic variables, education levels, etc.. ) we cannot possibly control for everything, in part because the sample is so small that I can’t include a lot of variables without losing all degrees of freedom. Empirically, the best I can conclude is that Western-style social democracies are different from poor countries. Well, duh. One aspect of that may be constraints on executives, but we cannot know that for sure.

Other indices of institutions are just as bad. The World Bank Governance indicators, commonly used, include sub-indices of “Governance”, “Accountability”, and “Voice”. Okay, and….what do I do with that? You want to tell me Governance is good in Switzerland and bad in Uganda, I guess I’d have to agree with you, not having any specific experience to draw on. But if I ask you what exactly you mean by that, what kind of answer would I get? These governance indicators are based on surveys of perceptions of the quality of institutions. The institutions that get coded as “good” are the institutions people find in rich countries, because those must be good institutions, right? These measures are inherently endogenous.

This problem holds to some extent even for modern measures of institutional quality like the Doing Business indicators. These have the virtue of measuring something tangible – the number of days necessary to start a business, for example – but it isn’t clear that this should enter linearly to a specification. Does going from 146 to 145 days to start a business have the same effect as going from 10 to 9? Why should it? Is there a threshold we should worry about, like getting the number of days under 30? And just because we can measure the number of days to register a business, does that mean it is important, or that it constitutes an “institution”?

Reading the cross-country empirical institutions literature is the equivalent of watching studio analysis of NFL games. You have a bunch of people “in the game” of economics sitting around making un-refutable statements that sound plausible, but have essentially zero content. “He’s got a real nose for the ball”. Okay, meaning what? How does one improve ones nose for the ball? Is there a machine in the weight room for that? Is this players nose better than that players nose? How could you compare? “Good institutions” is the equivalent of “having a nose for the ball”. It’s plausibly true, but impossible to quantify, measure, or define.

Another big problem with the empirical cross-country institutions work is courtesy of Glaeser et al (2004). Their point is that our institutional measures are generally measuring outcomes, not actual institutional differences. One example is Singapore, which scores (and scored) very high on institutional measures like risk of expropriation and constraints on executives. Except under Lee Kwan Yew, there were no constraints. He was essentially a total dictator, but happened to choose policies that were favorable to business, and did not arbitrarily confiscate property. But he *could* have, so there is no actual institutional limit there. The empirical measures of institutions we have are not measuring deep institutional, but transitory policy choices.

That leaves us with the whole issue of incredibly small sample sizes, often times in the 50-70 country range, eliminating the possibility of controlling for a number of other covariates without losing all degrees of freedom. And don’t forget publication bias, which means the only things we see in the literature are the statistically significant results that got thrown up in the course of running thousands of regressions with different specifications and measures of institutions.

In short, it may be that institutions do matter fundamentally for development. But the cross-country empirical literature is not evidence of that. There is a fundamental “measurement-before-theory” issue in this field, I think. We don’t know what we should be measuring, because we don’t have any good definition of an “institution”, much less a good theory of how they work, arise, collapse, or mutate. So we end up flinging things that sound “institution-ish” into regressions, without knowing what we are actually measuring.

Next up will be 2nd-generation cross-country empirical work that uses instrumental variables. Spoiler alert: those don’t work either.

29 thoughts on “The Skeptics Guide to Institutions – Part 1

  1. Great post. This reminds me of how some used to treat capital, and more recently human capital as largely homogeneous blobs: the more the better, with little care for the fact that trains are not cars and are not factories; in the same way that basic numeracy is not computer programming and is not a degree in English literature.

    It seems as though “good institutions” are the new homogenous blob for good social scientists to unravel.

    P.S. You have a typo – you put the plural of country as country’s rather than countries.

    • Important things first. Yes, that is a typo. Good catch.

      And that is essentially the issue with “institutions”. Defined broadly enough, of course they matter. But that doesn’t leave us with any useful information.

  2. It is always interesting to see analyses of the Cuban economy that don’t consider the attempts of its very large neighbor to quash it.

  3. Pingback: The Skeptics Guide to Institutions – Part 1 « La Pata de Cabra

  4. Pingback: The Skeptics Guide to Institutions – Part 2 | The Growth Economics Blog

  5. My primary objections to the elevation of “institutions” to the status of primary explanation have always been the same three: First, institutions are so obviously endogenous. So it’s not clear how one develops a clear causal pathway for anything. Second, “institutions” are so obviously heterogeneous. So how does one unpack that heterogeneity in any meaningful way? How can we even begin to measure the relative importance of different institutions? Third, the efforts to identify “good” institutions have all seemed to me to be so obviously political (see, e.g., the Heritage Foundation, for which education and health case institutions do not matter) that I almost immediately distrust the analyses.

    • I think that last point is particularly important. What are good institutions? Well, whatever rich Western countries do must be good, right? Nick Crafts made this point about Britain and the IR. You cannot say that *any* attribute of Britain led to the IR – any single attribute may have been bad for industrialization, but was overwhelmed by other good factors. We have more candidate explanations that observations.

      • DV: Speaking of the Nick Crafts and the IR, Robert Allen ran some simultaneous equations on the British and other countries “early modern” (that is, pre-IR) period, using institutions as a dummy variable – and they didn’t even register in explaining how early modern Britain had higher wages and per capita GDP than France or other areas. It seems that when you actually know about country specific events (like the “new draperies” in Britain), these tend to overpower vague cross-country judgments about institutions.

        BTW, many of the points you make in this post are similar to ones that have been made for years by Robert Solow against various cross-sectional studies. Good to see that you are taking him seriously! 🙂

      • I just finished reading Allen and was extremely disappointed with his handling of this matter. As best I can tell he attempted to reduce institutional complexity into a binary score of absolute monarchy vs all others.

        The oddest thing in reading Allen’s text is nonstop mention of innovation based upon limited time patents and venture capital and freedom of association of farmers or businessmen to get together and do whatever they choose without bureaucratic or religious or traditional or incumbent forces interfering. Yet, his models ignore all that, taking patents, absence of monopolies, freedom of association and freedom from state control for granted.

        Don’t get me wrong, I thought the book was fantastic, and there could have been more behind the scenes which he did not convey. But the take away I had on institutions was that he did a gross oversimplification that totally ignores the meat behind North and Weingast and found that after simplifying it no longer mattered,

  6. You write, “We don’t know what we should be measuring, because we don’t have any good definition of an “institution”, much less a good theory of how they work…”

    I think the problem is a little different. From Coase and the institutionalists, we can make a rather exact definition, and it turns out that Adam Smith was talking about it in Wealth of Nations, Ch.3

    But they have two kinds of relationships which compose them (i.e. rules, and the transactions that are covered by those rules) and these two kinds of relationships are different logical types, and the measures of these two kinds of relationships are always incommensurate to each other. Thus, institutions cannot be numerically evaluated. Among other things, their success depends upon the agreement on the rules, for example, and the comprehension of the actors involved, etc.

    • Lee – there is certainly a possibility that we cannot literally measure institutions. I’m sympathetic to that view, as they are a combination of formal rules, informal rules, norms, culture, etc.. How could you possibly quantify that?

    • Lee,

      Thanks for the links! What a treat.

      I have been thinking for a couple of months of how to use models which represent the dynamic network effects of markets and societies. I now see that you have done so. These are great.

      Related to the topic of institutions, I agree with your link that institutions are critical. Of course, dvollrath makes a good point that defining good ones is easier said than done.

  7. Pingback: Friday links: on live tweeting talks, measurement vs. theory, #myworstgrade, and more | Dynamic Ecology

  8. It is interesting to note that Rodrik in his recent work downplays the importance of institutions considerably. He does not argue that they don’t matter, who in their right mind could, but that they are not the primary causal determinants in explaining growth!

    • Dwayne – that’s about where I am on this. Until better evidence exists, I think we have to be careful with the institutions literature.

  9. Not sure what the take-away is from this series. If you are suggesting that institutional indices need to improve over time, or prove and refine themselves via experimental prediction, then I agree. Let the games begin.

    If you are suggesting that they are useless now and in the future then I disagree. Seems like a much more promising approach than mere rhetoric and argumentation.

    • Good point. I just posted the 3rd installment, and I think the last few paragraphs give a better summary of where I’m at on this. One, the measures are poor, so yes, better measurement is required. And I’m all for that. Let’s have at it.

      Two, by skeptical I mean that I don’t think the evidence is strong enough to support conclusions that institutions matter. Certainly not enough to make strong policy recommendations. It’s a “let’s be careful here” argument. But as I say in the new post, you show me good evidence, and I’m in. I’m not dismissing institutions as a potential reason.

      • I like your third installment and will add a comment there. Overall, I think you are being too harsh. As an example, let’s say we define three institutional archetypes — feudalism, capitalism and socialism. We can then score each state on where it lies. I am confident that we could find empirical institutional differences in economic growth or level based upon this system. Indeed, the answer is so obvious, that I can’t imagine anyone wasting their time publishing the results.

        To question whether institutions matter seems borderline nihilistic to me. Granted, there may be multiple paths to success (and multiple ways to define success), and the particular course may be influenced by contextual and historically contingent issues, but economic success is greatly a cooperative endeavor and thus any society succeeding to grow above Malthusian limits has to be at least partially solving the central dilemma of sociology and game theory — namely solving the “problem of cooperation” (prisoners dilemma and solving the issue of free riders, cheaters and predators). From here we can investigate how they solve the problem of cooperation. Certain patterns emerge from this. These patterns and regularities can be indexed with caution and care and used empirically to make predictions which can feed back into improved models.

      • Cardiff – I did write this very harshly, for the purpose of really questioning what I’ve learned from this literature. Being skeptical that we have good evidence institutions matter is not the same thing as saying they *don’t* matter. I think they probably do, but I’m not willing to say that we have any good evidence of this. It just sounds right. It fits intuition, basic historical understanding, etc.. But it isn’t empirically strong.

        I’m with you that the solving the coordination problem is the essence of things – but I don’t know that we have a good way of measuring this. It’s really a “we know it when we see it” kind of thing.

  10. Pingback: A week of links | EVOLVING ECONOMICS

  11. Actually, given the noise and the other problems with the indices, you make the pertinent point – we are dealign with “mere rhetoric and argumentation.” (he, he)

  12. Pingback: The Skeptics Guide to Institutions « Economics Info

  13. Pingback: The Skeptics Guide to Institutions – Part 3 | The Growth Economics Blog

  14. Pingback: All Institutions, All the Time? | The Growth Economics Blog

  15. Pingback: Is David Booth right to come out against good governance? | Devpolicy Blog from the Development Policy Centre

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s