Edward Conard

Top Ten New York Times Bestselling Author

  • “…a must-read for serious students of economic policy.” - Glenn Hubbard, Dean, Columbia Business School, and former Chairman of the Council of Economic Advisers
  • “Unintended Consequences offers deep and well-argued analyses on almost every issue.” - The New York Times
  • “…a fresh argument for the productive value of inequality.” - David Autor, Professor of Economics, Massachusetts Institute of Technology
  • “…a must-read for serious students of economic policy.” - Glenn Hubbard, Dean, Columbia Business School, and former Chairman of the Council of Economic Advisers
  • “Unintended Consequences represents the most cogent and persuasive analysis of the Financial Crisis to date.” - Andrei Shleifer, 1999 John Bates Clark Medal Winner
  • “There are an amazing number of good ideas and interesting points made in Unintended Consequences. The thinking underlying it, and the obvious depth of understanding of the author, are very impressive.” - Steven Levitt, coauthor of Freakonomics; 2004 John Bates Clark Medal
  • “…a comprehensive explanation of the modern economy.” - Julian Robertson, Founder, Tiger Management
  • “A full-throated defense of economic dynamism.” - The Wall Street Journal
  • “…serious thinking for serious thinkers. …a thought-provoking blueprint for growing middle- and working-class incomes.” - Mitt Romney, former Governor of Massachusetts
  • “…reminds us that inequality sends a signal of what society lacks most, in America’s case, entrepreneurship and risk taking.” - Lawrence Lindsey, CEO, The Lindsey Group, former Director of the National Economic Council
  • “Unintended Consequences should be read by anyone who takes for granted the superiority of progressive taxation and has not thought carefully about the trade-offs involved.” - The New Republic
  • “…challenges misconceptions that distort our economic debates.” - Arthur Brooks, President of the American Enterprise Institute
Upside of Inequality Oxford Unintended Consequences

Macro Roundup


Thursday, August 19, 2021

AI Summary. The inspection paradox causes samples to overrepresent larger groups and longer intervals: surveying students yields an average class size of 90 versus an actual 35, and passengers' average transit wait appears as 9.2 minutes versus an actual 7.8 minutes. Similar biases appear in social networks and prison sentence lengths.

Allen Downey Science
Date Posted:
Is Database:
Database

The inspection paradox distorts sample data by overrepresenting larger groups, leading to biased averages: e.g., average class sizes appear 90 vs. an actual 35, and prison sentences average 13 years vs. an actual 3.6 years, skewing perceptions.

Why do surveys overstate class sizes, wait times, and prison sentences?

Core argument: Surveying college students about class sizes yields an average of 90 students versus an actual average of 35, because the inspection paradox overrepresents larger classes in the sample.

The inspection paradox skews sample data towards larger groups, impacting mean returns across various domains. For instance, when surveying college students about class sizes, larger classes are overrepresented, leading to an average class size of 90 compared to the actual average of 35. Similarly, in public transportation, passengers experience longer wait times due to the overrepresentation of longer intervals, with the biased average wait time being 9.2 minutes versus the actual 7.8 minutes. In social networks, the friendship paradox shows that a random user's friends have more friends on average, with a probability of 76% that a friend is more popular. This paradox also affects perceptions in prisons, where longer sentences are overrepresented, skewing the perceived average sentence length from 3.6 years to nearly 13 years. Understanding this paradox is crucial for accurate data interpretation and avoiding statistical errors.

Takeaways by Macro Roundup® AI

  1. Surveying college students about class sizes yields an average of 90 students versus an actual average of 35, because the inspection paradox overrepresents larger classes in the sample.
  2. Public transportation riders experience a biased average wait time of 9.2 minutes against an actual 7.8 minutes, since longer intervals between vehicles are overrepresented among passengers.
  3. The friendship paradox gives a 76% probability that a randomly chosen user's friend is more popular, while overrepresentation of longer prison sentences skews perceived average sentence length from 3.6 years to nearly 13 years.

Allen Downey, "The Inspection Paradox is Everywhere," Towards Data Science, August 5, 2019, https://towardsdatascience.com/the-inspection-paradox-is-everywhere-2ef1c2e9d709

The Inspection Paradox is Everywhere

The inspection paradox is a statistical illusion you’ve probably never heard of. It’s a common source of confusion, an occasional cause of error, and an opportunity for clever experimental design.

And once you know about it, you see it everywhere.

How many students?

One of my favorite examples is the apparent paradox of class sizes. Suppose you ask college students how big their classes are and average the responses. The result might be 90. But if you ask the college for the average class size, they might say 35. It sounds like someone is lying, but they could both be right.

When you survey students, you oversample large classes: If there are 10 students in a class, you have 10 chances to sample that class; if there are 100 students, you have 100 chances. In general, if the class size is x, it will be overrepresented in the sample by a factor of x.

That’s not necessarily a mistake. If you want to quantify student experience, the average across students might be a more meaningful statistic than the average across classes. But you have to be clear about what you are measuring and how you report it.

By the way, I didn’t make up the numbers in this example. They come from data reported by Purdue University for undergraduate class sizes in the 2013-14 academic year.

From their report I estimate the actual distribution of class sizes (with guesses to fill missing data). Then I compute the “biased” distribution you would get by sampling students. Figure 1 shows the results.

The Inspection Paradox is Everywhere: Extended Excerpt Image 1


The student sample is less likely to contain classes smaller than 40 and more likely to contain larger classes.

Going the other way, if you are given the biased distribution, you can invert the process to estimate the actual distribution. You could use this strategy if the actual distribution is not available, or if it is easier to run the biased sampling process.

The same effect applies to passenger planes. Airlines complain that they are losing money because so many flights are nearly empty. At the same time, passengers complain that flying is miserable because planes are too full. They could both be right. When a flight is nearly empty, only a few passengers enjoy the extra space. But when a flight is full, many passengers feel the crunch.

Where’s my train?

The inspection paradox also happens when you are waiting for public transportation. Buses and trains are supposed to arrive at constant intervals, but in practice some intervals are longer than others.

With your luck, you might think you are more likely to arrive during a long interval. And you’re right: a random arrival is more likely to fall in a long interval because, well, it’s longer.

To quantify this effect, I collected data from the Red Line in Boston. Using the MBTA’s real-time data service, I recorded the arrival times for 70 trains between 4pm and 5pm over several days.

The shortest gap between trains was less than 3 minutes; the longest was more than 15. Figure 2 shows the actual distribution of time between trains, and the biased distribution that would be observed by passengers.

The Inspection Paradox is Everywhere: Extended Excerpt Image 2


The average time between trains is 7.8 minutes, but the average of the biased distribution is 9.2 minutes, almost 20% longer.

In this case the difference between the two distributions is moderate because the variance of the actual distribution is moderate. When variance is higher, as in the next example, the effect of the inspection paradox can be much bigger.

Are you popular?

In 1991, Scott Feld presented the “friendship paradox”: the observation that most people have fewer friends than their friends have. He studied real-life social networks, but the same effect appears in online networks: if you choose a random Facebook user and then choose one of their friends at random, the chance is about 80% that the friend has more friends.

The friendship paradox is a form of the inspection paradox. When you choose a random user, every user is equally likely. But when you choose one of their friends, you are more likely to choose someone with a lot of friends. Specifically, someone with x friends is overrepresented by a factor of x.

The Inspection Paradox is Everywhere: Extended Excerpt Image 3


To demonstrate the effect, I use data from the Stanford Large Network Dataset Collection, which includes a sample of about 4000 Facebook users. I compute the number of friends each user has, and the number of friends their friends have. Figure 3 shows both distributions.

The difference is substantial: In this dataset, the average user has 44 friends; the average friend has 104, more than twice as many. And the probability that your friend is more popular than you is 76%.

Road rage

Some examples of the inspection paradox are more subtle. One of them occurred to me when I ran a 209-mile relay race in New Hampshire. I ran the sixth leg for my team, so when I started running, I jumped into the middle of the race. After a few miles I noticed something unusual: when I overtook slower runners, they were usually much slower; and when faster runners passed me, they were usually much faster.

At first I thought the distribution of runners was bimodal, with many slow runners, many fast runners, and few runners like me in the middle. Then I realized I was being fooled by the inspection paradox.

In long relay races, runners at different speeds end up spread over the course; if you stand at a random spot and watch runners go by, you see a representative sample of speeds. But if you jump into the race in the middle, the sample you see depends on your speed.

Whatever speed you run, you are more likely to pass slow runners, more likely to be overtaken by fast runners, and unlikely to see anyone running at the same speed as you. Specifically, the chance of seeing another runner is proportional to the difference between your speed and theirs.

We can simulate this effect using data from a conventional road race. Figure 4 shows the actual distribution of speeds from the James Joyce Ramble, a 10K race in Massachusetts. It also shows the biased distribution that would be seen by a runner at 7 mph.

The Inspection Paradox is Everywhere: Extended Excerpt Image 4


In the actual distribution, there are many runners near 7 mph. But if you run at that speed, you are unlikely to see them. The observed distribution has two modes, with fast and slow runners oversampled and fewer runners in the middle.

Even if you are not a runner, you might have noticed the same effect on the highway. You are more likely to see drivers who go too fast or too slow, and less likely to see safe, reasonable drivers like yourself.

Orange you glad you asked?

A final example of the inspection paradox occurred to me when I read Orange is the New Black, a memoir by Piper Kerman, who spent 13 months in a federal prison. Kerman expresses surprise at the length of the sentences her fellow prisoners are serving. She is right to be surprised, but it turns out that she is the victim of not just an inhumane prison system, but also the inspection paradox.

If you arrive at a prison at a random time and choose a random prisoner, you are more likely to choose a prisoner with a long sentence. Once again, a prisoner with sentence x is oversampled by a factor of x.

But what happens if you observe a prison over an interval like 13 months? It works out that if your sentence is y, the chance of overlapping with a prisoner whose sentence is x is proportional to x + y.

Using data from the U.S. Federal Bureau of Prisons, I estimate the actual distribution of sentences for federal prisoners, as would be seen by a judge, the biased distribution as seen by a one-time visitor, and the partially-biased distribution seen by a prisoner with a 13-month sentence. Figure 5 shows the three distributions.

The Inspection Paradox is Everywhere: Extended Excerpt Image 5


In the unbiased distribution, almost 50% of prisoners serve less than one year. But short-timers are less likely to be observed than lifers. To a one-time visitor, fewer than 5% of prisoners have sentences less than a year.

The distribution seen by a short-time prisoner is only modestly less biased than the view of a one-time visitor. The mean of the actual distribution is 3.6 years; the mean of the biased distribution is almost 13 years. To a 13-month observer, the mean is about 10 years.

The dataset I used for this example is a snapshot of the prison population in July 2019. So the reported distribution is biased; I had to “unbias” it to estimate the actual distribution.

Also, federal prisoners typically serve 85% of their nominal sentence. I took that into account in the calculations.

Had enough?

In summary, the inspection paradox appears in many domains, sometimes in subtle ways. If you are not aware of it, it can cause statistical errors and lead to invalid inferences. But in many cases it can be avoided, or even used deliberately as part of an experimental design.

You can read more about these examples in my books, which are available as free PDFs from Green Tea Press and also published by O’Reilly Media.

I discuss the class size example in Think Stats, 2nd Edition (affiliate link),

Steve Comment: Re: Inspection Paradox, I was unfamiliar with phrase also. My understanding of it basically when you take a sample the sample will skew towards the larger groups, in turn that will impact the mean that you return.

To use the example from the article, “…How many students?One of my favorite examples is the apparent paradox of class sizes. Suppose you ask college students how big their classes are and average the responses. The result might be 90. But if you ask the college for the average class size, they might say 35. It sounds like someone is lying, but they could both be right. When you survey students, you oversample large classes: If there are 10 students in a class, you have 10 chances to sample that class; if there are 100 students, you have 100 chances. In general, if the class size is x, it will be overrepresented in the sample by a factor of x. That’s not necessarily a mistake. If you want to quantify student experience, the average across students might be a more meaningful statistic than the average across classes. But you have to be clear about what you are measuring and how you report it. By the way, I didn’t make up the numbers in this example. They come from data reported by Purdue University for undergraduate class sizes in the 2013-14 academic year. From their report I estimate the actual distribution of class sizes (with guesses to fill missing data). Then I compute the “biased” distribution you would get by sampling students. Figure 1 shows the results. The student sample is less likely to contain classes smaller than 40 and more likely to contain larger classes. Going the other way, if you are given the biased distribution, you can invert the process to estimate the actual distribution. You could use this strategy if the actual distribution is not available, or if it is easier to run the biased sampling process….”

And the friendship paradox in Think Complexity, 2nd Edition (affiliate link).

The Red Line example in Think Bayes (affiliate link),

Ed Comment:So if a college had two classes one with 100 students and one with one (or zero), would it report an average class size of 50? That would be highly misleading. Wouldn't the not-misleading answer be: ((100 x 100) + (1 x 1))/101? I would say moral people have an obligation to the truth and if they didn't use the latter calculation, they would have an obligation to both reveal it an explain why not. We have to be very vigilent about this. It's very deceptive. How does it effect monopsony studies? Wages fall if there is only one employer but perhaps hardly any employees finds themselves in that situation. Can one data point pivot the line if all the other data points cluster around a pivot point?

Ben Comment:Yes, one data point can pivot the line of best fit depending on the weight of that point. It’s also why most monopsony studies (including mine) employment-weight their regressions instead of equal weight the regressions. What you’re describing, and what this phenomenon is naming, is the difference between a weighted average and a simple average. In your school example, the average class size is 50 but the average student is in a class of ~99. The weighting makes it about the students.

Friday, August 20, 2021

AI Summary. The Living Wage Calculator estimates that one adult with two children in New York City needs an hourly wage of $49.18, assuming full-time work of 2,080 hours a year, to cover basic living expenses without public assistance. This exceeds the state minimum wage, which does not vary with family size or number of dependents.

Amy Glasmeier MIT
Date Posted:
Is Database:
Database

According to @AmyGlasmeier @MIT, a household of one adult and two children in NYC requires an hourly wage of $49.18 to meet basic living expenses.

What hourly wage does a single parent of two need?

Core argument: The Living Wage Calculator estimates that a household of one adult and two children in New York City needs $49.18 an hour to cover basic living expenses, assuming the sole provider works 2,080 hours annually.

The Living Wage Calculator indicates that a household of one adult and two children in NYC requires an hourly wage of $49.18 to meet basic living expenses. This figure assumes the sole provider works full-time, equating to 2080 hours annually. The living wage is calculated to ensure that individuals can support themselves and their families without relying on public assistance. This contrasts sharply with the state minimum wage, which remains constant regardless of family size or dependents. The data, updated annually, highlights the significant gap between minimum wage levels and the income necessary to maintain a basic standard of living in high-cost areas like NYC. This disparity underscores the challenges faced by low-income families and the potential need for policy interventions to address wage inadequacies and support economic stability.

Takeaways by Macro Roundup® AI

  1. The Living Wage Calculator estimates that a household of one adult and two children in New York City needs $49.18 an hour to cover basic living expenses, assuming the sole provider works 2,080 hours annually.
  2. The Living Wage Calculator defines a living wage as income sufficient to support oneself and one's family without relying on public assistance, in contrast to a state minimum wage that does not vary with family size or dependents.
  3. The Living Wage Calculator's annually updated data indicates a gap between minimum wage levels and the income required for a basic standard of living in high-cost areas such as New York City.

Example for Living Wage Calculation for New York-Newark-Jersey City, NY

The living wage shown is the hourly rate that an individual in a household must earn to support his or herself and their family. The assumption is the sole provider is working full-time (2080 hours per year). The tool provides information for individuals, and households with one or two working adults and zero to three children. In the case of households with two working adults, all values are per working adult, single or in a family unless otherwise noted. The state minimum wage is the same for all individuals, regardless of how many dependents they may have. Data are updated annually, in the first quarter of the new year. State minimum wages are determined based on the posted value of the minimum wage as of January one of the coming year (National Conference of State Legislatures, 2019). The poverty rate reflects a person's gross annual income. We have converted it to an hourly wage for the sake of comparison.

Amy Glasmeier, "Living Wage Calculator," MIT, Accessed August 20, 2021, https://livingwage.mit.edu/