data bitten
data science, business intelligence, and quantrepreneurship

• Home
• About
• RSS
• Subscribe:e-mail

• Recently Written

  • Learning Machine Learning: Beyond the Hype
  • After Hours Marketing podcast: “The Subtle Differences Between Reporting and Analytics”
  • Seven ways to avoid the seven month data science itch
  • The law of small numbers
  • The red and green rule
  • CRAPOLA design principles for quantitative data
  • Reporting versus analytics
  • Uber’s newest data visualization is uber misleading
  • How to hire data scientists – free white paper
  • Hiring data scientists: the data problem

• Categories

  • Uncategorized

• Archives

  • May 2019
  • July 2016
  • June 2016
  • May 2016
  • February 2016
  • March 2013
  • November 2012
  • October 2012
  • September 2012
  • June 2012

Nate Silver, Probabilistic Celebrity

Tuesday night was quite spectacular for Nate Silver. I’m referring not to my suspicion that the oft-bespectacled, apparently liberal-leaning statistician was pleased with the outcome of the presidential race – though there was that – but rather the accompanying mighty boost to his career as his highest likelihood prediction of the election results proved spot on.

Silver’s predictions were based on what might be termed a “sophisticated form of poll aggregation.” His calculations incorporated the results of many polls, which were scrubbed and weighted based on factors such as historical accuracy of each polling firm, and his probability forecasts were derived from repeated election simulations (biostatistician Bob O’Hara speculated on some of the details). Note that Silver wasn’t the only one who applied statistics to polling data with impressive results; for example, check out Princeton Election Consortium’s forecast.

In the end, while the presidential race was declared by many prominent pundits as a dead heat or favoring Romney, the non-pundit’s New York Times blog served as a Silver lining for Democrats. Designer Michael Cosentino of gdgt.com published impressive-looking side-by-side maps of Silver’s predictions and actual results once most contests had been called by the major networks:

For correctly predicting all fifty states, Nate Silver was proclaimed the winner of the election (or at least a Silver medalist), a “poster child” of data in politics, “king of the quants,” the “patron saint” of big data, and even “a near-god for nervous Democrats.” Sales of his book, The Signal and the Noise, experienced a healthy spike. Even before the election, Twitter had been exploding with love for Nate, resulting in among many other things a surge of activity in the Chuck Norris-esque hashtag #NateSilverFacts and the parody accounts @fivethirtygreat, @fivethirtynate, and @drunknatesilver. Multiple domains have been registered in his honor, including NateSilverFacts.com and IsNateSilverAWitch.com, whose homepage reads:

Blind hate (for Nate)

Not all of the press has been as positive as labeling him a magical being. But most of the vitriol was unsubstantiated and suffered from a poor grasp of statistics and reason. New York Times columnist David Brooks said that “pollsters tell us what’s happening now. When they start projecting, they’re getting into silly land” and wrote that “even experts with fancy computer models are terrible at predicting human behavior” (to which technosociologist Zeynep Tufekci had a nice retort: “But experts with fancy computer models are good at predicting many things in the aggregate”). MSNBC’s Joe Scarborough referred to Nate Silver as an “ideologue” and a “joke.” On National Review Online, Josh Jordan claimed without substantiation that Silver’s partisanship “shows in the way he forecasts the election” (note that the article Jordan cites to demonstrate that Silver was “openly rooting” for Obama in 2012 talks only of the 2008 election). Slate’s Daniel Engber completely missed the point when he wrote that “Nate Silver didn’t nail it; the pollsters did.”

One of the worst offenders was Politico’s Dylan Byers, who wrote:

Prediction is the name of Silver’s game, the basis for his celebrity. So should Mitt Romney win on Nov. 6, it’s difficult to see how people can continue to put faith in the predictions of someone who has never given that candidate anything higher than a 41 percent chance of winning (way back on June 2) and – one week from the election – gives him a one-in-four chance, even as the polls have him almost neck-and-neck with the incumbent.

Ezra Klein has a nice explanation of one of the many flaws in Byer’s logic:

If Mitt Romney wins on election day, it doesn’t mean Silver’s model was wrong. After all, the model has been fluctuating between giving Romney a 25 percent and 40 percent chance of winning the election. That’s a pretty good chance! If you told me I had a 35 percent chance of winning a million dollars tomorrow, I’d be excited. And if I won the money, I wouldn’t turn around and tell you your information was wrong. I’d still have no evidence I’d ever had anything more than a 35 percent chance.

MIT Knight Science Journalism Tracker’s media critic Paul Raeburn had a similar reaction:

Let’s compare Silver’s work to a weather forecast. As of Nov. 4, Silver gives Obama an 86.3 percent chance of winning the election. If a meteorologist said there was an 86 percent chance of rain – and it didn’t rain – Byers would presumably “not continue to put faith in the predictions” of the weather forecaster. But we know that’s not right. Forecasts are generally correct – but not always. That does not make them worthless. When there is an 86 percent change of rain, most of us grab an umbrella. And we should.

Liberal satire blog Wonkette also posted a direct reaction to Byer’s troubling piece:

we would like to urge everyone to go read this Politico piece again about how dumb and wrong Nate is and how math and numbers are ruining political punditry forever, and then laugh and laugh at how upset people were by the concept that you could tell how an election might turn out by asking people in advance how they’ll vote and then figuring a way to accurately assess the answers they give.

Silver simply critically interpreted people’s answers to the question of how they plan to vote in a scientific manner. Sounds like a prime example of turning information into knowledge to me. Data science for the win.

This shouldn’t have been upsetting to people. Tufekci eloquently sums up why:

We rely on statistical models for many decisions every single day, including, crucially: weather, medicine, and pretty much any complex system in which there’s an element of uncertainty to the outcome. In fact, these are the same methods by which scientists could tell Hurricane Sandy was about to hit the United States many days in advance. Dismissing predictive methods is not only incorrect; in the case of electoral politics, it’s politically harmful.

She also has a message for the haters:

Refusing to run statistical models simply because they produce probability distributions rather than absolute certainty is irresponsible…. We should defend statistical models because confusing uncertainty and variance with “oh, we don’t know anything, it could go any which way” does disservice to important discussions we should be having on many topics – not just on politics.

Blind love

All of this is not to say that articles that came out in favor of Silver were flawless. Quite the opposite was true; quantitative exceptionalism abounded on both sides of the debate. His predictions were perceived as exceptional (in some cases better, in others worse) because they came from science, which has a habit of making people’s brains turn off. As Paul Raeburn wrote, “Nate Silver’s rational approach to politics seems to provoke highly irrational responses.” Many voices on both sides of the debate conflated likelihood of winning and victory margin, for example. Silver predicted a high likelihood of Obama winning by a fairly slim margin, not a landslide victory.

One critical fact that is largely absent from the conversation is that it was partially luck that Silver’s predictions were spot on for all fifty states. If we assume that the state-by-state results as predicted in Silver’s November 6 presidential forecast were independent events, he had a 12.5% chance of getting all fifty correct (the product of all of his state win likelihoods). Granted, this represents much better odds than the approximately one in a quadrillion (1 in 1,000,000,000,000,000) chance of guessing all fifty correct by flipping a coin (50% raised to the 50^th power), but Silver’s perfect sweep can’t be attributed to his skill alone.

In effect, being celebrated for this accomplishment makes Silver a probabilistic celebrity. He seems like a pretty rational guy, so I’m guessing he’s more than mildly amused by how much credit he’s gotten for his probabilistic predictions ending up correct. Bowing down to him for this reason is at its core a quantitative exceptionalist act.

Silver’s biggest victory might have been for making a case by example for moving more toward science in politics, away from gut-instinct-based punditry. Pundits are only human, and humans are known for being biased and otherwise poorly calibrated estimators (e.g. see Douglas Hubbard’s How to Measure Anything); Silver himself characterizes subjective pundit predictions as “literally no better than flipping a coin.” Slate decried the poor accuracy of pundits’ predictions in this election, and several media outlets have questioned whether punditry is dead as a result of Silver’s superior methods. But keep in mind that prediction is hard; for some people, even predicting the past can be tricky. At the very least, some good humor has come out of the discussion, such as the election drinking game proposed by Brian Fadden of The New York Times: “Drink until you’re as dumb as a pundit.”

Statisticians would most likely have beaten pundits at predicting a domestic feline’s surprising third place finish in Virginia’s senate race had they been looking for it, but this is not to say that qualitative analysis lacks value. To the contrary, designing and interpreting polls requires it. It is the marriage of qualitative and quantitative analysis that makes statisticians like Silver so successful. Critical inquiry is the key ingredient. Blindly following anyone – either pundits or statisticians – can get you into trouble. In fact, many pundits get themselves into trouble for their own lack of skepticism. A Forbes.com article titled “Nate Silver’s Prediction Was Awesome – But Don’t Build Statues To ‘Algorithmic Overlords’ Just Yet” includes the fair and balanced passage:

Discrediting expert predictions [pundits] seems like real progress – but not if we believe the enduring lesson is to replace one group of fortune tellers with another.  We should certainly strive to use big data and rigorous math whenever we can – but let’s be careful not to fall into the trap of letting down our guard, and trusting all experts who come bearing algorithms.

This isn’t about picking the right side. It’s about being rational.

comments

• About

  Data Bitten aims to tell the story of the data revolution. More to come.

• Stay connected