“Distinguishing the signal from the noise requires both scientific knowledge and self-knowledge: the serenity to accept the things we cannot predict, the courage to predict the things we can, and the wisdom to know the difference.”

The Signal and the Noise got a lot of hype when it was released, partly because Nate Silver correctly predicted 49 of the 50 states in the American presidential election, again. The book itself is wide-ranging, covering not just politics but financial crises, baseball, the weather, economics, flu, chess, poker, global warming, and a host of other issues. In all of them, Silver examines the state of the industry in terms of its use of data and its ability to predict the future. His broadest lesson, though, is that the modern world is flooded with data, almost overwhelmed with it. As he correctly points out:

“The number of meaningful relationships in the data – those that speak to causality rather than correlation and testify to how the world works – is orders of magnitude smaller. Nor is it likely to be increasing at nearly so fast a rate as the information itself; there isn’t any more truth in the world than there was before the Internet or the printing press. Most of the data is just noise, as most of the universe is filled with empty space.”

At the core, his solution is humility about what we can predict and what we can understand about the world, no matter how much data we have. More formally, he argues for Bayesian thinking – when we make a prediction, he suggests, it should never be a single value, but instead a probability weighting of various outcomes. It is not possible, he argues, to make predictions with data abstracted from context: we must understand how the data works, not just observe it, as stock market quants might argue.

Statistics is not exactly a mass-market topic, and Silver has done an admirable job making his book accessible to the general public. I flipped through the chapter on baseball, but I imagine other readers might well do the same to the chapter on economic forecasting: to each their own. Generally, however, his examples are great fun, whether on how people predict (or don’t) earthquakes, how poker players calculate opponents hands, or the development of chaos theory by Lorenz as he sought to predict the weather. He is also a dab hand at phrasing, and there are almost too many good lines to pick one to quote. Even if you don’t directly work with data, your life is affected by it in a myriad of ways, and it’s worth understanding the limitations of our knowledge.