Painting By Numbers is a book I’ve wanted to write for a long time. BUT, as I begin the roll-out of its promotional campaign, I first want to acknowledge the books which should serve, with Painting By Numbers, as a syllabus of sorts. These are books which inspired me to write my own, expand on the topics I raise, and address the issues of numerical uncertainty in specific industries and sectors. All of them should be on your radar if you are passionate about this topic. My hope is that, after readers get acquainted with the concepts at the elementary, anecdotal level I present them, they will move on to the deeper and broader treatments available from these experts. Links to their Amazon pages are provided for convenience.
The Signal and the Noise, Nate Silver, Penguin Group, New York, New York, 2012.
This book should be considered a modern bible on the limitations of forecasting and prediction, but also on how prediction can be improved. I’ve recommended it to many friends and several have taken me up on it. If I ever teach a class on this subject, I will warm up the students with Painting by Numbers and then use The Signal and the Noise as the main text. The breadth of Silver’s topics and discussion points are, well, breathtaking. He tackles numerical analysis in baseball, election polling, climate change, gambling, weather forecasting (different from climate change), epidemics, financial markets, chess and much more. If my work is known for one thing, I hope it will be that it achieved more with respect to brevity and simplification. The Signal and the Noise is an investment of time and brain cells but well worth the sacrifice of both.
Mindware: Tools for Smart Thinking, Richard E. Nisbett, Farrar, Strauss and Giroux, New York, New York, 2015.
I reference Mindware in the text, because of the author’s unabashed warnings regarding the limitations of multiple regression analysis (MRA), perhaps the most prevalent numerical analysis conducted in research (especially the social sciences). Nisbett also observes that “our approach to hypothesis testing is flawed in that we’re inclined to search only for evidence that would tend to confirm a theory while failing to search for evidence that would tend to disconfirm it.” Nisbett’s book is very readable. While his focus is on reasoning in general, experiments, and the philosophy of knowledge, his central question is very similar to mine: How well do we know what we know?
The Laws of Medicine: Field Notes from an Uncertain Science, Siddhartha Mukherjee, TED Books/Simon & Schuster, New York, 2015.
This slim volume, by the Pulitzer Prize winning author of The Emperor of All Maladies, reveals why ‘the laws of medicine are really laws of uncertainty, imprecision, and incompleteness.” They are, in fact, the ‘laws of imperfection.’ Probably the greatest piece of wisdom I got from this book is that even a perfect experiment is not necessarily generalizable. In other words, even if all of your statistics prove that your experiment ran perfectly, that doesn’t mean your results can be extrapolated to larger or different populations or even repeated for an identical sample.
In my view, the medical profession is particularly rife with arrogance and inability to face the limits of certainty. Mukherjee courteously holds the collective profession up in front of a mirror, pointing out the flaws in what he concedes is a relatively young area of science.
Willful Ignorance: The Mismeasure of Uncertainty, Herbert Weisberg, John Wiley & Sons Inc, Hoboken, NJ, 2014.
Weisberg tackles the subject of uncertainty from the perspective of the general process of scientific discovery and uses engaging stories about scientists and “thinkers” throughout history to illustrate his points. Like Nisbett, he also thinks statistical analysis has approached “a crisis” (paraphrasing the back flap copy). One of his central tenets is that “this technology for interpreting evidence and generating conclusions has come to replace expert judgment to a large extent.
“Scientists no longer trust their own intuition and judgment enough to risk modest failure in the quest for great success.” And this corollary: “Instead of serving as a adjunct to scientific reasoning, statistical methods today area widely perceived as a corrective to the many cognitive biases that often lead us astray.” It isn’t the role of science to provide answers; it’s to refine the questions. It’s a readable text but falls squarely between an academic textbook and one attempting to popularize science concepts.
Automate This, Christopher Steiner, Portfolio/Penguin, New York, New York, 2012.
The book’s subtitle, “How Algorithms Came to Rule Our World,” suggests that Steiner’s focus is how human activities are being automated through bots governed by algorithms. “Algorithms,” he writes,
“operate much like decision trees, wherein the resolution to a complex problem, requiring consideration of a large set of variables, can be broken down to a long string of binary choices.”
Binary choices are ones computers can make. But this statement also shows that an algorithm is just another form of numerical analysis. Of all the books I recommend, Steiner’s scares me the most. Consider this:
“Of the nearly one billion users in Facebook’s system, the company stores up to a thousand pages of data, including the type of computer you use, your political views, your love relationships, your religion, last location, credit cards…” (Remember, it was published in 2012). Think about that with respect to the privacy and national security debate.
At one time, the federal government forced AT&T to cooperate for national security in ways no one wants to remember. Now, imagine when the social media sites of our modern world have your information wrong, when they have drawn the wrong conclusions from your digital footprints! Steiner also describes a company which has developed a bot that “sucks in box scores from sporting events, identifies the most relevant aspects, and writes a story built around those aspects of the game. Is this the end of sports journalism as we know it?
Models.Behaving.Badly.: Why Confusing Illusion with Reality Can Lead to Disaster, on Wall Street and in Life, Emanuel Derman, Free Press/Simon & Schuster, New York, New York, 2011.
Derman is a physicist turned Wall Street “quant” and was one of a plethora of authors weighing in on the financial crisis and great recession of 2007/2008. Derman brings into the discussion the idea of models and metaphors:
“Models stand on someone else’s feet. They are metaphors that compare the object of their attention to something else that it resembles. Resemblance is always partial, and so models necessarily simplify things and reduce the dimensions of the world.”
But this later quote is priceless in its utility for understanding: “Once you understand that a model isn’t the thing but rather an exaggeration of one aspect of the thing, you will be less surprised at its limitations.”
This is similar to what Nisbett is trying to convey about MRA, which limits the researcher to one aspect of the thing, and thus loses the context of all the other influences on that one thing (e.g., a measured, independent variable). Although Derman focuses (mostly) on financial models, he explains very well the limitations of models for economics, global climate, and other broad situations compared to those used in physics.
An Engine, Not a Camera, Donald Mackenzie, The MIT Press, Cambridge, Mass., 2006.
If more people read and understood Mackenzie’s account of his deep research into valuation models for financial derivatives and the inner workings of financial markets, the world of investment would probably be very different. Mackenzie shines a bright light on the purpose of most models—to create a version of reality and then capitalize on that reality. In this case, Mackenzie argues persuasively that the Black-Scholes model for options pricing, which did indeed by most accounts change the field of finance, was developed to drive a market (engine) rather than reflect a market (camera). His analysis lends evidence to a broader contention, that the “invisible hand” of the market is anything but, that markets are deliberately constructed for the entities which will participate in that market.
To my way of thinking, An Engine, Not a Camera is about uncertainty at its highest level, as it casts doubt on the entire notion of a “free market,” “Markets are not forces of nature, they are human creations,” he writes. To which I would add (as I suggest in the chapter on business models), models today are primarily used to create new markets and new realities, not expand our understanding of the human condition.
Useless Arithmetic, Orrin Pilkey and Linda Pilkey-Jarvis, Columbia University Press, New York, 2007.
This is an example of a book that focuses on a specific field of applications identified in the subtitle, “Why Environmental Scientists Can’t Predict the Future.” This quote sums up what you are going to learn from the Pilkeys: “The reliance on mathematical models has done tangible damage to our society in many ways. Bureaucrats who don’t understand the limitations of modeled predictions often use them.” Even if you consider yourself an environmentalist, Useless Arithmetic is very useful for understanding how math models are used and abused.
Merchants of Doubt, Naomi Oreskes and Erik M Conway, Bloomsbury Press, New York, 2010.
As I note, uncertainty is something used to create doubt. In particular, the authors take aim at scientists and researchers pressed into service (and well paid) to blow up what is left of scientific uncertainty on highly charged political and cultural issues to impede progress on the issues of the day. They go as far to accuse such experts as turning doubt into a “product.” The issues they tackle include smoking and cancer, the ozone hole, global warming, acid rain, and other ecological issues. Unlike many of the other books listed, the authors in particular assess the public and political debates around these issues, not the scientific method. Health effects of smoking were turned into a great debate, funded by “big tobacco,” after the scientific evidence was rapidly drawing the conclusion, assert the authors. Among the important tenets of wisdom imparted is that balance in reporting is not giving equal weight to both sides, but to give accurate weight to both sides. Some “sides” represent deliberate disinformation spread by well-organized and well-funded vested interests, or ideological denial of the facts.
Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy, Cathy O’Neil, Crown, New York, 2016.
You’ve probably inferred from the title that this book aims to be provocative and incendiary first. It certainly accomplishes that. O’Neil tackles data and modeling through the prism of social justice and power structures. But her metaphor is precious because it reveals how models evolve into WMD. One of her best examples is the US News & World Report college ranking system. Over several decades, it became the standard for college rank and therefore the object of intense manipulation so that colleges could improve on the rank. She observes (correctly in my mind) that all of the emphasis on the rating and its following among parents doesn’t do a damn thing for the quality of education. When a school’s objective becomes figuring out how to “game the ranking,” it’s no different than my attempt to game my rankings of colleges to favor the school I had already selected, as I illustrated in the opening chapter. O”Neil applies her analysis to getting insurance, landing a job, obtaining credit, on-line advertising and other aspects of unfairness in modern life.
An Introduction to Mathematical Modeling, Edward A Bender, Dover Publications, Mineola, NY, 1978.
Here, the term “introduction” refers to very mathematics-intensive theory and applications, optimization routines, and probabilities. The first chapter, “What is Modeling?” does a good job of laying the groundwork for those who wish to skip the math.
Measurements and Their Uncertainties, Ifan G. Hughes and Thomas P.A. Hase, Oxford University Press, Oxford, England, 2010.
This book, focused on error in physical sciences, also gets complicated in a hurry, but again, the first chapter is well structured and offers good foundational material. It starts with the overriding point that “there will always be error associated with that value due to experimental uncertainties.” It goes on to classify uncertainties as random errors, systematic errors, and mistakes. While most discussions of uncertainty and error (mine included) focus on extrapolation, or extending a curve fit to data past the original measured data (or making inferences into the future using data from the past), this book reminds us that interpolation can be just as insidious. Interpolation refers to assuming the shape of the curve or line or graph between the measured data points. While this is a textbook, it is graphically rich rather than mathematically intensive (authors assume that computers will be doing most of the math).
Interpreting Data, Peter M Nardi, Pearson Education Inc, Boston, 2006.
This book keeps to the straight and narrow of how data analysis is applied in experiments. It notes in the introduction that “it is written in non-technical everyday language…” With passages like “Pearson r correlations are for interval or ratio levels of measurement…Many researchers, however, use these correlations for dichotomies and for ordinal measures, especially if there are equal-appearing intervals,” I’m not convinced of the everyday language. Nevertheless, I found it useful as a refresher on how experiments are designed, data taken, results analyzed, and conclusions drawn.
A Demon of Our Own Design, Richard Bookstaber, John Wiley & Sons Inc, Hoboken, NJ 2007; and Lecturing Birds on Flying, Pablo Triana, John Wiley & Sons, Hoboken, NJ, 2009.
Both of these books are focused on financial engineering and were blessed in being well-timed with the collapse of financial markets and the world economy. They cover similar territory and both insinuate that financial markets are imperiled by the way modeling is applied. The subtitle for Demon is “Markets, Hedge Funds, and the Perils of Financial Innovation,” and the subtitle for Lecturing Birds is “Can Mathematical Theories Destroy the Financial Markets?” However, everything I read tells me that things have only gotten worse, so unless you are seeking recent historical perspective, I’d supplement these two books with some more recent titles.
20% Chance of Rain, Richard B Jones, Amity Works, Connecticut, 1999
This book wants to be “Your Personal Guide to Risk,” as its subtitle urges. Written by an industry colleague in my consulting work, who spent decades in the machinery insurance business, it’s not really about modeling or uncertainties, but about risk and how we measure risk through probabilistic assessment. Jones stresses the uncertainty boundaries around any risk assessment and that “perception creates risk reality.” He also offers this bit of timeless wisdom: “Statistics do not, and cannot, prove anything. The field of statistics is incapable of this. Statistics can provide information to help us make decisions, but the decisions are still ours to make.” Today, statistics and numerical analysis in general are being used so decisions can be made for us (automation, digital algorithms, market construction, even on-line dating and hookup). We’d better all have a thorough understanding of their limitations
If you enjoyed this article, please consider sharing it!
