Currently viewing the category:
"Painting By Numbers"
We need to bring error and uncertainty analysis forward in the public discourse. That’s what Painting By Numbers is for.
Consider these two statements.
- “Cuomo says 21% of those tested in NYC had virus antibodies”
- “Every 1% increase in unemployment results in 37,000 deaths”
The first is a headline in the New York Times April 23, 2020. The second is taken from a meme in my social media feed the same day.
In terms of numbers widely propagated and magnified in the public sphere, both suffer from a common deficiency. Quantified error and uncertainty bounds around the result are not reported. So, the public has no idea what the value of the numerical result really is.
Without a sober, quantified explanation of accuracy, validity, relevance, repeatability, bias, and certainty, both numerical results come across as sensationalist in their own way.
This gap is a constant source of misunderstanding for smoldering crises like climate disruption and social inequities but becomes dangerous, frankly, during immediate crises like the COVID-19 pandemic and the 2007-2008 financial crisis (lack of understanding of financial engineering models). Information, including results of countless numerical analyses, forecasts, and predictions, is disseminated fast furiously and peoples’ heads spin.
Memes propagated through social media aren’t going to improve in quality anytime soon. I understand that. I’m not going to even try deconstructing the unemployment meme.
But scientists, academics, political leaders, and journalists should be more careful.
It’s one thing to also report, or state from the podium, “these are early results and must be validated with more testing,” “preliminary results,” or “the testing method is still under development and is not 100% percent accurate.”
In fairness, a New York Times article about the 21% number does include many disclaimers. https://www.nytimes.com/2020/04/23/nyregion/coronavirus-new-york-update.html
The account does acknowledge that the accuracy of the test has been called into question. But what does that tell us? Not much. The article also takes the percentage and propagates it through another calculation, stating that “if the state’s numbers indicated the true incidence of the virus, they would mean that more than 1.7-million people in New York City…had already been infected” and “That is far greater than the 250,000 confirmed cases of the virus itself that the state has already recorded.”
So a numerical result with unquantified accuracy now implies that the infection rate is almost seven times higher than the confirmed cases. The error in the 21% number is now embedded in the next numbers!
Just because a bit of information is a number doesn’t necessarily mean it is telling us something meaningful, relevant, or useful at this time.
Determining and error or uncertainty is a rigorous, quantitative, analytical exercise and should be conducted for every numerical result, especially during times of manic public concern.
Many, though not enough, scientific journal papers at least will include a qualitative on error and uncertainty in the measurements, the models, statistics, assumptions, etc, especially around statistical analysis. Rarely do you see included in media reports a thorough answer to the question “how confident are we in the numerical result we just reported?”
We need to bring error and uncertainty analysis forward in the public discourse. That’s what Painting By Numbers is for.
Painting By Numbers: How to Sharpen Your BS Detector and Smoke Out the Experts earned Silver in the Foreword Reviews Indies Awards for 2016 in the category Social Sciences.
Recognition is always welcome, especially as an independent author through an independent publisher, and this latest award for Painting By Numbers sits alongside the GOLD IPPY won in the 2017 Independent Publisher Book Awards in the category Current Events: Social Issues/Humanitarian.
To everyone fighting numerical illiteracy, we salute you!
At the end of this paper is a tantalizing Best Practice, however. There are two sidebar text boxes: (1) What is already known on the subject?” and (2) What this study adds. Imagine if every article, every paper having numerical analysis or results had a third section, (3) What are the uncertainties around our results?
When I was a kid, I sometimes would write down lots of really huge numbers and add them up, subtract one from the other, or multiply them. Just for the fun of it. You might think, wow, a budding math genius (not even close), but then I’d have to add, sometimes I did this to keep myself awake so I could sneak out of my room at night and watch TV with my sister well past our bedtimes.
Now, just for kicks, I read through technical papers with complex numerical analysis and see if I can find the Achilles Heel in the analysis, a questionable assumption, or a variable with a high degree of error associated with it.
After reading an article about the total costs of bicycle injuries (I am an avid cyclist), I went to the original source, linked below. Calculating the total cost of something is always fraught with uncertainty. Let me reiterate that I’m not impugning the credibility of the authors; I’m pointing out common uncertainties in numerical analyses which should be more visible.
Well, it didn’t take long to find at least one Achilles Heel, and it’s a good one because I see it frequently. The “heel” is evident from the graph on page three of the paper. Without getting down into the weeds, the total cost has three principle components – medical costs, work loss costs, and lost quality of life costs.
It’s easy to see that the lost quality of life costs represent the largest of the three cost components. In fact, just eyeballing the bar chart, that component is two to three times the size of the other two components. So it makes the “total cost” of bicycle injuries appear much higher. What isn’t so easy to discern is that the lost quality of life costs are probably subject to a far greater error factor than the other two.
Estimating “quality of life” is more difficult, because it’s a more subjective variable. This is what I mean in commandment 7 of Painting by Numbers: “Don’t confuse feelings with measurements.” Medical costs of an injury are less squishy – someone had to pay the bills after all – as is work loss. Just multiple the wages or salaries by the lost time due to the injury.
To their credit, the authors point this out in the Discussion section: “Costs due to loss of life are challenging to estimate.” What would have been far more helpful in understanding the validity of this quant exercise is if the authors added error bands around the three variables in the figure I referenced above. Or ran the results with and without the very high error prone variable and compared them. Because, as stipulated by Commandment three in Painting By Numbers, “Find the Weakest Link,” the results are only as good as the most error prone variable.
At the end of this paper is a tantalizing Best Practice, however. There are two sidebar text boxes: (1) What is already known on the subject?” and (2) What this study adds. Imagine if every article, every paper having numerical analysis or results had a third section, (3) What are the uncertainties around our results?
http://injuryprevention.bmj.com/…/injuryprev-2016-042281.fu…
This is another entry at my Facebook Author Page on error, bias, numerical analysis, and all the topics in Painting By Numbers: How to Sharpen Your BS Detector and Smoke Out the Experts.
I’ve spent many hours in my career listening to technical papers, reviewing them for engineering associations and conferences, and editing them or extracting from them for publications and client reports. Over close to four decades, I’ve witnessed a deterioration in quality of these papers and presentations. Many of them today are thinly veiled marketing pieces for the authors’ companies.
So my eyeballs perked up when I read this headline at Retraction Watch: “Could bogus Scientific research be considered false advertising?” The opening sentence is, “Could a scientific paper ever be considered an advertisement?” Retraction Watch is a newly discovered website I’m now following through regular notices.
The questions were stimulated by a court case in Japan where a researcher for a top global pharmaceutical company was being tried, not for manipulating data and scientific fraud (that had already been acknowledged), but for criminal violation of Japan’s advertising laws. The article goes further to probe whether a similar court case in the US might find the researcher and/or his/her company guilty of false advertising when research shown to include falsified data is circulated with promotional material about the drug.
There’s a difference between a technical paper so weak it comes across as company marketing collateral and corrupted research data used to support pharmaceutical advertising. But my larger point here is that the general deterioration in technical information disseminated by “experts” to professionals and consumers creates a huge credibility gap.
It’s high time we call out data-driven BS for what it is in many cases – advertising, false or legitimate, for a product, company, specialist, researcher, author, or government policy maker disguised as legitimate information.
Retraction Watch is a fascinating site to follow (even if somewhat depressing). Someone has to do the dirty work of accentuating the negative. I’m glad I’m not alone!
http://retractionwatch.com/…/bogus-results-considered-fals…/
Could a scientific paper ever be considered an advertisement? That was the question posed to a…
RETRACTIONWATCH.COM
From a Painting By Numbers perspective, the article below is probably one of the most important you’ll read this month, maybe the next few months.
It does a great job expanding on my Commandment No. 10, “Respect the Human Condition,” probably the most sweeping of the twelve commandments in my book. It means that the foibles of us mere mortals – such as accentuating the positive, stretching for success, seeking reward and avoiding punishment – are almost always baked into every numerical result we see in the public sphere. And when they aren’t, you can bet it took lots of experts with plenty of patience for the foibles, or biases, to be extracted out.
Unless you are looking at primary research documents, every numerical result you see has two major components: the work of the analysts or researchers themselves and the work of those (journalists, communications professionals, policy aids, etc) who report them. The headline of the article below focuses on making the scientific method better account for less than positive results. But the authors also take reporters to task, who generally ignore critical research which doesn’t lead to a positive result.
The headline, “Dope a Trope shows modest cancer fighting ability in latest research,” is going to have higher readability than “Scientists find Dope a Trope has no effect on cancer patients.” The problem with this is, in the realm of research, there could be half a dozen experiments of the latter variety and only one of the former. And the half dozen who found no effect probably aren’t going to impress those who fund research.
The author, Aaron E. Carroll of the Indiana University School of Medicine, notes, rightly I believe, that the whole culture of professional scientific research has to change to address this endemic challenge. Thankfully, the author has a great blog site, The Incidental Economist, where he regularly expands on this broad but critical subject. For those interested in diving in even deeper, The Center for Open Science has tools and info for making research methods more transparent and results more reproducible. Only after many experts arrive at the same results should the rest of us even begin to take them seriously.
https://www.nytimes.com/…/science-needs-a-solution-for-the-…
Policy-makers are probably the worst offenders when it comes to using and abusing mathematical modeling and numerical analysis, the subject of my latest book, Painting By Numbers: How to Sharpen Your BS Detector and Smoke Out the Experts. When it comes to the administration’s rollback of the global climate change regulations and specifically the Clean Power Plan, however, one number which really matters is 3. That’s the number of times the Supreme Court has ruled in favor of the Environmental Protection Agency’s authority to regulate carbon dioxide under the Clean Air Act Amendments.
This means that the Administration cannot just end the Clean Power Plan, central to the EPA’s carbon regulation strategy, but must come up with an alternative regulatory framework. The EPA concluded, and the Supreme Court upheld, an endangerment finding for carbon pollutants, and therefore the agency is legally required to regulate carbon emissions.
Ironically, this is similar to the repeal, replace, repair problem with the Affordable Care Act. You can’t just “repeal” EPA’s carbon regulations, and it will be difficult to replace them. So, repair is probably going to be the sensible option.
Nothing is easy when it comes to federal regulations and that’s the way the framers of our Constitution intended.
Another in my continuing series on applying the “commandments” from my most recent book, Painting by Numbers: How to Sharpen Your BS Detector and Smoke Out the Experts…
“Americans ate 19% less beef from ’05 to ’14,” according to a recent headline. The article goes on to focus on the carbon footprint reduction associated with eating less beef and avoiding the methane emissions from the back ends of cattle and all the carbon-intensive stuff that has to happen to get that steak to your dinner table. The original study (hot linked in the article) the article is based on comes from the Natural Resources Defense Council (NRDC). It wasn’t just beef eating that fell during this period; chicken and pork were affected as well.
The article makes a valiant case for how Americans are “gradually changing their diets, driven by health concerns and other factors.” The original NRDC study’s aim is to show that the answer to the question, “Where’s the beef?,” is a victory in the war on climate change. It may indeed have something to do with that.
But the larger explanation for this quantitative result probably has as much to do with general economic forces. 2007 marked the beginning of the “great depression” and the economy has been on an anemic (relative) growth cycle since we came out of it. The beef ranch-to-table production and delivery cycle is not only carbon intensive, it’s expensive. Beef, for most people, is one of the most expensive foods they buy, and it would be natural to cut back on its consumption during bad economic times.
The article is a little more fair about all this than the original study. The author cites a survey in which 37% of Americans say price is the number one reason why they ate less beef. That’s somewhat out of step with the conclusion NRDC is trying to draw. The study avoids mention of ANY economic factors. It would have been more credible if there was even an attempt to “iron out” the affect of general economic conditions. Consumer economic activity generally has been substantially cut back since 2007.
Of course, the study also doesn’t explicitly CLAIM a correlation between consumer’s desire to impact carbon footprint by eating less beef, but you can bet it wishes to suggest one, it we invoke Commandment six in Painting By Numbers, “Understand the Business Model.” NRDC is an environmental policy organization (and an effective one at that).
What’s critical here is the aura around that 19% number (and I’m not even going to start in on the methodology to calculate it). In the study, it was all about an associated (not correlated) improvement in carbon footprint. In the article, it was about all the reasons except economic forces why Americans are reducing beef consumption (price being more of a footnote). In two iterations, there’s a great deal of political, cultural, and social “stuff” hanging off of that number. Imagine how laden it might be once it’s being discussed around the dinner table!
https://www.nytimes.com/…/…/beef-consumption-emissions.html…
I’ve been posting at my Facebook author page regular supplements to the examples from everyday life I use in Painting By Numbers: How to Sharpen Your BS Detector and Smoke Out the Experts. I figured it would be smart to post them here too. Here’s one from this week.
It’s always instructive to study the source when controversy swirls. I just read the official Congressional Budget Office (CBO) “scoring” of the Republican American Health Care Act (AHCA). While the “numbers,” $330-billion (reduction in federal deficit) and 24-million (forecasted number of uninsured Americans by 2026) are getting the headlines, the real conclusion of the report is this:
“…the great uncertainties surrounding the actions of the many parties that would be affected by the legislation suggest that outcomes of the legislation could differ substantially from some of the estimates provided here. Nevertheless, CBO and JCT are confident about the direction of certain effects of the legislation. For example, spending on Medicaid would almost surely be lower than under current law. The cost of the new tax credit would probably be lower than the cost of the subsidies for coverage through marketplaces under current law. And the number of uninsured people under the legislation would almost surely be greater than under current law.”
It’s that word, “direction,” I call to your attention. Those who pay economic modelers usually want hard numbers and ranges and low, high, and average scenarios. But most modelers will tell you that all they are able to provide with much confidence, especially on something like legislation affecting 1/6 of the US economy, is a few “directional” conclusions.
The CBO is bi-partisan so I am assuming they didn’t start with the answer (the AHCA is better than Obamacare!) and work backwards, commandment 11 in Painting By Numbers. However, you should more strongly consider commandment 6, understand the business model. The role of the CBO in this case is to provide “quantitative cover” for a controversial and convoluted piece of legislation. This helps legitimize the effort in the minds of politicians and us citizens, but how much the analysis informs the debate is questionable at best.
Here’s the original CBO report: https://assets.documentcloud.org/documents/3516452/CBO-Health-Care-Cost-Estimates.pdf
