At the end of this paper is a tantalizing Best Practice, however. There are two sidebar text boxes: (1) What is already known on the subject?” and (2) What this study adds. Imagine if every article, every paper having numerical analysis or results had a third section, (3) What are the uncertainties around our results?
When I was a kid, I sometimes would write down lots of really huge numbers and add them up, subtract one from the other, or multiply them. Just for the fun of it. You might think, wow, a budding math genius (not even close), but then I’d have to add, sometimes I did this to keep myself awake so I could sneak out of my room at night and watch TV with my sister well past our bedtimes.
Now, just for kicks, I read through technical papers with complex numerical analysis and see if I can find the Achilles Heel in the analysis, a questionable assumption, or a variable with a high degree of error associated with it.
After reading an article about the total costs of bicycle injuries (I am an avid cyclist), I went to the original source, linked below. Calculating the total cost of something is always fraught with uncertainty. Let me reiterate that I’m not impugning the credibility of the authors; I’m pointing out common uncertainties in numerical analyses which should be more visible.
Well, it didn’t take long to find at least one Achilles Heel, and it’s a good one because I see it frequently. The “heel” is evident from the graph on page three of the paper. Without getting down into the weeds, the total cost has three principle components – medical costs, work loss costs, and lost quality of life costs.
It’s easy to see that the lost quality of life costs represent the largest of the three cost components. In fact, just eyeballing the bar chart, that component is two to three times the size of the other two components. So it makes the “total cost” of bicycle injuries appear much higher. What isn’t so easy to discern is that the lost quality of life costs are probably subject to a far greater error factor than the other two.
Estimating “quality of life” is more difficult, because it’s a more subjective variable. This is what I mean in commandment 7 of Painting by Numbers: “Don’t confuse feelings with measurements.” Medical costs of an injury are less squishy – someone had to pay the bills after all – as is work loss. Just multiple the wages or salaries by the lost time due to the injury.
To their credit, the authors point this out in the Discussion section: “Costs due to loss of life are challenging to estimate.” What would have been far more helpful in understanding the validity of this quant exercise is if the authors added error bands around the three variables in the figure I referenced above. Or ran the results with and without the very high error prone variable and compared them. Because, as stipulated by Commandment three in Painting By Numbers, “Find the Weakest Link,” the results are only as good as the most error prone variable.
At the end of this paper is a tantalizing Best Practice, however. There are two sidebar text boxes: (1) What is already known on the subject?” and (2) What this study adds. Imagine if every article, every paper having numerical analysis or results had a third section, (3) What are the uncertainties around our results?
http://injuryprevention.bmj.com/…/injuryprev-2016-042281.fu…
If you enjoyed this article, please consider sharing it!
