Thoughts on Covid, pt 1: science-based decision making

The amateur scientist and economist in me wanted to explore how emerging scientific information impacts public policy.

Me hiking the CDT through Collegiate Peaks Wilderness in the Colorado Rocky Mountains (Sept 2018).

Below are some long-form thoughts on the evolving Covid crisis; these views do not represent anyone other than my own.

I like to start any Covid discussion by (a) thanking all crisis-agnostic workers who put themselves in various frontline roles for the public benefit of others (e.g. healthcare workers, firefighters, police officers, teachers, etc.) and (b) extending sincere sympathy to everyone lost or will lose a loved one or cherished experience due to Covid.

Science in the media

In lieu of a metascience-esque discussion on what science is, I like the way that a senior political writer (who’s not a scientist) at FiveThirtyEight put it in a recent podcast (538):

“There’s a difference between the way science works and how science is taught in school. If what you’ve done is gone through…regular American school system and you didn’t major in science in college, what you’ve gotten is a series of facts that you’ve had to memorize and you’ve gotten this sort of process taught into [you] that like, ‘and then we discovered a thing and then we knew it — the end!’ And you kind of come away with this idea that it is more absolute in its answers and that it’s simpler in getting there than I think it actually is. And things like [Covid] put us in a really intimate place with scientific uncertainty that most of us are not familiar with being in. And it’s a really uncomfortable place…[it’s] difficult to explain that, ‘this is the best that we can do in terms of, this is the most truth we have, but it’s still not absolute truth’…[it’s] a hard thing to really wrap your head around.”

People look to scientists for the comfort of 1+1=2 or F=m*a, but most science — especially things involving population health like medicine or economics — represents the process of discovery. As Carl Sagan once quipped, “Science is a way of thinking much more than it is a body of knowledge.”

Today’s spotlight on scientific research related to Covid has led to an “infodemic” in which emerging, singular research studies are being amplified in the media oftentimes in a selective manner that fits certain narratives. Without getting too cynical, it’s worth pointing out that the metrics being reported on like case counts, death counts, total deaths, log scales, model predictions, excess deaths, lethal risks of infection, and antibody tests are potentially all flawed, obscured by noise, and/or too uncertain to be useful (Guardian). It’s reminiscent of the famous adage, “A little knowledge is a dangerous thing. So is a lot.”

So what information is reliable? It’s less about what is fake versus real news, but more about how to interpret different data points to inform decisions. Early on, was WHO’s 3.4% estimated case fatality rate actionable (WHO)? Or does one non-peer reviewed article that claims 1.4% infected fatality rate (ICL) substantiate ‘the truth’? How about another non-peer reviewed article that implies 0.05% (medRxiv)? By March 12th, when most developed countries outside of Asia were facing the decision of whether to lock down society, we had 900 papers, preprints, and preliminary reports related to Covid (Nature). How do we decide which voices to elevate? How are world leaders deciding which voices to listen to?

When asked to discuss the virus’s trajectory, Art Reingold (head of epidemiology and biostatistics at UC Berkeley) responded, “As I like to say, all models are wrong, some models are useful” referencing the common aphorism in statistics (Guardian). While each research study might uncover useful nuggets of information, the findings are more directional than they are absolute, something the media oftentimes fails to acknowledge or provide context for.

But this is, of course, not to say that scientists cannot come to a consensus on complex topics.

Scientific consensus

How do we define “scientific consensus”? There are some metascience articles (e.g. AJPH), but Wikipedia has a good summary of how it’s defined in the scientific literature (Wiki):

“Scientific consensus is the collective judgment, position, and opinion of the community of scientists in a particular field of study. Consensus implies general agreement, though not necessarily unanimity. Consensus is achieved through communication at conferences, the publication process, replication of reproducible results by others, scholarly debate, and peer review.”

How do Covid policy responses fit that narrative? The question is confiscated by a lot of immediate noise on a controversial topic, but I would argue that we do not have the level of rigor warranted to merit a scientific consensus (e.g. as defined above) regarding what the prudent public policy response to Covid should be. Scientific consensus is based on scientific evidence which is insights gathered from extensive and time-intensive scientific research (good op-ed on this here). Could we claim in mid-March that the holistic efficacy of a lockdown to combat a disease of Covid’s nature or the appropriate models to measure that efficacy were built and had achieved “communication at conferences, the publication process, replication of scientists in a particular field of study, scholarly debate, and peer review”? I would argue no.

It’s fair to argue that this definition of scientific consensus is somewhat impractical; that the bar is so high that it prevents decision-makers from being able to act on science. And there’s merit to that, but something with as many implications as ‘scientific consensus’ on topics that are endlessly messy to measure should be a very high bar.

What if we don’t have the luxury of time, can surveys of scientists confirm scientific consensus? Yes, to the same extent that surveys can confirm anything (i.e. with consideration of the commonly known issues with surveys like response bias, survey population generalizations, assumptions of prior knowledge, leading questions, etc.). When Naomi Oreskes, the Harvard historian of science, wanted to prove that there was a scientific consensus on the existence of anthropogenic global warming, she published an article in Science showing that a survey of the abstracts (i.e. summary conclusions) of 928 peer-reviewed scientific articles published between 1993 and 2003 showed complete unanimous agreement. That is ten years of research to reach scientific consensus. And how does our understanding of the meta-impact of lockdowns compare to that? It pales. Fredrik Erixon, the director of the European Centre for International Political Economy in Brussels, echoed that perspective when he said (on April 1st), “The theory of lockdown, after all, is pretty niche, deeply illiberal — and, until now, untested. It’s not Sweden that’s conducting a mass experiment. It’s everyone else.” (Spectator)

The issue with surveys today of scientists on lockdown efficacy is that we’re generally either asking scientists for broad policy opinions rather than a scientific opinion pertaining to their specialty, or we’re extrapolating their responses on a subset of the issue to imply justification for a broad response. When we think about surveys of experts we need to be careful that the questions are clear and targeted so that the right experts are answering the right questions, otherwise, we’re asking for subjective opinions rather than research-backed science. For example, pro-lockdown advocates have pointed to subsets of the following studies (with my commentary):

• [Mar 27] IGM survey of a diverse group of expert economists with public policy interests found that 80%+ agree that abandoning lockdowns will lead to greater economic damage than sustaining lockdowns (this is a loaded question based on assumptions about lockdown epidemiology being asked of economists; survey responder comments include: “I am not an epidemiologist”, “This is primarily an epidemiological question”, and “We have insufficient data to assess”).
• [Apr 2] Sermo statistically significant study of global physicians who have personally treated Covid patients found a myriad of consensus findings, including that governments are weighing public and economic concerns appropriately (interesting perspective, but physicians are not the appropriate expert audience for economic policy questions).
• [Apr 2] Doximity survey of a large sampling of pan-specialty doctors found that the majority feel that the government is not responding adequately, that “social distancing” (not lockdown) is appropriate, and that shelter-in-place would most effectively flatten the curve (interesting perspective on one factor of the lockdown decision).
• [Apr 3] MDLinx survey of a relatively small sampling of doctors (assumably pan-specialties but not stated) found that 22% of doctors surveyed characterized Covid as “somewhat” or “not at all” a threat while 77% consider it an “extreme” threat, and that 70% support a national lockdown for ~1–6 months (interesting perspective, but again, doctors are one of many sets of expert opinions weighing the lockdown impacts).

John Jenkins (president of the University of Notre Dame) put it well: “There are, however, questions that a scientist, speaking strictly as a scientist, cannot answer for us. For questions about moral value — how we ought to decide and act — science can inform our deliberations, but it cannot provide the answer” (NYT). So, if not scientists, who’s responsible for making decisions?

Science-based decision making

Germany’s Ethics Counsel argues that “it was up to elected politicians, not scientists, to make the ‘painful decisions’ weighing up the lockdown’s effect on health and its other side effects” (IHE). (The article points out that Germany has a “particularly strong tradition of ethical debate” vs other European bioethics councils and that German academic philosophers have a stronger history of involvement in political discussion, hence the emphasis on Germany.) I personally agree that government decisions should be decided by democratically elected officials — that, along with checks and balances, is what a representative democracy is. Any empowered leader (prime minister, CEO, referee, etc.) is tasked with ingesting, digesting, and acting upon information on behalf of the group.

An interesting side point that the article makes is to focus on the composition of different global leaders’ Covid response task forces — what mix of experts are informing policy decisions? Looking at the US, the White House Coronavirus Task Force is composed of: 6 medical doctors (incl. 2 immunologists, 1 virologist, 1 anesthesiologist, 1 neurosurgeon, 1 radiation oncologist), 4 businessmen, 3 attorneys, 3 military officials, 2 lawyers, 1 telecoms advisor, 1 meteorologist, 1 economist, and 1 health policy consultant (Wiki). While the composition is largely made up of various heads of relevant government agencies, I would ask where the epidemiologists and behavior scientists are, or why it’s so underweighted on immunologists, virologists, and economists versus overweighted on businessmen and attorneys? That’s a separate discussion worth exploring.

So it’s the elected politicians, not scientists, who must make judgment calls on a Covid response. But handing over power to elected politicians can also be dangerous because authority can legitimize unsubstantiated claims, for example 2003 weapons of mass destruction and endless wars — should we hold congresspeople liable for voting in favor of the invasion of Iraq? They were simply listening to experts and people with authority feeding them supposedly substantiated information. But yes, they are certainly accountable for making a judgment call based on the available information.

And that’s what consumers do every day. We make personal health decisions based on existing scientific information but also accounting for other factors; I personally eat a balanced diet (even though fat, sugar, gluten, and dairy have each been the nutrition enemy at different times), choose to use birth control (even though the potential long-term side effects of invasive hormones are terrifying), and choose not to use most cosmetic products (because you never know if baby powder is carcinogenic, NYT).

But public policy is addressing population not individual health, and leaders must make prosperity decisions by weighing multiple risks against compounding side effects based on available scientific information; for a novel virus, that available information is still emerging.

Interpreting emerging research

On the topic of scientific research and looking for useful, directional findings, early studies on lockdown efficacy are emerging. A small subset of those studies include:

• [Apr 2] NPR overviews the early-April studies showing that ventilators were not treating Coronavirus as effectively as expected, which is worrisome given how widespread the treatment became and given the severe side effects of the invasive treatment.
• [Apr 3] Nature study found that surgical masks helped prevent the spread of Covid from sick patients, but then ACP study found that neither surgical nor cotton masks effectively filtered Covid particles coming from sick patients. This has led many Covid ward doctors to use face shields. WHO’s recommendation on face masks is: “Wear a mask if you are coughing or sneezing. If you are healthy, you only need to wear a mask if you are taking care of a person with COVID-19.” (WHO)
• [Apr 5] CSTE issued a statement recommending that Covid death counts should include both “Confirmed” and “Probable” cases of Covid. The CDC adopted this even though the precedent for virus-related deaths is to base them on direct count (JAMA) — for influenza, physicians do not test for it in the outpatient clinic nor every time a patient is hospitalized (and if the test results are not available by the time of death, it does not get counted).
• [Apr 9] The World Bank found that Covid could cause Sub-Saharan Africa’s first recession in 25 years, leading many African leaders to ease lockdowns despite the transmission risks of the virus (BBC).
• [Apr 17] A DFTB survey of scientific literature couldn’t find a single example of a child under 10 passing the virus on to someone else. That same week, Denmark reopened schools (NYT). A May 1st Lancet study found that “school closures alone would prevent only 2–4% of [COVID-19] deaths, much less than other social distancing interventions”.
• [May 12] Lancet study looked at the impact of global supply chain disruptions from Covid and found that for 118 low- and middle-income countries, under-5 child deaths per month are predicted to increase by 10–45% and maternal deaths by 8–39% (these are incremental and not overlapping with existing child and maternal deaths). The results were so harrowing that Unicef launched a campaign to “prevent the COVID-19 pandemic from becoming a lasting crisis for children, especially the most vulnerable children — such as those affected by poverty, exclusion or family violence.”
• [May 21] NBER study of SIR models found that targeted Covid policies perform better than uniform Covid policies, e.g. for the same economic cost a targeted policy results in lower mortality than a uniform policy.

These are by no means comprehensive and are selective based on my biases. And while I sympathize with the media’s impossible task of synthesizing and presenting various sources of technical perspectives, I’ve personally been frustrated with how the handling has been politicized and sensationalized. (I’ve experienced first-hand crises like the Boston Marathon bombings and 2017 NorCal fires, and felt as though the media fell prey to click-bait, “if it bleeds, it leads” reporting.)

Perhaps because I work in venture capital — where we gravitate toward maverick founders and where the general mentality is that things can and should be disrupted or at least constantly poked at — but I want to elevate a few expert scientists in particular who have received “personal attacks and general disparaging comments” (STAT). These aren’t conspiracies or fake news, they are just different scientific perspectives. Good overview in Bloomberg, but those include:

• Michael Levitt (Stan­ford professor of biology and computer science, and 2013 No­bel lau­re­ate in chem­istry) — good interview
• John Ioan­ni­dis (Stanford professor of medicine, epidemiology, and population health, and one of the most-cited scientists in the world) — infamous STAT article and subsequent interview
• Joel Hay (USC pro­fes­sor of phar­macy and health eco­nomics) — NR and NR studies and interview

I think it will be really interesting to see a post-crisis, documentary-style timeline of emerging information on the virus and how those data points resulted in public thinking and government response. That’s a gargantuan effort, but in my next post I will attempt to understand how different world leaders responded to the crisis and what I might have done in their shoes.

About me

Part of building empathy and having a respect-filled debate on a controversial topic is remembering what experiences color someone else’s opinions. On me:

• My undergrad education is in environmental sciences (physical science concentration) and environmental economics. I authored two academic research papers, one social survey on cooperative decision-making and one controlled experiment on how information affects consumer behavior (which was published in a Stanford journal). I studied biology, statistics, population health, and research methodology, among other things.
• My favorite book is “King Leopold’s Ghost” which recounts how the Belgian monarchy brutally colonized the DRC while convincing the world it was philanthropic. I generally have a visceral distrust of someone or something trying to tell a population how to think, especially in the context of using emotion to galvanize obedience.
• I was raised by two physician parents — my dad is an internal medicine, cardiology, and toxicology specialist who spent a decade in Italy’s healthcare system, and my mom is a general surgeon specialist. Both worked in New York City ICUs in the 1980s during the peak of the HIV/AIDS crisis when hospitals were flooded with patients.
• I’ve been a pinnacle decision-maker in “loudest-voice-wins” environments, most memorably as president of a 1300-person co-op (BSC).
• I was raised to be religious and attended Catholic mass on Sundays. While I’m not a practicing Christian, I believe that religion bestows a certain tranquility with respect to end of life; it is God’s (or Mother Nature’s) will.