Tracking coronavirus: Epic Testing Fail

There are, in the United States, as of this writing some 159,257 confirmed cases of COVID-19 infection. 2,937 are considered serious or critical, and 2,922 people have died from the disease.

Delving into the data further, we find that 66,497 cases are in New York, while 16,636 cases are in neighboring New Jersey. These two states, together, account for 52.2% of all confirmed COVID-19 cases in the US. New York by itself accounts for 41.7% of all confirmed COVID-19 cases.

That is a large number of cases. It is an especially large number of cases for a single state--a sobering reminder of the reality that the typical US state is, in terms of geographic expanse, population size, and economic size, the equal of many fully sovereign nations, and a compelling argument for evaluating each US state singly rather than collectively when comparing US COVID-19 statistics to the rest of the world.

It is also not the total number of COVID-19 cases we have in the United States, in New York, or in New Jersey. That number is unknown, just as the actual number of influenza A cases each year are unknown, and must be estimated.

With countries such as South Korea engaging seemingly successfully in mass testing, why has the US not been able to establish the true size of its COVID-19 patient population? Why was South Korea so much more successful than the US?

The short answer is: they weren't. The legacy media just tells you they were. The legacy media is wrong (of course), and I will leave it up to the reader to decide if that error is the product of malice, stupidity, or ignorance.

What The Legacy Media Gets Wrong About Testing

When we look at the testing regimes of other countries, South Korea in particular, the first thing we must acknowledge is how few cases of COVID-19 were uncovered. By mid-March, South Korea had administered some 290,000 tests, resulting in approximately 8,000 confirmed cases of COVID-19. In other words, only 2.7% of the tests administered were of people infected with the COVID-19 virus (better known as the CCPVirus, as it came into the world courtesy of the Chinese Communist Party). Moreover, those 290,000 tests amounted to roughly 0.5% of their population.

We do not have precise breakdowns between the number of tests administered to symptomatic vs asymptomatic individuals, but even if one assumes an even distribution between symptomatic and asymptomatic people being tested, either one must concede that South Korea never had many COVID-19 cases with which to contend, or their testing regime did not identify all carriers of the virus. Given recent upticks in new cases for South Korea, and fears of a second wave of infection, it is almost certain that not all cases of COVID-19 were identified.

Thus, not only did South Korea not have a "mass testing" program (testing only 0.5% of the population hardly qualifies as "mass" anything), but it did not reveal all hidden reservoirs of the disease. Their success in containing the disease must be primarily attributed to the willingness of South Korean citizens to follow protocols of "social distancing" and quarantine.

South Korea's low positive test rates have been largely replicated here in the US. Washington State had a marginally higher positive test rate at 7%, and only New York has had a sizable fraction of tests come back positive at over 40%.

The conclusion to be drawn from this is that testing, while an invaluable diagnostic tool, is simply unsuitable for any sort of epidemiological tracking. This is also the somewhat belated conclusion of the "experts", who have at last concluded that, for tracking purposes, confirmed case numbers are "meaningless". It was similar logic that led Los Angeles to alter its testing protocol to focus it on diagnosing patients for whom the positive diagnosis would impact the course of treatment.

Why Not Test?

I shall be clear: testing as a diagnostic tool is invaluable, and I am hardly going to challenge doctors on how to treat the sick patient.

However, as important as diagnostic testing is to patient care, neither it nor any other tool is fit to accomplish all things. Tests have their uses, and therefore they have their limitations; we should be mindful of those limitations and not push a tool to a purpose for which it is not fit.

Drawing on my own background of over a quarter century of experience in technology management, I am keenly aware of the limitations of various tools, tests, and metrics for gaining understanding about various phenomenon. In fact, the challenge of identifying the proper metrics has been one of the enduring tasks of technology managers the world over, leading to noted technology author and commentator Bob Lewis to coin "The First Law of Metrics": You get what you measure. There is a corollary to the First Law as well: what you mismeasure, you mismanage.

In the case of COVID-19 testing, attempting to extrapolate from testing to model disease spread was the mismeasurement that led, in multiple instances, to the curious presumed phenomenon of "cryptic transmission." In Washington State, the curious 6-week lag between the first positive test and the second positive test, with both tests showing the viral strain to be related (meaning both patients were on a common chain of transmission), led doctors to quite naturally wonder where the transmission had been occurring, and how had they failed to miss it.

The answer it both simple and brutal: they missed the transmission because, contrary to their presumptions, they were not actually looking for it.

Syndromic Surveillance: Going "Old School"

In assessing the utility of testing, we must remember one thing: illness does not wait on diagnostic tests. COVID-19 infection will work its will on the patient with or without a diagnostic test.

It comes as no surprise, therefore, that even for normal seasonal influenza, only 1.2 million lab tests for influenza have been performed through week 12, despite the CDC estimating at least 38 million cases of influenza and influenza like illness. As a further depiction of the limitation of testing beyond its diagnostic purpose, only 20% of lab tests for influenza yield a positive result; 80% of tests reveal the illness to be something other than influenza.

Yet these numbers illustrate how unnecessary testing is for disease surveillance. Consider: 1.2 million tests is at most 3% of all influenza and influenza like illness cases this flu season. For influenza we are not even attempting to use testing as a tracking tool. Yet we have very complete and precise statistics about influenza.

Enter what is known as "syndromic surveillance". Every hospital in every health district in every state in the country reports both numbers if patient visits for influenza like illness and hospitalizations. With or without a test, people who feel sick are going to go to the doctor, and if they are sick enough they are going to be hospitalized. Because so many of the symptoms of COVID-19 are also that of seasonal flu, with or without diagnostic testing, cases of COVID-19 severe enough to warrant a trip to the doctor are going to be capture via syndromic surveillance.

In the case of Washington State's mystery "cryptic transmission", scrutiny of that state's Weekly Influenza Report revealed that, from mid January until mid February, patient visits for influenza like illnesses actually declined. Thus, there was no "cryptic transmission"--the disease quite literally was not spreading in the state during that period.

While it may be "old school", syndromic surveillance is actually a better method for tracking disease spread for the simple reason that it does not encounter the challenges of estimating how many cases are missed. Where patient visits and hospitalizations rise, it is intuitively obvious that something has made them sick. As epidemiologist Eric Feigl-Ding belatedly noticed about California, using the Weekly Influenza Report to measure levels of infectious disease in California was a viable means of tracking COVID-19 without testing.

With proper measurement comes proper management, and by monitoring the pace of hospitalizations in various parts of the country it is possible for both epidemiologists and crisis management teams to assess which hospital systems are being strained, which ones need resources, and which ones may be plausibly placed on a lower priority.

Contrary to the fears of some doctors, lack of testing does not mean lack of options, nor lack of information. The alternatives are in some regards superior to testing.

How Many Cases?

Without testing, a question invariably arises: how are we to know how many cases of COVID-19 are out there? The answer, of course, is to estimate, in much the same fashion the CDC already does with influenza and influenza like illnesses. Much of the groundwork for building these estimates has already been done. In studying cases of COVID-19 in China, and their proximal origins, researchers have concluded that as many as 86% of new infections came from persons not previously identified as being infected. Viewed another way, 14% of all COVID-19 cases in China were likely documented. Whether that same percentage holds true for the United States is problematic, but if we assume that percentage holds (not unreasonable, given that 20% of all cases of influenza like illness are actually influenza, with the rest being something else), then the confirmed cases amount to 14% of all COVID-19 infections in the country. The 159,257 figure mentioned at the beginning translates into a probable 1.1 million cases of COVID-19 nationwide.

To put that number into historical perspective, on June 25, 2009, the CDC estimated there were at least 1 million cases of H1N1 "Swine" flu in the United States. This was 2 months and 10 days after the first H1N1 infection was reported in the US on April 15. With the first case of COVID-19 having been detected on January 19, we are at 2 months 12 days. In terms of broad case totals, the size of this outbreak within the United States is on par with the 2009 H1N1 pandemic. Moreover, given that New York, by far the epicenter of COVID-19 within the United States, has such a higher percentage of positive test results, it is highly likely that the percentage of undetected cases is lower, which would make this outbreak smaller at this point than 2009 H1N1. There is no reasonable projection grounded in real world data that puts the pace of COVID-19 infection in the US ahead of 2009 H1N1.

There is an additional perspective to consider. As we look at the increasing numbers of cases across the country, we should pay special heed to what has been said multiple times and in multiple places: as more testing is done, more cases will be found. Buried in this statement is an important acknowledgement about case numbers: rising case totals indicate the rate of detection far more than they indicate the rate of spread.

As matter of basic statistics, in order for the rate of detection to approximate the rate of spread one would have to randomly sample a population without regard to whether they were symptomatic or not. With testing criteria that seek to weight testing outcomes towards positive results, the rate of detection only broadly hints at the actual rate of spread. It is entirely possible, as has been reported, that New York's COVID-19 outbreak is already peaking, even though the new case totals continues to climb, simply because increasing testing is increasing case discovery (and lowering the number of undocumented cases).

Looking Ahead

While there has been a considerable increase in the availability of test kits for COVID-19, including a 5-minute rapid test from Abbott Labs, reliance on testing for disease tracking will remain problematic for all the reasons already described. Regardless of testing, syndromic surveillance remains the best data analytical tool for epidemiologists and crisis management personnel at both state and federal levels to assess the levels of infectious disease in various regions, and to gauge the burdens placed on hospitals in those regions over time.

As I have commented previously, a main priority in crisis management is to follow all the available data, to make maximum use of all the information that is available. When it comes to information, too much is never enough, particularly in a crisis. Numbers of confirmed cases is a valuable data point, and no one should ignore those totals. Similarly, no one should ignore hospitalization rates and influenza like illness patient visits in a particular area.

What no one should do, epidemiologists in particular, is focus on testing to the exclusion of all other information. If anything, testing should be given secondary importance and primacy placed on the far more comprehensive (and already captured) syndromic surveillance data. In the final analysis, the real concern is not what disease sends people to the hospital, but how many people are being sent to the hospital.

Testing does not answer that question, nor will it.