California’s self-driving car reports are public. Here’s what they don’t mean.

The California Department of Motor Vehicles released its annual cache of autonomous vehicle testing and disengagements data that, depending how one chooses to interpret the data, shows either stunning progress or stagnation.

The data, which every company testing autonomous vehicles on public roads in California must submit, tells a winding and sometimes contradictory tale of growth, consolidation and priorities. The total number of autonomous miles driven in 2019 rose 40%, to more than 2.87 million, thanks largely to a notable uptick in public on-road testing by Baidu, Cruise, Pony.ai, Waymo and Zoox, as well as newcomer Lyft.

And yet, the rise in total autonomous miles and permitted companies don’t tell the whole story. While the number of companies with testing permits grew to 60 in 2019, the percentage of companies actually testing on public roads fell to about 58%. In 2018, about 62% of the 48 companies that held permits tested on public roads.

Some companies scaled back public testing in California, either to move operations out of state or prioritize simulation. Aurora, for instance, saw its total on-road autonomous testing drop 59%, to 13,429 miles. Meanwhile, Aurora ramped up its simulation efforts, conducting more than 735,000 tests per day, an increase of over 100 times from 2018.

“While on-road testing is useful for collecting targeted data and performing late stage validation of self driving systems, we find that large-scale, on-road autonomous testing is a slow, and inefficient approach to development relative to more sophisticated, virtual techniques,” Aurora co-founder and CEO Chris Urmson wrote to the DMV.

Others, like Drive.ai, no longer exist. Two companies, Roadstar.ai and Ximotors.ai, failed to submit a disengagement report and have had their testing permits revoked.

The upshot: It’s not the who-is-winning-the-race narrative that many might expect or try to tell. Those kinds of rankings and comparisons are nearly impossible, for a number of reasons, including the fact that testing on public roads is conducted in areas with varying degrees of complexity. Additionally, companies aren’t required to report testing on private roads or tracks, out of state or in simulation, all of which provides a better assessment of an AV developer’s technology.