How much of a machine-learning earthquake catalogue can an expert verify?

Lingsen Meng , Hui Huang, Jinzhi Ma, Yang Ma

Machine-learning systems are becoming the instruments that generate the primary data of a scientific field, and what such a system asserts is not the same as what an expert would endorse. Earthquake detection is one of the few settings where the difference can be measured, because a detection can be checked blind against the waveforms that produced it. Four seismologists returned 4,800 blind verdicts on 1,200 detections from six catalogues. Of the events they were willing to judge they confirmed 60 to 98%, and three catalogues of the same Ridgecrest sequence, judged from the same waveforms, differed by ~20 points. A score fitted to seven quantities a catalogue already records reproduces the panel, transfers to a sequence it was not trained on, and ranks the machine-only additions lowest. Measuring this link, and re-measuring it as the systems change, keeps human judgement inside the scientific record rather than upstream of it.