{"pk":57199,"title":"[SoK] Evaluations in Industrial Intrusion Detection Research","subtitle":null,"abstract":"Industrial systems are increasingly threatened by cyberattackswith potentially disastrous consequences. To counter suchattacks, industrial intrusion detection systems strive to timelyuncover even the most sophisticated breaches. Due to its criticality for society, this fast-growing field attracts researchersfrom diverse backgrounds, resulting in 130 new detectionapproaches in 2021 alone. This huge momentum facilitatesthe exploration of diverse promising paths but likewise risksfragmenting the research landscape and burying promisingprogress. Consequently, it needs sound and comprehensibleevaluations to mitigate this risk and catalyze efforts into sustainable scientific progress with real-world applicability. Inthis paper, we therefore systematically analyze the evaluationmethodologies of this field to understand the current stateof industrial intrusion detection research. Our analysis of609 publications shows that the rapid growth of this researchfield has positive and negative consequences. While we observe an increased use of public datasets, publications stillonly evaluate 1.3 datasets on average, and frequently usedbenchmarking metrics are ambiguous. At the same time, theadoption of newly developed benchmarking metrics sees littleadvancement. Finally, our systematic analysis enables us toprovide actionable recommendations for all actors involvedand thus bring the entire research field forward.","language":"en","license":{"name":"Creative Commons Attribution-NonCommercial  4.0","short_name":"CC BY-NC 4.0","text":"Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.\n\nNonCommercial — You may not use the material for commercial purposes.\n\nNo additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.","url":"https://creativecommons.org/licenses/by-nc/4.0"},"keywords":[],"section":"Articles","is_remote":true,"remote_url":"https://escholarship.org/uc/item/43t0z18q","frozenauthors":[{"first_name":"Olav","middle_name":"","last_name":"Lamberts","name_suffix":"","institution":"","department":""},{"first_name":"Konrad","middle_name":"","last_name":"Wolsing","name_suffix":"","institution":"","department":""},{"first_name":"Eric","middle_name":"","last_name":"Wagner","name_suffix":"","institution":"","department":""},{"first_name":"Jan","middle_name":"","last_name":"Pennekamp","name_suffix":"","institution":"","department":""},{"first_name":"Jan","middle_name":"","last_name":"Bauer","name_suffix":"","institution":"","department":""},{"first_name":"Klaus","middle_name":"","last_name":"Wehrle","name_suffix":"","institution":"","department":""},{"first_name":"Martin","middle_name":"","last_name":"Henze","name_suffix":"","institution":"","department":""}],"date_submitted":"2023-10-31T13:35:28+05:30","date_accepted":"2023-10-31T13:35:28+05:30","date_published":"2023-01-01T05:30:00+05:30","render_galley":null,"galleys":[{"label":"","type":"pdf","path":"https://journalpub.escholarship.org/jsys/article/57199/galley/43396/download/"}]}