Field Performance of New Methane Detection Technologies: Results from the Alberta Methane Field Challenge

Emerging methane technologies promise rapid and cost-effective methods to measure and monitor methane emissions. Here, we present results from the Alberta Methane Field Challenge – the first large-scale, concurrent field trial of eleven alternative methane emissions detection and quantification technologies at operating oil and gas sites. We evaluate new technologies by comparing their performance with conventional optical gas imaging survey. Overall, technologies are effective at detecting methane emissions, with 8 out of 11 technologies achieving an effectiveness of approximately 80%. Importantly, results highlight the key differences in technology performance between those observed at controlled release tests versus those in field conditions. Intermittent emissions from tanks substantially affects detection and site-level quantification estimates and should be independently monitored while assessing technology performance. In this study, all technologies improved their effectiveness in detecting tank emissions when intermittency was considered. Truckand plane-based systems have clear advantages in survey speed over other technologies, but their use as effective screening technologies to identify high-emitting sites rests on their quantification effectiveness. Drone-based technologies demonstrated higher effectiveness than other technologies in identifying quantification rank compared to baseline OGI-based survey. Overall, quantification under in-field conditions is affected by several exogenous factors such as temporal variation in emissions and changing environmental conditions. We recommend that assessment studies of new methane detection technologies at oil and gas facilities include comprehensive, continuous, and redundant emissions measurement. Non-peer reviewed pre-print submitted to EarthArXiv


Introduction
Methane emissions across the oil and gas supply chain erode the potential climate benefits of 2 using natural gas over other carbon-intensive fuels such as coal [1]. The Intergovernmental Panel 3 on Climate Change (IPCC) in its recent report on 1.5°C of global warming highlighted the 4 importance of reducing short-lived greenhouse gases such as methane [2]. Methane, the major 5 component of natural gas, has a significantly higher global warming potential than carbon 6 dioxide. Recent research has shown that despite their short atmospheric lifetime, methane 7 emissions can contribute to decades of future sea-level rise [3]. Locally, reducing methane 8 emissions also reduces emissions of volatile organic compounds from oil and gas (O&G) 9 operations and improves air quality [4]. Beyond these local and global impacts, several recent 10 field campaigns to measure methane emissions have demonstrated a consistent underestimation 11 in official GHG inventories [5]- [8]. These discrepancies further underscore the need for   Recently, several new methane emissions detection technologies that promise faster and more 28 cost-effective leak detection than existing approaches have been developed [19]. These 29 technologies include continuous monitoring systems, mobile sensors mounted on drones, trucks, 30 and planes, handheld sensors, and satellite systems [20]. Most of these technologies are not 31 currently approved for use in regulatory LDAR programs. To enable widespread deployment, the 32 efficacy of new technologies must be validated through rigorous testing, modeling, and field 33 trials. 34 Field studies have been conducted as part of recent methane measurements campaigns. Mobile 45 truck-based platforms were deployed in British Columbia and Alberta to measure site-level 46 emissions, while plane-based systems were used to detect site-and basin-level emissions in the 47 [31]. More recently, scientists deployed drone-based systems for methane detection and 48 quantification at O&G facilities [29], [30], [32]. Finally, satellites have been used to study 49 regional and global methane emissions from anthropogenic and biogenic sources, and to identify 50 high-emitting methane sources associated with O&G activity [33]- [40]. However, despite the 51 use of alternative technologies in scientific studies for measuring methane emissions from O&G 52 operations, there has been no systematic field test of their performance. 53 In this paper, we report results from the Alberta Methane Field Challenge (AMFC)a large-54 scale, concurrent field trial of alternative methane emissions detection and quantification 55 technologies at operating O&G sites. We tested twelve different technology teams, including 56 fixed continuous monitoring systems, handheld devices, truck-mounted, drone-mounted, and prior testing experience, and deployment and scalability. In addition, the number of teams using 70 a specific platform (e.g., drone, truck, plane etc.) were also limited by the logistics of organizing 71 a safe, large-scale, blind, and concurrent field campaign. In all, 40 technologies applied to 72 participate, of which 12 were selected. A summary of the participating technology teams 73 (hereafter referred to as teams) is given in Table 1. The AMFC campaign was held in two phases 74 phase 1 and 2with truck teams participating in both. Detailed technical specification about 75 each participating team is provided in SI section 2. The fixed sensor analysis is included in SI 76 section S3 and not in the main text due to the nature of analysis required as compared to other 77 teams which participated in the AMFC. The Heath team did not report quantified emissions rates 78 or emissions attribution, and the analysis in the SI has been conducted by the authors of this 79 paper. To further improve our understanding of measurement uncertainty in QOGI-based quantification 188 estimates, we use Monte-Carlo analysis to estimate error as a function of sample size (SI section 189 4.1). Using a bootstrapped sampling technique (with replacement) and 10,000 Monte-Carlo 190 realizations, we find that the 5 th and 95 th percentile of the sample mean are -23% and +26%, 191 respectively, for a sample size of 50 (SI Figure S7). Similarly, at a sample size of 20 emissions -192 typically seen in production sitesthe 95% confidence bounds of the average emission rate is -34% and +39%. Thus, it is critical for QOGI measurements to be interpreted in an aggregate 194 context, as individual measurements can have higher error rates as shown in Figure 1

203
In this section we present results from both phases of the AMFC. A few caveats will help in 204 interpreting results.     Table 2 shows a summary of the site-level performance of the participating teams. The 221 comparison with baseline OGI survey is only made at overlap sites, which is limited by the 222 survey speed of the OGI team (3-6 sites/day). We make several important observations. Photonics, only 'tier-1' emissions -where the technology was able to localize and quantify 235 methane plumes were considered, leading to a 43% detection effectiveness. In addition, Bridger also identified 'tier-3' emissions that correspond to plumes that were observed but too weak to 237 localize or quantify. Including these 'tier-3' emissions, the detection effectiveness increased to 238 90%. However, 'tier-3' emissions detections cannot be used for follow-up emissions mitigation 239 action as the weak plumes could not be localized. Similarly, although Sander Geophysics' 240 detected emissions at 77% sites found by the OGI crew, they were only able to quantify 241 emissions from four sites because of unstable wind conditions.

242
Second, survey speed varied from 3 sites/day for Tecvalco to 15 sites/day for Altus Geomatics, 243 indicative of the range of survey methods employed. On average, aerial and truck-based systems 244 that measure at the site-level are at least three to five times faster than the baseline OGI survey.

245
For all technologies, survey speeds as part of an LDAR program deployment can be expected to 246 be somewhat higher than those observed in this study because of artificial constraints that  For most participating teams, the average baseline OGI site-level emissions rates were higher at 299 sites where the teams' also detected emissions compared to sites where the teams did not detect   Table 3 shows the detection effectiveness across five major equipment types at overlap sites for 310 teams that detected equipment level emissions. We make several observations.     Drone-and plane-based teams are reasonably effective at estimating the rank-order of site-level 436 methane emissions. Aerometrix and SeekOps demonstrated an accuracy of 57% and 63%, 437 respectively. While the overall correlation between the drone teams and OGI ranking is only moderate (Pearson's correlation coefficient 'r' = 0.5), both teams correctly identified 60% of the 439 top 10 highest and lowest emitting sites in comparison to OGI. Bridger was accurate for three of 440 the seven quantified overlap sites. However, this effectiveness could change as the two plane 441 teams have a relatively small sample size: Bridger with 7 and Sander with 2 sites.

442
The truck-based teams were between 39-55% accurate across both phase 1 and phase 2 443 campaigns with a relatively low correlation between their ranks and OGI ranks: Altus with r = 444 0.3 and r = 0.28 for phase 1 and 2 respectively, and UofC with r = 0.3 and r = 0.17 for phase 1 445 and 2, respectively. Tecvalco reported quantification data for component-level sources from 10 446 sites that were accessible and safe to measure. Of these, Tecvalco was within 20% of OGI ranks 447 for 7 sites, where it identically ranked the top two emitting sites similar to OGI.     and logistical considerations to allow for a safe, concurrent, large-scale field trial. Thus, the 10 number of teams that could be selected for the field campaign with similar platforms (e.g., aerial 11 systems) were limited by safety considerations, irrespective of the outcome of their evaluation.

S.1.2 Field campaign 13
The information in this section is provided to assist in the development and execution of future 14 field campaigns. It includes details on team orientation, in-field communications, site scheduling, 15 and data integrity and handling procedures.

16
Orientation: A day long orientation was mandatory for participating teams in both AMFC 17 campaigns. The orientation included information on site permitting, environmental health and 18 safety certifications, emergency protocols, appropriate safety gear, and site-access agreements 19 for participating teams to get on producing oil and gas sites for measurements.  Coordination of site visits between the OGI crew collecting baseline emissions data and 33 participating teams was essential to compare data. Researchers prepared a detailed field schedule 34 for each day of the campaign, which accounted for several factors including travel time between 35 sites, survey speed, and site size. Every morning, the participating teams attended a briefing led 36 by the researchers and the OGI team to go over field schedule, plan the order of team site visits, 37 and discuss any issues that might have come up on prior field days. Each day, the two OGI teams 38 surveyed a pre-selected list of 3-6 sites from the field schedule that were 'mandatory' for all 39 teams to visit on the same day. Teams could survey additional sites after completing surveys at 40 the mandatory sites. teams were asked to also submit their typical field survey data reports as would be provided to 45 customers, along with any supporting evidence for the data (e.g., kmz files or images and videos 46 etc.). The teams also submitted daily field logs with information on survey times, sites visited, 47 and any safety or logistical observations which could be useful for other teams visiting those 48 sites.  Table S1 shows the technology specifications for each participating team as reported in their process. Given that the main goal of the study is to compare the performance of technologies under pre-determined criteria, a thorough understanding of the underlying sensor is not essential. 56 Regulators, operators, and other academics may find the data in this study valuable in making 57 decisions about technology choices, irrespective of specific sensors used by the participating 58 teams. However, specifications for each technology may differ based on more recent testing.

59
In addition to information on the team, technology performance, testing data, and business 60 information, data on the following critical parameters were requested.   inlet port mounted externally on the survey aircraft and pumps the air through a particle separator filter to an off-axis integrated cavity output spectroscopy (OA-ICOS) analyzer to 20 Hz (1 Hz commonly used). The initial processing results in a measure of the methane concentration enhancement along the flight lines. Inversion is then performed to calculate the ground locations and mass emission rates of sources 500' and 1000' above ground level (AGL), ideally conduct surveys at 500' AGL. In survey configuration, Sander estimates a lower detection limit of approximately ~30kg/hr for a source at ground level when flying at 500' AGL Tecvalco Ltd. Hand-held Component Tunable diode laser absorption spectroscopy Gas-Trac LZ30fixed at 100ms Gas-Trac LZ50 -fixed at 100ms Gas-Trac FPL -collected at 1Hz. Data output is adjustable: 1 second, 10second average, and 1minute average.

University of Calgary
Truck Equipment Open-path wavelength modulated spectroscopy System collects data at 10 Hz as the vehicle drives The methane sensor has no absolute minimum or maximum detection limit and can function from 0.0 ppmv to approx 40.0 ppmv, at which point it 'ranges out' and no longer produces accurate information The dynamic range of the sensor is not well       The NTOC staff and researcher on site monitored flow rates continuously during all releases, and 166 data were logged electronically at a 15-second interval. These release rates were chosen to mimic 167 both equipment-level and site-level emissions typically observed at operating oil and gas sites.

168
The rates were based on the frequency for various release classes recorded by the OGI crew in 169 the June AMFC campaign. The test rates spanned three orders of magnitude as shown in Figure   170 S4 and ranged from a low of 30 standard cubic feet per hour (scfh) to a maximum of 2000 scfh. The distribution of controlled release rates was such that only 20% of all releases were greater 175 than 1000 scfh. These release rates were further randomly assigned to two different release  Each team participated in 3 to 5 controlled releases per day. Teams were asked to start measuring 180 when the release rate was stable and had a maximum of 15 minutes for quantification per release.

181
Multiple teams could measure a release simultaneously provided they did not interfere with each 182 other (e.g., a ground team and an aerial team). The CRT was conducted in a single-blind manner, 183 with only the researchers and personnel from NTOC aware of the release rates.   Effect of Thermographer Operation: Figure S6 shows the CRT results for the two OGI field 212 teams. We find that, on average, there are differences between the two field teams where team 1 213 demonstrated a parity chart regression slope of 0.89 compared to team 2 with a regression slope 214 of 0.76. However, the overlap in the 95% confidence intervals of the two crews indicate that this 215 difference might not be statically different. Even so, recent studies with several OGI camera 216 operators conducted at the METEC test site in Colorado have demonstrated that operator 217 experience plays a role in the effectiveness of leak detection [57]. We note that the two teams did 218 not measure CRT simultaneously and thus the difference might also be attributable to differences 219 in atmospheric conditions. Figure S10: Parity chart of CRT emission rate and QOGI measured emission rate as a function of the two 222 field crews that were deployed as part of the baseline OGI team in AMFC phase 2. 223 Uncertainty analysis: Figure S7 Figure S9 shows the quantification accuracy parity charts between controlled release rates and 251 estimated rates by the participating teams. Only teams that quantified emissions rates during the 252 controlled release testing are included in this analysis.  Emissions quantification is a challenging problem influenced by several factors such as 263 atmospheric conditions, data processing algorithms, instrument sensitivity, survey method, and 264 gas composition. Therefore, it is not possible to attribute observed differences in these limited 265 controlled release tests to any specific influencing factor. However, we note that in both the   Figure S10 shows the flow rate quantification accuracy at the site-level as a parity chart of OGI 294 site-level emissions estimate and the estimates from participating teams for overlap sites.

295
Discussion of the results of Figure S10 have also been included in the main text where relevant.

296
If the quantification measurements between OGI and the participating team were identical, the  All teams significantly underestimated emissions by over 60% compared to QOGI-based 319 estimates, except for the UofC team in phase 2. Altus underestimated emissions on average 320 between 75% and 92%, while UofC underestimated on average between 17% and 58%. In  Tecvalco only measured component-level emissions that were accessible thus site-level 327 aggregation will necessarily be lower than that determined by OGI.

328
If we consider those sites where QOGI estimated emission estimates are less than 350 scfh, 329 quantification accuracy for most teams, as measured by the slope of regression, increases (Table   330   S2). According to QOGI estimates, 75% of all site-level emissions were below 350 scfh across 331 both phases.  Figure S11 shows the fraction of total emissions detected by a team as a function of the size-

351
In this section we discuss some of the challenges faced during AMFC field work and potential 352 solutions that can be implemented in future field campaigns.