Sensitivity Part III: Probability of Detection, The Value and Limitations of Controlled Release Testing, and Why Minimum Detection Limit Is Insufficient
In our previous Detection Sensitivity Part I and Part II articles, we dove into what a detection sensitivity is and why it’s important. Here, we explain more on why a probability of detection (PoD) is a critical component for describing a detection sensitivity, how it’s determined, and how it differs from a minimum detection limit.
The “emission rate detection sensitivity,” or how small of a leak a technology can detect, is perhaps the most important performance metric available to gauge the effectiveness of a methane detection solution. After all, you can’t fix what you don’t detect. And more importantly, you can’t develop a robust strategy for emissions reduction without a comprehensive view of emission sources across an asset portfolio. But specifying, or interpreting, a detection sensitivity isn’t as simple as it sounds!
Here's the problem: Say a technology provider states that their detection sensitivity is 10 kg/hr. If you’re like most people, you’d interpret that to mean that the technology will detect leaks with an emission rate of 10 kg/hr or greater, and miss leaks smaller than 10 kg/hr. Sounds reasonable, right? Unfortunately, describing the ability to detect leaks is not so clear-cut. Here’s why:
- Leak detection is statistical and probabilistic, so a leak of a given size might be detected on a first scan, yet the same leak could be missed on a second scan. The likelihood of catching (or missing) a leak of a given size is captured by including a PoD in the specification of detection sensitivity.
- Results from controlled release studies are a great first step to assess a technology’s detection sensitivity and PoD, but they are usually only indicative of the technology’s performance under “ideal” conditions. There are many operational and environmental parameters that can cause a technology’s detection sensitivity to be poorer when deployed in the field.
- Sometimes an emission rate, say 10 kg/hr like the example above, is described as a “minimum detection limit,” or MDL. This isn’t a helpful descriptor since the minimum leak size a technology can detect is dependent on operational and environmental conditions, and typically represents an extremely rare event.
Let’s dive into these issues in more detail.
Emission Rate Detection Sensitivity Means Little without a Probability of Detection
Simply stating an emission rate (e.g. 10 kg/hr) is insufficient to specify an emission rate detection sensitivity because detecting leaks is statistical and probabilistic. This means there is a chance that a technology might detect one leak on one scan, and miss a leak of the same size on another scan. We can capture this statistical nature by including a PoD for a given emission rate, stated as a percentage. For an emission rate of 10 kg/hr, there is a huge difference between detecting that leak size 1% of the time (1% PoD) versus 90% of the time (90% PoD).
An example of the use of PoD comes in the EPA’s November 2022 supplemental proposal for new methane standards. One of the proposed detection tiers for a methane detection technology requires a detection sensitivity of 10 kg/hr with 90% PoD. This means, that to meet this requirement, a technology must detect, statistically, at least 9 out of 10 emitters with an emission rate of 10 kg/hr. Emission rates larger than this (e.g. 25 kg/hr) will be detected with an even higher probability (i.e. more than 90% of the time). And plenty of emitters smaller than 10 kg/hr will also be detected, but their detection becomes less likely as the emission rate becomes smaller. By choosing 90% PoD, regulators and operators have high confidence that an emitter of that size will be detected.
How Is Probability of Detection Determined in Practice?
PoD is most often determined by measuring repeated controlled releases at different emission rates. As a real-life example of how PoD is determined, the plot below is from a study of Gas Mapping LiDAR (GML) performed by Colorado State University’s Methane Emissions Technology Evaluation Center (METEC) and the Stanford Natural Gas Initiative, where they recently assessed detection limits and highlighted how PoD is determined.
The black dots clustered on the top of the graph represent controlled releases that were detected (i.e. true positives) and are assigned a value of one. Those on the bottom axis represent the controlled emissions that were missed (i.e. false negatives) and are assigned a value of zero. Data was taken at Bridger's standard operational AGL in the production field. The data shown was scaled by wind speed to remove that dependence. Based on those “hits” (detected) and “misses” (not detected), a mathematical regression was performed to generate the blue PoD curve. The curve tells us what the probability of detection is for a given emission rate with a 1 m/s wind speed. From the curve, 0.9 on the left axis represents a 90% PoD, which occurs for an emission rate of 0.41 kg/hr per m/s wind speed on the bottom axis. So, if the average wind speed in the Permian Basin is 6 m/s, then you would expect the technology to be able to typically achieve a detection sensitivity of 2.46 kg/hr [=0.41 kg/hr/(m/s) * 6 m/s] with 90% PoD, assuming a linear dependence on windspeed.
Additionally, a recent study by the Energy and Emissions Research Laboratory (EERL) at Carleton University developed a more generalized mathematical formula for establishing a PoD, based on controlled releases The study indicates the assumption of linearity with wind speed isn’t perfect, and also incorporates flight altitude dependence into the determination of PoD.
All of that said, while controlled release studies facilitated by an independent third-party can be hugely valuable for getting a solid baseline understanding of PoD, it’s how a technology performs in the field that really matters. There are many operational and environmental parameters that can cause a technology’s detection sensitivity to be poorer than what is reported based on controlled releases when the tech is deployed across the full spectrum of conditions in the field.
And What about “Minimum Detection Limits” or MDLs?
Occasionally, you’ll find a technology provider state a “minimum detection limit,” or MDL. When detecting methane leaks, the MDL represents the smallest leak detected by the technology, regardless of the probability of the detection (i.e. it could be an extremely rare detection event enabled by “perfect” conditions). Specifying leak detection sensitivity with an MDL doesn’t capture the statistical and probabilistic nature explained above and provides little information regarding how a technology will perform. Does a detection event at the MDL represent a 1% PoD (one out of one hundred) or a 0.0001% PoD (one out of one million)? If just an MDL is reported, we don’t know what to expect as far as likelihood of detecting leaks of a given size. We discourage the use of MDLs to describe a methane leak detection technology, and instead encourage the use of emission rate detection sensitivity with an accompanying PoD.
Takeaways
Emission rate detection sensitivity with a PoD is perhaps the most important metric available to describe the performance of a methane detection solution.
Simply stating an emission rate or an MDL as a detection sensitivity doesn’t provide enough information about how a technology will perform. Without the PoD, no one knows whether the technology detects a given size leak 9 out of 10 times or one out of a million! When assessing a methane detection technology, be sure to ask the technology provider for this information. Here’s a guide for this and other important questions to ask. We discourage reliance on an MDL as a metric for determining actual emissions detection performance.
PoD can be characterized by reputable independent third parties (e.g. METEC) through controlled release studies. Most often, the results are normalized to a wind speed of 1 m/s. This can be handy for making apples-to-apples comparisons between technologies, but remember to multiply by the anticipated field wind speed (in the same units) as an initial estimate of what performance to expect. Also, beware that controlled release studies are often performed under ideal conditions, or under a very limited set of environmental conditions. This means that field performance can diverge dramatically from controlled release results. Read our next blog article in this series to understand how to navigate this issue.
To learn more, check out our Detection Sensitivity Part I and Detection Sensitivity Part II articles.