Everyone wants to know where the new audit program stands and when audits will be available. I know there are still ongoing discussions within the committee on how the audits should be handled and what acceptance criteria should be applied. Until there is a final decision on how the audits will be performed, there can be no determination on when they will be available. I do know the cost could be significant for each of the audit cylinders required.
Some of the uncertainty involves what will and will not be included in the cylinder and why. This is tied to deciding what level of acceptance criteria would be required based on the historical data available at this time. The problem is the number of changes which have been undertaken since the audit program was started more than two decades ago and the lack of correlation as to how those changes improved the pass/fail rate over that time.
Beginning with the audit data Dr. Jayanty published in his 1990 paper as a base line and comparing it to the information from the last few years of the audit program, we can see how things have improved significantly.
In the 1990 data, there were 45 audits in total, ranging from a low of 96.1 ppmC to a high of 1982 ppmC. These audits were a mixture of different organic compounds, along with a low percent level of CO2. This was prior to the establishment of TES, and within the period where there seemed to be some significant problems. This is supported by the fact 17 of these 45 audits failed the criteria of + 20% between the reported and known concentrations in the audits. That is a 37.8% overall failure rate, but not the worst of information. Of the 7 audits at or below 100 ppmC, all failed the criteria. Of the audits at or below 150 ppmC, 7 of 8 (87.5%) failed. Of the audits at or below 200 ppmC, 12 of 24 (50%) failed. There were even 4 failures in the 800-1000 ppmC range, which were most likely due to gross contamination issues. The lower levels, of course, had higher failure rates because the allowed range is naturally smaller and the problems have a larger impact at the lower levels.
Now compare that data to the audits from the most recent program in the < 150 ppmC range. Out of 157 audits from 46 - 150 ppmC, 65 failed the same criteria of + 20% This represents a 20 fold increase in the number of audits in this concentration range, but a 90 fold increase in the number of passing audits. The failure rate drops to 41.4% for this new set, from 87.5% for the older set. The differences in QC for the equipment and analysis instituted in the interim had a lot to do with the improvement as I noted in the newsletters in the later 1990s, but since there was also CO2 added to the older set and not the newer set, the value attributed to that particular variability is not known at this time. Of the very low end audits, which I consider between 46 and 60 ppmC, only two of the 11 passed the + 20% criteria, and the lowest which passed was a 50.4 ppmC. I remember getting the phone call when that audit was confirmed as passing because I was so sure there would not be a real audit that close to the use limit. I was pleasantly surprised to find out there was an audit at that level and that it had passed. There was another at that same concentration level which failed by 2.62%, which seems to support the idea the “single use” audit cylinders may have been reused, since such a duplication is not very common in the rest of the lower level data. According to the data, there were also 4 of these audits with a 0 ppmC reported concentration, which would seem to be an error of some sort, as there should be something reported as a concentration. There would also be several audits which failed due to audits being taken after samples, which had contaminated the sample trains. These obvious problem audits were not removed from the totals presented above because there was no way to document the situation surrounding these audits. If the more obvious outliers such as the 0% or 1000+% recoveries are eliminated, the fail rate drops to 36.1%. There were 12 audits reported failing at >200% and 16 audits reported as failing below the 80% recovery range. There were 17 more reported in the range between 140% and 200% recovery, which would leave only 16 possible “close” high bias failures with the 92 passing audits.
One of the concerns raised was blank subtraction, which is not allowed for the samples and also not being considered for audits. Nevertheless the concept was reviewed, and some surprising information resulted from that review. Since there is no way to know what level of blank one might see, varying levels were chosen and all of the audits were adjusted accordingly and the impact determined. The subtraction of a blank of 10 ppmC, which would be a reasonable level assumed for any blank in our experience, slightly increases the fail rate from 41.4% to 42.0% for the audits < 150 ppmC. With our MDL and a minimal sample volume relating to the 60 cc/min for an hour sampling rate, the MDL adjusted for the sample would approach 10 ppmC. If one increases the blank subtraction, the fail rate is also increased for these samples. For the blank subtraction of 15 ppmC, the fail rate jumps to 50.3%, and for the blank subtraction of 20 ppmC, the fail rate moves up to 100%. Thus, it shows there is not a very large difference in the concentration range of those which passed, but a much larger range for most of those which failed. This means a blank subtraction would more likely make the failure rate worse than better. This confirms something I had suspected from seeing several sets of very low sample concentrations compared to the average blank analyses. The samples were consistently lower than the blanks, which indicated the blank concentration was not in addition to the low level samples, but possibly instead of that impact. If the obvious outliers are removed, the fail rate still increases with each level, to a failure of 100% at the 20 ppmC subtraction level.
The one positive aspect of the new audit program is that we plan to provide an audit collection manifold, including a regulator for each audit to be taken, as part of our rental equipment. This will help to standardize the way audits are collected and ensure everyone can meet the instructions for collecting the audits in the future.
Triangle Environmental Services, Inc.