SIR #: 10012
Standard: 2003 NELAC
Link to relevant standard
1. ILAC Guide 13 in section 184.108.40.206 requires the PT provider to have procedures for dealing with small data sets that may be inappropriate for statistical evaluation. APG has protocol in place for all non-NELAC PT programs that deals with this issue. However, in the case of the NELAC PT program, APG feels strongly that since NELAC evaluation limits are regulatory and are written into State laws that we have no option but to apply the NELAC FOT requirements as written without exception regardless of sample size.
However, the A2LA auditors are requiring us to use an alternative evaluation technique based upon our own technical judgment, or prior studies on a case by case basis. While is would be simple to implement a criteria based upon professional judgment it would raise issues of objectivity. Such a procedure would lead to variability in laboratory evaluations, and be in conflict with the NELAC level playing field concept. Such practices would lead to arbitrary and inconsistent evaluations. It would furthermore transfer responsibility for setting laboratory evaluation criteria to the PT provider and removes it from the NELAC PT Board who are responsible party.
The NELAC 2003 Standard in Chapter 2 Section 2.6 says: "PT providers shall evaluate results from all PT studies using NELAC mandated acceptance criteria described in Appendix C." It continues: "The PT Board shall provide, and update as necessary, the data acceptance criteria that all providers shall use for all PT studies". Based upon this section APG believe that ILAC Guide 13 Section 220.127.116.11 is not relevant to the NELAC program until the NELAC PT Board provides the necessary acceptance criteria.
TNI FINAL RESPONSE:
(Proficiency Testing Board / NELAP Board, 10-11-09)
The TNI PT Board thinks that the acceptance criteria listed in the various Fields of Proficiency Testing Tables should be adequate to meet ILAC G13 requirements in most cases. For those analytes where the acceptance criteria are based on fixed limits or upon regression equations, these limits and criteria are based on aggregate PT data spanning several years from multiple PT providers.
Of course, the NELAP Program requires PT results to be scored acceptable or unacceptable based on these published limits. If the number of participants in the PT study is small, the acceptance limits published in the Tables still need to be used. However, since these limits are based on the aggregate scientific and statistical analyses, the TNI PT Board thinks that using these limits would satisfy ILAC G13 requirements for small data sets. The PT Provider should not have difficulty using this as a justification, and this justification should carry more tangible, defensible weight compared with any other alternatives that could be considered.
Nevertheless, there are Fields of Proficiency Testing where the acceptance limits are still based on consensus participant mean and a PT-study specific standard deviation. In these cases, the PT provider would definitely need to formulate an alternate procedure to handle small data sets. However, the TNI PT Board cannot really provide or advocate a specific protocol to use in these instances. In fact, it may be scientifically unsound to do so, since other procedures and statistical models (e.g., Lorentzian, Maxwellian, chi-squared, or Poisson, as opposed to Gaussian) may work better. In addition, the PT Provider may need to adapt or change models and procedures used to accommodate individual circumstances for a given PT study.
The TNI PT Board thinks the important thing to do is to document the preferred procedure(s) chosen (to satisfy ILAC G13), implement this procedure for the small data sets as needed, and be prepared to revise the SOP if the results do not work out as expected.