Cardwell, K. 2018. Diagnostic Assay Validation Terminology. The Plant Health Instructor. DOI: 10.1094/PHI-I-2018-0709-01
Validation of diagnostic assays refers to metrics and definitions that help frame the performance characteristics of the assay. Validation metrics are designed to understand how reliable an assay is under various conditions. Decisions about some of the validation metrics in an assay may be different for different purposes, depending upon factors such as the consequences for a false positive or negative test result. However, the research needed to fully validate an assay can take time, so there may be events where diagnosticians have to use an assay before all of the performance metrics are fully known. Diagnostic assay developers whether in the private sector or in a University lab, must be familiar with the terminology and statistics of diagnostic assay validation. Plant disease diagnosticians may find the following definitions helpful when describing their confidence in the outcome of an assay. The following glossary of terminology has been adapted from multiple sources for Plant Pathologists, and is considered a work in progress.
NATIONAL CLEAN PLANT NETWORK (NCPN): Created to protect U.S. specialty crops such as grapes, fruit trees, citrus, berries, roses and sweet potatoes from the spread of economically harmful plant pests and diseases, the NCPN ensures the global competitiveness of U.S. specialty crops by creating high standards for clean germplasm. http://nationalcleanplantnetwork.org NATIONAL SEED HEALTH SYSTEM (NSHS): A program authorized by USDA-APHIS and administered by the Iowa State University Seed Science Center to accredit both private and public entities to perform activities needed to support the issuance of Federal phytosanitary certificates for the international movement of seed. Through the NSHS, new seed health testing methods are incorporated into the accreditation program to maintain the program on the cutting edge of technology. www.seedhealth.org.
PREDICTIVE VALUE OF A POSITIVE TEST RESULT: # True positive/(True positive + False positive) x 100;
PREDICTIVE VALUE OF A NEGATIVE TEST RESULT: # True negative/(True negative + False negative) x 100; Note: positive and negative predictive values are influenced by the prevalence of disease in the population. At low disease prevalence the chance for false positive is higher, whereas for high disease prevalence, the chance for a true positive is higher. PERFORMANCE CHARACTERISTIC: An attribute of a test method critical to defining its performance. Characteristics can be analytical sensitivity and specificity, accuracy and precision, diagnostic sensitivity and specificity and/or repeatability and reproducibility. PRECISION: The degree of dispersion (such as variance, standard deviation or coefficient of variation) within a series of measurements of the same sample tested under specified conditions (ICH Q2, 2005). Low variance between test measurements indicates higher precision. PRESUMPTIVE POSITIVE: A sample that has tested positive with a screening method, but requires additional testing to complete the diagnostic determination. For regulatory purposes, presumptive positive samples require additional test results to authorize regulatory actions. PROFICIENCY TESTING: Measure of laboratory competence using inter-laboratory comparisons against pre-established criteria; implied in this definition is that participating laboratories are using the same test methods, reagents and controls and results are expressed qualitatively (ISO/IEC 17043, 2010). QUALITATIVE METHOD: Method that detects presence of an analyte based on chemical, biological, or physical properties. Some qualitative methods can be made to be “semi-quantitative” to provide rough estimates of amount present. QUANTITATIVE METHOD: Method that produces a specific measured quantity of an analyte, such as copies per reaction or colony forming units per milliliter. Quantitative methods must have a well characterized range supported by robust statistical analyses that brackets the action limit. QUANTITATION LIMIT: Lowest amount of analyte in a sample which can be quantitatively determined with suitable probability of precision and accuracy (ICH Q2, 2005). The quantitation limit is critical for measuring analytes in complex sample matrices, determining pass or fail decisions, and establishing rejection criteria for a test. Quantitative assays are commonly used for measuring analytes, impurities and/or degradation products (excipients). RANDOM ERROR: Irreproducibility in replicate measurements resulting from random changes in experimental conditions. Random error is the dispersal or variance of measurements collectively evaluated to characterize the precision of an assay. This is in contrast to systematic errors derived from innate error in the system, such as the measurement uncertainty of pipettes, balances, and cycling/thermoblock temperatures (GUM, 2008). RANGE: The range of an analytical procedure is the interval between the upper and lower concentration (amounts) of analyte in the sample for which it has been demonstrated that the analytical procedure has a suitable level of precision, accuracy and linearity (ICH Q2, 2005). REFERENCE LABORATORY: Laboratory of recognized scientific and diagnostic expertise for a particular plant disease or testing methodology; includes capability for characterizing and assigning values to reference reagents and samples. REFERENCE MATERIAL: Reagent validated to define its homogeneity, stability, and reproducibility to standardize or calibrate tests or equipment (VIM, 2007). Reference material should first be sought through an International Reference Laboratory, but can also be obtained from a laboratory that can demonstrate the material has been validated and determined fit for its intended use. REFERENCE STANDARD: Traceable item with a defined measurement uncertainty such as weights used to calibrate a weigh balance. Reference standards are metrologically traceable to a source such as NIST (VIM, 2007). REFERENCE STRAIN: a. A verified pure culture of a target organism (preferably housed in an established and recognized culture collection); b. Purified nucleic acids from a verified target organism or verified infected host for positive control in nucleic acid-based assays; c. Purified or expressed proteins from a verified target organism or infected host for positive control in immunoassays; and d. For Forensics, genomic and/or SNP information from a worldwide collection of isolates or strains of the target organism. (b. and c. for when the organism is either unculturable or difficult to maintain in culture). REPEATABILITY: Level of agreement between replicates of the same sample in the same exact conditions by the same operator, equipment and reagents. For example, the test repeated by analyst A on instrument ABC using reagent lot XYZ on the same day (VIM, 2007). REPRODUCIBILITY: Ability of a test method to provide consistent results for the same sample tested by the same method in different laboratories (VIM, 2007). RING TEST: Evaluation of assay performance or reagent integrity by two or more laboratories. One laboratory may act as the reference in defining test sample attributes, or a consensus of performance may be determined. ROBUSTNESS: Assessment of an assay or material to produce expected results when subjected to testing outside its verified range of use (ICH Q2, 2005). Changes in variables such as temperature, humidity, and stability can be observed to verify whether the assay or material will maintain its validated characteristics when mishandling occurs or if there is a moderate risk that test conditions cannot be adequately controlled. SCREENING METHOD: Method intended to identify, detect or measure the presence or absence of an organism in a sample as a preliminary assessment. Screening is also referred to as triage, to eliminate negative samples and move presumptive actionable samples forward for confirmation by a certified authority. SELECTIVITY: The capability to discriminate between the organism of interest and other organisms and components of the sample, such as host tissue. In binary analysis, selectivity is the equivalent of global accuracy taking into account all false reactions, both positive and negative. SENSITIVITY (ANALYTICAL): Synonymous with “Limit of Detection”; smallest detectable amount of analyte that can be measured with a defined certainty; analyte may include antibodies, antigens, nucleic acids or live organisms. SENSITIVITY (DIAGNOSTIC): Proportion of known infected reference samples that test positive in the assay; infected plants that test negative are considered to have false-negative results. SENSITIVITY (RELATIVE): Proportion of reference samples defined as positive by one or a combination of test methods that also test positive in the assay being compared. SPECIFICITY (GENERAL): A measure of the host range of a pathogen ranging from an extreme specialist for a single species or strain of its host to a generalist with many hosts ranging over several groups of organisms (ISPM 5, 2017). SPECIFICITY (ANALYTICAL): Analytical specificity refers to the ability of an assay to identify a specific organism or analyte, rather than any other. The higher the analytical specificity, the lower the level of false positives. SPECIFICITY (DIAGNOSTIC): Proportion of known uninfected reference plants that test negative in the assay; uninfected reference plants that test positive are considered to have false-positive results. STRINGENCY: Applying rigorous standards of performance and validity, such as: stringent laboratory controls; also, conscientious attention to rules and details. SYSTEMATIC ERROR: Measurement error innate in an instrument, reagent or testing a specific well-characterized sample matrix. Systematic errors collectively contribute to the minimum dispersion of measurements possible before proceeding to test the sample (GUM, 2008). THRESHOLD: Measured value distinguishing between negative and positive test results. The threshold may be adjusted to reduce the 5% uncertainty inherent in a system, under the assumption or evidence collected that sample results follow the standard distribution curve as concentration decreases. TRUENESS: Degree of agreement of the expected value with the true or reference value over a series of evaluations (GUM, 2008). VALIDATION: Process that determines the fitness for purpose of an assay, which has been properly developed, optimized and standardized, for an intended use. VERIFICATION: Confirmation by examination of objective evidence that specified requirements have been fulfilled.
Bustin, S. A., Benes, V., Garson, J. A., Hellemans, J., Huggett, J., Kubista, M.,...Wittwer, C. T. (2009). The MIQE Guidelines - Minimum Information for Publication of Quantitative Real-Time PCR Experiments. Clinical Chemistry, 55(4), 611-622.
GUM. (2008). ISO/IEC Guide 98-3:2008 Uncertainty of measurement -- Part 3: Guide to the expression of uncertainty in measurement. International Organization of Standards.
ICH Q2. (2005). Validation of Analytical Procedures: Text and Methodology. ICH Harmonised Tripartitie Guideline. International Conference on Harmonization.
ICH Q5. (2004). Comparability of Biotechnological/Biological Products Subject to Changes in their Manufacturing Process. ICH Harmonised Tripartitie Guideline. International Conference on Harmonization.
ISO 16577. (2016). ISO 16577:2016 Molecular biomarker analysis -- Terms and definitions. International Organization of Standards.
ISO/IEC 17043. (2010). ISO/IEC 17043:2010 Conformity assessment -- General requirements for proficiency testing. International Organization of Standards.