Forensic Metrology: Its Importance and Evolution in the United States

Forensic measurements play a significant role in the U.S. criminal justice system. Guilt or innocence, or the severity of a sentence, may depend upon the results of such measurements. Until recently, however, forensic disciplines were largely unaware of the field of metrology. Accordingly, proper measurement practices were often, and widely, neglected. These include failure to adopt proper calibration techniques, establish the traceability of results and determine measurement uncertainty. These failures undermine confidence in verdicts based upon forensic measurements. Over the past decade, though, the forensic sciences have been introduced to metrology and its principles leading to more reliable measurement practices. The impetus for this change was driven by many forces. Pressure came initially from criminal defense lawyers challenging metrologically unsound practices and results relied upon by government prosecutions. Litigation in the State of Washington led this movement spurring action by attorneys in other jurisdictions and eventually reform in the measurement practices of forensic labs around the country. Since then, the greater scientific community, other forensic scientists and even prosecutors have joined the fight. This paper describes the fight to improve the quality of justice by the application of metrological principles and the evolution of the field of forensic metrology.


Beginnings
Forensic measurements play a prominent role in the criminal justice system. They are relied upon to investigate and prove charges ranging from murder to simple traffic offenses. Some crimes and punishments, for example driving with a breath alcohol concentration (BrAC) in excess of a prohibited level, even require forensic measurements to be established. As ubiquitous as forensic measurements are, though, historically they have often been poorly understood and misrepresented by forensic and legal professionals alike. This undermines the ability of fact finders to properly weigh such evidence and the public's confidence in verdicts based upon it.
The fight to change this state of affairs through the imposition of a complete and coherent metrological framework upon forensic measurements was taken up by attorneys in Washington State defending individuals charged with the crime of Driving Under the Influence of Alcohol and Drugs (DUI). DUI defense was a natural place for such a movement to take hold as almost all DUI prosecutions include the results of forensics measurements as evidence. These include breath and blood alcohol test results as well as measurements of the amount or concentration of drugs in an individual's blood. The lion's share of the litigation undertaken in this cause centered on BrAC tests as they are by far the most frequent forensic measurement performed.
Breath alcohol tests are used in the investigation and prosecution of DUI throughout the United States. Their purpose is to measure either an individual's breath or blood alcohol concentration. An important element of many breath test machines is a simulator. A simulator is a device that is filled with an aqueous-alcohol solution, referred to as a simulator solution. The simulator heats the solution to a specified temperature producing a vapor of a specified alcohol concentration. This vapor is used to both calibrate breath test instruments as well as to check their accuracy during the performance of a breath test. In order to ensure that the vapor created is the correct concentration, a thermometer is used to monitor the temperature of the solution it is a product of.
In early 2001, the Washington State Toxicology Lab claimed that the accuracy of the temperatures reported for the solutions were determined by a specified "margin of error" attributed to the thermometers used to measure them. An investigation by the defense turned up several problems with the State's claim, however. First, the uncertainty of the temperatures measured by the thermometers used was found to be significantly greater than the margin of error that had been claimed. Second, the thermometers themselves were not being used in a manner consistent with their validation rendering any values reported unreliable even had the claimed margin of error been correct. After a day-long hearing wherein these issues were addressed, a Court suppressed the breath test results. [1] The victory wasn't about "just trying to get another guilty person off" as is so often lamented by critics, though. It was about preventing the government from using flawed science to deprive Citizens of their liberty. Every one of us is innocent until proven guilty. That is one of the safeguards against tyranny provided by our Constitution. When the government tells a judge or jury that science supports claims that it does not, it is tantamount to committing a fraud against our system of Justice. It doesn't matter whether the deception is purposeful or not because the result is the same: a Citizen's liberty is imperilled by a falsehood.
The government sought to fix the problems that had been encountered in the use of the thermometers by requiring that the temperatures measured by such thermometers must, going forward, be traceable to standards maintained by NIST. Unfortunately, the government neither knew the definition of traceability nor understood how to establish it. The Defense Bar put an end to this bad government science by showing what was required to establish the traceability of the temperature reported by these simulator thermometers. The Washington State Supreme Court suppressed breath tests state-wide in the first published decision explicitly recognizing metrology in a forensic context. In doing so it enunciated a clear and common sense principle: "If the citizens of the State of Washington are to have any confidence in the breath testing program, that program has to have some credence in the scientific community as a whole." [2] 2. Not Just Bad Science, Bad Tribunals But bad government science doesn't necessarily arise from bad government scientists. Nor is the desire to ensure that science is used correctly to discover truth in the courtroom confined to defense attorneys. Forensic scientists, prosecutors and judges have sought the same goals.
In 2004, the head of the Washington State Breath Test Program helped the defense to keep the government from administratively suspending a women's driver's license based upon a misleading test result. The woman had submitted to a breath test that yielded duplicate results both in excess of the legal limit. Through the State expert, the defense was able to show that the uncertainty associated with the results proved that there was actually a 56.75% probability that her true BrAC was less than the legal limit.
The bad government science in this case was not that done by forensic scientists. To the contrary, it was one of the State's top forensic scientists who used metrology to establish that this motorist had most likely not violated the law. Rather

Metrology: A Tool for the Critical Analysis of All Forensic Measurements
Measurements involving the determination of a person's breath or blood alcohol concentration are quite common because the crime of DUI is defined by the results of these measurements. They are by no means the only type of forensic measurements, though. Determining the weight of seized drugs using a scale; the speed of a motor vehicle using a radar; the angle at which a bullet entered a wall using a protractor; and even the distance between a drug transaction and a school using a measuring wheel; these are just a few of the many types of forensic measurements that are performed. The same underlying metrological principles that allowed for the analysis of the breath alcohol measurements discussed above apply to every other forensic measurement as well. This leads to an astonishing conclusion. Since the science of metrology underlies all measurements, its principles provide a basic framework for the critical evaluation of all measurements, regardless of the field they arise out of. Accordingly, given a familiarity with metrology, scientists and police officers can better perform and communicate the results of the forensic measurements they perform; Lawyers can better understand, present and cross-examine the results of forensic measurements intended to be used as evidence; Judges will be better able to subject testimony or evidence based on forensic measurements to the appropriate gatekeeping analysis; and each of these participants will be better prepared to play their role in ensuring that the misuse of science doesn't undermine the search for truth in the courtroom.

Reform Through Litigation
This was the idea attorneys had in mind when, in the Summer of 2007, a forensic scientist within the Washington State Toxicology Lab was discovered committing perjury, claiming to have performed measurements that she had not. Upon further investigation, though, the defense discovered that the Lab's problems went far deeper than perjury. The Lab's process for creating simulator solutions for the calibration and checking of breath test machines was in a state of disarray. Failures to validate procedures, follow approved protocols, adhere to scientifically accepted consensus standards, properly calibrate or maintain equipment, and even to simply check the correctness of results and calculations were endemic.
In a private memo to Washington's Governor, the State Toxicologist explained that the measurement procedures in question "had been in place for over twenty years and had gone unchallenged, leading to complacency." What allowed the defense to find what others had missed over the years was, again, metrology. Viewed through the appropriate metrological framework, it became clear that what complacency had led to was the systemic failure of the Lab to adhere to fundamental scientific requirements for the acquisition of reliable measurement results. After a seven-day hearing that included testimony from nine experts, declarations from five others, as well as one hundred and sixtyone exhibits, a panel of three judges issued a thirty-page ruling suppressing all breath test results until the Lab fixed the problem's identified. [4] Under new leadership, the Lab subsequently used the same metrological framework to fix its problems that had been used to discover them. It did so by implementing fundamental metrological principles and practices and obtaining accreditation under ISO 17025, the international standard that embodies these principles and practices. Because of this, the Washington State Toxicology Lab has one the best Breath Test Calibration programs in the country. The same metrological principles that can be such effective tools in the hands of legal professionals can be even more powerful when employed by competent forensic scientists.
Criminal defense attorneys in other states, including Michigan, California, Minnesota, New Mexico, Pennsylvania and Arizona, have since begun to follow Washington's lead.

The National Academy of Sciences
It was at about this time, in February of 2009, that the National Academy of Sciences released a report on the state of forensic science in America. [5] The Report was very critical of the practices engaged in by many of the forensic sciences identifying problems in the areas of method validation, adherence to appropriate practices as evidenced by consensus standards, the determination and reporting of measurement uncertainty and many others. A majority of the scientific issues identified by the Report, though, are those that metrology addresses. What had been done in Washington State with respect to forensic measurement was to not only beat the Academy to the punch in the discovery of these issues, but also to the identification of the appropriate framework for their solution.
In the forensics community, metrology is now being relied upon to address many of the issues identified by the National Academy of Sciences. Its principles are helping to improve how forensic measurements are developed, performed and reported. Accreditation and adherence to international scientific standards are restoring confidence that forensic measurements comply with the same rigorous methodology followed in other sciences. And it is providing a common language for all those engaged in making or relying upon forensic measurements to communicate about them regardless of application.

A Growing Movement
Despite having had it's breath test calibration program accredited, by the Summer of 2009, the Washington State Toxicologist had decided that the uncertainty associated with breath test results obtained in Washington would neither be determined nor provided to Citizen's who were being prosecuted on the basis of those results. Accordingly, Washington defense lawyers engaged the fight again. Their argument was that breath test results could not be properly weighed by fact finders absent their uncertainty. In other words, judges and juries could only understand the conclusions supported by a result if they were provided the uncertainty that revealed the range of values that could reasonably be attributed to an individual's BrAC based on that result.
The Defense drove home it's point during cross-examination of the State's primary expert. The witness was handed a breath alcohol test ticket containing results of 0.081 g/210L and 0.080 g/210L. Assuming proper quality assurance procedures and testing protocols were followed, all parties agreed that these were the results of an "accurate and reliable" test. The expert was then asked, given these results, could he state beyond a reasonable doubt that this individual's BrAC exceeded a 0.080 g/210L (the per se limit in Washington State). The expert responded: "I would have to say yes based on these results here." Similar evidence and testimony, concerning a range of forensic measurements, is introduced in courtrooms around the country every day. And based on such evidence and testimony, citizens accused of all manner of crimes plead or are found guilty. The problem is that an accurate and reliable test doesn't necessarily mean what most lay factfinders presume. In fact, as the Court later noted, even an expert may be misled by what is deemed an accurate and reliable result. This was demonstrated by subsequently providing the expert with the breath test result's uncertainty and asking whether he still believed that these results supported the conclusion that this individual's BrAC exceeded a 0.080 g/210L that beyond a reasonable doubt. His response was no.
The expert conceded that, despite the fact that the test was accurate and reliable, based on the result's uncertainty there was actually a 44% likelihood that this individual's BrAC was below 0.080 g/210L! Far more than a reasonable doubt, these "accurate and reliable" test results barely established the conclusion as more likely than not! After a five-day hearing that included testimony from four experts as well as 93 exhibits, a Panel of three judges issued a thirty-page order explaining that breath test results would henceforth be inadmissible unless they were accompanied by their uncertainty. The rulings from this and other Washington cases concerning measurement uncertainty garnered nationwide attention. Lawyers, judges, forensic scientists and scholars from around the country began discussing and writing about the importance of providing a measured result's uncertainty when the result will be relied upon as evidence at trial. [7] Thomas Bohan, former president of the American Academy of Forensic Sciences, declared it to be "a landmark decision, engendering a huge advance toward rationality in our justice system and a victory for both forensic science and the pursuit of truth." [8] The battle was subsequently taken up by defense attorneys in several other State and Federal Courts, and continues to spread as of the time of this writing.
Unfortunately, in a decision that defies reason, the Washington State Court of Appeals decided that it understood science better than scientists. It found that: 1) Proper science does not require that the uncertainty of measured results be either determined or provided to the users of those results; and 2) despite the State's top expert having been fooled by breath test results that were not accompanied by their uncertainty, that measured results were best understood without their uncertainty. [9] Given the embarrassment suffered by the State Toxicology Lab during the lower Court proceedings, however, it now "voluntarily" determines and provides the uncertainty of the results of all breath tests performed within the State of Washington.

The Prosecution Begins to See the Light
It's not just defense counsel who are concerned about the effects of bad and misleading forensic evidence in the courtroom, though. In a 2013 paper published in the Santa Clara Law Review, a California prosecutor provided the rational for why all those advocating on behalf of the state should also be fighting for the imposition of metrological requirements on state administered breath and blood tests. In fact, after a trial court denied a defense motion to require the reporting of uncertainty and traceability with blood test results, he worked with that jurisdiction's lab to make sure that this was done for all future test results despite the court's ruling. And he subsequently worked to make this a mandatory regulatory requirement. Why? Because he wants to ensure that the science presented by the state in court is "the best science regardless of what the law requires." [10]

Back to Fundamentals: The Measurand
The latest battles in this saga concern the most fundamental aspect of the measurement process: identification of the measurand. The problem at issue is identification of measurands in statutes by scientifically unsophisticated legislators that are either so significantly under-defined, or vary by jurisdiction in such a way, that they create confusion for both forensic science and legal professionals.
An example of the first type of problem arises under controlled substance statutes that prohibit certain substances and their "analogues." An analogue is typically defined as having a chemical structure "substantially similar" to the named controlled substance. What constitutes substantially similar, however, is a matter of opinion on which forensic scientists in the same laboratories often disagree on. The second type of issue arises primarily in the DUI context where statutes of different jurisdictions specify distinct measurands for a common breath alcohol test. [11] Depending upon the measurand identified, the result of a breath test means something very different between these jurisdictions and the uncertainty of the results varies significantly as well.
Statutes suffering from such infirmities create a confusing landscape. Given the role of lawmakers, as opposed to scientists, in the specification of a test's measurand, though, they are representative of the type of problems that arise at the intersection of law and science. With a lack of scientific sophistication, many lawmakers fail to realize the necessity of, or required detail concerning, the measurand in a measurement. In fact, many have never even heard the term measurand before. The truth about any scientific measurement is that it can never reveal what a quantity's true value is. The power of metrology lies in the fact that it provides the framework by which we can determine what conclusions about that value are supported by measured results. It tells us how to develop and perform measurements so that high quality information can be obtained. It helps us to understand what our results mean and represent. And finally, it provides the rules that guide our inferences from measured results to the conclusions they support. Whether you are a prosecutor or defense attorney, judge or forensic scientist, or even a law enforcement officer who performs measurements as part of investigations in the field, forensic metrology provides a powerful tool for determining the truth when forensic measurements are relied upon. Forensic science, legal practice and justice itself are improved by a familiarity with the principles of forensic metrology. [12]