More Junk Criminal Science

The newly renamed Maryland Supreme Court has ruled that ruled that ballistics evidence is basically junk science.

They ruled that a so-called ballistics expert can rule a bullet of shell casing out, but not in, because they are wrong between 30% and 50% of the time. (Not much better than a coin flip)

In Maryland, firearms experts will no longer be allowed to testify that a specific gun fired a specific bullet, the state’s highest court ruled in an opinion published Tuesday.

Authored by Chief Justice Matthew J. Fader of the Supreme Court of Maryland, the opinion imposes limits in the courtroom on the practice known as firearm “tool mark” analysis. The forensic technique postulates that machines used to make guns leave tiny imperfections on their components, and that those components imprint unique marks on ammunition — composed of softer metal — when fired.

Until now, it was commonplace for firearms examiners — usually employed in police crime labs — to testify that a gun recovered by law enforcement fired bullets or used casings found at a crime scene, if they believed that to be true based on their observations under a microscope.

But four of seven justices on the state Supreme Court found that the scientific methodology is not reliable enough to allow examiners to testify that a particular gun fired a particular bullet. Examiners can, however, testify “that patterns and markings on bullets are consistent or inconsistent with those on bullets fired from a particular known firearm,” the opinion said.

………

With the opinion, Maryland becomes one of the nation’s first jurisdictions where an appellate court has recognized shortcomings in the forensic practice and imposed limits on its use in court.

Maneka Sinha, an associate professor at the University of Maryland Francis King Carey School of Law who studies forensic sciences, said the justices “came to the conclusion that scientists, academics and others seriously studying the discipline already have: that conclusions claiming they can say a specific gun fired a specific item of ammunition are simply unreliable.”

As to the studies, when said studies are not run by the the Association of Firearm and Tool Mark Examiners (ATFE), the result are unreliable:

………

Leading off its criticism of assuming ballistics evidence is good evidence, the court cites a 2009 report by the National Research Council (NRC) of the National Academies of Science. That report said the standards created by the Association of Firearm and Tool Mark Examiners (ATFE) were faulty because so much of what was assumed to be scientifically sound was little more than examiners’ subjective interpretations of marks found on bullets.

………

The government cites studies showing minuscule error rates by ATFE examiners, with one study showing a near-zero rate of false positives. But the court says these were tests controlled by the ATFE where examiners knew they were being tested and every test set included a test bullet fired by the test gun.

In “black box” studies (only two appear to meet this description), ATFE examiners fared much worse. Test sets did not always contain a match. Examiners didn’t know they were being tested. In those tests, the error rate was exponentially higher: more than a third of the matches were declared “inconclusive.” In the other test, positive results (i.e., supposed matches) varied as much as 15% between sets of examiners. Negative results (non-matches) varied nearly as much: 13-14% between sets of examiners over two rounds of testing.

Is being right 74-80% of the time the evidentiary standard in the US criminal justice system? Obviously, it shouldn’t be. But it has been because examiners routinely overstated the confidence of their findings and the US court system basically never bothered to wonder if the experts might be wrong.

There was a similar case to this a few years ago about fingerprints.  There is no consensus on the numbers of points of similarity.

If there is a field of study where tests should be double blind in all instances, it is forensic science.

The incentives are for getting convictions, not getting it right.