Dispatches From The Junk Science Front

Politics & Current Events

In 2008 I pointed out that the TSA's pseudo-scientific "behavior detection" program seemed almost indistinguishable from random chance. Five years and millions of gropes-by-government-agents later, the General Accounting Office agrees:

The program called Screening of Passengers by Observation Techniques (SPOT) trains TSA officers to identify suspicious behavior that could reveal a terrorist. While it has been criticized for years for alleged racial profiling, TSA officials say it is a key part of screening airline passengers.

The Government Accountability Office reviewed 400 studies over 60 years that found people are only slightly better than chance at spotting deceptive behavior. And a Department of Homeland Security study in April 2011 intended to validate the program was unable to demonstrate its effectiveness because of unreliable data, according to the new GAO report.

The program has cost a billion dollars. The TSA can't demonstrate that it works using accepted scientific means. The TSA's reaction is unsurprising: "yeah, well, our other methods are even worse:"

Behavior Detection Officers also operate a program called Managed Inclusion which evaluates passengers at the checkpoints and allows some to enter the faster Pre-Check lanes.

"Defunding the program is not the answer," Pistole said. "There would be fewer passengers going through expedited screening, there would be increased pat downs, there would be longer lines, and more frustration by the traveling public."

Or, put another way, a piece of shit is better than no piece of anything:

The union representing TSA officers defended the program.

"An imperfect deterrent to terrorist attacks is better than no deterrent at all, " said American Federation of Government Employees National President David Cox, speaking in a conference call after the hearing. "Is it a perfect program? No, but until we have a better program, we shouldn't just trash and burn this program."

That's so sciency! "Well, I can't prove this hypothesis. But until I come up with a better hypothesis, I think we should stick with this one."

Meanwhile, in Texas . . .

. . . did you just say "aw, shit, this is gonna be awful, because it's Texas?" Perhaps you did. Perhaps you are not completely unjustified in leaping to that conclusion. But you're wrong. Texas, it turns out, passed an innovative law to allow prisoners to attack convictions premised on discredited junk science spouted by prosecution "experts." Last week, using that law, a Texas court overturned the convictions of four women caught up in the "ritualized child abuse" scare of the 1980s and 1990s:

Indeed, at the original trial of the San Antonio Four, a pediatrician testified that the victims exhibited physical signs of sexual abuse. This expert testimony provided the prosecution with much needed corroboration of the two girls' stories. Such medical testimony, however, has now been debunked by new understandings in the field of pediatrics. If the two girls had been physically examined using today's standards, the medical testimony would no longer corroborate the allegations of sexual abuse.

Like many of the defendants in ritualized-abuse cases, the San Antonio Four faced bizarre and fanciful claims that should have triggered skepticism — had not "think of the children!" drowned out all critical thought. Like many such defendants, junk science and bizarre and facially questionable allegations combined with innate identity-based hostility:

A witness for the prosecution, pediatrician Nancy Kellogg, testified that the two young girls’ injuries were used in satanic rituals prevalent among lesbians.

I don't claim to be a scientist. I'm functionally scientifically illiterate. But I know enough to understand that science is about questioning and proving, and that when it's the government that shows up with the snake oil, we too often accept it without scrutiny. That may be because the government usually packages junk science with fear.

Last 5 posts by Ken White

40 Comments

40 Comments

  1. Dan Hill  •  Nov 25, 2013 @8:52 am

    What's the old saying. Yeah, that's right. "Good enough for government work."

  2. Peter English  •  Nov 25, 2013 @9:02 am

    "Science is about questioning and proving".

    Well sort of. Or half right.

    Science isn't so much about proving as disproving. It's hard to prove that things always fall downwards. What you can do is lots of experiments that fail to show that things don't fall downwards.

    The hypothesis is "things always fall downwards". The opposite – the so called null-hypothesis – is the opposite; that "things don't always fall downwards".

    If you can do an experiment in which the null hypothesis is demonstrated to be correct, then you've disproven your hypothesis, and you'll have to throw it out or refine it.

    In this case, the studies have failed to support the hypothesis that the "behaviour detection programme" – sorry, "behavior detection program", given that you're American and all that – works any better than chance.

    When studies fail to support a hypothesis there are several possibilities:

    - The hypothesis is incorrect.
    - The hypothesis is correct, but the studies were too small or badly designed to detect this.

    Often in the latter case, however, the hypothesis may be correct, but only a bit. If you're looking for a small effect – the program is better than chance, but only just; or MMR vaccine does cause idiopathic thrombocytopaenic purpura (a mild and self-limiting condition) but only in no more than 1 in 23,000 people, and only after the first dose of vaccine) – then you need to do a massive study to detect this.

    Similarly, if they've done decent studies of a reasonable size and failed to find evidence that the behavior detection program works any better than chance, you can be fairly confident that – if it works at all – it's not a lot better than chance.

    (Incidentally, the bit about MMR is true! We've known this for years. That's part of the reason we can be so confident it's safe – we've know for decades that it causes a mild side-effect in a tiny proportion of people; and despite looking really, really hard, we can't find anything more serious than that to worry about.)

  3. JonasB  •  Nov 25, 2013 @9:16 am

    How would you test a TSA program using scientific means, anyway?

  4. Speed  •  Nov 25, 2013 @9:27 am

    In the economic sphere an act, a habit, an institution, a law produces not only one effect, but a series of effects. Of these effects, the first alone is immediate; it appears simultaneously with its cause; it is seen. The other effects emerge only subsequently; they are not seen; we are fortunate if we foresee them.

    Bastiat, Frédéric
    (1801-1850)

    http://www.econlib.org/library/Bastiat/basEss1.html

  5. TM  •  Nov 25, 2013 @9:32 am

    I wonder if there was ever a (failed and useless to the general public) government program where the government officials thought that defunding it was the answer.

  6. Craig  •  Nov 25, 2013 @9:44 am

    It's ridiculous that anyone even asks a TSA union representative to comment on issues like these. A person in that position is there for no reason but to advocate for TSA employees, which in this case means arguing against anything that might threaten to de-fund a program that pays their salaries. If this person worked for a railroad worker's union and the question was, "Should the government stop mass-arresting Jews and sending them by train to concentration camps?", he would argue against that too, not necessarily out of any personal antagonism towards Jews but simply because the Final Solution is "good" for railroad workers (well, as long as they aren't Jewish).

  7. I was Anonymous  •  Nov 25, 2013 @10:02 am

    Only 6 comments, and already Godwinned.

  8. A Different William  •  Nov 25, 2013 @10:06 am

    deleted.

  9. Marconi Darwin  •  Nov 25, 2013 @10:18 am

    When there is disdain for real science, and even more disdain for complexity, people will always prefer simplicity and certainty over complexity and doubt because, hey look, common sense!

  10. Dion starfire  •  Nov 25, 2013 @10:50 am

    I was going to crack a joke about Ken being psychic with his ". . . did you just say "aw, shit, this is gonna be awful, because it's Texas?" " line, got distracted by Peter English's enlightening and interesting post and had to add one point to it.

    I vaguely recall a type of theory/hypothesis from my geometry class that we accepted and used as true, but wasn't technically true because nobody had been able to prove when it was false. and his hypothesis/null-hypothesis lines up with what I recall from a science lecture about the various flavors of proving a hypothesis (proved, failed to disprove, failed to prove the null-hypothesis, and one other I can't remember at the moment).

  11. En Passant  •  Nov 25, 2013 @10:50 am

    My quick, and admittedly pessimistic, reading of the statute is that there is slightly less to it than might meet the ear on hearing about it. The meat is in Section (b)(2) [emphasis mine]:

    (b) Authorizes a court to grant a convicted person relief on an application for a writ of habeas corpus if:

    (2)the court makes the findings described by Subdivisions (1)(A) and (B) and also finds that, had the scientific evidence been presented at trial, on the preponderance of the evidence the person would not have been convicted.

    So, the new evidence must show actual innocence by a preponderance of evidence. That's a heavier burden than just raising massive reasonable doubt.

    That, and Section 2 provides that the act is not retroactive to habeus filings made earlier than the effective date of the act.

    But small favors are better than none at all.

  12. Maria  •  Nov 25, 2013 @10:51 am

    @JonasB: You could randomly select people for increased scrutiny at a rate similar to that of TSA agents in that airport. Or perhaps have one checkpoint agent use his "SPOT" training in one checkpoint, and everytime someone is stopped, have someone stopped in a nearby checkpoint. It can't be truly blind, because the agents will know who made the choice to stop, but it's a start. I'm sure someone with more experience in research can come up with a way to make it blind.

  13. En Passant  •  Nov 25, 2013 @11:05 am

    Dion starfire wrote Nov 25, 2013 @10:50 am:

    I vaguely recall a type of theory/hypothesis from my geometry class that we accepted and used as true, but wasn't technically true because nobody had been able to prove when it was false.

    Or maybe you're recalling that the parallel postulate of Euclidean plane geometry is false in Lobachevskian hyperbolic geometry?

  14. Peter H  •  Nov 25, 2013 @11:15 am

    En Passant,

    I read that statute to mean that the court has to find that acquittal on the beyond a reasonable doubt standard would have to be the more-likely-than-not outcome. I may be wrong though, so if anyone has an authority to cite I'd appreciate it.

  15. Kelly  •  Nov 25, 2013 @11:20 am

    This quote: "A witness for the prosecution, pediatrician Nancy Kellogg, testified that the two young girls’ injuries were used in satanic rituals prevalent among lesbians." Reminds me of what a fire investigator said to help convict a young black man of arson in Tucson, AZ. He said that young black men tended to start fires. Mr. Talyor spent 42 years in prison, and was only released recently when modern fire investigation techniques were applied to the facts of the case.

  16. En Passant  •  Nov 25, 2013 @11:22 am

    Peter H wrote Nov 25, 2013 @11:15 am:

    I read that statute to mean that the court has to find that acquittal on the beyond a reasonable doubt standard would have to be the more-likely-than-not outcome.

    You very well may be right, and I hope you are. I wear pessimistical goggles, and also find statutory construction mystical.

  17. Pedant  •  Nov 25, 2013 @11:27 am

    "A witness for the prosecution, pediatrician Nancy Kellogg, testified that the two young girls’ injuries were used in satanic rituals prevalent among lesbians."

    I am a straight white man. But I know (and am friendly with) several lesbians. These must be "fringe" lesbians, as I have never been invited to a "satanic" ritual, even though they are "prevalent."

    Some "pediatrician"!

  18. I was Anonymous  •  Nov 25, 2013 @11:39 am

    I'm certain that some people from the GAO are now on the "No-Fly" list.

  19. rsteinmetz70112  •  Nov 25, 2013 @11:46 am

    Some of my best friends are lesbians and only a couple of them worship the devil. They both work for TSA.

  20. JeffM  •  Nov 25, 2013 @12:01 pm

    I think most working scientists subscribe to Popper's theory of what constitutes a scientific hypothesis: it is one that can in principle be shown to be false. That is, a scientific hypothesis can NEVER be confirmed, but it may be falsified. The scientific ideal is that there are just two verdicts: guilty (meaning falsified) and not proven (meaning not falsified as of yet). The verdict of innocent (meaning confirmed) is not allowed.

    Kuhn, Lakatos, and others have pointed out that science does not actually work in that idealized fashion. For centuries, scientists effectively viewed the Newtonian theory of gravitation as confirmed despite what appeared to be known falsifying exceptions. It was only when presented with a better theory, namely Einstein's, that scientists rejected Newton's theory (although they still accept it as a convenient approximation for many purposes). The question of what is a "better" theory is still not completely answered, but scientists seem to believe that they can recognize a better theory in practice even if they cannot agree in principle about what exactly defines "better."

    The fact remains that scientists use the best theory that they have even if they know it's wrong until they can replace it with a better theory. So the TSA's behavior is quite "sciency." But a theory, once developed, is a sunk cost. The TSA's program incurs costs on an ongoing basis. The question is not whether continuing that program is "sciency" but whether its apparently meager if not entirely illusory benefits outweigh its substantial costs.

  21. Marzipan  •  Nov 25, 2013 @12:19 pm

    tl;dr this comment: A major problem with these kinds of detection efforts is that the base rate of a terrorist's attack happening is so low that a truly scientific decision outside of "let them all through" is extraordinarily difficult to obtain. This is the bane of anyone whose job entails predicting extremely infrequent events.

    Long answer: Even with the best tests available, it's a bitch to do the job of ferreting out terrorists. Let's assume that you have a test (e.g., a single test, a configuration of behavioral results) that can detect 99.9% of all terrorists who would otherwise harm the traveling public. That test has a .999 sensitivity. Let's also say that test is able to weed out 99.9% of those who are not terrorists. The test thus has a .999 specificity. Anyone working in assessment knows that these are ludicrously optimistic estimates for a real-world measure, but they help to make the point.

    According to the FAA, there were 732,000,000 passenger flyers on US airlines in 2012. Let's assume that of these, 1000 were terrorist flyers (a single terrorist might fly more than once per year), leaving 731,999,000 non-terrorists flying. Applying our super-spiffy test, all but 1 of the terrorists would have been detected (99.9% sensitivity), so 999 terrorist flights would have been detected (so much for them flying more than once!). So far, the TSA looks great!

    However, with a 99.9% sensitivity rate, (1-.999) * 731,999,000 false positives occur, so 731,990 false alarms happen (or people are detained for additional questioning). Thus, the total number of people who the test calls terrorists is 732,998.

    The positive predictive value of the TSA test is thus 999/732998, or .00136. More precisely, that means that 99.834% of the people who are detained and likely subjected to incredibly invasive checks (because we have super-test, remember!) will have been inconvenienced for no good reason.

    In this case, we'd actually be more accurate (i.e., have a higher test hit rateby calling everyone a non-terrorist. Yes, we'd let all 1000 terrorists through, but we'd be right 731,999,000/732,000,000 times, for a hit rate of 99.99986%. With our test, the TSA's hit rate is 731,267,009 (remember the one missed terrorist!)/732,000,000, or 99.89986%. That means the TSA makes two orders of magnitude more classification errors than doing absolutely nothing.

    And this is in an idealized situation. In reality, no affordable test is even close to that sensitive (i.e., good at actually catching terrorists), nor is it even close to that specific (i.e., good at screening out non-terrorists). Thus, the numbers are far, far worse in actual practice. And that's why we call it security theater.

  22. AlphaCentauri  •  Nov 25, 2013 @12:40 pm

    Some of my best friends are lesbians and only a couple of them worship the devil. They both work for TSA.

    Hey, maybe you're hit on something here. Let it be known among Islamists that getting on a plane involves being felt up by devil-worshipping homosexuals and they might all decide that's not what they want to do during their last hour alive. ;)

  23. Stephen  •  Nov 25, 2013 @1:14 pm

    Which is more disturbing — that someone with a medical degree can make the statement "satanic rituals prevalent among lesbians" and be entirely serious about it, or that most of the American population does not hear that statement and immediately denounce the speaker as a nutjob?

  24. InnocentBystander  •  Nov 25, 2013 @1:58 pm

    Dion starfire, the famous example of an "unproven but accepted theorem" was Fermat's Last Theorem which said a^n +b^n cannot equal c^n where a,b, and c are positive integers and n is an integer greater than 2. Unfortunately, this example from my childhood was shattered in 1995 when Andrew Wiles proved it mathematically.

  25. Jonathan Corbett  •  Nov 25, 2013 @2:14 pm

    The TSA really snorts the taint :)

  26. James Pollock  •  Nov 25, 2013 @4:29 pm

    I wonder if there was ever a (failed and useless to the general public) government program where the government officials thought that defunding it was the answer.

    Well, the Democrats in Congress tried voting several times to defund the war in Iraq, and the Republicans in Congress have voted about 300 times to defund the ACA. I leave it as an exercise in value judgment as to whether either qualifies as "failed and useless to the general public".

  27. James Pollock  •  Nov 25, 2013 @4:36 pm

    It can't be truly blind, because the agents will know who made the choice to stop, but it's a start. I'm sure someone with more experience in research can come up with a way to make it blind.

    The problem isn't the blindness of the study. The problem is in controlling the variables. Did the TSA stop all terrorist attacks because they caught all the terrorists, or because the terrorists stopped sending people to airports? Subquery: If the latter, did the terrorists stop sending terrorists to the airport because they were afraid of being detected by the TSA, or was there another reason? (Ran out of terrorists willing to die for the cause; had a better attack planned; achieved their goals and no reason to keep trying; other)

    To go all Rumsfeldian, you can tell whether the TSA catches the explosives that you try to sneak past them, but you can't know if they caught any of the explosives that the terrorists tried to sneak past them, because they (the terrorists) won't say.

  28. Drewski  •  Nov 25, 2013 @5:01 pm

    I haven't read that law yet, but s'pose Texans can use it to challenge drug dog evidence post-conviction, even if they couldn't at trial? That would be interesting.

  29. Matthew Cline  •  Nov 25, 2013 @6:02 pm

    Leaving aside the "satanic rituals prevalent among lesbians" bit (which I won't deign to call even "junk science"), the claim that the girls showed physical signs of sexual abuse might or might not have been junk science. It might very well have been junk science, but the mere fact that it was wrong isn't a guarantee that it was junk.

    And while on the subject, from the Huffington Post:

    In the intervening years, while they languished in prison, three of the four women took and passed polygraphs administered by a respected expert.

    I find that ironic, given that if polygraph evidence was allowed in court, the same law might be used to challenge the validity of that evidence.

  30. En Passant  •  Nov 25, 2013 @7:57 pm

    Matthew Cline wrote Nov 25, 2013 @6:02 pm:

    Leaving aside the "satanic rituals prevalent among lesbians" bit (which I won't deign to call even "junk science"), the claim that the girls showed physical signs of sexual abuse might or might not have been junk science. It might very well have been junk science, but the mere fact that it was wrong isn't a guarantee that it was junk.

    A good point. In the words of the statute, "relevant scientific evidence that … contradicts scientific evidence relied on by the state at trial" doesn't always mean the trial science was "junk". It may mean the science relied on at trial was superseded by more recent fundamental scientific findings, as JeffM noted above about Newtonian gravity.[1]

    But most bad forensic "science" is just bogus, as in "satanic rituals prevalent among lesbians", or a medical examiner basing a tox screen on homeopathic medicine.

    One indicator of good science (distinct from junk "science") is that good science generally produces (or can produce) error estimates with findings. Any "science" that cannot produce some kind of error estimates should be viewed with suspicion.

    FN 1: But I wouldn't rely on the general theory of relativity to fight a speeding ticket.

  31. babaganusz  •  Nov 25, 2013 @9:37 pm

    Sam Harris had thoroughly convinced himself that all it should take is a baseline of closely searching anyone and everyone in ~traditional Muslim garb~ and their possessions. (Sam has some hard-to-shake notions about how utterly and invariably beliefs dictate behaviors.) Bruce Schneier patiently and repeatedly explained to Sam along the lines of "sorry, random searches are a more robust system than that."

    (consider: the Terrorist.org's workaround for the TSA groping and upending the luggage of everyone who "looks Muslim" = golly, if only human endeavor had hit upon some form of… disguise… 'workaround' for random searches = basically outspend the TSA and flood individual flights with armed/primed operatives – if you're even confident with the assumption that at least one gets detained/prosecuted.)

    not that that matters anymore. Marzipan and Peter English are my newest heroes. i can't wait for statistics classes.

  32. Matthew Cline  •  Nov 25, 2013 @9:58 pm

    But I wouldn't rely on the general theory of relativity to fight a speeding ticket.

    No, for that you should use the uncertainty principle.

  33. babaganusz  •  Nov 25, 2013 @10:01 pm

    oh, and once again:
    @ I was Anonymous: wrong. the Captain Obvious pullquote you're looking for is "HEY GUYS LOOK, NAZI REFERENCE". nobody and no position was being compared to or equated with Nazis. Craig's framework was entirely apt (and arguably not even tasteless). don't hide your dim in-joke-glee behind the ongoing dilution of the entire point of Godwin's Law – going blithely out of bounds of the premise, there is no point whatsoever in even giving it a name.

  34. Christopher  •  Nov 26, 2013 @12:40 am

    "An imperfect deterrent to terrorist attacks is better than no deterrent at all, "

    Uh… no. If the deterrent is operating at chance levels, then you've literally shown that an imperfect deterrent is no better than no deterrent at all.

  35. Anony Mouse  •  Nov 26, 2013 @1:41 am

    On second thought, nevermind.

  36. Kathleen  •  Nov 26, 2013 @5:52 am

    I just read a blog post by the child of a woman who ran a daycare center and got caught up in that hysteria. Even people who never got charged with anything had their lives destroyed by that witch hunt. http://killer-martinis.squarespace.com/my-wordpress/2013/11/24/its-time-i-really-introduced-myself

  37. babaganusz  •  Nov 26, 2013 @6:43 am

    Anony Mouse • Nov 26, 2013 @1:41 am

    On second thought, nevermind.

    thank goodness someone is thinking twice…

  38. Tam  •  Nov 26, 2013 @10:01 am

    "[P]ediatrician Nancy Kellogg, testified that the two young girls’ injuries were used in satanic rituals prevalent among lesbians."

    LOLWUT?

    Anybody who listened to that gem with a straight face should die of embarrassment.

  39. KR  •  Nov 26, 2013 @10:15 am

    I don't believe I've seen a lesbian engage in any Satanic rituals, though I do know one who roots for the Yankees, which is similar.

  40. Brad Hicks (@jbradhicks)  •  Nov 26, 2013 @12:07 pm

    [Long string of expletives deleted.] Today I Learned that we still had people in jail for Satanic Ritual Abuse. I thought the last of those poor people got freed years ago.

    Don't mind me, I'll be over in the corner twitching for a while. The Satanic Ritual Abuse moral panic was a very bad time for me; I narrowly escaped going to jail for it myself.