Dispatches From The Junk Science Front

Print This Post

You may also like...

40 Responses

  1. Dan Hill says:

    What's the old saying. Yeah, that's right. "Good enough for government work."

  2. "Science is about questioning and proving".

    Well sort of. Or half right.

    Science isn't so much about proving as disproving. It's hard to prove that things always fall downwards. What you can do is lots of experiments that fail to show that things don't fall downwards.

    The hypothesis is "things always fall downwards". The opposite – the so called null-hypothesis – is the opposite; that "things don't always fall downwards".

    If you can do an experiment in which the null hypothesis is demonstrated to be correct, then you've disproven your hypothesis, and you'll have to throw it out or refine it.

    In this case, the studies have failed to support the hypothesis that the "behaviour detection programme" – sorry, "behavior detection program", given that you're American and all that – works any better than chance.

    When studies fail to support a hypothesis there are several possibilities:

    - The hypothesis is incorrect.
    - The hypothesis is correct, but the studies were too small or badly designed to detect this.

    Often in the latter case, however, the hypothesis may be correct, but only a bit. If you're looking for a small effect – the program is better than chance, but only just; or MMR vaccine does cause idiopathic thrombocytopaenic purpura (a mild and self-limiting condition) but only in no more than 1 in 23,000 people, and only after the first dose of vaccine) – then you need to do a massive study to detect this.

    Similarly, if they've done decent studies of a reasonable size and failed to find evidence that the behavior detection program works any better than chance, you can be fairly confident that – if it works at all – it's not a lot better than chance.

    (Incidentally, the bit about MMR is true! We've known this for years. That's part of the reason we can be so confident it's safe – we've know for decades that it causes a mild side-effect in a tiny proportion of people; and despite looking really, really hard, we can't find anything more serious than that to worry about.)

  3. JonasB says:

    How would you test a TSA program using scientific means, anyway?

  4. Speed says:

    In the economic sphere an act, a habit, an institution, a law produces not only one effect, but a series of effects. Of these effects, the first alone is immediate; it appears simultaneously with its cause; it is seen. The other effects emerge only subsequently; they are not seen; we are fortunate if we foresee them.

    Bastiat, Frédéric
    (1801-1850)

    http://www.econlib.org/library/Bastiat/basEss1.html

  5. TM says:

    I wonder if there was ever a (failed and useless to the general public) government program where the government officials thought that defunding it was the answer.

  6. Craig says:

    It's ridiculous that anyone even asks a TSA union representative to comment on issues like these. A person in that position is there for no reason but to advocate for TSA employees, which in this case means arguing against anything that might threaten to de-fund a program that pays their salaries. If this person worked for a railroad worker's union and the question was, "Should the government stop mass-arresting Jews and sending them by train to concentration camps?", he would argue against that too, not necessarily out of any personal antagonism towards Jews but simply because the Final Solution is "good" for railroad workers (well, as long as they aren't Jewish).

  7. I was Anonymous says:

    Only 6 comments, and already Godwinned.

  8. A Different William says:

    deleted.

  9. Marconi Darwin says:

    When there is disdain for real science, and even more disdain for complexity, people will always prefer simplicity and certainty over complexity and doubt because, hey look, common sense!

  10. Dion starfire says:

    I was going to crack a joke about Ken being psychic with his ". . . did you just say "aw, shit, this is gonna be awful, because it's Texas?" " line, got distracted by Peter English's enlightening and interesting post and had to add one point to it.

    I vaguely recall a type of theory/hypothesis from my geometry class that we accepted and used as true, but wasn't technically true because nobody had been able to prove when it was false. and his hypothesis/null-hypothesis lines up with what I recall from a science lecture about the various flavors of proving a hypothesis (proved, failed to disprove, failed to prove the null-hypothesis, and one other I can't remember at the moment).

  11. En Passant says:

    My quick, and admittedly pessimistic, reading of the statute is that there is slightly less to it than might meet the ear on hearing about it. The meat is in Section (b)(2) [emphasis mine]:

    (b) Authorizes a court to grant a convicted person relief on an application for a writ of habeas corpus if:

    (2)the court makes the findings described by Subdivisions (1)(A) and (B) and also finds that, had the scientific evidence been presented at trial, on the preponderance of the evidence the person would not have been convicted.

    So, the new evidence must show actual innocence by a preponderance of evidence. That's a heavier burden than just raising massive reasonable doubt.

    That, and Section 2 provides that the act is not retroactive to habeus filings made earlier than the effective date of the act.

    But small favors are better than none at all.

  12. Maria says:

    @JonasB: You could randomly select people for increased scrutiny at a rate similar to that of TSA agents in that airport. Or perhaps have one checkpoint agent use his "SPOT" training in one checkpoint, and everytime someone is stopped, have someone stopped in a nearby checkpoint. It can't be truly blind, because the agents will know who made the choice to stop, but it's a start. I'm sure someone with more experience in research can come up with a way to make it blind.

  13. En Passant says:

    Dion starfire wrote Nov 25, 2013 @10:50 am:

    I vaguely recall a type of theory/hypothesis from my geometry class that we accepted and used as true, but wasn't technically true because nobody had been able to prove when it was false.

    Or maybe you're recalling that the parallel postulate of Euclidean plane geometry is false in Lobachevskian hyperbolic geometry?

  14. Peter H says:

    En Passant,

    I read that statute to mean that the court has to find that acquittal on the beyond a reasonable doubt standard would have to be the more-likely-than-not outcome. I may be wrong though, so if anyone has an authority to cite I'd appreciate it.

  15. Kelly says:

    This quote: "A witness for the prosecution, pediatrician Nancy Kellogg, testified that the two young girls’ injuries were used in satanic rituals prevalent among lesbians." Reminds me of what a fire investigator said to help convict a young black man of arson in Tucson, AZ. He said that young black men tended to start fires. Mr. Talyor spent 42 years in prison, and was only released recently when modern fire investigation techniques were applied to the facts of the case.

  16. En Passant says:

    Peter H wrote Nov 25, 2013 @11:15 am:

    I read that statute to mean that the court has to find that acquittal on the beyond a reasonable doubt standard would have to be the more-likely-than-not outcome.

    You very well may be right, and I hope you are. I wear pessimistical goggles, and also find statutory construction mystical.

  17. Pedant says:

    "A witness for the prosecution, pediatrician Nancy Kellogg, testified that the two young girls’ injuries were used in satanic rituals prevalent among lesbians."

    I am a straight white man. But I know (and am friendly with) several lesbians. These must be "fringe" lesbians, as I have never been invited to a "satanic" ritual, even though they are "prevalent."

    Some "pediatrician"!

  18. I was Anonymous says:

    I'm certain that some people from the GAO are now on the "No-Fly" list.

  19. rsteinmetz70112 says:

    Some of my best friends are lesbians and only a couple of them worship the devil. They both work for TSA.

  20. JeffM says:

    I think most working scientists subscribe to Popper's theory of what constitutes a scientific hypothesis: it is one that can in principle be shown to be false. That is, a scientific hypothesis can NEVER be confirmed, but it may be falsified. The scientific ideal is that there are just two verdicts: guilty (meaning falsified) and not proven (meaning not falsified as of yet). The verdict of innocent (meaning confirmed) is not allowed.

    Kuhn, Lakatos, and others have pointed out that science does not actually work in that idealized fashion. For centuries, scientists effectively viewed the Newtonian theory of gravitation as confirmed despite what appeared to be known falsifying exceptions. It was only when presented with a better theory, namely Einstein's, that scientists rejected Newton's theory (although they still accept it as a convenient approximation for many purposes). The question of what is a "better" theory is still not completely answered, but scientists seem to believe that they can recognize a better theory in practice even if they cannot agree in principle about what exactly defines "better."

    The fact remains that scientists use the best theory that they have even if they know it's wrong until they can replace it with a better theory. So the TSA's behavior is quite "sciency." But a theory, once developed, is a sunk cost. The TSA's program incurs costs on an ongoing basis. The question is not whether continuing that program is "sciency" but whether its apparently meager if not entirely illusory benefits outweigh its substantial costs.

  21. Marzipan says:

    tl;dr this comment: A major problem with these kinds of detection efforts is that the base rate of a terrorist's attack happening is so low that a truly scientific decision outside of "let them all through" is extraordinarily difficult to obtain. This is the bane of anyone whose job entails predicting extremely infrequent events.

    Long answer: Even with the best tests available, it's a bitch to do the job of ferreting out terrorists. Let's assume that you have a test (e.g., a single test, a configuration of behavioral results) that can detect 99.9% of all terrorists who would otherwise harm the traveling public. That test has a .999 sensitivity. Let's also say that test is able to weed out 99.9% of those who are not terrorists. The test thus has a .999 specificity. Anyone working in assessment knows that these are ludicrously optimistic estimates for a real-world measure, but they help to make the point.

    According to the FAA, there were 732,000,000 passenger flyers on US airlines in 2012. Let's assume that of these, 1000 were terrorist flyers (a single terrorist might fly more than once per year), leaving 731,999,000 non-terrorists flying. Applying our super-spiffy test, all but 1 of the terrorists would have been detected (99.9% sensitivity), so 999 terrorist flights would have been detected (so much for them flying more than once!). So far, the TSA looks great!

    However, with a 99.9% sensitivity rate, (1-.999) * 731,999,000 false positives occur, so 731,990 false alarms happen (or people are detained for additional questioning). Thus, the total number of people who the test calls terrorists is 732,998.

    The positive predictive value of the TSA test is thus 999/732998, or .00136. More precisely, that means that 99.834% of the people who are detained and likely subjected to incredibly invasive checks (because we have super-test, remember!) will have been inconvenienced for no good reason.

    In this case, we'd actually be more accurate (i.e., have a higher test hit rateby calling everyone a non-terrorist. Yes, we'd let all 1000 terrorists through, but we'd be right 731,999,000/732,000,000 times, for a hit rate of 99.99986%. With our test, the TSA's hit rate is 731,267,009 (remember the one missed terrorist!)/732,000,000, or 99.89986%. That means the TSA makes two orders of magnitude more classification errors than doing absolutely nothing.

    And this is in an idealized situation. In reality, no affordable test is even close to that sensitive (i.e., good at actually catching terrorists), nor is it even close to that specific (i.e., good at screening out non-terrorists). Thus, the numbers are far, far worse in actual practice. And that's why we call it security theater.

  22. AlphaCentauri says:

    Some of my best friends are lesbians and only a couple of them worship the devil. They both work for TSA.

    Hey, maybe you're hit on something here. Let it be known among Islamists that getting on a plane involves being felt up by devil-worshipping homosexuals and they might all decide that's not what they want to do during their last hour alive. ;)

  23. Stephen says:

    Which is more disturbing — that someone with a medical degree can make the statement "satanic rituals prevalent among lesbians" and be entirely serious about it, or that most of the American population does not hear that statement and immediately denounce the speaker as a nutjob?

  24. InnocentBystander says:

    Dion starfire, the famous example of an "unproven but accepted theorem" was Fermat's Last Theorem which said a^n +b^n cannot equal c^n where a,b, and c are positive integers and n is an integer greater than 2. Unfortunately, this example from my childhood was shattered in 1995 when Andrew Wiles proved it mathematically.

  25. The TSA really snorts the taint :)

  26. James Pollock says:

    I wonder if there was ever a (failed and useless to the general public) government program where the government officials thought that defunding it was the answer.

    Well, the Democrats in Congress tried voting several times to defund the war in Iraq, and the Republicans in Congress have voted about 300 times to defund the ACA. I leave it as an exercise in value judgment as to whether either qualifies as "failed and useless to the general public".

  27. James Pollock says:

    It can't be truly blind, because the agents will know who made the choice to stop, but it's a start. I'm sure someone with more experience in research can come up with a way to make it blind.

    The problem isn't the blindness of the study. The problem is in controlling the variables. Did the TSA stop all terrorist attacks because they caught all the terrorists, or because the terrorists stopped sending people to airports? Subquery: If the latter, did the terrorists stop sending terrorists to the airport because they were afraid of being detected by the TSA, or was there another reason? (Ran out of terrorists willing to die for the cause; had a better attack planned; achieved their goals and no reason to keep trying; other)

    To go all Rumsfeldian, you can tell whether the TSA catches the explosives that you try to sneak past them, but you can't know if they caught any of the explosives that the terrorists tried to sneak past them, because they (the terrorists) won't say.

  28. Drewski says:

    I haven't read that law yet, but s'pose Texans can use it to challenge drug dog evidence post-conviction, even if they couldn't at trial? That would be interesting.

  29. Matthew Cline says:

    Leaving aside the "satanic rituals prevalent among lesbians" bit (which I won't deign to call even "junk science"), the claim that the girls showed physical signs of sexual abuse might or might not have been junk science. It might very well have been junk science, but the mere fact that it was wrong isn't a guarantee that it was junk.

    And while on the subject, from the Huffington Post:

    In the intervening years, while they languished in prison, three of the four women took and passed polygraphs administered by a respected expert.

    I find that ironic, given that if polygraph evidence was allowed in court, the same law might be used to challenge the validity of that evidence.

  30. En Passant says:

    Matthew Cline wrote Nov 25, 2013 @6:02 pm:

    Leaving aside the "satanic rituals prevalent among lesbians" bit (which I won't deign to call even "junk science"), the claim that the girls showed physical signs of sexual abuse might or might not have been junk science. It might very well have been junk science, but the mere fact that it was wrong isn't a guarantee that it was junk.

    A good point. In the words of the statute, "relevant scientific evidence that … contradicts scientific evidence relied on by the state at trial" doesn't always mean the trial science was "junk". It may mean the science relied on at trial was superseded by more recent fundamental scientific findings, as JeffM noted above about Newtonian gravity.[1]

    But most bad forensic "science" is just bogus, as in "satanic rituals prevalent among lesbians", or a medical examiner basing a tox screen on homeopathic medicine.

    One indicator of good science (distinct from junk "science") is that good science generally produces (or can produce) error estimates with findings. Any "science" that cannot produce some kind of error estimates should be viewed with suspicion.

    FN 1: But I wouldn't rely on the general theory of relativity to fight a speeding ticket.

  31. babaganusz says:

    Sam Harris had thoroughly convinced himself that all it should take is a baseline of closely searching anyone and everyone in ~traditional Muslim garb~ and their possessions. (Sam has some hard-to-shake notions about how utterly and invariably beliefs dictate behaviors.) Bruce Schneier patiently and repeatedly explained to Sam along the lines of "sorry, random searches are a more robust system than that."

    (consider: the Terrorist.org's workaround for the TSA groping and upending the luggage of everyone who "looks Muslim" = golly, if only human endeavor had hit upon some form of… disguise… 'workaround' for random searches = basically outspend the TSA and flood individual flights with armed/primed operatives – if you're even confident with the assumption that at least one gets detained/prosecuted.)

    not that that matters anymore. Marzipan and Peter English are my newest heroes. i can't wait for statistics classes.

  32. Matthew Cline says:

    But I wouldn't rely on the general theory of relativity to fight a speeding ticket.

    No, for that you should use the uncertainty principle.

  33. babaganusz says:

    oh, and once again:
    @ I was Anonymous: wrong. the Captain Obvious pullquote you're looking for is "HEY GUYS LOOK, NAZI REFERENCE". nobody and no position was being compared to or equated with Nazis. Craig's framework was entirely apt (and arguably not even tasteless). don't hide your dim in-joke-glee behind the ongoing dilution of the entire point of Godwin's Law – going blithely out of bounds of the premise, there is no point whatsoever in even giving it a name.

  34. Christopher says:

    "An imperfect deterrent to terrorist attacks is better than no deterrent at all, "

    Uh… no. If the deterrent is operating at chance levels, then you've literally shown that an imperfect deterrent is no better than no deterrent at all.

  35. Anony Mouse says:

    On second thought, nevermind.

  36. Kathleen says:

    I just read a blog post by the child of a woman who ran a daycare center and got caught up in that hysteria. Even people who never got charged with anything had their lives destroyed by that witch hunt. http://killer-martinis.squarespace.com/my-wordpress/2013/11/24/its-time-i-really-introduced-myself

  37. babaganusz says:

    Anony Mouse • Nov 26, 2013 @1:41 am

    On second thought, nevermind.

    thank goodness someone is thinking twice…

  38. Tam says:

    "[P]ediatrician Nancy Kellogg, testified that the two young girls’ injuries were used in satanic rituals prevalent among lesbians."

    LOLWUT?

    Anybody who listened to that gem with a straight face should die of embarrassment.

  39. KR says:

    I don't believe I've seen a lesbian engage in any Satanic rituals, though I do know one who roots for the Yankees, which is similar.

  40. [Long string of expletives deleted.] Today I Learned that we still had people in jail for Satanic Ritual Abuse. I thought the last of those poor people got freed years ago.

    Don't mind me, I'll be over in the corner twitching for a while. The Satanic Ritual Abuse moral panic was a very bad time for me; I narrowly escaped going to jail for it myself.