Puff the Magic Dragnet

Politics & Current Events, Technology

Talking around the edges of what's classified is all the rage these days. See, for example, the commercial for the NSA that ran on 60 minutes tonight.

In that vein, a former employee of Tailored Access Ops explains (within Info Assurance guidelines) what he did at the NSA and why he's ok with it.

Insufficiently discussed in most rants about the NSA is this question: if the only way to find the needles in a haystack is to store the entire haystack, and if you're against storing the entire haystack, and if you insist that it's vital to find the needles, then given the size and growth rate of the haystack, how do you propose doing that?

Some are ok with storing the haystack. That's the status quo.

Some are against the haystack and also don't think finding the needles is all that important. After all, more die at the hands of swimming pools and ladders, etc….

But for those who think proactive action against malevolent actors is desirable, how (apart from surveilling a subset of exhaustive data) shall we winnow them out of an ever-increasing crowd and discern their voices in an ever louder din?

If not this way, then how?

Last 5 posts by David

93 Comments

93 Comments

  1. Jim Tyre  •  Dec 15, 2013 @10:00 pm

    David,

    First, you're right to call tonight's 60 minutes piece a commercial for the NSA. But you conclude with:

    If not this way, then how?

    Would you care to show any actual evidence that "this way" helps even a little, assuming the goal is laudable? The NSA, certain members of Senate Intel (notably Diane Feinstein) and others have made the claim, on many occasions. that having and searching the haystack has yielded many positive results. But virtually all the claims have been refuted rather thoroughly, often by the same persons who made them later saying that their earlier statements were not entirely correct.

    As one of the lawyers on EFF's NSA domestic spying team continuously since 2006, it's my business to know these things. Your premise has little to no support in fact.

  2. SIV  •  Dec 15, 2013 @10:12 pm

    We need to fight our own " malevolent actors" over here so we don't have to fight the other ones over there.

  3. Sami  •  Dec 15, 2013 @10:14 pm

    There are a couple of options that I can see, offhand.

    One of the easier ways to split the baby of privacy-versus-protection is to pass the haystack through a mechanical thresher. Going through all of everyone's data is beyond the scope of human enterprise anyway, so filter it through computer systems if you really want, but give no human being access to it without a warrant issued for probably cause.

    Because privacy is relevant largely to what other humans see. I would have a problem with a stranger reading my e-mail, but by the very nature of the medium, every e-mail I send and receive passes through multiple computers, and I don't care.

  4. Xenocles  •  Dec 15, 2013 @10:30 pm

    I'm not aware of any examples of the species you seem concerned with, David. Everyone I've read on this matter falls into either the camp that says "Any price is worth protecting our country from these people" or the one that says "We cannot cross these lines, even if doing so might save lives."

  5. Kittens R. Horrid  •  Dec 15, 2013 @10:40 pm

    Not much wrong with storing the haystack. It's the dredging without oversight—LOVEINT without consequences, 50% probability of foreignness checkboxes—that's a problem. Storing the haystack, though, also means it can get dredged through again in the future, whenever oversight is lowest.

  6. Zack  •  Dec 15, 2013 @10:46 pm

    Problem is we have precious little evidence it works, even if we allow the collection of the whole haystack. Let's set that aside for the moment and consider the final question. There are other ways, but they all involve offending the constitution in some form or fashion. The minimally invasive and least offensive option I can conceive of would be something along the lines of No Fly Listing and/or surveiling anyone without a security clearance who has been to/in Iran, North Korea, Cuba, Yemen, Somalia, Lebanon, Syria, or Oman since the Iranian Revolution. We make deals with Pakistan, Saudi Arabia, Egypt, Afghanistan, and Iraq that as long as they surveil within their countries and work to catch and prosecute terrorists, and share that data and those prosecutions with us, that we will not conduct surveillance or extraordinary rendition within those countries. If they fail to meet their burden, the deal's off, and we treat them like the rest. Otherwise we'll pay them the costs they incur for the surveillance and prosecutions we demand, which should (in theory) be a pittance compared to what we're spending on the NSA and to give them aid already.

    We could handle internal US issues by letting US citizens whom this would apply to that they are being surveiled and will continue to be surveiled until they submit to an extensive background check to clear their name. After they pass or fail it, they can go on their merry way.

    Again: almost certainly runs into a cavalcade of constitutional and international law issues, but that's the best solution I can think of given the parameters of the problem.

  7. Larry  •  Dec 15, 2013 @10:47 pm

    What is the cost to keep and filter the haystack. and what do we get for it?
    How many billion do we pay per terrorist found? How many citizens are saved per year to make the cost acceptable?
    Or is it J. Edgar Hoovers ghost? What does the NSA know about certain members of congress that they are such obvious lackeys?

  8. Aaron Spink  •  Dec 15, 2013 @11:05 pm

    The only problem I have with the modern NSA is the legitimizing of it. The NSA isn't fundamentally doing anything they haven't been doing for the past 25+ years. The main issue is that what they have been doing and how they have been doing it has become legitimized both legally and morally in many ways.

    The NSA was much better off when their sources of information and the existence of that information itself was almost completely clandestine. Instead of asking AT&T for the data, they simply dug into the ground and put a shunt onto the AT&T cable quietly at night. Same for undersea cables. But the important thing was that the NSA had no legitimate access to the data.

    Not having legitimate access to data is important as it constrains those with the data from revealing or using the data. This is actually a good thing for both our liberty and security. The NSA didn't want to reveal its sources of information even with other internal security agencies and were purposely vague and limited with info. It also made it so that internal access to the data was heavily restricted and monitored.

    This new modern legitimized NSA is far far too open and free with information and its all because they actually have legitimate access to the majority of said information.

  9. Carl 'SAI' Mitchell  •  Dec 15, 2013 @11:10 pm

    First, you assumed that the "store the haystack and sort through the whole thing" method actually works. That's not been shown to be the case.

    The thing about terrorism is that you can't stop it from happening, you can only respond to it. Terrorists can always change their targets, and the methods of attack are so varied that to remove them all would severely impact the average person's life. For example, a terrorist could drive their car the wrong way on a highway just before rush hour, killing a few and causing a massive traffic jam. Or drive a car into a police station. Etc. You can't just ban cars, and you can't remove the targets.

    So you remove some of the targets which are easy to attack and have high consequences for being attacked (eg strengthen the cockpit doors on aircraft) but you don't try to stop every terrorist getting onto a plane because that won't work. Even if you strip & cavity searched every passenger at some point an agent would miss something.

    Instead, you focus on disaster response, and old-fashioned police work to bring perpetrators to justice. You make sure you have trauma teams, good hospital ERs, etc, and you minimize the damage. You refuse to be terrorized, and you carry on with life. You never negotiate with terrorists, and you don't publish their names or their organization's demands in the news. You let them be just as anonymous as the (vastly more damaging) drunk drivers everywhere. Just as you can't prevent drunk driving completely you can't prevent terrorism completely, so you live with it and try to convince people to be better. When they're not convinced you have the police arrest them.

    Terrorists are just criminals, we need to stop pretending they're special. Treat them like drunk drivers, and stop glorifying them.

  10. David  •  Dec 15, 2013 @11:13 pm

    @Jim Tyre

    David,
    …But you conclude with:
    >>If not this way, then how?
    Would you care to show any actual evidence that "this way" helps even a little, assuming the goal is laudable? …As one of the lawyers on EFF's NSA domestic spying team continuously since 2006, it's my business to know these things. Your premise has little to no support in fact.

    I'm sure you're an excellent lawyer. For that reason, I expect you to avoid fallacies rather than commit them.

    One you commit is to suppose that an expression of the form "if not by means of X, then how?" affirms the efficacy of X. Suppose I had been discussing the use of healing crystals in Sausalito and I had asked those who think healing is desirable, "If not by means of amethyst on the chakras, then how?"

    Would you leap to the conclusion that I endorse this use of crystalline structures, and would you infer that I affirm the claims of those who do? (If so, then you're only an excellent lawyer by accident. ;) )

    Another you commit is to suppose that I've offered a premise that has little to no support in fact. To have done that, I'd have had to affirm a premise at all. As noted above, I have not. But even if I had done so, your rebuttal is weakly dependent on the inaccessibility of classified data.

    "Virtually all the claims have been refuted rather thoroughly", you assert. That's fascinating, since no data that would decide such questions have been, at this juncture, declassified or leaked. Nobody has declassified and presented datum Y, so you deny Y, accuse of credulity any who in ignorance are open to the possibility of Y, and call those who are in a position to know of Y liars. But constraints on speech are part and parcel of the classification system.

    That fact throws your advocacy on the horns of a dilemma: if you don't know whether classified cases of the system's efficacy could be adduced, then you literally don't know what you're talking about; but if you do know whether such cases could be adduced and you affirm that there are none, then you're either a rogue (violating the terms of your clearance) if telling truth or a scoundrel if telling falsehoods.

    I think you're neither a rogue nor a scoundrel; rather, I think you don't know and therefore assert blindly, calling for evidence from those who are legally constrained not to give it. In short, your unverifiable assertion is no more reliable than, say, Sidney Blumenthal's representation on the courthouse steps concerning his own closed testimony that he thought would remain sealed.

    Your comment may count as persuasion, but it isn't persuasion by valid argumentation.

    All that noted, you seem like a sharp and friendly guy, so I'll put the question to you again: if not by these means (which you're free to take as inefficacious if it help you over the hump), then how?

  11. Demosthenes  •  Dec 15, 2013 @11:21 pm

    "Insufficiently discussed in most rants about the NSA is this question: if the only way to find the needles in a haystack is to store the entire haystack…"

    Yes, I want the needles found. And yes, I don't want the whole haystack stored. But this statement here…you know, it's been a while since the logic course I took in college, but I still recognize a potentially false antecedent when I see one.

    Which is my fancy way of saying: I do not accept that the NSA must go about things in the manner it has chosen. Perhaps I could be convinced. But I haven't seen a case yet that would do so.

  12. David  •  Dec 15, 2013 @11:25 pm

    @Carl 'SAI' Mitchell

    First, you assumed that the "store the haystack and sort through the whole thing" method actually works.

    Nothing in my post takes a position on that issue, and taking a position on that issue isn't a precondition of considering the question I posed. Looks like the assumptions are all yours.

  13. A Lurker  •  Dec 15, 2013 @11:32 pm

    I have thought the NSA's methods of collecting the entire haystack basically idiotic. The problem is the vast majority of the traffic will be mundane such as ordering a pizza that almost certainly not to have any significance. What they should be doing shrinking the haystack to much more manageable size.

    Also, in terms of relative peace most signal intelligence is not likely to have any operational information. The US Navy's reading of the Japanese diplomatic codes in the Fall 1941 did not yield any clues as to actual Japanese military plans. In fact the analysts had trouble trying to determine when and where the Japanese were going to attack. The problem is that a competent terrorist organization will try to avoid sending mission plans via electronic media as much as possible. And if they do they will try to hide it inside innocuous sounding chatter similar to the Japanese "flower code" or the US Marine Corps Windtalker codes. These are well documented methods that have used before and the basic premise behind them is still valid.

  14. David  •  Dec 15, 2013 @11:44 pm

    "Insufficiently discussed in most rants about the NSA is this question: if the only way to find the needles in a haystack is to store the entire haystack…"

    Yes, I want the needles found. And yes, I don't want the whole haystack stored. But this statement here…you know, it's been a while since the logic course I took in college, but I still recognize a potentially false antecedent when I see one.

    Which is my fancy way of saying: I do not accept that the NSA must go about things in the manner it has chosen. Perhaps I could be convinced. But I haven't seen a case yet that would do so.

    I called attention to the size and growth rate of the haystack to suggest the impossibility of a scenario in which the winnowing keeps pace with the burgeoning. But maybe that was too subtle.

    Perhaps an analogy will help. Suppose I say that without leads, the only way to find that signed and dedicated copy of Miracle Mongers and Their Methods that I accidentally gave to Goodwill 20 years ago is to start in Barrow and search every home until you reach Key West. If the book exists in one of those homes, then following this Cat-in-the-Hat method should work. But that doesn't mean that it has ever worked, nor does it mean that it's an efficient way to search, nor does it mean that it's a good idea.

    Given sufficient time, all sorts of telling connections would surely be discerned. But nobody– not even the NSA– has that much time, and the flow of information is growing rapidly.

    My conjunctive antecedent, "if the only way to find the needles in a haystack is to store the entire haystack… given the size and growth rate of the haystack", was meant to dismiss this methodology for the sake of argument in order to call for alternatives– which I then proceeded to do.

    I still recognize a potentially false antecedent when I see one

    Not shown, but I welcome the opportunity to clarify my rhetoric. If my statement was awkward, I apologize for that.

    That said, what's your proposed alternative to the current attempt at haystack-winnowing?

  15. Marconi Darwin  •  Dec 15, 2013 @11:50 pm

    The size of the haystack has an upper bound. Sounds callous to say this but when the cost of preventing a terror attack is way more than the cost of the lives that would be lost, the trade-off will be made.

    The haystack accumulation has not been costly enough, so for now, it'll continue growing.

  16. Demosthenes  •  Dec 16, 2013 @12:01 am

    Would you leap to the conclusion that I endorse this use of crystalline structures, and would you infer that I affirm the claims of those who do?

    I can't speak for Jim Tyre, but I can speak for myself.

    In fact, yes, I would. As would most people. Because this isn't a formal logic classroom — it's a blog, where rhetoric (a closely related but hardly identical subject) generally rules. It's a common rhetorical tactic to counter someone by saying "You say this solution won't work; well, what's yours?" as a means of endorsing, if only by default, the solution your interlocutor has opposed. As an added bonus, the burden is now on him to make a case that you can pick apart, leaving your case looking all the stronger if you succeed. Plus the skilled rhetorician can always claim, truthfully enough, that he did not necessarily imply any argument whatsoever, placing his rhetoric behind an impenetrable shield of logic and allowing him to accuse his interlocutor of fallacious reasoning should he go after the rhetoric alone.

    I know from having read your posts that you're smart enough to know all of that, David. What I can't decide is whether the tension between your use of a common rhetorical strategy — and your insistence that you're not using it in the way it normally is used — is intentional, or merely a happy accident in your favor.

    That said, what's your proposed alternative to the current attempt at haystack-winnowing?

    Since this is the second time you have asked this question in the comments, I'm leaning toward the "intentional" interpretation. However, as I don't know that, I'll assume good faith and answer.

    I don't have an alternative. (I know enough to know that I don't know enough to design a solution, or even propose one.) Nor do I need an alternative in order to object to the way things are currently being done. Security is a valuable good, but it is neither the only good nor the most valuable one. Our government having so much data on us, data which could be abused in ten thousand ways with very little accountability for the abusers, is simply too high a price to pay for even an absolute level of security against terrorist threats — which our current system does not and cannot produce in any event.

  17. Jim Tyre  •  Dec 16, 2013 @12:13 am

    David,

    I don't think you're a rogue or scoundrel either. '-)

    We could engage in a battle of words on what you meant (or didn't) in your post. But I won't, because I credit you for knowing what you meant. So I will respond only to one specific statment in your comment in response to mine:

    "Virtually all the claims have been refuted rather thoroughly", you assert. That's fascinating, since no data that would decide such questions have been, at this juncture, declassified or leaked

    That statement is incorrect factually. Watch, for example, the December 11 Senate Oversight hearing at http://www.c-spanvideo.org/program/OversightofS It's more than two hours long, but for a quick highlight, listen to Senator Leahy starting just after 1:09 – paraphrasing, rather than quoting exactly, Leahy says that we've already established that what the NSA was referring to as 54 separate instances of the successful use of the section 215 bulk collection program was really just one instance. (Not that the government ever would exaggerate.)

    There's backup for what Leahy says. I have no more access to classified material than do you. But I study the public record, I know what's there.

  18. David  •  Dec 16, 2013 @12:14 am

    In fact, yes, I would. As would most people. Because this isn't a formal logic classroom — it's a blog, where rhetoric (a closely related but hardly identical subject) generally rules.

    I'll cop to being logical even at the expense of rhetoric.

    I don't have an alternative. …Nor do I need an alternative in order to object…. Our government having so much data on us, data which could be abused in ten thousand ways with very little accountability for the abusers, is simply too high a price to pay.

    Note that the folks who are actually tasked with solving the problems of preemption don't have the luxury of lacking a solution; they have to implement something, and then refactor, refine, or replace it along the way while actually pursuing their objective.

  19. Tarrou  •  Dec 16, 2013 @12:20 am

    @ Marconi,

    Human psychology doesn't work that way. That "callous calculation" cannot and will not be made until such time as no one cares about the issue. Humans are terrible instinctive judges of risk. The public will never accept "well, we did a CBO study and we figured out we could save money by allowing terrorist attacks". What they want is a long series of expensive and opaque programs to give them the illusion of security combined with an acceptable level of discomfort. People don't trust medicine that tastes too good, hence the interminable bitching but overwhelming acceptance of the TSA.

    When muslim terrorism goes the way of anarchist terrorism, we'll make that calculation about fifty years later.

  20. Christopher  •  Dec 16, 2013 @12:28 am

    I'm flabbergasted. I'd think here, of all places, the authors would notice that this is an argument against, minimally, the Fourth Amendment.

    After all if we need the whole "haystack" (Haystack being code for "reams of private information, acquired on little suspicion and/or by accident") then why should we force our intelligence agencies to go through the elaborate rigmarole of warrants and probable cause? If we need the whole haystack, why not just put a telescreen in every home? Why have any privacy at all? Why have a fourth amendment?

  21. David  •  Dec 16, 2013 @12:29 am

    Jim Tyre, Thanks for your productive reply. There's no doubt that exaggeration is rampant in DC.

    Claiming 54 hits instead of 1 hit with 54 nodes in the graph is an example. It's similar in kind to claims that KSM was waterboarded 183 times rather than 5 times with 183 pours.

    It all depends on what the meaning of a "hit" or a "session" is, right?

    I don't think the evidence you mention damages my point: specific instances in which intelligence discoveries enable preemption remain classified… partly because they're subject to gross misinterpretation of the sort in the Leahy example or in the many examples of early press coverage relating to NSA leaks.

    The 60-minutes piece, for example, included declassified information about the detection and preemption of state-sponsored cyber-attacks targeting zero-day BIOS vulnerabilities. While claims about the gravity of that threat and the course of its detection and prevention should invite reasonable, skeptical evaluation, confusion about whether it was one BIOS or 54 or 256 tells us nothing about how many such examples that could be adduced remain classified despite their persuasive potential. Likewise, semantic squabbles over one incident mentioned in open testimony to Congress doesn't really touch the deeper issue of arguments from ignorance.

  22. David  •  Dec 16, 2013 @12:50 am

    @Christopher

    I'm flabbergasted.

    Probably you're just tired or sleepy.

    I'd think here, of all places, the authors would notice that this is an argument against, minimally, the Fourth Amendment.

    To what does "this" refer? To what does "argument" refer? Remember, an argument is a connected series of statements intended to establish a proposition, and the original post lacks some of those characteristics.

    After all if we need the whole "haystack" (Haystack being code for "reams of private information, acquired on little suspicion and/or by accident") then why should we force our intelligence agencies to go through the elaborate rigmarole of warrants and probable cause?

    You seem to be conflating the concept of "needing to store it" with the concept of "needing to rifle through and inspect it". On the present scheme, the need to store it arises not because it's likely to be relevant. (As @A Lurker points it, it consists mostly of calls to Papa John's and texts about picking up the kids from soccer.) Rather, it arises because nestled somewhere in the midst of it all is a salient detail indistinguishable from its neighbors unless/until linked by hops and probabilities to some known malefactor.

    Mom can i stay at Jenns 4 sup? luvu

    Hey, sorry I haven't replied. We've been crazy busy...

    Les sanglots longs des violons de l’automne blessent mon cœur d’une langueur monotone.

    Your lucky Powerball numbers today are...

    If we need the whole haystack, why not just put a telescreen in every home? Why have any privacy at all? Why have a fourth amendment?

    Because there's a conceptual and material difference between what you say and do within the walls of your castle and what you fling unencrypted from your parapets onto wires and waves…. Sending a digital communication (email, phone call, text) is different in kind from activities such as hanging a Kinkade over your couch.

    That doesn't mean that comm is automatically fair game; but it does mean that you have only yourself to blame if you don't become cryptoliterate.

  23. Sun Tzu  •  Dec 16, 2013 @12:59 am

    As someone critical of the NSA, I'm a little confused how this came off as a blog piece critical of critics of the NSA.

    It read to me as a question posed to people within the security state apparatus asking what on earth they thought they were doing with that gigantic haystack over there to just keep throwing hay on it rather than start sifting through it.

    Though I'd have to agree, pretty much everyone who comments with some level of informed statement on this issue falls into the "kill Edward (Snowden)" or "pardon Edward" camps in some kind of elaborate prank on tweens rather than possessing nuanced positions that somehow liberty and security may be in some idealized balance point or that one may be crafted. There may be a real audience for that debate, but so far it seems to be limited to people who have no idea what's going on and little interest in it.

  24. David  •  Dec 16, 2013 @1:03 am

    @Sun Tzu, That's either a great point or a terrible point. :)

  25. Christopher  •  Dec 16, 2013 @1:06 am

    Sending a digital communication (email, phone call, text) is different in kind from activities such as hanging a Kinkade over your couch.

    I find that to be a fairly facile argument. The government is perfectly capable of bugging your home, or looking into it with infrared cameras.

    I mean, you're just sending those heat waves out there, where any random schmoe could pick them up, so what right do you have to complain?

    I feel like you're kind of just question-begging here; Can the NSA legally open your mail? Could they search your vehicle without a warrant? After all, you're the one putting it out there, in plain view where anybody could see it.

    And on the other hand, surely, surely the things that terrorists do in the privacy of their own homes are deeply relevant to National Security? Surely we wouldn't put the telescreen in for the millions of people hanging kinkades; it's to see the people who are building fertilizer bombs.

    I mean, my question back to you is, if not through the telescreen, then how do we find those people building the fertilizer bombs, those needles in the haystack of life?

  26. David  •  Dec 16, 2013 @1:14 am

    @Christopher, you seem to be confused about the differences between the FBI and the NSA.

    Surely we wouldn't put the telescreen in for the millions of people hanging kinkades; it's to see the people who are building fertilizer bombs.

    I remain undecided about which is the greater offense.

  27. Bear  •  Dec 16, 2013 @1:21 am

    Well… I'm an RKBA/2A activist. I carry for personal defense. Funny thing though, I'm not allowed to defend myself by listening in on all my neighbors' phone calls and email, just in case they might be plotting against me (and if I ever try, you can be sure that I finally tipped over into full-blown paranoia). I don't even get to shoot people who approach on the street with a troubled look upon their face on the assumption that they're about to attack me. I have to wait until they do something that the mythical "reasonable man" would reasonably consider a threat to life or limb.

    Funny thing: One of the alleged jobs of the US government is protecting my right to personal defense. That was the idea behind the Second Amendment which codified a preexisting right. But the government takes that a little further by [allegedly] protecting everyone through "reasonable" limitations on my right to shoot threats.

    Rest easy; this isn't meant as a RKBA screed. In fact, I won't respond to anyone who wants to argue that aspect, since it distract readers from the topic at hand: the NSA. What this is… is noting the depressingly ironic contradiction of what our "protective" government supposedly does for and to us.

    "If not this way, then how?" Maybe the NSA all government agencies should have to comply with the same limitations imposed on the citizens. You know… probably cause, reasonable expectations, appropriate applications of force, that sort of thing.

    If they can't, maybe we should be asking much more basic questions regarding what the government really is there to do.

  28. Vagrant  •  Dec 16, 2013 @2:44 am

    It seems to my largely uninformed eyes that the only other viable options for searching the haystack fall into two general categories:

    Constellation searching is the first, where a particular email or piece of text is highlighted to a human operator if it contains multiple target features; I.E. an IP address activated within the last month (I) is used to purchase a firearm (II), or some similar combination of factors thereof. It seems to me as if this would be some kind of integrated search function that security personnel could establish a search term list for and run almost every day, with whatever recent clues they had obtained from other sources added in. Such a search might require a warrant but as it fails to target a specific person it'd likely be much easier for law enforcement to obtain.

    Not that warrants are remembered much in the current environment, one admits.

    To follow your metaphor, this would be searching the haystack with a magnet instead of just shoving one's hands into the hay and hoping to not get pinched.

    The other concievable method is that the identity tagging of suspects via warrant to follow what recognizable information they send – anything that touches a social network, sent out through a known IP address, or used on a chat service or mail service with a known username. Diagramming their movements, essentially, in hopes of following the web strands back to the spider.

    I presume both of these are already used to some extent, but following them to their logical extent instead of the current data hoarding fetish of the NSA would seem the logical move.

  29. Cat G  •  Dec 16, 2013 @2:52 am

    David,

    First, excellent name.

    Your post suppositions that searching for needles within the haystack is a worthy endeavor, but that our methods are flawed. And indeed, our methods are flawed – storing the haystack is inefficient, as are the search methodologies. My position is that the haystack (and needles therein) are irrelevant to the concept pre-emptive prevention of getting poked by needles.
    The more efficient and permanent solution is not to focus on the needles, but the manufacturers of needles. All needles must be created and forged in some way – the methods by which raw material is turned to a needle are known. We do not need the haystack to discover them – in many cases we helped to create the forges and provided the hammers and tongs which were used to create the needles. It is more efficient to concentrate on these manufacturers of needles – from there, discovering the needles becomes more efficient. It's as if each needle still bears threads tying it back to its manufacturer, which can be followed with relative ease through the haystack.

    I think I've overstrained the metaphor for the time being – but imagine a world of velcro, adhesives, and transdermal patches. What need has such a world for needles? Perhaps the answer is not to search the haystack, but to make needles obsolete.

  30. Catty  •  Dec 16, 2013 @3:28 am

    It is also important to note at what cost. the current way of sorting the haystack is inefficient because the efficient ways didn't funnel enough money to contractors.

    Computers, analysits and so forth don't work for free. Assuming for a moment that even all the exagerations about stopped plots where true (and they are not) you would still have had better outcomes if you instead funneled half of the money to FEMA and blew the rest on hookers and crack.

    In fact internal NSA analysts have complained about some of the taps that drown them in a deluge of low quality data, ie making the haystacks bigger without adding more needles. But contractors get paid well for them so they get expanded.

    The only thing these programs seem good for is funneling taxpayer money to crooks.

  31. Tribal  •  Dec 16, 2013 @4:29 am

    You spread out the haystack and run a powerful magnet over it.

    This is what Sami suggested, and is something, at least on a smaller scale, that we can do. My wife works in e-discovery, and her software uses human-guided optimized searches to find the requested data faster, and even demonstrably more accurately, than a room full of paralegals and attorneys. As a lawyer, I was skeptical, but she showed it to me in practice (using the publicly-available Enron data) and it works as advertised. The problem is that you really do need a few highly-skilled technicians (I'm a prosecutor, and as a non-lawyer she makes twice my salary) per several ten million documents, plus low-level data entry people.

  32. JdL  •  Dec 16, 2013 @4:30 am

    There won't be any more needles to find if we stop roughing up the world's haystacks. We are creating the problems we claim to be trying to fix.

    I guess saying this makes me NOT one of the people who "…think proactive action against malevolent actors is desirable…", so you really aren't addressing the question to me, but I answered anyway. ;-)

    I'm content to live with the existing risks of terrorism in exchange for stopping our government from being the biggest danger to world peace on the face of the globe. Not to mention being the biggest criminal organization on the face of the globe.

  33. Wade  •  Dec 16, 2013 @5:10 am

    I wrote a long comment about the lack of exceptions contained within the text of the Fourth Amendment, but then I reread the post and reminded myself what the author was asking. Namely, how do we prevent terrorism without violating the Constitution?

    The answer is simple and unpleasant. We can't. We can be free or safe. Not both.

  34. cpast  •  Dec 16, 2013 @5:25 am

    @Wade: Do you mean we can't be both completely free and completely safe? Or do you mean if we're even slightly free, we're not at all safe, and if we're even slightly safe, we have no freedom? If the latter, then you have a debate between max freedom and max security, with nothing in between. If it's only the former, though, then a) I don't think anyone disagrees – you can't be completely safe *ever*, for one thing; and b) it still leads to the question, where to draw the line?

  35. Damon  •  Dec 16, 2013 @5:50 am

    I'm all for this NSA program…when they can provide clear convincing documental evidence that it's worked here in the states. Oh wait, that's classifed isn't it? Guess you know what you need to do….declass it and prove your assertions or STFU and end the program.

  36. JdL  •  Dec 16, 2013 @5:50 am

    @Wade:

    The answer is simple and unpleasant. We can't. We can be free or safe. Not both.

    Being unfree does not make us safe. I would challenge you to name a single country in which lost freedoms resulted in greater security for the populace.

  37. J@m3z Aitch  •  Dec 16, 2013 @6:27 am

    @David, responding to @Carl 'SAI' Mitchell.

    [SAI] First, you assumed that the "store the haystack and sort through the whole thing" method actually works.</em?

    "Nothing in my post takes a position on that issue,"

    Doesn't
    if the only way to find the needles in a haystack is to store the entire haystack
    assume that? The question is incoherent unless the method actually works. That is, storing the haystack can't be the "only" way to find needles unless it is "a" way to actually find needles.

    And there's the problem, I think. You're addressing a crowd that largely rejects the premise of the question.

    But the question also rigs the game against anyone who might try to actually answer your actual question. "If this is the only way [then] if not this way, then how?" You've asked a question that you've denied has an answer. That's why everyone's answering different questions or questioning the question itself (well, that, and the fact that such is the nature of blog comments).

  38. Crusty the Ex-Clown  •  Dec 16, 2013 @6:46 am

    Holy guacamole, if it's true we each commit three felonies a day, and if the NSA accumulates and stores indefinitely all of our email, metadata, financial transactions, medical records, etc., etc., then we are all so hosed. Right there has the foundation been laid for the modern police state. Should anyone become a serious annoyance to the government just a quick sift of the haystack would produce a number of possible reasons for incarceration.

    Ben Franklin war right…

  39. Steve  •  Dec 16, 2013 @7:06 am

    Proactive action against malevolent actors is desirable.

    The solution that does not employ the Haystack-Sift in its present form is a solution that relies on the creation and implementation of more tools germane to the venue of observation. Essentially, in any dense set containing a potentially non-empty subset of entities we wish to identify, which are indistinct from the general set entities, it is necessary to implement a filter, usually a mechanical (reliable) one, to cause the differences between the entities we wish to identify and the other entities in the set to become more clear.

    In most cases heretofore, this has been implemented through the use of spot-checks of a randomly selected subset of the general set. (Example: Crowd surveillance using Markov Random Fields — http://goo.gl/yyt2Dt — apologies, I cannot link to the full content of this paper.)

    The better solution is to develop the correct tool to cause the communications from the malevolent actors to become clear at a purely mechanical level, and turn the task over to a machine and react appropriately to any alerts it generates. (I'm simplifying here, we'd want multiple redundancies to safeguard against tampering, but that's a logistical problem, not a technical one.)

    I understand that the people tasked with ensuring security in this manner don't have the option of saying "it'll be 5 to 10 years before we can develop this technology" to their scared political masters, but that was not the question that was asked.

    A machine-executed sort is better than a human one for purely security-focused reasons.

  40. ZarroTsu  •  Dec 16, 2013 @7:10 am

    My answer, 90% sarcastically and 10% seriously, is "a magnet".

    I guess I don't have what it takes to work for the NSA.

  41. Rhonda Lea Kirk Fries  •  Dec 16, 2013 @7:11 am

    @the universe at large

    I'll cop to being logical even at the expense of rhetoric.

    If only that were not such a rare practice.

    @Cat G

    …imagine a world of velcro, adhesives, and transdermal patches. What need has such a world for needles?

    Transfusions. Infusions.

    It's a nice idea, but the technology doesn't exist. We have to work with what we've got.

    @David

    If not this way, then how?

    With whatever tools remain after everyone has vented their collective spleen about the tools currently in use.

    We'll bitch about the methods until the next failure of intelligence, and then we'll scream bloody murder that our government did not protect us. Rinse and repeat.

    As a rule, I don't watch TV, but now and then, I'll give an hour to mindlessness and useless agita. I find myself appalled by the tactics of law enforcement, and I bemoan the abuse of the innocent, even as I feel relief that those same tactics result in caged criminals.

    I have no idea how to resolve the conflict.

  42. Quiet Lurcker  •  Dec 16, 2013 @7:15 am

    'How do we stop terrorism before is happens' presumes that we can do so. I put it to you, we can't. Period. Throw out the 4th Amendment. We still can't. Throw out the 5th Amendment. Same thing.

    We can't, because the systems we have in place; the systems that seem to work to stop terrorism, are built around authorities/society at large having some indication that something is going to happen.

    Columbine happened because no one knew or could predict what those kids would do. Same thing with Newtown. Same with the navy yard shooting. Same with this last one this last week or so.

    Those shooters were not on the radar.

    The shooting in Aurora, Co., could have been stopped – IF the system had worked as designed. It didn't.

    And it is literally impossible to change or re-engineer the system so that things like that will be caught. It just can't be done.

  43. Duncan  •  Dec 16, 2013 @7:33 am

    It's an interesting question, and one that will not be answered here. Or by our children. Or grandchildren. Etc. Etc. An inherent problem is that we aren't even sure in some cases what the "needle" is until after the needle has stuck in someone's foot. What is "terror?" @Carl"SAI"Mitchell was right – a terrorist can simply drive a car into other cars. There are as many ways to kill, maim, and terrorize people as there are people in the world. And in many cases, the acts themselves only worked as a result of chance events. Archduke Ferdinand would not have been killed by Princip if not for a comedy of random events. Look at 9/11: would it be at all possible today for a tiny group of men to take over an airliner with nothing more than some boxcutters? Filled with wary U.S. citizens?

    So you can't stop it completely, and can't even know what most of the "needles" are until after the fact. But the problem isn't the needles. It's the haystack. No matter how good a magnet you develop that is ostensibly tuned only to the needles and no other metals in the haystack, you still have to run the magnet through the haystack.

    I don't believe that the vast majority of folks at the NSA are intent on reading my love letters. I think they really honestly want to protect us all with the tools available. But at some point they have to look through the dumpster filled with our communications and, so long as we want the government to do SOMETHING, the haystack will be searched. Perhaps I'm just not looking at this creatively enough, but at this point I think the only thing that we can do is what humans are particularly good at – muddle through as best we can. Demand that the magnets get better and more precise, demand oversight, and demand that the hay be burned after a period of time (remember that we have to keep it around for at a least a while to fine tune our magnet; think Google's search algorithm). We WILL lose information because the hay must be burned. We WILL lose information because we're fine tuning the magnet after the fact in many cases. Some people will be hurt or killed because we missed a needle. Innocent people will have their dirty laundry looked through. And yes, some people in power will use the magnet for nefarious and unlawful purposes. I don't like the system, and I'm not defending it. But I feel the same way about it that Sir Winston Churchill felt about democracy. It's the worst form of surveillance except all the others that have been tried.

  44. David C  •  Dec 16, 2013 @8:06 am

    if the only way to find the needles in a haystack is to store the entire haystack, and if you're against storing the entire haystack, and if you insist that it's vital to find the needles,

    Well, if you make THOSE assumptions, it's obviously impossible. You say X is the only way to do Y, but you're against X, but Y is vital. So either you need to find a way to do Y that doesn't involve X, or you decide whether not doing X is more important than doing Y. Those are pretty much the only choices.

  45. OrderoftheQuaff  •  Dec 16, 2013 @8:07 am

    The haystack confers power on those who possess and can access it. You'll vote how I say or I'll disclose all your kinky fetishes to the world.

  46. Rob R.  •  Dec 16, 2013 @8:41 am

    NSA. I think NSA is just a word so young men like you can wear a suit and have job.

  47. Brad Hutchings (@BradHutchings)  •  Dec 16, 2013 @9:10 am

    David, if "this way" actually worked, Facebook would be worth 10x what it is. The fact is that there is a terrible amount of noise built into the social graph. Modeling it with the goal of catching bad guys is necessarily going to overlook a lot of bad guys and catch a lot of innocent people in nets of suspicion. Look at the ads Facebook serves up, and they actually have people willfully contributing information to their social graph model.

  48. Shane  •  Dec 16, 2013 @9:44 am

    The trouble with haystacks is that they have many needles. When you are looking for needles usually you don't find the needle that you are looking for but another needle. Maybe said needle isn't as big as the needle that you are looking for, maybe said needle isn't as shiny as the one that you are looking for, but it is a needle none the less.

    The question becomes should we remove these needles from the haystack. Hell, should we even be concerned about a specific needle or should we just search the haystack generically for needles because we all know that every haystack contains at least one needle.

    That IMHO is the danger of storing the haystack.

  49. Ron Larson  •  Dec 16, 2013 @9:53 am

    I like the "Store the haystack in our barn" analogy.

    The problem is that once your neighbors hear you have the haystack, want to start combing through it for their own agendas. That is what scares me most. Already the MPAA (private industry) is demanding access to the NSA's haystack to find those dreaded movie pirates. Seriously. Then want the NSA to help them find kids who ripped and shared "GrownUps 2".

  50. Aaron  •  Dec 16, 2013 @9:58 am

    David, my proposed alternative:

    Develop suspicion via traditional investigative methods. When you have reasonable, articulable suspicion about a person, articulate it reasonably to a judge and get a warrant for that person's communications or phone records or what-have-you. Follow leads from that to find collaborators and other terrorists.

    I don't think our government should be in the business of finding needles in haystacks, or any other kind of suspicionless dragnet collections.

  51. Shane  •  Dec 16, 2013 @10:01 am

    @Carl 'SAI' Mitchell

    Instead, you focus on disaster response, and old-fashioned police work to bring perpetrators to justice. You make sure you have trauma teams, good hospital ERs, etc, and you minimize the damage.

    It seems police tactics need to come up with the times also. I can't find the article, but it seems in the Arapahoe High School the police have changed tactics, instead of cordoning off they now rush in. This may seem like a small change but it's impact will be huge at least in active shooter situations. Many lives will be saved with this one small change.

    I agree with you, we don't need to empower more legal criminals we need to change with the times and use ingenuity to solve problems instead of taking the lazy liberty destroying path of guilty until proven innocent.

  52. mud man  •  Dec 16, 2013 @10:04 am

    We COULD try introducing civilization to the places where we think of as terrorist incubators. Like, drones dropping nutritional supplements and medical kits in remote parts of Pakistan, rather than home delivery of explosive ordinance.

    Course, that wouldn't work for treehuggers, animal rights hate mongers, Occupy 99-percenters, folks like that who walk among us like normal citizens, but we haven't been hearing much from them lately anyway. The other problem is it puts a lot of analyst types back on the street.

  53. melK  •  Dec 16, 2013 @10:19 am

    Well, one alternate way is to convince a piece of straw to turn into a needle then pick up the needle. But you hardly need a haystack to do that.

  54. Shane  •  Dec 16, 2013 @10:28 am

    @Tarrou

    … the illusion of security …

    Hammer meet nail.

  55. Shane  •  Dec 16, 2013 @10:29 am

    @melK

    Your link is dead. Try again :)

  56. Shane  •  Dec 16, 2013 @10:37 am

    @Cat G

    Perhaps the answer is not to search the haystack, but to make needles obsolete.

    Human ingenuity precludes this.

  57. Not stupid  •  Dec 16, 2013 @10:38 am

    Rather than store a haystack just use a really strong magnet. All needles found. No hay touched. Very easy.

  58. Shane  •  Dec 16, 2013 @10:48 am

    @Quiet Lurker

    The shooting in Aurora, Co., could have been stopped – IF the system had worked as designed. It didn't.

    And it is literally impossible to change or re-engineer the system so that things like that will be caught. It just can't be done.

    This reminds me of the business cycle, and the attempts by an all knowing body that is trying to override it.

    You are right, we will pass laws trying to stop the next financial meltdown from happening, but they won't. The irony is that the new laws actually insure the next meltdown is worse, and so and so forth ad nausem.

  59. ZarroTsu  •  Dec 16, 2013 @10:50 am

    If only we had some sort of device that could turn hay into needles, then we could ban hay altogether!

  60. Jesse from Tulsa  •  Dec 16, 2013 @11:08 am

    How? NOT MY PROBLEM.

    As a citizen of the United States, land of the free, my primary duty is to protect my freedom. In rare instances that may mean sacrificing portions of it, but always balanced. In the War on Terror, we are sacrificing enormous amounts of freedom to fight a vague and minimal threat.

    Evening including the 911 attacks, terrorism over the last 15 years, 20 years, or 30 years has been a minimal threat to Americans. Not even registering on the most likely cause of death lists. Dog bite, accidental death by toddler… higher than terrorism. We have created a massive industry to "fight" terrorism, with no cost benefit analysis.

    So the government can stop compiling information on citizens just in case it needs it later. How it does its job without violating the 4th Amendment is not my problem.

  61. C. S. P. Schofield  •  Dec 16, 2013 @11:53 am

    The problem is that, for assorted reasons, we have taken common sense off the board. Heaven forfend we we actually make a distinction between "exercising freedom of Religion" and preaching violent jihad, and throw the jihadis out of the country so hard they skip across the Atlantic like so many flat rocks. For one thing, doing so might cause people to look closely at the expressed opinions of the Reverend Jeremiah Wright, and that would Never Do. God forbid we tell Syria "the very next time we trace a terrorist act to your money or training, we decapitate your government", and then follow up on it. We Couldn't Possibly simply allow upstanding citizens to carry their personal firearms onto airplanes, so that the next time a handful of camel-pestering swine try to hijack a plane they get shot to doll rags. And if we actually traced, prosecuted, and jailed every dim-bulb who gave terrorists financial support we'd have to imprison a significant portion of the Professoriat and most of the Kennedy family.

    And so we end up playing high tech super spy games. Because the low tech solutions that history tells us would do so much to suppress Jihad would give the Liberal Left hysterics.

  62. Demosthenes  •  Dec 16, 2013 @11:59 am

    Note that the folks who are actually tasked with solving the problems of preemption don't have the luxury of lacking a solution…

    Which is actually beside the point, and is a line that will make people feel empowered to engage in equivocation if seriously advocated. To wit: "Something must be done; this plan is something; therefore…"

    I admit that there are many areas in which I lack the detailed knowledge necessary to come up with a plan that would solve a pressing problem. I don't know how to reduce violent crime rates, for example. That wouldn't make it acceptable for the state and its authorities to place cameras in everyone's homes as a deterrent.

  63. Jim Tyre  •  Dec 16, 2013 @12:06 pm

    BREAKING:

    In one of the several cases challenging the section 215 bulk metadata collection program, a court in D.C. has just issued a preliminary injunction in favor of the plaintiff. (Though the court also stayed enforcement of the injunction pending appeal by the government.)

    68 page ruling at http://legaltimes.typepad.com/files/obamansa.pdf Plaintiff has shown a substantial likelihood of success on the merits on his Fourth Amendment claim.

  64. SimpleMachine  •  Dec 16, 2013 @12:23 pm

    If and when you get pricked with a needle, remove it from the haystack.

  65. tom  •  Dec 16, 2013 @12:33 pm

    "That security we had is gone. North Korea has nuclear weapons and is threatening to fire them at the US."

    BAHAHAHA! OMG, an actual intelligence professional thinks that NK seriously has the capacity to nuke us? I'm shaking in my boots! Quick! Take my rights!

  66. JdL  •  Dec 16, 2013 @1:00 pm

    @C. S. P. Schofield:

    God forbid we tell Syria "the very next time we trace a terrorist act to your money or training, we decapitate your government", and then follow up on it.

    Are you saying you'd advocate the U.S. "decapitating" Syria, subject to such "tracing"? Could you please clarify:

    . Exactly why is it our role to "decapitate" anybody anywhere in the world? Since government and other criminal entities large and small exist everywhere, what, if anything, should limit the number of "decapitations" we righteously carry out?

    . What does the word "trace" mean? What standard of proof, if any, will be employed? Are you at all concerned with the possibility of incorrectly attributing guilt and then following up with bombs? What, if anything, should be the punishment for somebody who made such a mistake?

  67. James  •  Dec 16, 2013 @1:01 pm

    a) I don't think anyone disagrees – you can't be completely safe *ever*, for one thing; and b) it still leads to the question, where to draw the line?

    The line is this: Does this search/query/storage/dragnet affect any thing/person/datastore not specifically named in a warrant, issued by a judge based on a reasonable articulation of a crime?

    No? You can't touch it.

    Yes? You may investigate/search/examine it. If there is specific, articulable reason to retain/detain/store it, do so. If not, release or destroy it.

    In short: the government may not enter my home, examine my belongings, nor detain my person without cause.

    Will bad guys go free, and terrorists have more opportunity to blow stuff up? Yep.

    Will the government have less power to cherry pick the haystack, looking for needles to poke innocent people with?

    Yep.

  68. Owen  •  Dec 16, 2013 @1:34 pm

    @David:

    Note that the folks who are actually tasked with solving the problems of preemption don't have the luxury of lacking a solution; they have to implement something, and then refactor, refine, or replace it along the way while actually pursuing their objective.

    I am reticent to put words in your mouth, so could you expand on this for a moment and explain a) what you are asserting and b) how that assertion fits into this overall discussion?

    It reads as if you are implying that because someone is "tasked" with solving a problem, their solution is better than the baseline simply because it is intended to be a solution, rather than due to an actual benefit or improvement derived therefrom. If you are not implying or asserting any position, however, I wonder what the purpose of the comment could be, other than an appeal to prejudice or pathos ("something MUST be done!").

  69. Xenocles  •  Dec 16, 2013 @1:45 pm

    Something has to be done.
    This is something.
    Therefore, this has to be done.

  70. StewBaby911  •  Dec 16, 2013 @2:38 pm

    Regarding the asserting that there is no evidence that collecting
    the haystak has led to any prevention(s)..

    " Judge Leon also emphasized that he was unpersuaded by the government’s claims that the program served the public interest, pointedly noting that it failed to cite “a single instance in which analysis of the N.S.A.'s bulk metadata collection actually stopped an imminent attack, or otherwise aided the government in achieving any objective that was time-sensitive.”

  71. James  •  Dec 16, 2013 @2:57 pm

    Suppose I had been discussing the use of healing crystals in Sausalito and I had asked those who think healing is desirable, "If not by means of amethyst on the chakras, then how?"

    Would you leap to the conclusion that I endorse this use of crystalline structures, and would you infer that I affirm the claims of those who do?

    Absolutely, and I think the majority of people would. Similarly, if someone asked, "How you can eat at McDonalds?" I would think they did not think eating at McDonalds was a good idea, even though their question is purely about the mechanics of eating.

    "If not by X, then how?" is a rhetorical device usually employed to convince people of X. If that was not your intention then I would submit that is an odd fact about how you communicate, not a fault of the reader.

    The reason for this, of course, is because it is a very silly question if you don't believe in X. "If not cyanide, then what will I eat for dinner?" would rightly prompt anyone to ask, "Why on Earth are you bringing cyanide into this?" (or "Are you okay?"). Similarly, I don't ask, "If not by time machine, then how should I travel to work?" even though it is logically equivalent to "How should I travel to work?" based on my current understanding of physics.

    The thing is, the question is a silly one if X is not an effective method. In my above example of cyanide for dinner, a perfectly good answer is, "Well, if your only option is cyanide, I'd recommend no dinner at all." So that is my answer to your question, if the only option is storing the haystack then I'd recommend simply not doing it, and I can hold that position even though I think that we should, in general, try to find those needles rather than not. Even with nothing else to eat, and with a strong interest in eating, I still wouldn't have the cyanide.

  72. Quiet Lurcker  •  Dec 16, 2013 @3:01 pm

    @James -

    I would change your question from

    The line is this: Does this search/query/storage/dragnet affect any thing/person/datastore not specifically named in a warrant, issued by a judge based on a reasonable articulation of a crime?

    to
    "Does what I am searching/looking at/handling/taking match in all specifics the description contained in a search warrant from a court of competent jurisdiction?"

    If my answer is 'no', then I have no business knowing it even exists.

    If my answer is 'yes', then I will search or handle or take in the manner specifically described in the same warrant, or if that does not exist, then following a procedure that is set forth as being commonly acceptable for such circumstances.

  73. Xenocles  •  Dec 16, 2013 @3:23 pm

    ""If not by X, then how?" is a rhetorical device usually employed to convince people of X. If that was not your intention then I would submit that is an odd fact about how you communicate, not a fault of the reader."

    Thank you for your entire post, but especially this, James. Good rhetoric is founded on rigorous logic and should include a good deal of it, but by its very nature it rounds off some of that rigor in order to communicate more effectively. At best David employed a false rhetorical cognate, much like the novice Spanish speaker who increases his embarrassment by attempting to express it.

  74. Dion Starfire  •  Dec 16, 2013 @4:06 pm

    The haystack approach is a result of targeting (people with) the intent to commit a terrorist act.

    The best alternative I'd suggest is either target a different leg of the Motive-Opportunity-Means triangle.

    Some other, harder to swallow options include:
    * Accept that level of destruction as a natural consequence living in a massive concentration of people (there aren't many cities on the entire planet where tens of thousands of people can be affected by a single human attack)
    * Adjust our national policy so we're less of a leader (and thus a target) on the world stage.
    * Provide better protection (i.e. armoring structures, anti-intruder measures, etc.) for probable targets to reduce the impact of attacks when they do succeed.

  75. Rhonda Lea Kirk Fries  •  Dec 16, 2013 @4:50 pm

    If not by X, then how?" is a rhetorical device usually employed to convince people of X.

    Unless it's employed as a starting point to brainstorm alternate tactics to achieve the same goal.

  76. Xenocles  •  Dec 16, 2013 @4:51 pm

    "Unless it's employed as a starting point to brainstorm alternate tactics to achieve the same goal."

    Even then, the implication is that the resulting ideas have to be better than X.

  77. AlphaCentauri  •  Dec 16, 2013 @5:03 pm

    If the data collection had the capability of identifying terrorists in a prompt enough fashion to prevent an attack, we wouldn't all be getting robocalls with spoofed caller ID numbers for Cardmember Services, GE Home Security, the free cruise to the Bahamas, etc.

    It's not just a case of whether we should pursue the capability we have. We clearly don't have the capability. We are drowning our intelligence staff in useless data, just as Stasi did, and likely making it harder to find criminals.

    We might be able to analyze data in retrospect to find out who was responsible for an attack so we could all start removing some other article of clothing before getting on some other mode of public transportation in hopes of averting a similar attack in the future, but I see zero evidence it can reliably prevent attacks in the first place.

    Prevention of attacks/keeping us safe is the reason used to justify the program. We're not all that interested in the names of the people who died in a suicide attack. And even when we know their names and the names of their superiors in a terrorist organization, it took us years to find Obama (despite his being, what, 6' 6" in a country full of short people?) and we have no manageable way to bring such people to trial.

    So there is little preventive benefit. What are the harms? In addition to the expense, and in addition to the "ick" factor of someone pawing through our personal information, there is the important issue that the NSA has capabilities that exceed those of their supposed overseers. They are capable of gathering data on politicians and military officers without those persons knowing about it or being able to control it. The NSA can and perhaps already has become a supergovernment without checks and balances. It may well be that they have enough dirt on enough people and their friends and campaign donors that no one dares try to reign them in. And if it hasn't happened yet, there needs to be a damn good mechanism in place to prevent it, because otherwise it will happen. Absolutely guaranteed.

    As far as the excuse that those who have nothing to hide have nothing to fear, the fact that everyone knows the NSA has this quantity of data means that they can claim anything they want about you and people will believe them. So it's not a question of whether you have something to hide, but whether you have something worth taking. If you own valuable property, the government would not have to obtain your property legally if they can simply claim they scanned your internet traffic and learned you were running an international child porn distribution server. A wealthy person who wanted your land could bribe the right person and arrange for law enforcement to seize your assets and throw you in jail, and you would have no way to prove your innocence.

    I am saying this, and I'm the flaming liberal around here. It's not just a question of whether you believe in government or not, it's understanding basic human behavior and how things have happened over and over in history when there were not adequate controls on the acquisition of power in any one person or group of people.

  78. C. S. P. Schofield  •  Dec 16, 2013 @5:34 pm

    @JdL

    Maybe I'm mistaken, but my impression is that pretty much nobody other than Syria denies that Syria funds (and directs) a fair amount of the terrorism in, for example, Lebanon. Sort of like, nobody really denies that in the 1970's a significant proportion of the funding of the IRA came from Boston.

    All I'm suggesting is that, assuming that we want to curb international terrorism – or even simply terrorism at home – making believable threats against the States that sponsor same is likelier to have a positive effect than listening to everybody's phone calls.

  79. Aaron  •  Dec 16, 2013 @6:10 pm

    @J@m3z Aitch

    Doesn't if the only way to find the needles in a haystack is to store the entire haystack assume that? The question is incoherent unless the method actually works. That is, storing the haystack can't be the "only" way to find needles unless it is "a" way to actually find needles.

    This is the best summary I've seen so far of what's wrong with David's questions and why I reject the very premise of them.

  80. Tom  •  Dec 16, 2013 @7:29 pm

    "Even then, the implication is that the resulting ideas have to be better than X."
    And if you'd taken even half a minute to think through the post, that is the point – we are supposing that we must perform "proactive action against malevolent actors" and asking if not what we do now, for David elliptically implied that we all know that's not acceptable, what ought we do?

    That this is not the instant interpretation of any careful reader is shocking to me.

  81. Erwin  •  Dec 16, 2013 @7:29 pm

    …well…I'd ask…how did we deal with espionage and terrorism before the internet? Then, I'd just do that. We survived. There is very little evidence for a new need for massive surveillance dragnets and a very real danger arising from developing that capability in our government.

    …that said…I'm flexible enough to wonder if bioengineering advances are getting to the point where they might justify considerably more draconian measures. Ten+ years ago, we were at the point where a not terribly talented graduate student could get stuck with what was arguably biological weapons work (developing an air-stable viral shell for AIDS) at major public medical institution.* Time has marched on. At some point, Uncle Jimmy's 5k custom virus printer is going to cause trouble. I don't really care if you bring machine guns to the local nursery, but I'm a bit nervous about you testing smallpox v3.0.

    –Erwin

    *Oddly, it sailed right past the IRB. They'll nail you for using unclear descriptions of tiny radiation doses, but, eh, build AIDS dust as part of an invitro study and you're golden.

  82. Rhonda Lea Kirk Fries  •  Dec 16, 2013 @8:16 pm

    Even then, the implication is that the resulting ideas have to be better than X.

    Yes.

  83. Xenocles  •  Dec 16, 2013 @8:29 pm

    You must spend most of your time shocked, Tom.

    As it turns out, "those who think proactive action against malevolent actors is desirable" includes nearly enough everyone in the world. But nothing is done in a vacuum. A custom-built sports car is desirable too, but the six-figure car loan that comes with it and the costs of ownership are not. Thus for most people it applies to, the question "For those who desire a custom-built sports car, if not the McLaren F1, then what?" has "Nothing at all, including the F1" as an acceptable answer.

    As it happens, the people who think that "it's vital to find the needle" almost to a man are absolutely fine with any costs they have to bear to do it. That's what vital means. They might wish the costs were lower, but they don't really grumble about paying them. To a slightly lesser degree the people who reject the current security scheme do so with a reasonable consideration of the cost. Many of them deny there is a substantial cost; the remainder seem willing to accept one if it exists. I am familiar with all three kinds, and within their frameworks their arguments are reasonably consistent. This other sort, the kind David seems to be addressing, is only theoretical for me. I'm not familiar with anyone of prominence on this issue who has expressed this earnest desire for limitless cake without the understanding as to why it can't be done.

    And it turns out that nothing is a perfectly acceptable answer. It's okay to not have a solution – you are perfectly within your rights to do nothing but shoot down proposals; it is your duty to do so for truly unacceptable ones. To suggest otherwise is to advance a sort of dishonest argument by default, a shell game of burden shifting and false choices. If there is no apparent acceptable solution, then you reevaluate your premises. You change some or all of your desires (you do without), your aversions (you put up with it), or your available resources (accept a different price and make up the difference somewhere else).

  84. David  •  Dec 16, 2013 @9:47 pm

    For those who may be confused on this point: I am by no means an advocate of the Store All The Things approach and its Cat-in-the-Hat counterpart. Given infinite space and time, the data yield; lacking those luxuries of opportunity, we have a poor understanding of what swags, flags, and tags may actually buy us.

    Tickets to security theater? That's an idea too readily entertained by folks who want to be seen seeming to do something useful.

    A demurral and deferral on cracking the conundrum? One might argue that the stakes aren't really high enough until nukes or bugspray come into play. Maybe they won't; maybe they will. It seems as if the "do nothing" approach implicitly carries an expiration date.

    One problem is shiny object syndrome. Having tasted the sweet Orwellian technonectar, some busy bees are loath to give it up whether the plant yields fruit in season or not. Another problem is overclassification to the point that review becomes ingrown, insular, and something else beginning with 'i'.

    To the folks playing on the periphery of "Something must be done; this is something; …": that's witty but obtuse. Pointing out that someone somewhere is trying something isn't an apologia; it's a way to underscore just how larval and uncertain the whole business remains despite its scale and energy. The current system is a pre-adolescent Pantagruel.

    That's precisely why it's worth pondering the alternatives, thinking outside the safe harbor, sailing the SCIF to a blue ocean….

  85. Jim Tyre  •  Dec 16, 2013 @9:52 pm

    That's precisely why it's worth pondering the alternatives, thinking outside the safe harbor, sailing the SCIF to a blue ocean….

    Nice, David. '-)

  86. Cat G  •  Dec 17, 2013 @4:06 am

    @Rhonda – Yes, as I was saying the metaphor was getting to be rather strained at that point. There are legitimate uses for actual needles. The metaphorical needles (aka terrorists) on the other hand have few if any "legitimate uses" in the context of our societal compact. The core tenet in play is that said "needles" are going outside of acceptable redress to employ tactics which instill fear in the hope of driving change. (With the small exception, of course, of men/women like the fictional Joker, who just want to watch the world burn. Which I would argue are not terrorists, but deranged psychopaths.)
    Breaking from the metaphor, we do actually have the ability to take steps which mitigate the rationales and reasons that drive people to take such extreme views and actions. Others above have mentioned a few ways, and there are (rather terrible and unethical methods) that could also be employed (although they would render us as little better and in some ways worse).
    But rather than to focus on the needles after the point of manufacture, I believe it would be better to remove the fires of the forge that allows them to become hardened, sharp steel points bloodying us. Attacking the root cause, to me, will always be more efficient than attempting to triage the symptoms.

    @Shane
    Human ingenuity is indeed vast, but so too are the often overlooked traits of compassion and understanding. And laziness. If given the choice of planning and executing an attack or maybe just going to work, most people will go with the less active option. It takes a strong stimulus to push people into active plotting and action.

    On furtherance of the topic, I think it's incrdisingenuousgenious to include spree killers in with terrorists as "needles". The motivations and actions are not the same. The results may be similar, but they aseparateeperate problems. If searching the haystack for needles, you are not often going to find the strands of hay that have broken and become sharp on their own. The two have some overlap (I believe self-radicalization may be included in both subsets) but it would require a far greater intrusion to provide "perfect" safety against people whose mental illnesses progress to dangerous violence.

    Going back to the haystack and needle metaphor – a magnet may help, but I think the Mythbusters have already tackled this and it was still horribly inefficient. Even if you also burn the hay to get to the needles. And the current dragnet style approach seems to me to be burning the barn, the hay, and still losing needles in the remaining ashes.

  87. Will Ollie  •  Dec 17, 2013 @10:09 am

    I am surprised that people continue to be distracted by hashing and rehashing the red herring of 'terrorism' and preventing it, to senselessly waste energy on that fabrication. God its an effective tactic to get people from pulling back the right curtain, get people up in bunches by exploiting fear.

    I have always thought it, though now confirmed..well, lets let mr. snowden explain succinctly:

    "These programs were never about terrorism: they're about economic spying, social control, and diplomatic manipulation. They're about power."

    Any social upheaval/protest/push back now is met with a militarized police.

    So, why should we waste time discussing how to solve the red herring? We have a system in place in which the shameful private habits or scandal are hung over the heads of leadership and judges, and it doesn't have to be their mistake, but someone close to them that they would try and protect. You don't get to run for president or have an influential position in gov't without being threatened. That is what should be the prime discussion to some degree.

  88. mud man  •  Dec 17, 2013 @10:24 am

    The metaphorical needles (aka terrorists) on the other hand have few if any "legitimate uses" in the context of our societal compact.

    There's a problem here with definition by example. OK flying airplanes into skyscrapers is beyond "legitimacy" however defined. How many peeps on the no-fly (etc) lists are up for that? Are you willing to bear any burden, etc, to extinguish people who take video cameras into slaughterhouses? who camp in public parks as political theater? who chain themselves to redwood trees? who take pictures of cops doing what cops do? All those things are interfering with the smooth functioning of society as we know it.

  89. James  •  Dec 17, 2013 @10:35 am

    That's precisely why it's worth pondering the alternatives, thinking outside the safe harbor, sailing the SCIF to a blue ocean….

    I do appreciate that we need to find better ways of dealing with terrorism, but I still think that the comparison to the haystack sets us on the wrong path. The whole data collection thing is a way to find terrorists, but the idea that we can stop terrorism by finding terrorists isn't necessarily a good one. We haven't stopped the drug trade by finding drug users or even drug dealers.

    The solution to terrorism is more complicated than the solution to drugs (legalizing bombing things probably won't work). But I think we can be far more effective at reducing it by looking at how the terrorist system works than by looking for individuals who are about to commit acts. Reducing the number of people interested in terrorism by half is just as good at stopping plots as finding half the would be terrorists.

    Terrorism from within is much more of a threat than terrorism by foreign bodies. Mass killings are already relatively commonplace in the US. If you really want to reduce terrorism you need to make the government work for the people.

    Many of the people who praised Mandela after his death were the same people who called him a terrorist decades ago. The thing is, he pretty much was a terrorist by any sensible definition. Now we say, "Well, apartheid was so bad we can understand why someone would be moved to violent action." That's the main way to reduce terrorism – try not to be so bad that people feel the need to blow things up and kill people to make a point. I think the US could improve a *lot* in this regard.

    As you note, eventually there will be terrorists who can get their hands on nukes. When that day comes, the only real protection is not being the kind of place that they want to nuke.

  90. kgb999  •  Dec 17, 2013 @3:30 pm

    if the only way to find the needles in a haystack is to store the entire haystack, and if you're against storing the entire haystack, and if you insist that it's vital to find the needles, then given the size and growth rate of the haystack, how do you propose doing that?

    That's a pretty big "granted" to start this thought experiment with. Suppose the most efficient way to safeguard stacks of hay – which may or may not have needles in them – does not involve storing the haystacks at all? The purpose of collecting hay is to feed the cattle, after all. You can't just keep it forever without modifying the reason hay is being gathered in the first place.

    In that case, one must ask if the perversion of purpose inherent in keeping all the haystacks does more or less damage to the system fed by hay than the increased risk of missing a needle incurred by sticking with long-established techniques for detecting and removing needles without disrupting the core system that is theoretically being protected in the first place.

    It's not that more people die from car wrecks than from eating needle-hay … it's that needle-hay has been a problem since well before the Victorian-era anarchists ran around bombing stuff. It's not like we've sat in darkness up to this point helpless and unable to pick a few flipping needles out of our hay.

    Sure, a needle gets missed from time to time. But. Boston? Let's get real here. They locked down the whole damn city for a whole day and still didn't find the guy … who it turns out was hiding in a boat at the end of a trail of blood from his abandoned vehicle. If they had just processed it like any other violent crime scene, they'd have tracked the guy down in no time. To me all evidence is that this crap doesn't even work. Never mind being unconstitutional as all hell.

    Maybe the key isn't to try and extract scary voices from an incomprehensible din at all. Maybe it's better to employ proven investigative technique to identify the specific voices belonging to people breaking the law and then get a warrant to listen in only on the voices we've established need listening to. And to do it better than just OK. Then turn all those other resources wasted on hoarding hay over to doing *stuff like that* instead of stroking themselves collecting dossiers on the detailed comings and goings of *#$&!~ World of Warcraft.

    Big data falls victim to the post hoc fallacy. Every terrorist in America could well be listed in the phone book – this does not mean that reading the phone book long and hard enough will identify America's terrorists. The whole framework is based on a purely unproved theory.

  91. Mark - Lord of the Albino Squirrels  •  Dec 17, 2013 @10:41 pm

    A series of modest stratagems for the acquisition of sewing implements from animal feed.

    1) Just look for the damned needle
    – difficult, tedious, old fashioned, and sometimes successful.

    2) Purchase a strong magnet, then look for the needle using that.
    – like a sting operation, but with pins and dried grass.

    3) Burn the hay
    - the more hay you can ignore, the better. Filter, filter, filter.

    4) Throw the hay in a pool and stir. Then look at the bottom of the pool.
    - study and focus on your target in other words.

    5) Destroy ALL needles.
    - might require a heightened military budget

    6) Destroy ALL hay
    - that hay is probably a needle in disguise anyway

    7) Store all the hay
    - a rat attracting fire hazard. Besides, the needle might be back where you got the hay in the first place – not recommended

  92. Cat G  •  Dec 18, 2013 @2:41 am

    @Mud man

    All those things are interfering with the smooth functioning of society as we know it.

    None of those things are terrorism. There is a difference between activism and terrorism although some activists may also be involved in terrorism. Interfering with, as you say "the smooth functioning of society" (I am unaware that society has ever functioned smoothly, as a whole) is not terrorism. By common definition, terrorism is the attempt of a group to force change on another group or to enforce political change through the use of violence to create fear in the larger population. Mass casualty attacks, threats of mass casualty attacks, etc.

    The scenarios you put forth are not in this mold – someone camping in the redwoods, someone chaining themselves to a tree, videotaping or photographing anything… none of these things are threatening violence against the cause they wish to change. They may be inconvenient, they may be irritating, but they are not inspiring fear or causing death to the opposition of the cause they are promoting.

    The only activism that I can come up with commonly (maybe) is the act of malicious sabotage from some environmental groups against logging equipment – but not the vast majority. (Pouring sugar into a gas tank is not the same as firebombing the equipment. Spiking a tree… I dunno.) "Freeing" lab animals wouldn't count, most likely, unless you're knowingly releasing animals that are carrying biological agents for the purpose of causing a large death toll.

    One of the problems I see is that suddenly everyone winds up being a terrorist, even when they are not. It confuses the issue, perhaps deliberately, and helps no one.

    As for the red herring, Will ollie – it's the public justification for the haystack. Whether it's the real motivator or not, if you strip them of that fig leaf it becomes an even harder proposition for anyone to support. It strips any legitimacy from the action. So yes, it's a "red herring" to go after. Because when you eliminate the herring, it's harder for someone to claim that you're just a wackjob conspiracy theorist and marginalize you. It's not about what you believe. It's about what most people believe.

  93. kgb999  •  Dec 18, 2013 @2:15 pm

    @Cat G
    The entire point of spiking trees is to make loggers afraid that cutting into a tree may result in physical injury or death. Not every tree … but maybe that next one. The tactic uses intermittent reinforcement to place workers in a state of constant fear with the purpose of forcing them to stop logging activity. I think there needs to be a degree of perspective when crafting public policy responses, but it's definitely a violent act that falls under most definitions of terrorism.

    The other problem is spiking a tree can end up being like leaving a land mine in a war zone. Nobody ever comes back and pulls them out. Years later, there is no way a forestry professional trying to hold a fire line or address an infestation of invasive species could ever know they are about to encounter the unexploded ordinance of eco-battles past.

    I'd put it above any form of equipment monkeywrenching in terms of being violent. Burning equipment is mighty dramatic, sure … but it's still just damaged equipment. Doing something that's expected to at least maim people is on a whole different level.

    It doesn't change your underlying point, but I thought it worth noting.