The Future of Mass Dossiers
The Future of Mass Dossiers

JURIST Guest Columnist Christopher Slobogin of Vanderbilt University Law School says that a set of new guidelines for the National Counterrorism Center’s use of information contains provisions which are troubling from a privacy standpoint, and should be modified to require more congressional oversight…

The US Department of Justice (DOJ) recently issuedGuidelines for Access, Retention, Use, and Dissemination by the National Counterterrorism Center and Agencies of Information in Datasets Containing Non-Terrorism Information” [PDF]. As this prolix title implies, the National Counterterrorism Center (NCTC), the key agency for organizing and analyzing national security intelligence, routinely acquires and accesses datasets about US citizens that contain personal details having nothing to do with terrorism. These datasets could contain information about credit card transactions, airline reservations, phone and ISP communications, bank, tax and social security records and perhaps even medical consultations. Nowhere do the guidelines mention these datasets by name, but previous intelligence practices make clear they are among the sources the NCTC wants to consult. While the government was probably collecting much of this information before the events of September 11, 2001, since that time it has clearly been aggressively engaged in doing so, through a series of programs known successively as Total Information Awareness, Terrorism Information Awareness, ADVISE (Analysis, Dissemination, Visualization, Insight and Semantic Enhancement) and, most recently, “fusion centers,” where information from various sources is “fused” together.

Before promulgation of the guidelines, the government could legally retain information that is not presently linked to national security concerns for no longer than six months. The guidelines extend that period to at least five years. The impetus for this change was undoubtedly the belief that effective counter-terrorism is possible only if intelligence agencies have access to every conceivably relevant piece of information, which is less likely to occur if datasets are frequently purged.

The public has very little information about the correctness of this assumption. They do not know how many “terrorists,” if any, have been caught as a result of burrowing through these huge datasets, nor whether prolonging the time all of this personal information remains in government hands is likely to improve whatever the hit rate was under the old regime. We do know that, as a theoretical matter, algorithms designed to detect terrorist activity through mining mammoth databases are extremely unlikely to work, given the needle-in-the-haystack nature of searching for terrorists among the general population. We also know that too much information can be as damaging to investigative efforts as too little.

Let us assume, however, that data-mining can sometimes work in the national security context or, more plausibly, that the availability of large amounts of information in a single government repository facilitates following up on investigative leads about terrorism. If the government has the money to spend on this type of surveillance, isn’t it better to be safe than sorry? Indeed, why limit government retention of this information to five years? Why not collect every possible tidbit of information on everyone who lives in or has any connection with the United States and maintain it forever, or at least until death?

After all, the guidelines direct that individuals who use the datasets be trained in how to access them and possess the relevant security clearance, and insist that auditing systems track who accesses the datasets and for what purpose. The guidelines further declare that dissemination of data to other entities may occur only for “lawful” purposes and require that procedures be established for ensuring the security of the datasets and for correcting erroneous information. The guidelines also set up enforcement mechanisms. The entire process is subject to the oversight of a Civil Liberties Protection officer, who must conduct periodic reviews, as well as spot checks, to ensure compliance with these various safeguards. The NCTC itself must also conduct periodic reviews of its adherence to the rules, and if a “significant failure” is discovered it must submit a report describing the problem to various government entities, including the DOJ and the Inspector General. Finally, the Privacy and Civil Liberties Oversight Board is to have access to all of these records and reports.

Ideally, only those interested in counter-terrorism will see the information contained in the datasets, they will use it only to combat terrorism, and any divergence from that goal will be discovered and reported to those higher up in the hierarchy. A similar set-up has occasionally resulted in embarrassing disclosures about FBI failures to follow its own rules. Maybe it will work with the national intelligence community as well.

But probably not. The problem starts with the breadth of information the NCTC can legally maintain and access. The guidelines define “terrorism information” as any information relating to “terrorist groups or individuals,” as well as information about “groups or individuals reasonably believed to be assisting or associated with such groups or individuals.” That definition of the data government may gather is already pretty broad. But the guidelines also authorize the NCTC to collect datasets that contain “non-terrorism information,” which is defined, believe it or not, as information “that has not been identified as terrorism information.” In other words, the NCTC can collect, access and retain any piece of information about anyone, and hold it for five years. While the Guidelines state that “analysts may not browse through records in the dataset that do not match a query with terrorist datapoints, or conduct pattern-based queries or analysis without terrorism datapoints,” that language applies only when the NCTC is seeking access to information contained in a database maintained by another entity. With datasets that it acquires and maintains on its own, the NCTC “may conduct (i) queries that do not consist of, or do not consist exclusively of, terrorism data points, and (ii) pattern-based queries and analyses.”

Then there are the generous rules governing dissemination of information in the NCTC’s possession. Terrorism information, as well as information that “reasonably appears to be necessary to understand or assess terrorism information,” may be sent to “a federal, state, local, tribal or foreign or international entity or to another appropriate entity that is reasonably believed to have a need to receive such information for the performance of a lawful function.” Note that the “function” that triggers information disclosure need not be related to counter-terrorism; it only needs to be “lawful.” Moreover, NCTC dissemination of information may also occur if reasonably necessary to determine whether it “constitutes” terrorism information.

Given the mindset of the government post-9/11, the breadth of this language is understandable. Once the premise is accepted that any and all information can help discover terrorist activities, these types of rules are the natural outcome. Unfortunately, they provide the government with enormous discretion. Perhaps the Civil Liberties Protection officer will detect any egregious actions on the part of intelligence authorities, but that assumes that these officers will be given sufficient access to the doings of intelligence officials, whose entire modus operandi is covert; in any event, the amount of information the NCTC collects and the number of people who will have access to it create a situation that is likely to overwhelm even the most diligent monitors. It doesn’t help that the guidelines recognize no specific remedy, either systemic or against individuals, for any “significant failures” that are discovered. In fact, the guidelines state that the rules they establish

are not intended to, and do not, create any rights, substantive or procedural, enforceable at law or in equity, by any party against the United States, its departments, agencies, or entities, its officers, employees, agents or any other person, nor do they place any limitation on otherwise lawful investigative or litigation prerogatives of the United States.

Even with superior oversight and enforcement, mistakes and mishaps will occur. The databases will be hacked. The wrong people will be targeted, interviewed and arrested because of data errors. Mission creep — the use of “terrorist” and “non-terrorist” information to detect domestic crime and simple regulatory problems — is inevitable. All of this has already occurred under current programs, as I and others have documented. It will only be more likely to happen with the mega-sized and mega-prolonged data collection process now contemplated.

The know-it-all state is one that tends to be a state that oppresses, because those with knowledge are tempted to use it. Congress recognized that fact when it defunded the Total Information Awareness (TIA) program back in 2003; undoubtedly it was heavily influenced by TIA’s eerie icon, depicting an all-seeing eye on top of a pyramid accompanied by the logo “Knowledge is Power.” Even the late Chief Justice William Rehnquist, no enemy of the government, was leery of panvasive investigative techniques like TIA; as he wrote in 1974 soon after he joined the Court, “most of us would feel that … a dossier on every citizen ought not to be compiled even if manpower were available to do it.”

So how can the country be protected from terrorist threats without making all of us feel (to borrow a concept from the Fourth Amendment) less “secure” from government intrusion? One solution is to ban the creation and maintenance of databases devoted to ensuring the government has comprehensive dossiers on all of us. Another is to keep any databases that are developed under lock and key until the government develops articulable reasons to suspect a particular person or upcoming event is associated with terrorism. A third solution combines aspects of these two approaches: members of Congress must be told the precise databases the NCTC plans to accumulate and maintain, and must also be made aware that the NCTC will be gathering this type of information about them as well as everyone else. If, under this condition, Congress is still willing to sanction the NCTC’s data-gathering practices, then something like the guidelines, together with judicial oversight to ensure that data accumulation is not exercised in a discriminatory or pretextual manner, could be allowed. But if Congress remembers the reasons it defunded the TIA program and acts accordingly, the NCTC should not be able to create or maintain databases of its own, but rather should have to rely on databases created by other entities for other purposes. In either case, the NCTC should only be able to access database information if it has developed a demonstrably plausible terrorist profile, an articulable suspicion about the terrorist designs of individuals or groups it targets, or an articulable suspicion that a terrorist act is about to take place that the data can help pinpoint.

Christopher Slobogin is the Milton Underwood Professor of Law at the Vanderbilt University Law School, where he serves as Director of the Criminal Justice Program. Slobogin’s research focuses on criminal law, criminal procedure, mental health law and evidence. He served as reporter for the American Bar Association’s Task Force on Law Enforcement and Technology and is the author of Privacy at Risk: The New Government Surveillance and the Fourth Amendment.

Suggested citation: Christopher Slobogin, The Future of Mass Dossiers, JURIST – Forum, Apr. 11, 2012,

This article was prepared for publication by Caleb Pittman, an assistant editor for JURIST’s academic commentary service. Please direct any questions or comments to him at

Opinions expressed in JURIST Commentary are the sole responsibility of the author and do not necessarily reflect the views of JURIST's editors, staff, donors or the University of Pittsburgh.