I thought the WSJ made an interesting point in an editorial this morning:
The NSA is collecting “metadata”—logs of calls received and sent, and other types of data about data for credit card transactions and online communications. Americans now generate a staggering amount of such information—about 161 exabytes per year, equal to the information stored in 37,000 Libraries of Congress. Organizing and making sense of this raw material is now possible given advances in information technology, high-performance computing and storage capacity. The field known as “big data” is revolutionizing everything from retail to traffic patterns to epidemiology.
Mr. Obama waved off fears of “Big Brother” but he might have mentioned that the paradox of data-mining is that the more such information the government collects the less of an intrusion it is. These data sets are so large that only algorithms can understand them. The search is for trends, patterns, associations, networks. They are not in that sense invasions of individual privacy at all.
If the NSA isn’t scrubbing vast amounts of data, then it can’t discover who is potentially a threat. The alternative to automated sweeps is more pervasive use of lower-tech methods like wiretaps, tracking and searches—in a word, invasions of persons rather than statistical probabilities. The political attack on data-mining could increase rather than alleviate the risk to individual rights.