Justice Department paid AT&T for access to 4 billion call records a day for federal, local drug investigations

Hemisphere project, revealed by NYT, has AT&T employees sit alongside drug units to aid access to data in exchange for payment

US law enforcement officers working on anti-drugs operations have had access to a vast database of call records dating back to 1987, supplied by the phone company AT&T, the New York Times has revealed.

The project, known as Hemisphere, gives federal and local officers working on drug cases access to a database of phone metadata populated by more than four billion new call records each day.

Unlike the controversial call record accesses obtained by the NSA, the data is stored by AT&T, not the government, but officials can access individual's phone records within an hour of an administrative subpoena.

AT&T receives payment from the government in order to sit its employees alongside drug units to aid with access to the data.

The AT&T database includes every phone call which passes through the carrier's infrastructure, not just those made by AT&T customers.

Details of the program – which was marked as law enforcement sensitive, but not classified – were released in a series of slides to an activist, Drew Hendricks, in response to freedom of information requests, and then passed to reporters at the New York Times.

Officials were instructed to take elaborate steps to ensure the secrecy of the Hemisphere program, a task described as a "formidable challenge" in the slide deck, which detailed the steps agencies had taken to "try and keep the program under the radar".

The instructions added that the system should be used to generate leads towards new material, with call records obtained through standard subpoenas then used to provide evidence. The "protecting the program" section concluded that "all requestors are instructed to never refer to Hemisphere in any official document".

A key purpose of the Hemisphere database appears to be tracking "burner" phones used by those in the drug trade, and popularized in the long-running drama "The Wire." Slides published in the Times reveal details on how Hemisphere traces "dropped" phones and "additional" phones used by law enforcement targets.

A Justice Department spokesman said in a statement given to the New York Times that "subpoenaing drug dealers' phone records is a bread-and-butter tactic in the course of criminal investigations."

He added the program "simply streamlines the process of serving the subpoena to the phone company so law enforcement can quickly keep up with drug dealers when they switch phone numbers to try to avoid detection" – a similar explanation to those given by intelligence officials of the NSA's PRISM program, which grants the NSA access to information held on the servers of some of the largest internet companies.

The DoJ spokesman refused to disclose the cost of the program, telling the Times the figure was not immediately available.

Jameel Jaffer of the ACLU told the Times the program raised "profound privacy concerns".

"I'd speculate that one reason for the secrecy of the program is that it would be very hard to justify it to the public or the courts," he said.

Separately, the Washington Post published fresh details from the US National Intelligence Budget request for 2013, released by the former contractor Edward Snowden.

The document details a multi-million dollar program to prevent "insider threats" from intelligence officers, with plans to launch more than 4,000 investigations into unusual staff activity at the agencies, including downloading large numbers of documents or accessing material which they would not need in the normal course of their duties.

Such steps were in motion before Snowden disclosed any material to journalistic organisations, or the publication of such details in the Guardian and Washington Post. The NSA has yet to establish what material was taken by Snowden, according to press reports.

An NSA spokeswoman told the Post contractors were not included among the 4,000 planned investigations of security clearances. More than 500,000 US contractors hold top-secret clearances.

guardian.co.uk © Guardian News and Media 2013

["Stock Photo: A Visibly Angry Police Officer" on Shutterstock]