From Turing to Snowden: How a WWII intelligence pact between the U.S. and UK forged modern surveillance

Revelations show transatlantic intelligence pact started in second world war is expanding beyond states' ability to control it

There haven't been too many moments of levity over the past four months for those intimately involved in the story of Edward Snowden. It hasn't been a laughing matter for the man himself, who is now stuck in Russia, the intelligence agencies whose secrets he has disclosed, or the governments that have had to deal with the consequences.

But the impasse between the opposing forces in this unprecedented and complex saga has been broken on occasion. One of these moments came at the Guardian's London headquarters, near King's Cross station, on Wednesday 17 July.

The scene was a second-floor office overlooking Regent's Canal, the time 11am. On one side of a large, round wooden table sat two senior officials from the Cabinet Office, nursing cups of coffee and unconcealed irritation. Facing them were two journalists from the Guardian.

After hollow pleasantries and firm handshakes, the conversation turned to the right to freedom of speech on issues that might affect national security. And, though no voices were raised, the message – which had come directly from the prime minister – was loud, clear and intended to unnerve.

The Guardian had become a target for every intelligence service in the world, intoned the grey-suited official. His colleague nodded. She took notes. Hostile foreign agencies would be using all manner of low tricks and high technology to get hold of the classified files gifted to us by Snowden.

The tactics might include anything from pointing long-range lasers at plastic cups used by our reporters (very good for eavesdropping apparently), to bribing members of our staff. Had we recruited anyone in the past few weeks, they inquired? Any Chinese, perhaps?

The Guardian had to be alert to such dangers because the threat could come from anyone, anywhere, at any time.

"Some of the best assets of the best intelligence services in the world will be interested in you," said the bespectacled official. At precisely that moment, and with implausibly good timing, two window cleaners slowly dropped into view on the outside of the building. In a cradle hanging from the roof, they soaped, swiped and polished, moving slowly up and down just a few feet away.

"Are they yours or the Chinese?" said one of the editors. Even the mandarins managed a smile.

For a few moments the absurdity of the situation overwhelmed the seriousness; and if it wasn't quite football between the trenches at Christmas, the meeting ended cordially, with both sides recognising the difficulties of the other.

Since then, the debate has become rather more polarised, entrenching views at a time when a more rounded and less doctrinal discussion might be better for the people who really matter in all this.

Those people do not include Snowden or the reporters working on the stories; or the directors of intelligence who have been so affronted by the disclosures; or even the presidents and prime ministers on whose watches surveillance has entered a new, remarkable, era. The principal characters in this drama are not the giant computers used for storage, analysis and codebreaking, or the technicians who built them.

The people who really count are the millions who send emails or search on Google or use mobile phones – and expect privacy. Those who use Skype, send direct messages on Twitter, post on Facebook or rely on the internet to buy groceries also make up the cast. Because all this information, whatever safeguards you have taken, can be swept up, decoded and analysed by British and American intelligence agencies.

Arguments over Snowden's motives, whether he is a whistleblower or a traitor, whether his disclosures have damaged the agencies or just embarrassed them, may never be reconciled. But on one matter there is no doubt. The highly classified files have shown that espionage has changed.

The world immortalised by John le Carré drew a distinction between those who were in "the business" and those who were not. This withered with satellite communications and died when everyone began to research and speak to each other online. We are all part of "the business" now.

Without offering details to anyone outside their inner circles, western intelligence agencies embarked on a new strategy – data trawling.

When the first of the Snowden revelations was published in the Guardian in June – revealing that the NSA was secretly storing and analysing details of millions of phone calls made in the US – the transformation was recognised immediately.

"The administration is saying that without any individual suspicion of wrongdoing, the government is allowed to know whom Americans are calling every time they make a phone call, for how long they talk and where," the New York Times said in an editorial.

"Through a series of legal contortions Obama has argued that Congress, since 9/11, intended to implicitly authorise mass surveillance. But this strategy mostly consists of wordplay, fearmongering and a highly selective reading of the law."

As the focus of the stories turned to the UK over the following weeks, it would soon become clear that all those criticisms could be levelled at Britain too. The UK hasn't just been a partner in this technological adventure, it has been a pioneer, with the two countries working more closely in the field of intelligence-gathering than in perhaps any other since the second world war.

The bombe and the Purple machine

The gardens at Woking crematorium in Surrey are neat and peaceful, and full of well-tended rows of red, pink and yellow autumn flowers. Under the blooms sit further rows of small, clean white postcards, on which mourners have written their thoughts and prayers for the deceased.

The crematorium was founded in 1878 (to the indignation of residents who didn't want the town to become the first in the country to have such a godless facility) and sits directly opposite the Winston Churchill sports centre.

A few of the dead have memorials here; many more have their names recorded in the books of remembrance that are stored in the vaults. Alan Mathison Turing is not one of them.

Turing was cremated here on Saturday 12 June 1954, five days after he died. His ashes were spread by his elder brother John, in Tennyson Lake Garden North, a secluded 1.6-hectare (four-acre) garden next to a pond where their father's remains had also been scattered.

Archives at the crematorium only record that Turing was 41 and that his occupation was "university reader". There is nothing else to mark his death.

The Times obituary, which appeared on the same day, noted Turing was a mathematician and logician who had branched into "the design and use of automatic computing machines".

The piece bemoaned how the second world war had "interrupted Turing's mathematical career for six critical years between the age of 27 and 33" – and his death had deprived the world of a man who could have "made much greater discoveries". Few people knew the truth: Turing was a master codebreaker, and during those "lost" war years, he had made one of the most important discoveries in British military history.

He had enabled the codes used by the Nazis to send messages to and from their commanders to be cracked.

The story of Turing and the team at Bletchley Park was not one GCHQ wanted to boast about in the 1950s or in the decades thereafter; his work was top secret, his private life complex and, for the time, scandalous.

Turing was gay. He killed himself by eating an apple laced with cyanide two years after he had been convicted of having a sexual relationship with a young man from Manchester.

Just about everything Turing did in his professional and personal life, and the environment in which he did it, has changed since his death.

But if you are tracing the roots of the relationship between GCHQ and the NSA, to understand why the agencies work so closely together, and why they seem so genuinely perplexed (and angry) by the furore now surrounding them, then it is to Turing and his contemporaries that you have to turn.

Looking back is also the only way to appreciate how the intelligence agencies have ended up on the path to mass surveillance, and managed to travel a long way down it, without facing the kind of public scrutiny they are confronted with now. The fathers of the institutions that have become security behemoths were men such as Turing and Wolf Friedman, an American cryptologist who was as brilliant as his British counterpart although somewhat less eccentric.

There had been codebreakers before these two, but their work was on the verge of a technological revolution that is still going on today.

Working in Hut 8 at Bletchley Park, then the home of GCHQ's forerunner, the Government Code and Cypher School, Turing found a way of reading messages sent by the Germans, using a codebreaking machine called the bombe. Across the Atlantic, Friedman developed a way of cracking the Purple machine, the device used by Japan to code wartime messages.

British and American spies had worked closely during the first world war, but the bonds were pulled tighter in the spring of 1941, when four US officials travelled to Bletchley Park to deliver a model of the Purple machine.

They received intelligence gifts in return and the exchange of information has been going on ever since.

During the second world war, codebreaking was the endeavour of small teams of gifted individuals, working with crude machines that hummed and whirred as cogs and wheels of varying sizes turned at different speeds.

The enemy was known and the purpose of the interceptions clear – to seize the initiative in a global war between countries who threatened each other's existence.

But in the years that followed, the enemies changed, intelligence capabilities developed, and the newly renamed GCHQ and newly formed NSA, created in October 1952, began an inexorable rise, intertwining interests and capabilities that insiders say would now be almost impossible to untangle. Professor Anthony Glees, who has written about this relationship, says it is one of Britain's last claims to global power status.

"In large part this position stems from three facts of British life, each directly connected in purpose to the other: our nuclear deterrent capability, our armed forces and our secret intelligence community," he said.

"Yet what gives us our critical mass as a power is one single, overarchingly important reality: our intense and intimate security relationship with the USA. Intelligence co-operation is its throbbing heart."

One senior member of Britain's intelligence community told the Guardian: "The relationship between the NSA and GCHQ is unique. Most intelligence agencies compete with each other. The CIA, for instance, sees MI6 as a competitor. They work with each other, but there is always some tension. The NSA and GCHQ are not like that.

"When you get a GCHQ pass it gives you access to the NSA too. You can walk into the NSA and find GCHQ staff holding senior management positions, and vice versa. When the NSA has a piece of intelligence, it will very often ask GCHQ for a second opinion. There have been ups and downs over the years, of course. But in general, the NSA and GCHQ are extremely close allies. They rely on each other."

This symbiosis developed during the attritional years of the cold war. It continued to evolve as signals intelligence – Sigint, as the agencies call it – became as important as human intelligence (Humint) – recruiting "moles" and informers.

And with the onset of the internet and cyberwarfare, GCHQ and the NSA, always in lockstep, achieved pre-eminence among their agency peers.

The Snowden files revealed intelligence-gathering is now being conducted on a grand scale, with the NSA and GCHQ exploiting advances in technology to tap into, store and analyse more and more information.

Without Snowden, we would not have known that the amount of personal data available to GCHQ from internet and mobile traffic increased by 7,000% between 2008 and 2012.

That is just what the UK collects; the files also revealed that 60% of all Britain's refined intelligence comes from the NSA.

From humble beginnings in rickety wooden huts, GCHQ has become the keystone of Britain's spy agencies, and its "doughnut" headquarters in Cheltenham is probably the most remarkable building ever constructed in the UK.

Covering more than 92,000 sq metres (1m sq ft), it is packed with supercomputers operated by codebreakers and data miners who work behind concrete and limestone walls that are up to 2.5 metres (8ft) thick.

With a staff of 6,400, it is the biggest of Britain's intelligence agencies. It is rather smaller, however, than the NSA.

At its headquarters in Fort Meade, Maryland, the NSA has more than 1,000 buildings on a 2,000-hectare (5,000 acre) site and employs an estimated 40,000 people. That figure does not include the analysts at different bases across the world, or the army of subcontractors needed to keep the agency running smoothly. Edward Joseph Snowden was one of them.

Snowden

For a man who has been at the centre of worldwide attention for five months, surprisingly little is known about Snowden.

After "outing" himself in June, insisting he didn't want to hide behind a cloak of anonymity, he has spoken on just a few occasions, and only when circumstances seemed to demand it.

He avoided everyone he didn't want to see when he was in Hong Kong, the first place he escaped to, and for several weeks he remained beyond the reach of the world's media, and doubtless a small army of spies, while holed up in a hotel room in the transit area of Moscow's Sheremetyevo airport.

In the absence of regular pronouncements, there has been much animated speculation about why he did what he did, who has the material he took, and what kind of damage he has done.

And most of it remains just that – speculation, albeit of a kind that has fuelled character assassinations from those circling the wagons around the intelligence agencies.

Traitor was a barb he must have expected; he has also been branded a self-serving twerp (by the former head of MI5 Stella Rimington), a naive narcissist, and perhaps strangest of all, a cross-dressing Little Red Riding Hood (in the Washington Post).

What we do know about Snowden suggests he doesn't easily fit into any of those categories, or indeed, any stereotype. He does not look like a computer boffin, nor does he speak in the manner of a tortured ideologue.

There is no Julian Assange-like messiah complex for cod-psychologists to dissect, and money doesn't appear to matter much to him. He hasn't asked for, or received, any payment from the Guardian.

He remains something of an enigma, happy to stay out of the limelight.

Born on 21 June 1983, Snowden spent his early years in Elizabeth City, North Carolina. His parents, Lonnie and Elizabeth Snowden, were teenage sweethearts who met at Northeastern high school and married in a "back garden" wedding in 1979, when they were 18. They had a daughter, Jessica, and then Edward. They split up in 2001 and Lonnie, who retired from the US coastguard, has since remarried and moved to Pennsylvania.

In 1993, when Snowden was 10, the family moved to Crofton, Maryland, near the NSA's HQ. Neighbours who spoke to US newspapers said he was polite, quiet and seemed to spend too much time looking at computers. Dawn Whitmore, a former classmate, remembered him as a shy and serious boy, who sported a pudding-bowl haircut and thick glasses. "He was very well thought-out with what he was trying to say. He was always very, very nice with me, but I was also a very nerdy, shy girl."

Snowden went on to Anne Arundel community school, but dropped out in his second year. He dropped out again in 2004 but must have studied at home to earn a GED (General Educational Development) – the equivalent of a high-school certificate.

Snowden might have ended up in the army – he had four months in the reserves in mid-2004. "I enlisted shortly after the invasion of Iraq and I believed in the goodness of what we were doing," Snowden said. But he broke both legs in an accident and never completed his training.

Though he may not have had a string of formal qualifications, sitting in front of a screen honed something inside him, because in 2005, after a spell as a security guard, he was taken on by the CIA as an IT analyst. He was obviously good at it, because within two years he was one of the agency's operatives working in Geneva, Switzerland. It was not an experience he enjoyed.

"Much of what I saw in Geneva really disillusioned me about how my government functions and what its impact is in the world," he said. "I realised that I was part of something that was doing far more harm than good."

Snowden had a variety of jobs with confusing titles. Essentially the posts gave him privileged access to internal networks: ironing out problems, making systems work more efficiently and, ironically, making sure they were secure.

He was a troubleshooter, which is the principle reason why he managed to see so many documents – and spirit them away without leaving an electronic fingerprint.

"When you're in positions of privileged access, you're exposed to a lot more information on a broader scale than the average employee," he said. "And because of that you see things that may be disturbing. You recognise that some of these things are actually abuses. And when you talk to people about them in a place like this … people tend not to take them [the abuses] very seriously.

"Over time that awareness of wrongdoing builds up. And the more you talk about it the more you're ignored, the more you're told it's not a problem, until eventually you realise these things need to be determined by the public and not by somebody who was simply hired by the government."

In 2009 Snowden left the CIA to work in the private sector and four years later he got a job with Booz Allen Hamilton, a consulting company that supplies computer specialists to the NSA. Initially he was posted to Japan, then Hawaii. Snowden has admitted he only took the $122,000-a-year (£75,000) post to get access to certain material. By 20 May 2013, he had gathered what he wanted.

He boarded a flight to Hong Kong, leaving behind him a bewildered girlfriend, a boss who thought he needed time off to treat epilepsy (his cover story), and any chance he could ever again lead a normal life.

In interviews over the past four months, he has attempted to answer the questions that have been thrown at him, particularly by those who have made the most damaging claim – that he must have given his secrets to foreign governments.

He insists this is completely untrue. "This is a predictable smear that I anticipated before going public, as the US media has a kneejerk 'Red China!' reaction to anything involving Hong Kong or China, and is intended to distract from the issue of US government misconduct.

"Ask yourself: if I were a Chinese spy, why wouldn't I have flown directly into Beijing? I could be living in a palace petting a phoenix by now. No. I have had no contact with the Chinese government. I only work with journalists."

In his most recent interview in the New York Times, Snowden said he hadn't taken any secret files with him to Russia, where he has been given asylum for a year. "What would be the unique value of personally carrying another copy of materials onward? There's a 0% chance the Russians or Chinese have received any documents."

The four computers Snowden has been carrying with him since he left Hawaii were not jam-packed with secret files after all; they were a decoy. US officials who have been to Moscow have since said this is correct.

The other accusation made against Snowden is that he revealed secrets that would put people's lives in danger. He denies this emphatically. "I did not reveal any US operations against legitimate military targets. I pointed out where the NSA has hacked civilian infrastructure such as universities, hospitals and private businesses because it is dangerous. These nakedly, aggressively criminal acts are wrong no matter the target. The public needs to know the kinds of things a government does in its name, or the 'consent of the governed' is meaningless."

But the revelations must have affected national security? Think again, he asks.

"US officials say this every time there's a public discussion that could limit their authority. US officials also provide misleading or directly false assertions about the value of these programmes.

"Journalists should ask a specific question: since these programmes began operation shortly after September 11, how many terrorist attacks were prevented solely by information derived from this suspicion-less surveillance that could not be gained via any other source? Then ask how many individual communications were ingested to achieve that, and ask yourself if it was worth it.

"Bathtub falls and police officers kill more Americans than terrorism, yet we've been asked to sacrifice our most sacred rights for fear of falling victim to it."

Mastering the internet

Snowden was still a teenager when 2,996 people were killed on 9/11. The failure to detect the plot sent the intelligence services into panic; they recruited as many Islamic specialists and linguists as they could, redirected operations towards Osama bin Laden – and began the deliberate, elaborate process of building an intelligence machine that could feed information to sustain the wars in Iraq and Afghanistan.

It was the first glimpses of this evolving apparatus that so worried Snowden and led him – seven years later – to undertake the biggest ever theft of material stamped with the highest classification levels, called Strap 1 and Strap 2.

The wars against the terrorists coincided with a period of huge technological innovation, making it more difficult for the agencies to detect the important "noise" they were listening for amid all the chatter from the rest of us.

In Britain, the challenge thrown down by the explosion in use of the web, mobile phones and social media was met by a GCHQ programme that showed remarkable ambition. It was called Mastering the Internet (MTI).

In America, similar projects were just as audacious. The files released by Snowden show the agencies have been, and remain, determined to eavesdrop on every possible method of communication, regardless of how much extraneous material they gather in the process; they have put taps on the cables that carry raw internet traffic across the world; they have gone further "upstream" and devised ways of getting material from the computers which run the big internet service providers; they have refined their relationships with mobile phone companies so they can get details of every call made and received; and they found ways of defeating encryption software too, setting supercomputers to crack codes, or by inserting secret "back doors" into the software itself.

The scale of this technological achievement is admirable, the logic behind it clear; but all this impressive architecture has been built without any political discussion about whether this is the right thing to do, or any endorsement from millions of members of the public, whose personal lives are now being recycled through giant databases.

A sense of the anxiety that was driving GCHQ to do this was revealed in an internal memo, dated Tuesday 19 May 2009, which was written jointly by the director in charge of the MTI project and a member of the agency's cyber-defence team. The memo was a "prioritisation and tasking initiative" to another senior member of staff, who was being urged to come up with new ideas, fast.

"It is becoming increasingly difficult for GCHQ to acquire the rich sources of traffic needed to enable our support to partners within HMG [Her Majesty's government], the armed forces and overseas," they wrote.

"The rapid development of different technologies, types of traffic, service providers and networks, and the growth in sheer volumes that accompany particularly the expansion and use of the internet, present an unprecedented challenge to the success of GCHQ's mission." The memo continued: "We would like you to lead a small team to fully define this shortfall in tasking capability [and] identify all the necessary changes needed to rectify it."

The two chiefs said they wanted "potential quick-win solutions not currently within existing programme plans".

Those existing programmes might have been a reference to the NSA's Prism project, and Tempora, GCHQ's crown jewel, which was in development. The former was started in 2007 as a way for the NSA to get access to the computer systems which run Microsoft, Google, Facebook, Apple and other giants of the web world.

A 41-slide PowerPoint presentation dated April 2010 was among the Snowden documents, and it revealed the NSA was very pleased with the information it was receiving, which included search histories, the content of emails, videos, photos and live chats.

The NSA hailed Prism as "one of the most valuable, unique and productive accesses", and said it had generated 24,005 reports in 2012 (from a total of 77,000 over the previous five years).

Some of this information was being shared with GCHQ, though it is still unclear how much of this material was sought – requiring a warrant signed by a minister – rather than offered by the Americans, which would not.

Using secret court orders, the US was already bulk-collecting the telephone records of millions of US customers of Verizon, one of America's largest telecoms providers. With Prism, the NSA had another way of accessing vast amounts of information about people who were not under any suspicion at all.

But this was nothing compared with Tempora.

The British programme was far more ingenious, a real breakthrough in capability, albeit one which critics say meant an even bigger potential intrusion into the private lives of ordinary people.

GCHQ had been tapping the undersea cables that carry the internet in and out of the UK for years, but Tempora provided the ability to analyse the information in almost real time, rather than dumping the data into vast but simple electronic storage bins that take time to sift through.

The buffering that Tempora allows acts like Sky+ television, slowing down or briefly halting the stream of information to make it easier for other filters to search for keywords, names, or patterns of behaviour.

GCHQ keeps the content of messages for three working days, and the simple "metadata" – which includes details of who sent and received them – for up to 30 days. The programme was trialled at GCHQ's station in Bude, Cornwall (which is partly funded by the NSA) and was an instant success.

British delight at having scooped the Americans was matched by the NSA's desire to get its hands on the project; an internal US guide to using the system, which became fully functional in 2011, described it as "an exciting opportunity to get direct access to enormous amounts of GCHQ's special source data". When the NSA was offered access to Tempora for a trial period, the agency told its analysts to be on best behaviour.

"[We] need to be successful!," a memo urged. "We're depending on you to provide the business case required to justify expanded access."

"We need to prove that NSA's access is necessary to prosecute our mission and will greatly enhance the production of the intelligence. The success of this three-month trial will determine expanded NSA access to internet buffers in the future."

The strategy seems to have worked. By May last year, an internal GCHQ memo said it had 300 analysts working on intelligence from Tempora, and the NSA had 250.

The documents show Tempora gave the UK "the biggest internet access" of any member of the Five Eyes electronic eavesdropping alliance, comprising the US, UK, Canada, Australia and New Zealand.

They also show that by 2012 GCHQ was handling 600m "telephone events" each day. It had tapped more than 200 fibreoptic cables and was able to process data from at least 46 of them at a time.

With each cable carrying data at a rate of 10 gigabits a second, the tapped cables could, in theory, deliver more than 21 petabytes a day – equivalent to sending all the information in all the books in the British Library 192 times every 24 hours. "This is a massive amount of data!" one of the documents said. "You are in an enviable position – have fun and make the most of it."

All this activity was approved in the UK by a subsection of a law which was introduced in 2000 – the Regulation of Investigatory Powers Act (Ripa) – a time when the agencies could not have envisaged being able to conduct surveillance on such a massive scale, when buffering was not even possible. Ken Macdonald, the former director of public prosecutions, said these "blinding transformations" had rendered Ripa and other intelligence legislation "anti-modern".

"The spooks, who once sat in cubicles steaming open the glued-down flaps of a few dozen suspect envelopes, now have more fertile plains to furrow and the marvellous means to do it. Now they can steam open everything."

The law

Mr Justice Michael Burton seems an amiable fellow: ruddy-faced and quick-witted, he is a specialist in commercial law and well known around the Inns of Court for his interest in amateur dramatics.

He is also the newly appointed president of the investigatory powers tribunal (IPT), which assesses complaints against Britain's intelligence agencies.

A speech he gave to lawyers over lunch on Monday 14 October was a first; no other head of the tribunal has ever spoken in public before, though Burton assured his audience in the City of London that he had no intention of marking this historic occasion by telling them anything of great interest. However, he did give a glimpse into the problems faced by the IPT, without offering any particular solutions.

"We do receive a large number of applications from individuals about their belief, or often their paranoia, that they are being targeted," the judge explained after finishing a roast beef sandwich.

"I am afraid we get quite a lot of complaints from members of the public who say, for example, 'When I was 14, I had my tonsils removed and I believe that MI5 implanted electronic equipment in me.' "Very often the sign is whether they are resident in a mental institution. It ranges from that to very serious complaints."

The tribunal is one of the cornerstones of the regime designed to scrutinise the agencies, a regime described by William Hague, the foreign secretary, as one of the best in the world. But Burton's court has taken a fair degree of flak over the years, for reasons he described himself.

"The media describes the IPT using terms such as 'a secret closed court', 'a little-known complaints body', 'one of the most secretive judicial bodies in the country', 'the UK's most secret court most of whose cases are held in closed session'," Burton said. "Almost all of that is untrue."

Really? The IPT is certainly unlike any other court; it does not publicise a list of when it is holding cases or where; almost all of its hearings are in private – there will be no public sessions for the rest of this year. And it almost always – in more than 99% of cases – fails to uphold complaints against the secret services or local authorities.

Snowden's documents show GCHQ declaring the tribunal has never ruled against a British agency since the court was established 13 years ago. The IPT will not say whether this is true or not. It is a secret.

The tribunal will also not even say where it is based. Its website refers to a PO box in central London. The Guardian tracked this down to a post office delivery office in the centre of an enormous building site near MI6 headquarters in Vauxhall, south London.

Apparently the IPT doesn't have a permanent home. In his speech, Burton acknowledged the court, which has eight part-time members, moves around various buildings in Whitehall. This is not done to bamboozle reporters or members of the public, he said, but because the tribunal is shunted around to where-ever there is free space.

The Snowden files suggest GCHQ is not unduly worried about the IPT; it is easy to see why. Over the past 13 years, it has heard 1,469 cases. It has upheld complaints against councils and police forces on 10 occasions – that's 0.68% of the total.

The figure is actually even lower because two of the successful complaints referred to the same matter in 2010.

The case in question illustrates another criticism often levelled at the IPT. It involved Poole council and its unlawful surveillance operation on Jenny Paton and her three children. It took the IPT more than two years to rule the council had no business using anti-terrorist laws to establish whether the Patons lived outside a school's catchment area.

But while there is some transparency in the way local authorities can use and abuse their power, there is nothing similar for the agencies.

This is the essential conundrum of the tribunal. How can people bring credible cases to the tribunal, if nobody knows what the agencies are doing – even in the broadest sense? With too little in the public domain, it is not surprising that many cases fall at the first hurdle for being "frivolous or vexatious".

When the IPT does decide to take on a complaint, it can ask for "all such documents and information as the tribunal may require". But this exchange relies on trust. The court has no way of checking whether it is getting what it needs. The panel will then make an assessment – in secret – before making a ruling, whose details will almost certainly never be published.

In eight years up until 2010, the IPT had only disclosed findings from five cases, a situation that even the police have been embarrassed by.

Giving evidence to parliament, the police lead on surveillance, Chief Constable Nick Gargan, admitted the IPT was largely anonymous and needed to be more transparent.

"[It] ought to be encouraged to be more publicly visible both in terms of encouraging people to use it and, where meaningful claims have been made, to actually publicise those findings," he said.

Burton said he was open to ideas about how the IPT could make itself more accessible, but only to a point. He said he wouldn't approve anything that meant "all-important secrecy was lost".

The parliamentary intelligence and security committee (ISC) is the other essential plank of oversight cited by the government, but it has also been bedevilled by criticism since it was established in 1994.

Too weak, too close to government, too reluctant to criticise the agencies are some of the recurring jibes. Earlier this year, it won new powers to force the agencies to hand over material, and a small increase in the staff to review it. But while doubling of its budget to £1.3m will give it more clout, it is starting from a low base.

With a team of part-time staff, the committee's nine MPs have a huge job to provide credible oversight of the three spy agencies, which have a combined staff of more than 10,000 and a combined annual budget of £2bn.

The ISC is now also responsible for reviewing the work of the military's "defence intelligence" unit, which will add to the mountain of documents available for review.

The committee's announcement on Thursday 17 October of a broad inquiry into surveillance, which will include some public hearings, is a watershed moment for the ISC, and its chair, Sir Malcolm Rifkind. He has insisted his team is capable of doing the job, but even former heads of the committee have acknowledged what a formidable task this is.

"The ISC is underfunded. It needs more people working for it," said Kim Howells, a former Foreign Office minister who chaired the committee for two years until 2010. "It needs more investigators and it needs to be able to call on expertise whenever it is required. It might be a no-brainer, but you try to persuade the Cabinet Office to part with the cash."

Howells also revealed how Gordon Brown tried to influence the ISC when he was prime minister, even though it is supposed to be entirely beyond this kind of executive interference.

"I had the most terrible battles with him and the head of the civil service. They didn't really understand the committee must be absolutely scrupulously independent of government and the agencies."

The former MP, who retired at the last election, called for parliament to determine whether the laws governing the agencies are "too swingeing and give too much leeway to executive action". "That is where the debate should be. And if parliament thinks there is too much power then parliament should change the law."

He conceded the job of the ISC becomes extremely difficult if it is "not trusted by either parliament or the media, and if people don't accept what it says. Then the suspicion of collusion starts to grow." But he defended the MPs who have served on the committee, saying they could be trusted to do a good job, if they had more support.

"Generally they are people who are either at the end of their political life or near the end, they have quit climbing the greasy pole, or been dragged off it."

Paul Murphy, an ISC chairman during Labour's years in power, remembered his committee had nobody to help them make sense of vast numbers of technical documents. He was in charge between 2005 and 2008 – when many of the current surveillance programmes were likely signed off by ministers. Yet Murphy and his team were often flying blind.

"We didn't have an investigator. You'd effectively have to rely on the word of the agencies and if there was any dispute you'd have to go through the documents yourself, which we did. Reams of it.

"The documentation is so vast now that the ISC may need more than investigator. It's not an easy position to fill and the person has to have carte blanche. Even then you have to be selective because it is such a huge task."

'The golden age'

Alan Turing remains the one true superstar of Britain's intelligence community. His feats during the second world war are now being celebrated in a way that was unimaginable when he died.

There are statues in honour of him in Guildford and Manchester, where he lived and worked, and he was the subject of a rare speech on 4 October last year by Sir Iain Lobban, the current director of GCHQ, to mark the 100th anniversary of the codebreaker's birth.

He rattled through the stories of Turing's peculiarities – burying his silver bullion and then forgetting where; chaining his mug to his radiator; cycling in his gas mask to ward off hay fever. Lobban credited Turing with starting the "irrevocable change" that led to the formation of GCHQ and its evolution into "the highly technological intelligence organisation that it is today".

He said that if Turing were alive he would be working on threats from cyberspace, a clever way of co-opting the codebreaker and his achievements into surveillance programmes that would have been inconceivable to him. "Our challenges come from the explosion in the volume of communications as well as the relentless increase in new ways of accessing and processing that volume," said Lobban. "Then, code related simply to the encryption of communications; today, code refers to the way in which we program IT systems. Then, the challenge was to identify German and Japanese communications; today, the challenge can simply be to cope with the number of different communications options. Today the internet provides the virtual global landscape for an analogous struggle."

But the struggle is different, and surveillance strategy has been turned on its head to deal with it.

Snowden's files revealed the mouth of the intelligence funnel has been stretched wide open over the last decade. Using technologies that are becoming more powerful and sophisticated, GCHQ and the NSA have been undertaking government-authorised data trawling; the secret services have quietly ushered in the age of the digital dragnet.

The agencies say they cannot do their work without these capabilities and they want to expand them; critics say they should not have been able to acquire them without a proper debate, and without more muscular accountability.

Perhaps it was this tension that led James Clapper, the director of US national intelligence, to concede that a fork in the road had been reached with the Guardian revelations, and not before time. "As loth as I am to give any credit to what's happened here, which is egregious, it's clear that some of the conversations this has generated, some of the debate, probably needed to happen," he said.

The arguments are delicately poised, the issues could not be more important: how to balance privacy and security in the 21st century. The Snowden files make it clear that GCHQ and the NSA have turned Turing's niche pursuit into intelligence-gathering on an industrial scale. An internal memo to analysts at GCHQ dated late 2010 summed up the mood about the powers now available to them: "We are in the golden age."

guardian.co.uk © Guardian News and Media 2013