Ted Cruz using firm that harvested data on millions of unwitting Facebook users

Ted Cruz’s presidential campaign is using psychological data based on research spanning tens of millions of Facebook users, harvested largely without their permission, to boost his surging White House run and gain an edge over Donald Trump and other Republican rivals, the Guardian can reveal.

A little-known data company, now embedded within Cruz’s campaign and indirectly financed by his primary billionaire benefactor, paid researchers at Cambridge University to gather detailed psychological profiles about the US electorate using a massive pool of mainly unwitting US Facebook users built with an online survey.

As part of an aggressive new voter-targeting operation, Cambridge Analytica – financially supported by reclusive hedge fund magnate and leading Republican donor Robert Mercer – is now using so-called “psychographic profiles” of US citizens in order to help win Cruz votes, despite earlier concerns and red flags from potential survey-takers.

Documents seen by the Guardian have uncovered longstanding ethical and privacy issues about the way academics hoovered up personal data by accessing a vast set of US Facebook profiles, in order to build sophisticated models of users’ personalities without their knowledge.

It means you have a wild west, where the campaigns can do whatever they want and get away with it

Christopher Soghoian, ACLU

In the race to advance data-driven electioneering strategies pioneered by successive Obama campaigns, Cruz has turned to Cambridge Analytica for its unparalleled offering of psychological data based on a trove of Facebook “likes”, allowing it to match individuals’ traits with existing voter datasets, such as who owned a gun.

Analysis of Federal Election Commission (FEC) filings shows Cruz’s campaign has paid Cambridge Analytica at least $750,000 this year. The “behavioural microtargeting” company has also received around $2.5m over the past two years from conservative Super Pacs to which Mercer or members of his family have donated.

Related: Ronald Reagan and ... Barack Obama? Ted Cruz reveals presidential role models

In an interview with the Guardian last month , Cruz said his funding and outreach apparatus “is very much the Obama model – a data-driven, grassroots-driven campaign – and it is a reason why our campaign is steadily gathering strength”.

Cruz is increasingly seen as a leading Republican contender, uniting factions within the party beyond his evangelical and Tea Party base. In Iowa – the first state to vote in the presidential primary, in less than two months – the outspoken Texas senator dethroned Trump in a poll for the first time this week as the mogul became ensnared in yet more controversy.

Cambridge Analytica has also worked with the Republican candidate Ben Carson, receiving $220,000 from his campaign earlier this year. But the company is more closely involved in Cruz’s presidential bid, with a team of its data scientists currently working at Cruz campaign headquarters in Houston.

Having donated $11m to the main pro-Cruz Super Pac, Keep the Promise I, Mercer is Cruz’s top financier – and the largest individual donor to Super Pacs or outside groups during the presidential election cycle thus far, according to data compiled by the political transparency website Open Secrets.

Mercer’s connections to both the Cruz campaign and the data firm that is apparently helping to power the senator’s advantages were previously reported by Politico and Bloomberg . But political strategists and privacy advocates agreed that Mercer’s parallel funding channels, combined with concerns over the surreptitious, commodified Facebook data – reported here for the first time – represented an intensified collision of billionaire financing and digital targeting on the campaign trail.

“If people begin to be turned off by Trump, the Cruz campaign will likely have a better strategic understanding of the implications and how to capitalize upon them,” said Bud Jackson, a Democratic specialist in digital grassroots campaigning, when asked to review the relationships. “Where a candidate’s campaign may be afraid to go outside the boundaries of ethical behaviour because of a potential public backlash, an outside group may be less afraid.”

In an interview, Cruz spokesman Rick Tyler said the campaign had contracted Cambridge Analytica “because they’re a market leader and best in the field” but that officials had “done our due diligence”.

“My understanding is all the information is acquired legally and ethically with the permission of the users when they sign up to Facebook,” he said, referring questions about previous research to Cambridge Analytica.

The Guardian contacted Cambridge Analytica prior to publication via email with its findings and a list of detailed questions. Repeated calls were also made to its offices in Washington and New York, as well as its parent company in London. A representative in New York refused to speak with the Guardian and hung up the phone.

Related: Ted Cruz rejects demands to revive NSA surveillance after San Bernardino

Christopher Soghoian, principal technologist and senior policy analyst at the American Civil Liberties Union, described the Guardian’s findings as “troubling” and highlighted shortcomings by the FEC to adequately regulate campaigns’ use of data.

“What it essentially means is there is no one regulating campaigns’ privacy data and security practices,” he said. “So it means you have a wild west, where the campaigns can do whatever they want and get away with it.”

Research that seeded data on millions

Documents seen by the Guardian show Cambridge Analytica’s parent, a London-based company called Strategic Communications Laboratories (SCL), was first introduced to the concept of using social media data to model human personality traits in early 2014 by Dr Aleksandr Kogan, a lecturer at Cambridge University’s renowned psychology department.

Kogan established his own company in spring that year and began working with SCL to deliver a “large research project” in the US. His stated aim was to get as close to every US Facebook user into the dataset as possible.

The academic used Amazon’s crowdsourcing marketplace Mechanical Turk (MTurk) to access a large pool of Facebook profiles, hoovering up tens of thousands of individuals’ demographic data – names, locations, birthdays, genders – as well as their Facebook “likes”, which offer a range of personal insights.

This was achieved by recruiting MTurk users by paying them around one dollar to take a personality questionnaire that gave access to their Facebook profiles. This raised the alarm among some participants, who flagged Kogan for violating MTurk’s terms of service. “They want you to log into Facebook and then download a bunch of your information,” complained one user at the time.

Crucially, Kogan also captured the same data for each person’s unwitting friends. For every individual recruited on MTurk, he harvested information about their friends, meaning the dataset ballooned significantly in size. Research shows that in 2014, Facebook users had an average of around 340 friends.

Kogan assured the MTurk users their Facebook data would “only be used for research purposes” and remain “anonymous and safe”.

However, the Facebook data was then used to generate sophisticated models of each of their personalities using the so-called “big five” personality traits and characteristics – openness, conscientiousness, extraversion, agreeableness, neuroticism (known as the OCEAN scale).

By summer 2014, Kogan’s company had created an expansive and powerful dataset. His business partner boasted on LinkedIn that their private outfit, Global Science Research (GSR), “owns a massive data pool of 40+ million individuals across the United States – for each of whom we have generated detailed characteristic and trait profiles”.

Documents show SCL agreed to a contract with GSR, whereby it would pay its data collection costs in order to improve “match rates” against SCL’s existing datasets or to enhance GSR’s algorithm’s “national capacity to profile capacity of American citizens”.

In an email, Kogan said he was unable to explain in detail where all the data came from, as he was restricted by various confidentiality agreements. He said SCL is no longer a client.

He said that while GSR often used MTurk for data collection, it “never collected more than a couple thousand responses on MTurk for any one project, or even across all projects for a single client – the vast majority of our MTurk data collection as a company is in the form of surveys only”. He said GSR stores Facebook data anonymously.

Kogan explained that separate from his university role, his private company undertook various commercial ventures relating to data analysis. He said that when GSR collect Facebook data, the terms detail the use that information collected will be put to and make clear to participants that they are giving GSR full permission to use the data and user contribution for any purpose.

He said Cambridge University had “no knowledge of the clients or projects GSR had worked on” and that GSR has never used any data collected as part of his university activities.

‘Packaging voters like they’re consumers’

Today, Cambridge Analytica’s central offering to US politicians is to enable them to use the OCEAN scale in shaping highly targeted campaign messages. This allows candidates like Cruz to campaign on specific issues, but communicate them in multiple ways to different audiences depending on the personal information the company holds about them.

As the company’s CEO, Alexander Nix, explains in a promotional video : “The more you know about someone, the more you can align a campaign with their requirements or their wants and needs.” He did not respond to a request for comment.

Michael Zimmer, an associate professor at the University of Wisconsin, Milwaukee, where he specialises in privacy and internet ethics, described this as a “particularly problematic” kind of voter targeting that raised broader concerns in the US about “packaging voters like they’re consumers”.

“It’s one thing for a marketer to try to predict if people like Coke or Pepsi,” he said, “but it’s another thing for them to predict things that are much more central to our identity and what’s more personal in how I interact with the world in terms of social and cultural issues.”

Prior to its relationship with Cruz, Cambridge Analytica worked with a handful of 2014 midterm candidates, according to FEC filings. The firm also secured hundreds of thousands of dollars of business with John Bolton’s Super Pac, formed by the hawkish former UN ambassador to support conservative candidates campaigning on national security issues.

Ahead of the midterms, Cambridge Analytica reportedly developed a series of TV ads for candidates supported by Bolton , each aimed at different personality types and aired at times when viewers with personalities it aimed to reach were most likely to be watching.

Last week, the Cruz campaign launched a combative TV ad timed to air during a major college football game in Iowa. In the 30-second ad, Cruz vows: “We’ll rebuild our military, we’ll kill the terrorists and every Islamic militant will know – if you wage jihad against us, you’re signing your death warrant.”

Tyler, the Cruz spokesman, said Cambridge Analytica did not work on television advertising for the campaign. “They’re helping with online targeting and messaging, and that’s how we find people online,” he said.

It remains unclear when Mercer’s involvement with Cambridge Analytica began, but FEC filings show the company started working with Super Pacs that the secretive conservative donor has backed during the second half of 2014.

On Wednesday in Washington, Cruz was the only member of the Senate armed services committee to skip a hearing on US strategy to combat the Islamic State. Instead, he travelled to attend a fundraising luncheon in New York hosted by the president of Keep the Promise I, the Super Pac funded by Mercer that, one day earlier, had announced a $600,000 digital advertising strategy to be rolled out via social media in early voting states.

A spokesman for Mercer declined to comment for this article. A spokeswoman for Facebook declined to comment.

Additional reporting by Ben Jacobs in Amana, Iowa

guardian.co.uk © Guardian News and Media 2015