'Audits from hell': Obama's IRS tried to take on the ultra wealthy ten years ago — It didn’t go well

The magnate, Georg Schaeffler, was the billionaire scion of a family-owned German manufacturer and was quietly working as a corporate lawyer in Dallas. Schaeffler had extra reason to fear the IRS, it seemed. He wasn’t in the sights of just any division of the agency but the equivalent of its SEAL Team 6.

In 2009, the IRS had formed a crack team of specialists to unravel the tax dodges of the ultrawealthy. In an age of widening inequality, with a concentration of wealth not seen since the Gilded Age, the rich were evading taxes through ever more sophisticated maneuvers. The IRS commissioner aimed to stanch the country’s losses with what he proclaimed would be “a game-changing strategy.” In short order, Charles Rettig, then a high-powered tax lawyer and today President Donald Trump’s IRS commissioner, warned that the squad was conducting “the audits from hell.” If Trump were being audited, Rettig wrote during the presidential campaign, this is the elite team that would do it.

The wealth team embarked on a contentious audit of Schaeffler in 2012, eventually determining that he owed about $1.2 billion in unpaid taxes and penalties. But after seven years of grinding bureaucratic combat, the IRS abandoned its campaign. The agency informed Schaeffler’s lawyers it was willing to accept just tens of millions, according to a person familiar with the audit.

How did a case that consumed so many years of effort, with a team of its finest experts working on a signature mission, produce such a piddling result for the IRS? The Schaeffler case offers a rare window into just how challenging it is to take on the ultrawealthy. For starters, they can devote seemingly limitless resources to hiring the best legal and accounting talent. Such taxpayers tend not to steamroll tax laws; they employ complex, highly refined strategies that seek to stretch the tax code to their advantage. It can take years for IRS investigators just to understand a transaction and deem it to be a violation.

Once that happens, the IRS team has to contend with battalions of high-priced lawyers and accountants that often outnumber and outgun even the agency’s elite SWAT team. “We are nowhere near a circumstance where the IRS could launch the types of audits we need to tackle sophisticated taxpayers in a complicated world,” said Steven Rosenthal, who used to represent wealthy taxpayers and is now a senior fellow at the Tax Policy Center, a joint venture of the Urban Institute and Brookings Institution.

Because the audits are private — IRS officials can go to prison if they divulge taxpayer information — details of the often epic paper battles between the rich and the tax collectors are sparse, with little in the public record. Attorneys are also loath to talk about their clients’ taxes, and most wealthy people strive to keep their financial affairs under wraps. Such disputes almost always settle out of court.

But ProPublica was able to reconstruct the key points in the Schaeffler case. The billionaire’s lawyers and accountants first crafted a transaction of unusual complexity, one so novel that they acknowledged, even as they planned it, that it was likely to be challenged by the IRS. Then Schaeffler deployed teams of professionals to battle the IRS on multiple fronts. They denied that he owed any money, arguing the agency fundamentally misunderstood the tax issues. Schaeffler’s representatives complained to top officials at the agency; they challenged document requests in court. At various times, IRS auditors felt Schaeffler’s side was purposely stalling. But in the end, Schaeffler’s team emerged almost completely victorious.

His experience was telling. The IRS’ new approach to taking on the superwealthy has been stymied. The wealthy’s lobbyists immediately pushed to defang the new team. And soon after the group was formed, Republicans in Congress began slashing the agency’s budget. As a result, the team didn’t receive the resources it was promised. Thousands of IRS employees left from every corner of the agency, especially ones with expertise in complex audits, the kinds of specialists the agency hoped would staff the new elite unit. The agency had planned to assign 242 examiners to the group by 2012, according to a report by the IRS’ inspector general. But by 2014, it had only 96 auditors. By last year, the number had fallen to 58.

The wealth squad never came close to having the impact its proponents envisaged. As Robert Gardner, a 39-year veteran of the IRS who often interacted with the team as a top official at the agency’s tax whistleblower office, put it, “From the minute it went live, it was dead on arrival.”

Most people picture IRS officials as all-knowing and fearsome. But when it comes to understanding how the superwealthy move their money around, IRS auditors historically have been more like high school physics teachers trying to operate the Large Hadron Collider.

That began to change in the early 2000s, after Congress and the agency uncovered widespread use of abusive tax shelters by the rich. The discovery led to criminal charges, and settlements by major accounting firms. By the end of the decade, the IRS had determined that millions of Americans had secret bank accounts abroad. The agency managed to crack open Switzerland’s banking secrecy, and it recouped billions in lost tax revenue.

The IRS came to realize it was not properly auditing the ultrawealthy. Multimillionaires frequently don’t have easily visible income. They often have trusts, foundations, limited liability companies, complex partnerships and overseas operations, all woven together to lower their tax bills. When IRS auditors examined their finances, they typically looked narrowly. They might scrutinize just one return for one entity and examine, say, a year’s gifts or income.

Belatedly attempting to confront improper tax avoidance, the IRS formed what was officially called the Global High Wealth Industry Group in 2009. “The genesis was: If you think of an incredibly wealthy family, their web of entities somehow gives them a remarkably low effective tax rate,” said former IRS Commissioner Steven Miller, who was one of those responsible for creating the wealth squad. “We hadn’t really been looking at it all together, and shame on us.”

The IRS located the group within the division that audits the biggest companies in recognition of the fact that the finances of the 1 percent resemble those of multinational corporations more than those of the average rich person.

The vision was clear, as Doug Shulman, a George W. Bush appointee who remained to helm the agency under the Obama administration, explained in a 2009 speech: “We want to better understand the entire economic picture of the enterprise controlled by the wealthy individual.”

It’s particularly important to audit the wealthy well, and not simply because that’s where the money is. That’s where the cheating is, too. Studies show that the wealthiest are more likely to avoid paying taxes. The top 0.5 percent in income account for fully a fifth of all the underreported income, according to a 2010 study by the IRS’ Andrew Johns and the University of Michigan’s Joel Slemrod. Adjusted for inflation, that’s more than $50 billion each year in unpaid taxes.

The plans for the wealth squad seemed like a step forward. In a few years, the group would be staffed with several hundred auditors. A team of examiners would tackle each audit, not just one or two agents, as was more typical in the past. The new group would draw from the IRS’ best of the best.

That was crucial because IRS auditors have a long-standing reputation, at least among the practitioners who represent deep-pocketed taxpayers, as hapless and overmatched. The agents can fritter away years, tax lawyers say, auditing transactions they don’t grasp. “In private practice, we played whack-a-mole,” said Rosenthal, of the Tax Policy Center. “The IRS felt a transaction was suspect but couldn’t figure out why, so it would raise an issue and we’d whack it and they would raise another and we’d whack it. The IRS was ill-equipped.”

The Global High Wealth Group was supposed to change that. Indeed, with all the fanfare at the outset, tax practitioners began to worry on behalf of their clientele. “The impression was it was all going to be specialists in fields, highly trained. The IRS would assemble teams with the exact right expertise to target these issues,” Chicago-based tax attorney Jenny Johnson said.

The new group’s first moves spurred resistance. The team sent wide-ranging requests for information seeking details about their targets’ entire empires. Taxpayers with more than $10 million in income or assets received a dozen pages of initial requests, with the promise of many more to follow. The agency sought years of details on every entity it could tie to the subject of the audits.

In past audits, that initial overture had been limited to one or two pages, with narrowly tailored requests. Here, a typical request sought information on a vast array of issues. One example: a list of any U.S. or foreign entity in which the taxpayer held an “at least a 20 percent” interest, including any “hybrid instruments” that could be turned into a 20 percent or more ownership share. The taxpayer would then have to identify “each and every current and former officer, trustee, and manager” from the entity’s inception.

Taxpayers who received such requests recoiled. Attacking the core idea that Shulman had said would animate the audits, their attorneys and accountants argued the examinations sought too much information, creating an onerous burden. The audits “proceeded into a proctology exam, unearthing every aspect of their lives,” said Mark Allison, a prominent tax attorney for Caplin & Drysdale who has represented taxpayers undergoing Global High Wealth audits. “It was extraordinarily intrusive. Not surprisingly, these people tend to be private and are not used to sharing.”

Tax practitioners took their concerns directly to the agency, at American Bar Association conferences and during the ABA’s regular private meetings with top IRS officials. “Part of our approach was to have private sit-downs to raise issues and concerns,” said Allison, who has served in top roles in the ABA’s tax division for years. We were “telling them this was too much, unwieldy and therefore unfair.” Allison said he told high-ranking IRS officials, “You need to rein in these audit teams.”

For years, politicians have hammered the IRS for its supposed abuse of taxpayers. Congress created a “Taxpayer Bill of Rights” in the mid-1990s. Today, the IRS often refers to its work as “customer service.” One result of constant congressional scrutiny is that senior IRS officials are willing to meet with top tax lawyers and address their concerns. “There was help there. They stuck their necks out for me,” Allison said.

The IRS publicly retreated. Speaking at a Washington, D.C., Bar Association event in February 2013, a top IRS official, James Fee, conceded the demands were too detailed and long, telling the gathering that the agency has “taken strides to make sure it doesn’t happen again.” The Global High Wealth group began to limit its initial document requests.

The lobbying campaign, combined with the lack of funding for the group, took its toll. One report estimated that the wealth team had audited only around a dozen wealthy taxpayers in its first two and a half years. In a September 2015 report, the IRS’ inspector general said the agency had failed to establish the team as a “standalone” group “capable of conducting all of its own examinations.” The group didn’t have steady leadership, with three directors in its first five years. When it did audit the ultrawealthy, more than 40 percent the reviews resulted in no additional taxes.

The inspector general also criticized the IRS broadly — not just its high-wealth team — for not focusing enough on the richest taxpayers. In 2010, the IRS as a whole audited over 32,000 millionaires. By 2018, that number had fallen to just over 16,000, according to data compiled by Syracuse University. Audits of the wealthiest Americans have collapsed 52 percent since 2011, falling more substantially than audits of the middle class and the poor. Almost half of audits of the wealthy were of taxpayers making $200,000 to $399,000. Those audits brought in $605 per audit hour worked. Exams of those making over $5 million, by contrast, brought in more than $4,500 an hour.

The IRS didn’t even have the resources to pursue millionaires who had been hit with a hefty tax bill and simply stiffed Uncle Sam. It “appeared to no longer emphasize the collection of delinquent accounts of global high wealth taxpayers,” a 2017 inspector general report said.

In recent years, the number of Global High Wealth audits has been higher — it closed 149 audits in the last year — but tax lawyers and former IRS officials say even that improvement is deceptive. A major reason is that the audits are much less ambitious. “They were longer at the beginning and shorter as the process moved on,” Johnson, the tax attorney, said.

Inside the IRS, agents seethed. “The whole organization was very frustrated,” Gardner said. “They were just really not sure what the hell their mission was, what they were supposed to be accomplishing.”

Georg Schaeffler, 54, has flowing salt-and-pepper hair that makes him look like he could’ve been an actor on the 1980s TV show “Dynasty.” The impression is offset by the wire rim glasses he wears and by the bookish disposition of a person who, as a teenager, once asked for a copy of the German Constitution as a present.

As a younger man, Schaeffler tried to escape his legacy. He left Germany and the family company at a young age and lit out for the American West. He was trying to make it on his own “where people don’t know who you are,” as he would tell a reporter for a magazine profile years later. Some might escape to Texas to live a bit wild. Schaeffler became a corporate lawyer.

Schaeffler’s law firm colleagues didn’t know much more than that he spoke with an accent, and certainly not that he was vastly wealthy. That is, until he landed on the Forbes list of global billionaires. Rueful at the loss of his privacy, Schaeffler once declared: “I hate Forbes.”

The family’s riches stemmed from ball bearings and other automobile parts manufactured by the Schaeffler Group, which was founded by Schaeffler’s father and then passed to his mother after his father died. By 2006, Georg (pronounced GAY-org) owned 80 percent of the enterprise and his mother the remaining 20 percent. (As a Texas resident at that time, Schaeffler was required to pay U.S. income taxes.)

He very nearly lost it all. In 2008, Schaeffler Group made a big mistake. It offered to buy Continental AG, a tiremaker, just days before the stock and credit markets experienced their worst crisis since the Great Depression. Even as Continental’s stock price crashed, Schaeffler was legally obligated to go through with its purchase at the much higher pre-crash price.

Schaeffler Group flirted with bankruptcy and pleaded for aid from the German government. The media began to pay closer attention to the private company and the low-profile family that ran it. German press accounts dismissed Schaeffler’s mother as the “billionaire beggar” for seeking a bailout and pilloried her for wearing a fur coat at a ski race while seeking government help.

No German government aid came. The Schaeffler Group teetered, and the family’s fortune plummeted from $9 billion to almost zero. Amid the crisis over Continental, Georg accepted his fate and took up a more prominent role at the company; he’s now the chairman of its supervisory board.

To pay for Continental, Schaeffler Group borrowed about 11 billion euros from a consortium of banks. At the time, Schaeffler’s lenders, including Royal Bank of Scotland, were desperate, too, having suffered enormous losses on home mortgages. They wanted to avoid any more write-downs that might result if the company defaulted on the loans. So in 2009 and 2010, Schaeffler’s lenders restructured the debt in a devilishly complex series of transactions.

By 2012, these maneuvers had caught the eye of the Global High Wealth group. Paul Doerr, an experienced revenue agent, would head the audit. Eventually, the IRS discerned what it came to believe was the transaction’s essence: The banks had effectively forgiven nearly half of Schaeffler’s debt.

To the IRS, that had significant tax implications. In the wealth team’s view, Georg Schaeffler had received billions of dollars of income — on which he owed taxes.

The auditors’ view reflects a core aspect of the U.S. tax system. Under American law, companies and individuals are liable for taxes on the forgiven portion of any loan.

This frequently comes up in the housing market. A homeowner borrows $100,000 from a bank to buy a house. Prices fall and the homeowner, under financial duress, unloads it for $80,000. If the bank forgives the $20,000 still owed on the original mortgage, the owner pays taxes on that amount as if it were ordinary income.

This levy can seem unfair since it often hits borrowers who have run into trouble paying back their debts. The problem was particularly acute during the housing crisis, so in late 2007, Congress passed a bill that protected most homeowners from being hit with a tax bill after foreclosure or otherwise getting a principal reduction from their lender.

Tax experts say the principle of taxing forgiven loans is crucial to preventing chicanery. Without it, people could arrange with their employers to borrow their salaries through the entire year interest-free and then have the employer forgive the loan at the very end. Voila, no taxable income.

The notion that forgiven debt is taxable applies to corporate transactions, too. That means concern about such a tax bill is rarely far from a distressed corporate debtor’s mind. “Any time you have a troubled situation, it’s a typical tax issue you have to address and the banks certainly understand it, too,” said Les Samuels, an attorney who spent decades advising corporations and wealthy individuals on tax matters.

But the efforts to avoid tax, in the case of Schaeffler and his lenders, took a particularly convoluted form. It involved several different instruments, each with multiple moving parts. The refinancing was “complicated and unusual,” said Samuels, who was not involved in the transaction. “If you were sitting in the government’s chair and reading press reports on the situation, your reaction might be that the company was on the verge of being insolvent. And when the refinancing was completed, the government might think that banks didn’t know whether they would be repaid.”

This account of the audit was drawn from conversations with people familiar with it, who were not authorized to speak on the record, as well as court and German securities filings. The IRS declined to comment for this story. Doerr did not respond to repeated calls and emails.

A spokesman for Schaeffler declined to make him available for an interview. “Mr. Schaeffler always strives to comply with the complex U.S. tax code,” the spokesman wrote in a statement, saying “the fact that the refinancing was with six independent, international banks in itself demonstrates that these were arm’s length, commercially driven transactions. The IRS professionally concluded the audit in 2018 without making adjustments to those transactions, and there is no continuing dispute — either administratively or in litigation — related to these matters.”

Schaeffler’s lenders never explicitly canceled the loan. The banks and Schaeffler maintained to the IRS that the loan was real and no debt had been forgiven.

The IRS came not to buy that. After years of trying to unravel the refinancing, the IRS homed in on what the agency contended was a disguise. The banks and Schaeffler “had a mutual interest in maintaining the appearance that the debt hadn’t gone away,” a person familiar with the transaction said. But the IRS believed the debt had, in fact, been canceled.

In the refinancing, the banks and Schaeffler had agreed to split the company’s debt, which had grown to 12 billion euros at that point, into two pieces: A senior loan, to be paid back first, worth about 7 billion euros and a junior piece worth about 5 billion euros.

Schaeffler’s income-producing assets were placed into the entity that held the senior debt. Schaeffler was required to repay the debt according to a schedule and to pay a meaningful interest rate: 4.25 percentage points above the rate his lenders charge each other to borrow money. In short, it appeared to be a relatively straightforward debt transaction.

The junior debt was another matter — and its provisions would raise the hackles of the IRS. To begin with, the entity that held the junior debt did not directly hold income-producing assets. There was no schedule of payments that Schaeffler had to make on the junior debt. He wasn’t obligated to make principal payments until the end of the loan’s term. And it carried a nearly nonexistent annual interest rate of 0.1 percentage points above prevailing interbank lending rates, plus an additional 7 percent per year, which Schaeffler could choose to defer and pay at the end of the term.

The banks attached two other provisions to the refinancing: A “Contingent Remuneration Payment” and a “Contingent Upside Instrument,” according to German securities filings. The two additions called for Schaeffler to make payments to the future performance of the company.

The IRS and Schaeffler’s team fought especially over the Contingent Upside Instrument. Its value was tied to the Schaeffler Group’s future profitability, just like a share of stock would be. The IRS argued that not only was this an equitylike sweetener to the banks, but that it tainted the entire junior portion of the debt. To the IRS, it looked like the banks had a claim on future payments from Schaeffler, but they didn’t know when they’d receive it — or even if they would ever get anything.

To the IRS, these steps all added up to the effective cancellation of about $5 billion worth of debt, for which the banks had received something in return. That something looked and acted very much like equity.

The Schaeffler audit was one of the biggest for the Global High Wealth group. The IRS assigned a larger than normal team to the exam. The agency would send 86 separate document requests to Schaeffler through July 2013.

But there were problems almost from the beginning, according to people familiar with the audit, who provided this account and chronology. The IRS examiners disagreed with one another over strategy. The debates sometimes spilled into the view of Schaeffler’s team. “I remember a tremendous amount of turnover from the exam team and infighting. They were not presenting a coherent message,” a person in the Schaeffler camp said.

By contrast, Schaeffler’s team of lawyers and accountants was large and unified. “These taxpayers aren’t exactly represented by H&R Block,” Gardner, the retired IRS official, said.

Schaeffler’s advisers threw as much as they could back at the agency. Document requests are typically voluntary at the outset. But at one point, an IRS auditor was frustrated at what the team saw as the Schaeffler team’s resistance and delays and demanded, “Would a summons help?” according to a person familiar with the exam. Schaeffler’s team complained about the perceived threat. The IRS scolded its employee, and Doerr, the lead auditor, apologized to the Schaeffler side, according to the person.

In another instance, the IRS could not get information it sought from Ernst & Young, the accounting firm, related to its advice to Schaeffler. So it sued the accounting firm in early 2014. Ernst & Young contended the material was privileged because it was prepared in anticipation of litigation. The IRS won in the U.S. District Court for the Southern District of New York, but Ernst & Young appealed.

In early November 2015, with the Ernst & Young appeal unresolved, top IRS officials gave the Schaeffler audit team the permission it was looking for. They allowed the auditors to notify Schaeffler that they believed he’d failed to disclose about $5 billion in income and that he could expect a $1.2 billion tax bill. That included some $200 million in penalties because the agency viewed the transaction as abusive.

Only days later, the IRS was dealt a defeat that would further hamstring its ability to press its case. On Nov. 10, the 2nd U.S. Circuit Court of Appeals reversed the district judge, slapping down the IRS’ efforts to get the Ernst & Young documents, ruling they were in fact protected by privilege. The IRS had no choice. It would have to proceed without the documents.

The IRS took solace that despite the adverse ruling on the documents, the appeals court appeared to bolster the IRS’ view of the transaction. Describing it as a “complex and novel refinancing,” the court said the consortium of banks “essentially insured” Schaeffler “by extending credit and subordinating its debt.” The opinion found that Schaeffler’s team had known that litigation over the transaction was “virtually inevitable,” underscoring the sense that the billionaire’s lawyers and accountants knew they were pushing legal limits.

The two sides wrangled even over routine procedural matters. The statute of limitations was about to run out. Usually the taxpayer voluntarily agrees to extend the time limit rather than antagonize the agents doing an audit. But Schaeffler’s team raised the prospect of refusing an extension. They ultimately relented, but succeeded in amping up the pressure on the auditors.

Even as the antagonism built between the two sides, the IRS showed deference to the Schaeffler camp. Doerr gave Schaeffler’s attorneys a heads-up that the agency was going to deliver bad news, an action that was viewed as overly solicitous, according to one person. It gave an opening for Schaeffler’s lawyers to raise their concerns with the audit team’s bosses. They expressed how wrongheaded they thought the IRS’ position was and how inappropriate its actions had been.

In June 2016, the IRS sent Schaeffler the official notice that the agency would seek unpaid tax and penalties.

Schaeffler’s attorneys continued to argue, often above the heads of the audit team, that the auditors’ interpretation was incorrect. They held conference calls with top IRS officials, saying the audit team had given the Schaeffler side mixed messages. Some on the team had assured Schaeffler’s attorneys that he would not face a large tax bill or be subject to a penalty. Top officials then met with the Global High Wealth team to discuss the issues. “The pushback is incredible,” one knowledgeable person recalled.

The pushback worked — and here’s where an audit is radically different from a court case. Court cases are typically accompanied by publicly available decisions and rulings that explain them in detail. By contrast, audits are shielded by the secrecy of the IRS’ process. They can end with no scrap of publicly available paper to memorialize key decisions. In August 2016, in Schaeffler’s case, officials several rungs up the IRS hierarchy told the Global High Wealth team to withdraw the penalty from its request.

Even without a penalty portion, Schaeffler would still owe the original $1 billion in taxes if the IRS maintained its contention that the banks had cancelled his debt. Schaeffler’s team then went to work on that, too. It succeeded. By 2017, the IRS had abandoned its assertion that debt had been transformed into equity. After six years on a hard-fought case, the agency had effectively given up.

The IRS had a few stray quibbles, so the agency said it required a payment in the “tens of millions,” according to two people familiar with the audit. There the trail goes dark. Tax experts say Schaeffler’s team would likely have appealed even that offer, which in many instances leads to further reductions in money owed, but ProPublica could not ascertain that that occurred.

Thanks in part to the U.S. government’s bailout of the auto industry and the global economic recovery, the Schaeffler Group’s business rebounded. Despite a recent dip in the car market, things have turned out OK for Georg Schaeffler. Today, Forbes estimates his fortune at over $13 billion.