Sen. Rand Paul's (R-KY) controversial aide, who believes in Southern states should secede and has a history of wearing Confederate flags, has resigned to return to his career as a pundit.

Jack Hunter told The Daily Caller that he was quitting his job to clear his name and to avoid being a distraction for the senator. Hunter -- who was known on his South Carolina radio show as the “Southern Avenger” -- said he had been unfairly called a racist because he often wore a Confederate flag mask and was previously the chairman of the League of the South, which the Anti-Defamation League has called “an implicitly racist group.”

"I’ve long been a conservative, and years ago, a much more politically incorrect (and campy) one," Hunter wrote in an email. "But there’s a significant difference between being politically incorrect and racist. I’ve also become far more libertarian over the years, a philosophy that encourages a more tolerant worldview, through the lens of which I now look back on some of my older comments with embarrassment."

The Daily Caller's W. James Antle III noted that Hunter was a "personal friend" and had contributed columns to the website.

In an interview with The Huffington Post earlier this month, Paul said that Hunter should be forgiven for wearing Confederate flag masks because his mistakes were similar to using marijuana.

"If I thought he was a white supremacist, he would be fired immediately," the Kentucky Republican said at the time. "If I thought he would treat anybody on the color of their skin different than others, I’d fire him immediately."

"It was a shock radio job," Paul added. "He was doing wet T-shirt contests. But can a guy not have a youth and stuff? People try to say I smoked pot one time, and I wasn’t fit for office."

For his part, Hunter now blames "neoconservatives" for the controversy.

"I look forward to returning to just being a pundit and fighting these battles on my own," he said in his email to The Daily Caller. "The neoconservatives, who first ran and promoted this story, would much rather argue about the Civil War than the Iraq War."