Few things concentrate the mind towards anxiety like being informed by one of the world's most famous rock stars – famous not merely for selling 250m records, but also for having a bit of a temper – that it was all your idea that he play a festival where it is currently pouring with rain and the average audience member is, as he puts it, "young enough to be my grandchild". Sir Elton John is not, one suspects, one of life's great advocates for the pleasures of spending a weekend sleeping under canvas, queuing to use the composting toilets and making the best of it while the elements lash the main stage and the ground turns to slurry beneath your feet.
As has frequently been pointed out in the runup to his appearance at Bestival, the last time he performed at a festival, the Beatles had yet to break up, Jimi Hendrix was still alive and John was so little-known that the reviewer from Melody Maker had to accost him backstage and ask him what the songs he played were called. Having deliberately avoided them for the past 43 years, here he is, sitting in a portakabin in the Isle of Wight's Robin Hill County Park, resplendent in a jacket with the title of his 1971 album Madman Across the Water picked out in crystals on the back, muttering darkly about the mud, mentioning that he had second thoughts en route to the event – "When I drove down into the campsite, I thought: 'Oh God, I don't know about this'" – and troublingly persistent in his belief that "it's only because of you that I'm fucking doing this".
It goes without saying that this is not quite my interpretation of events. I interviewed Elton John a year ago, backstage in Las Vegas, where his Million Dollar Piano Show was packing them in. He was charming and chatty and "shitting myself" about a forthcoming performace in Ibiza with Pnau, the Australian electronic duo whose reworking of his 70s back catalogue, Good Morning to the Night, turned out to be his first UK No 1 album in more than 20 years. I wondered aloud why, alone among his peers, he hadn't played a big UK festival: as anyone who has witnessed a headlining set at Glastonbury knows, the secret is having songs that the audience immediately recognise. He dismissed the idea out of hand – "I'm too scared that people will throw things at me, I'm like the Queen Mother" – but something about the conversation apparently stuck with him and, when Bestival approached him for the umpteenth time, he agreed.
Had all this happened 25 years ago, I might be leaving the dressing room at high speed, pursued by flung crockery: his tantrums were so legendary that his friend, the late Dusty Springfield, advised him that the secret was not to smash anything expensive. But, if he's not exactly proposing we all pull our wellies on and follow him in an expedition up to the Wishing Tree Field, there is no hint of diva behaviour, nor even a flicker of irritability. John is quite the nicest international rock superstar you could wish to meet, if a little more preoccupied by the ongoing state of the weather than your average interviewee. He greets a succession of visitors warmly – upcoming R&B band the Strypes, who out of their stage uniforms look less like musicians than a group of fifth formers unaccountably separated from the rest of their school trip; and Chic's Nile Rodgers, who regales him with tales of his battle against prostate cancer, a topic he seems to have turned into something approaching a standup routine.
As is usual, John's conversation revolves around his current taste in music, which, as ever, is substantially more wide-ranging and leftfield than you might expect: he has just bought the debut album by punishing electronic collective Factory Floor ("punishing is good"), is particularly taken by the arrangements on These New Puritans' Field of Reeds, and seems delighted that 16-year-old New Zealand singer-songwriter Lorde has asked to perform with him.
It's not that people have got the wrong idea about his temper, he says. He certainly used to have his moments, particularly in his cocaine years, as evidenced by the famous incident when, awoken in his hotel room by a storm outside, he rang his management office and demanded they do something about the weather. It's partly fatherhood, he says, that has changed him: in January, he and his partner David Furnish became parents for a second time, when Elijah Joseph Daniel Furnish-John was born to the same surrogate mother as his two-year-old brother, Zachary. "I always thought I might be really irritable with kids, but no way. Both my parents had tempers and it frightened the shit out of me. I don't ever want to display that in front of the children. In the past, particularly when I was on drugs, there was a monstrous side to me, but I'm not really like that. I can be like that briefly, but everyone can. People say: 'Look, he's a fucking monster.' Hang around anybody in fucking showbiz, baby, and you'll see! We all have our sides like that."
And he heads off into the rain, up the stairs to the stage, where thousands of people young enough to be his grandchildren appear to be chanting his name.
Two days later, I meet him again at Woodside, the Windsor home he bought in 1975: in one of those asides that reminds you that, however much he seems like your average music obsessive when he's raving about US singer-songwriter Phosphorescent or the new Artic Monkeys album, John is very different from you or I, he casually mentions that he sold the house's entire contents at Sotheby's in 1988, netting £20m, then started collecting again. For a man who must have heard vast crowds singing along to Rocket Man and Your Song thousands of times, he seems genuinely and rather sweetly moved by his reception at Bestival. "I honestly think that's one of the best shows we've ever played, and one of the most enjoyable shows I've ever done in my life. I've played to big crowds like that before, and usually there's a big roar and you're on. This one … it just kept on going. It's very touching, for someone of my age and where I am in my career, to see that many young people enjoying what you do. And as you know," he smiles, "I've always been dead set against festivals, really suspicious and wary. I was really ecstatic on the way home."
Bestival comes at the end of a 12-month period that has been tumultuous even by John's standards. His second child was born in January. He recorded a remarkable new album, The Diving Board, which sounds unlike anything else in his back catalogue, and performed with a fairly mind-boggling array of other artists (he is presumably the only person who can claim to be on Queens Of The Stone Age's latest album … Like Clockwork and the new one by Engelbert Humperdink). He started work on a "surrealistic" biopic of his life, slated to star Tom Hardy, which apparently begins with the diverting image of John entering rehab in the early 90s dressed in a chicken costume. And he prompted the Chinese culture minister, Cai Wu, to demand that only stars with university degrees be allowed to play in China, after dedicating a show in Beijing to "the spirit and talent" of dissident artist Ai Weiwei.
"Ten thousand people went 'Eugh!', but how could you not?" he says. "He's become a friend, and what he's suffered as an artist is unbelieveable. During and after the show, my tour manager was interrogated for three hours, and what he said to them was true: I bought a beautiful Ai Weiwei piece, I was grateful and we've become friends. I didn't feel in any danger. Listen, I went to Russia in 1979 and I knew we were being watched all the time: I had an interpreter that they'd clearly set up." He laughs. "Actually, I ended up having sex with him on the roof of my hotel."
And then, this summer, he nearly died when appendicitis was misdiagnosed as a colon infection. "I did nine shows, 24 flights and played the White Tie and Tiara Ball [the annual fundraiser for his Elton John Aids Foundation] with Coldplay with a burst appendix. I was so lucky, because usually if it bursts, it causes peritonitis. If that happens, you have to be in hospital within an hour, otherwise you're brown bread. I'm lucky to be alive. They did the operation and it was not a walk in the park. They said: 'Oh, you'll be out the next day.' But it turned out to be a lot more complicated than that. I was in a lot of pain, and I was hallucinating from the morphine and the anaesthetic. I didn't sleep for four days, it was like being on speed. I was hallucinating like mad, which was, you know, amusing for the people around me."
He says his illness has caused him to reassess his life. "It was a sign. I'm addicted to working. I mean, I have a list of 100 countries I want to play in. I'm basically killing myself by travelling so much, for no reason whatsoever. As you know, I can't have one pair of shoes, I can't have one CD, I can't have one bunch of flowers, one car, one ornament, I mean, that's my mindset. It's like, if I do 120 shows in a year, I want to do 130 the next year. I'm very proud of doing over 200 flights a year. Why are you so proud? It's fucking killing you. After I was ill, I sat down with David and had to admit to my addiction: 'David, I can't stop working.'" He shakes his head. "I always run away with it. The horse bolts, like with the cocaine and everything like that. So it's just a wonderful wakeup call."
Besides, he says, cutting down on touring will enable him to make more music: he's planning on doing "something" with Pharrell Williams, and writing a musical about Tammy Faye Bakker – the late Christian singer and flamboyant wife of disgraced televangelist Jim Bakker who is an improbable gay icon – with Scissor Sisters frontman Jake Shears. Then there is his ongoing collaboration with lyricist Bernie Taupin, which he cheerily describes as "probably the strangest relationship in pop history": in 46 years, the two have never written a song in the same room, and John never reads Taupin's lyrics before setting them to music. "I just go into the studio, look at the lyrics for the first time when I put them on the piano, and go. If I haven't got it within 40 minutes, I give up. It's never changed, the thrill has never gone, because I don't know what I'm going to get next. I don't know what's going to land in front of me."
He is justifiably proud of The Diving Board, a stark, sombre, haunting album, filled with Taupin's bleak ruminations on mortality and ageing, and improvised instrumental piano pieces. It is, by some distance, the best album John has made in decades, sounding nothing like an attempt to recapture his 70s glory years and everything like a man embarking on a new musical chapter. It's tempting to wonder what the audiences who pay to hear him play The Bitch Is Back and I'm Still Standing will make of it. "They're miserable songs," he concedes, "and I love to sing miserable songs. If misery is done well, it's fantastic." He laughs. "I love misery. I don't know how the record company are going to sell it. I know it's not a commercial record, it's not going to sell zillions. I'm not Elton John from the past. I'm still very driven, but not in a bad way. I've never been jealous of anybody's success. I've been flummoxed by it because I don't understand it, but I'm not jealous of it."
He wants to make another album with the same producer, T-Bone Burnett, which leads him to talk about his other unfulfilled ambitions. He's always wanted to play in front of the Pyramids of Giza, "but they banned me from going there a couple of years ago because I'm gay". It wasn't the first time his sexuality proved an issue: a church in Tobago organised a protest against his appearance there, while he received threats from an Islamic group when he played in Morocco. "I'm supposed to be going to Moscow in December," he says. "I've got to go. And I've got to think about what I'm going to say very carefully. There's two avenues of thought: do you stop everyone going, ban all the artists coming in from Russia? But then you're really leaving the men and women who are gay and suffering under the anti-gay laws in an isolated situation. As a gay man, I can't leave those people on their own without going over there and supporting them. I don't know what's going to happen, but I've got to go."
He says, a little improbably, that he'd like to go back to China and talk to the government about Aids – the country's leading Aids activist has claimed that the government's estimate of 650,000 HIV cases is barely a 10th of the real figure. And he's never visited Albania, or Vietnam, or Cambodia, or Armenia. For a moment, he doesn't sound much like a man cutting his workload in half, then he catches himself. "But my priorities have changed. I probably won't be going to Albania any time soon. I don't want to tour all the time. I want to spend time with David and the kids. I've got musicals to write, films to produce. None of us know how long we've got left. I could go out tomorrow and have a car crash. But you can take precautions if you want to have a longer life. And I want to grow old with my children."
The Diving Board is released on Mercury on 16 September
© Guardian News and Media 2013
