As allegations of child molestation resurface, the reception for his latest film confirms the 78-year-old director's status as an American treasure

Exiting the successful New York premiere last week of his latest film, Woody Allen was delighted with the reception (and the A-list guests). But even he was aware that this level of prolific production has to end some time. "It's hard to find a story with a good beginning, middle and end that'll be entertaining to an audience," he told the Observer. "Eventually the supply will run out. I'll just keel over in my room with a stroke or something."

Allen is 78, and the reception accorded to Magic in the Moonlight, the fourth film by him to be released in three years, confirmed that he is an American treasure of the first order, despite the dark clouds over his private life.

The turnout of grandees at the premiere, from fashion's Anna Wintour to former New York police commissioner Ray Kelly, seemed to settle the debate over whether support for him would be withheld after allegations of child molestation were renewed in the New York Times earlier this year. Allen is more consistently successful now than at any point in his five-decade career.

Nevertheless, it has been a rocky year. In February, Allen was publicly challenged by his adoptive daughter Dylan Farrow, who reiterated claims – first levelled by Mia Farrow in the early 1990s – that Allen had sexually molested her. The claims came as Allen was about to receive a lifetime award at the Golden Globes. Allen rejected the accusation, saying that a previous police investigation had cleared him and his statement would be his "final word on this entire matter".

Last week, in an interview with Dave Itzkoff of the New York Times, he dismissed a suggestion that his audience might be affected. "I don't think anyone has ever not come to a film of mine that they thought they would enjoy," he said. "Nothing keeps them away if they think they'll enjoy the film. And if they don't think they'll enjoy the film, nothing we can do ever brings them in."

He seems to have won the public relations battle, however. One veteran film agent said last week that it was significant that New York society had turned out to support him. The original accusations had been a professional disaster for Allen, but their repetition had seemed to damage ex-wife Mia Farrow – who is now perceived by some as the vengeful ex-partner – more.

Meanwhile, Allen shows no sign of reducing his workload. Magic in the Moonlight, set on the Côte d'Azur in the 1920s and starring Colin Firth as a celebrated illusionist and Emma Stone as a would-be medium, comes in the wake of Midnight in Paris and To Rome With Love. Then there was Blue Jasmine, which earned Cate Blanchett a best actress Oscar. Each film was comfortably profitable, with Midnight in Paris taking $173m at the box office.

The movie industry has ignored calls from Farrow to boycott Allen's films; he is currently at work on a new picture starring Stone and Joaquin Phoenix and more are being readied.

While critical notices won't be published until next week, the latest film is clearly highly personal. Allen briefly performed as an illusionist. He told the Observer the film was inspired by a story he had read about the magicians and fraudulent spiritualists that the illusionist Harry Houdini was committed to exposing.

Allen is a storyteller, says David Thomson, critic and author of The New Biographical Dictionary of Film. While this may not be one of his best, "that doesn't mean that next year he won't make a very good one. Like any storyteller, some work and some don't.

"But why on earth do actors line up to work with him? The answer is, of course, because he's worked so steadily and become such an institution that it has become a badge of credit – almost a rite of passage – to work with him."

According to Firth, it would be mad not to exploit the years of craft contained in one directorial brain. "His writing, how he conceives characters and the plots he places them in, flatter actors," he said. Magic in the Moonlight, with Cole Porter's You Do Something To Me as its repeating soundtrack, also revisits the director's nostalgia for the elegance and aesthetics of past times.

For a director as dedicated to his work as Allen, an investigation – at 78 – into questions of spirituality and rationality is what makes the film more interesting, says Thomson. Allen, he senses, is simply not distracted by life around him.

"His mind is occupied with projects like this. It's a condition of creative work that creative people have great difficulties with life, so they find something that seems lifelike.

"He has a niche and an audience, and he makes films for a certain budget that gets enough of an audience so he can do what he wants."

That may also go to explain why Allen has never embraced Hollywood and its potential for greater riches.

"A natural storyteller like him could have earned a lot more money working in Hollywood, and I'm sure he enjoys it when a film does well, but I don't think that's what pushes him on."

Allen, says Thomson, has restored himself professionally. In spirit, he's the same sceptical, neurotic character he's always been.

"I'm like Blanche DuBois," he told Itzkoff at the Times. "I hope in life that there's a certain amount of magic. Unfortunately, there's not enough. There are little, sporadic things one could think of as magical. But for the most part, it's grim reality." © Guardian News and Media 2014