The problem of proof
I am embarrassed to make such a minor, pedantic point in discussing a question as big as the Singularity, but—well, it's just that, quite simply, there is no evidence to support it.
None. It is speculation informed by fiction. The Singularity is fantasy covered by a patina of rational thought, but. It. Is. Not. Rational. PowerPoint-informed imagination is not sufficient cause for physical existence. Belief in approaching AI, or even mildly human-like AI, requires considerable delusion. Our spreadsheets and databases possess no mindfulness. And the Singularity has all the problems of AI, coupled with the manic imagination of people who spend too much time looking at graphs. Technology does not create itself any more than boy bands do.
The argument behind Singularity amounts to bare extrapolation, and that's it. Look at how fast computers are getting! Wow, what if we extend the line out to eternity? Singularity thinking is like a man who finishes watching the Star Wars trilogy in 1998, right before the Phantom Menace comes out. Then he goes home and plots out the sequels. Allowing for advances in moviemaking, his chart tells him that this next movie will be the greatest event in cinematic history.
Here comes the great engine
Suppose you're at a tech conference. A steam engine engineer sits down across from you. He explains that the great steam engine is on the way.
You scoff. But the engineer is emphatic. He has charts: Hasn't steam engine tech evolved since the beginning of the 20th century? Modern materials are constantly improving. The Swedish company Energiprojekt AB has a five-cylinder engine with 30 percent efficiency. Arguably, modern steam engines now are the best they've ever been. Moreover, steam engines will one day improve themselves, without human interference. After all, many steam engines are capable of self-regulating behavior, using a governor. By the law of technological evolution, engines will upgrade themselves constantly. Faster engines will make faster engines. Eventually they'll hit light speed, and travel in time.
At some point in the future, a time-traveling steam engine will learn all secrets, discover the cure to death, and rule us justly. The Steam-ingularity will arrive.
For this reason, you have obligations to Time-Traveling Steam Engine you may not be aware of. The Engine will know who did, and did not, support research into steam. And even if you aren't afraid of the Engine's vengeance, you have a moral obligation to contribute to organizations that are researching how to build the Engine. Since the Engine will solve death, if you don't support the Engine, you are letting people die unnecessarily.
You point out that steam engines have no agency. They don't steer themselves. At best, they follow the tracks we lay down. "Well, for now," the engineer says.
The funny thing about the singularity…
Believing in the Steam-ingularity would require committing the error of projecting a graph to its illogical conclusion — observing an upward trend and making some unscientific extrapolation. Yet graphs of steam engine efficiency are just that — graphs. Simple models. Not pictures of the world. Human beings are terrible about predicting the future, mostly because we forget the strangeness of history, the limitation of math, and the glacial slowness of societal change.
If there is no rational evidence to believe in Singularity, what explains its apostles? As Daniel Kahneman used to ask when confronted with an irrational belief, "What might this be true of?" The Singularity is different from other pipe dreams, like the perpetual motion machine or Elon Musk. Singularity believers are legion, and well-funded.
The most interesting feature of Singularity is not the promise of AI, but the inability of self-labeled rational people to understand probability. The Singularity is the image of rationality, and not rationality itself.
Reader, suppose your identity came from being a Very Smart Person: consider the kind of God you would dream up. If the God of soldiers is a man of war, and the God of the poor is a generous master, then the God of Techies would be a Supremely Intelligent Machine. A machine that shares their preferences. A machine fashioned, in a sense, from their hand. The Singularity is the perfect religion for the Valley. Techies convinced people of their genius by reinventing everything. And here they are, reinventing religion.
And not reinventing well. Singularity is a belief in a silicon deity. But it's not a mature faith. Most thoughtful monotheists struggle with the problems of an omnipotent God. They devise theologies and arguments concerning the deity. Even if they're wrong, they show a seriousness the Singularity fans lack.
Worse, the Singularity does honor to a monstrous notion: the benevolent despot. Ancient elites distrusted the rabble, so they invented the idea of the perfect king. During the Renaissance, political writing was obsessed with the notion of creating The Good Prince. With enough education, the right upbringing, the perfect words, an ideal ruler could be shaped. In modern terms, the benevolent despot would see all of the data, figure out the proper response — and the people would be just and happy. In 10,000 years of recorded history, no king has been worthy of absolute power. No machine ever will be.
Even the divinely favored kings of the Bible run afoul of Heaven. But the Singularity is divine right without any brake. And that should give us pause. If you want a Good Shepherd, look inside yourself, read philosophy or go to church. Nobody else is selling.