US District Court Judge Denny Chin on Tuesday said in a ruling 13 months after the parties had their day in his Manhattan courtroom that the proposed settlement is "not fair, adequate and reasonable."
"While the digitization of books and the creation of a universal digital library would benefit many, the (settlement) would simply go too far," Chin said in his 48-page decision.
The settlement would grant Google "rights to exploit entire books, without permission of the copyright owners," he said, and reward it "for engaging in wholesale copying of copyrighted works without permission."
The 2008 settlement resulted from a class action lawsuit filed in 2005 by the Authors Guild and the Association of American Publishers (AAP) charging Google with copyright infringement over its huge book-scanning project.
The settlement called for Google to pay $125 million to resolve outstanding copyright claims and to establish an independent "Book Rights Registry," which would provide sales and advertising revenue to authors and publishers.
While rejecting the settlement, the judge left the door open for the parties to go back to the negotiating table.
Google said it was studying the judge's ruling while the Authors Guild and the publishers said they were willing to try to reach a new agreement.
Supporters of the settlement argued that Google's proposed digital library and e-bookstore would make millions of out-of-print books available and provide a new avenue for authors to profit from their works.
Opponents urged the judge to reject the deal on antitrust, copyright and privacy grounds and said it would give Google exclusive rights to digitize "orphan works" -- out-of-print books which remain under copyright but whose authors cannot be traced.
The US Justice Department criticized the settlement for including books unless an author expressly opted out of the deal and Chin also expressed concern about the opt-out aspect of the settlement.
"Many of the concerns raised in the objections would be ameliorated if the (proposed settlement) were converted from an 'opt-out' settlement to an 'opt-in' settlement," the judge said.
The Justice Department welcomed Chin's ruling. "We believe the court reached the right result on this complex, proposed settlement," said Gina Talamona, a Justice Department spokeswoman.
"The settlement proposed by Google and the authors and publishers exceeded the scope of the underlying lawsuit on which it was based and created concerns regarding antitrust, class certification and copyright issues," she said.
Google counsel Hilary Ware said the ruling was "clearly disappointing" and the company would "consider our options."
"Like many others, we believe this agreement has the potential to open-up access to millions of books that are currently hard to find in the US today," Ware said. "Regardless of the outcome, we'll continue to work to make more of the world's books discoverable online through Google Books and Google eBooks."
Google opened a Google eBookstore in December, a venture that is separate from Google Books, which was launched in 2004 and has digitized over 15 million books from more than 100 countries.
The Open Book Alliance, a coalition of groups and companies opposed to the settlement, said Judge Chin's ruling is "a victory for the public interest and for competition in the literary and Internet ecosystems."
Gary Reback, counsel for the Open Book Alliance, which includes Google rivals Amazon, Microsoft and Yahoo!, said in an interview with AFP that the ruling is "exactly what we hoped for."
"It does firmly and unequivocally reject what Google proposed," Reback said. "I suppose they could appeal but they got slapped down so badly I don't expect them to do that."
"If this goes back to first base I think a lot of authors are going to ask why the case was settled so cheaply," Reback added.
John Simpson of settlement opponent Consumer Watchdog said the ruling "should send the message to the engineers at the Googleplex that the next time they want to use someone's intellectual property, they need to ask permission."
Macmillan chief executive John Sargent, speaking on behalf of the AAP member publishers, said they are "prepared to modify the settlement agreement to gain approval."
Scott Turow, president of the Authors Guild, said "readers want access to these unavailable works, and authors need every market they can get.
"There has to be a way to make this happen," Turow said. "We'll be studying Judge Chin's decision and plan on talking to the publishers and Google with the hope that we can arrive at a settlement within the court?s parameters that makes sense for all parties."
Follow Yahoo! News on Twitter, become a fan on Facebook
