In a rare move, some Secret Service agents sent to Colombia to help protect US President Barack Obama at a regional summit have been sent home amid accusations of a sex scandal, officials and reports said on Friday.

"There have been allegations of misconduct made against Secret Service personnel in Cartagena, Colombia prior to the president's trip," Secret Service special agent in charge Edwin Donovan said in a statement.

"Because of this, those personnel are being relieved of their assignments, returned to their place of duty, and are being replaced by other Secret Service personnel."

He did not specify what allegations had been made against the Secret Service staff, who were in Cartagena ahead of Obama's arrival late on Friday for the Summit of the Americas which opens on Saturday.

But at least one of the agents had been involved with prostitutes in the Colombian resort city, the Washington Post said quoting Jon Adler, president of the Federal Law Enforcement Officers Association.

News of the scandal broke just as four home-made explosive devices went off, two in Cartagena and two in the Colombian capital Bogota late Friday near the US embassy without causing any casualties or damage, police said.

But Donovan stressed the Secret Service staffing changes "will not affect the comprehensive security plan that has been prepared in advance of the president's trip."

"The Secret Service takes all allegations of misconduct seriously. This entire matter has been turned over to our Office of Professional Responsibility, which serves as the agency's internal affairs component," he added.

He did not say how many agents had been sent home. But the Washington Post said there have been other incidents involving Obama's security detail, including an agent charged with second-degree murder after an incident in Hawaii in November ahead of an APEC summit, and one charged with drunk-driving while helping to arrange security for an Obama bus-trip in Iowa in August.

During a stopover in Tampa, Obama said he would be thinking about US workers during his talks.

"I am going to be thinking about how we can get more business, access to more markets and more customers in the region," he said. But "I want us selling stuff and I want to put more Americans back to work."

Two issues -- the pros and cons of drug legalization and Cuba's continued exclusion from the summit -- were expected to dominate the agenda, highlighting the growing disconnect between Washington and an increasingly assertive and independent Latin American bloc led by powerhouse Brazil.

Before leaving Washington, Obama made it clear in an interview with an association of leading Latin American newspapers, that he rejects the idea of decriminalizing drugs.

Washington would not "legalize or decriminalize drugs because doing so would have serious negative consequences in all our countries in terms of public health and safety," he said.

On Cuba, he insisted that Havana authorities "have shown no interest in changing their relationship with the United States, nor any willingness to respect the democratic and human rights of the Cuban people."

Cuba has never taken part in a Summit of the Americas. And in early March, Colombian President Juan Manuel Santos traveled to the Cuban capital to explain that a lack of consensus had prevented Cuba from being invited this time.