"Now we must move forward," Obama said in a written statement accepting Daschle's request to be taken out of consideration. A day earlier, Obama had said he "absolutely" stood by Daschle.
Daschle, the former Senate Democratic leader, said he would have not been able to operate "with the full faith of Congress and the American people."
"I am not that leader, and will not be a distraction" to Obama's agenda, he said.
His stunning statement came less than three hours after another Obama nominee also withdrew from consideration, and also over tax problems. Nancy Killefer, nominated by Obama to be the government's first chief performance officer, said she didn't want her bungling of payroll taxes on her household help to be a distraction.
Obama, Vice President Joe Biden and Daschle's former Democratic colleagues had rallied to his defense in the wake of questions about a series of tax issues. Last month, Daschle paid $128,203 in back taxes and $11,964 in interest.
"Tom made a mistake, which he has openly acknowledged," Obama said. "He has not excused it, nor do I. But that mistake and this decision cannot diminish the many contributions Tom has made to this country."
Daschle also was facing questions about potential conflicts of interests related to the speaking fees he accepted from health care interests. Daschle also provided advice to health insurers and hospitals through his post-Senate work at a law firm.
The withdrawal comes after Republicans and major newspapers questioned Obama's decision to stick with Daschle.
Republican Sen. Jim DeMint of South Carolina said Obama was "losing credibility" with his statements in support of Daschle. "Part of leadership is recognizing when there has been a mistake made and responding quickly," the Republican said.
In an editorial, The New York Times described Daschle's ability to move "cozily between government and industry" as a cloud over any role he might play in changing the nation's health care system.
The Chicago Tribune opined that "Daschle is dispensable" and suggested that "to proclaim high standards and then suspend them exposes Obama to charges that he is either hypocritical or obtuse."
© 2009 The Associated Press. All rights reserved. This material may not be published, broadcast, rewritten or redistributed.