A Mississippi lawmaker representing DeSota County apologized on Tuesday after complaining to a newspaper that “all the blacks are getting food stamps” and other remarks people are calling racist.

In a 30-second speech to the Mississippi House of Representatives, Rep. Gene Alday (R-Walls), who represents a district that sits just over the state border from Memphis, Tenn., said he was “deeply sorry” for his recent statements, according to an Associated Press report.

“I was wrong to say what I did, and there is no excuse for my behavior,” he said, receiving polite applause from the floor before leaving.

The furor started over comments he made the were reported by The Clarion-Ledger, which published a story on his views of the town of Walls, where Alday served as police chief and mayor before earning a seat in the state House in 2011.

He said that he came from a town where “all the blacks are getting food stamps and what I call ‘welfare crazy checks.’ They don’t work.” He also said that he was once taken to a hospital emergency room, where he laid “for hours because they [blacks] were in there being treated for gunshots,” according to the report.

Republicans quickly distances themselves from Alday, with both Gov. Phil Bryant and House Speaker Philip Gunn publicly condemning the remarks as racist. Alday said that same day he hadn’t meant his remarks to be published and that he is not a racist, and said he has taken steps to help black people. He rejected calls for his resignation.

Robert Johnson III (D-Natchez), who is black, led a news conference on Monday stating that he was concerned Alday’s comments indicate there were some racial motivations behind some of his policy decisions, although he said the next day he accepted Alday’s apology, while reiterating concern remains about how it affected certain policies decisions.