Oxfam chief executive Mark Goldring should resign and allow the charity to "look to the future", according to Priti Patel.
The former international development minister was speaking after claims that some of Oxfam's aid workers used prostitutes in Haiti following the 2010 earthquake.
The charity has been criticised for the way it handled the investigation into the allegations, something for which it has issued an "unreserved apology" to the Government, donors, supporters and the people of Haiti.
Ms Patel told Sky News on Wednesday, however, that the resignation earlier this week of Oxfam's deputy director Penny Lawrence was not enough and that Mr Goldring should also stand down.
She said: "The leadership - in this case a lack of leadership - has been just extraordinary.
"They've put themselves, their reputation, probably their job titles, before the victims of these individuals.
"They're an organisation that campaigns on women, on girls' rights, on transparency, and they have utterly failed.
"I think now, for the thousands of people that volunteer, that donate to Oxfam, for their aid workers even today that are working in very difficult circumstances around the world, he owes it to them basically to resign and enable Oxfam to look to the future, to focus on their core activities of poverty reduction and alleviating suffering...and I think that requires a change of leadership."
Earlier on Wednesday, it was announced that the current International Development Secretary will meet the National Crime Agency on Thursday over the scandal.
Penny Mordaunt said: "While investigations have to be completed and any potential criminal prosecuted accordingly, what is clear is that the culture that allowed this to happen needs to change and it needs to change now."
Charities receiving UK aid have been told to show "full transparency" and outline assurances about their safeguarding processes.
"If our standards are not met, then the British taxpayer will not continue to fund them," Ms Mordaunt said.
An NCA spokesman said: "The National Crime Agency have closely monitored the events occurring in respect of allegations of abuse overseas.
"There are a range of powers available to the NCA which enable us to investigate certain sexual offences committed outside the UK."
Meanwhile, Paris-based aid group Doctors Without Borders (MSF) said it had acted on 24 cases of harassment or sexual abuse last year.
The charity said it had received 146 complaints or alerts last year, identifying among them 24 cases of harassment or sex abuse and firing 19 employees as a result.