The business secretary has appealed to his party to give Boris Johnson the "time and space to deliver" on the government's promises as pressure continues to mount on his leadership.
The PM had another difficult week, with a critical report into No 10 lockdown parties and five senior aides quitting.
More Tory MPs also declared publicly they had no confidence in him.
But Kwasi Kwarteng said the PM was carrying out his commitment for a reset in No 10 and focused on policies.
And he told the BBC's Sunday Morning programme the "ongoing focus on 'partygate' is not helping anyone".
Former Tory leader Sir Iain Duncan Smith also told the programme he believed Mr Johnson should stay in his role as "the prime minister is the one that has to sort this out", adding: "[He] bears the overall responsibility".
And he warned other MPs who could be vying for a leadership bid to "temper their ambition".
But Sir Iain said the government "must do more" to tackle the cost of living crisis and "restore the sense [in the public] there is a government they can trust and respect".
The turmoil in Mr Johnson's government has led to growing questions from Conservatives about the PM's future.
On Monday, Sue Gray's initial findings around parties held in Downing Street during Covid lockdowns said there had been "failures of leadership and judgement".
More detail is also expected to come out as part of a Metropolitan Police investigation into 12 of the gatherings.
The PM also faced criticism - and the distancing of some of his ministers - after making a false claim that Labour leader Sir Keir Starmer failed to prosecute Jimmy Savile when he was director of public prosecutions.
This was followed by a string of resignations of senior aides - three of whom were caught up in the lockdown parties row, including senior civil servant Martin Reynolds who sent out an invitation to a "bring your own booze" party.
But one of his longest standing colleagues, Munira Mirza, quit over the Savile remarks and the PM's refusal to apologise.
Mr Johnson announced the appointment of two of their replacements over the weekend, with Cabinet Office minister Steve Barclay becoming chief of staff and his former colleague, Guto Harri, taking over as director of communications.
The PM said the pair would "improve how No 10 operates", and it is understood he is considering further changes to his top team.
Mr Harri tweeted that he was "delighted" to take up the role, alongside a picture of him with Peter Capaldi, who played Malcolm Tucker, No 10's director of communications in satire The Thick of It, and said he had "taken advice" from him.
Labour's deputy leader Angela Rayner called the new appointments a farce, adding the prime minister had "clearly run out of serious people willing to serve under his chaotic and incompetent leadership".
Mr Kwarteng told the BBC the new hires were "a step in the right direction", adding: "The prime minister has been very clear that he wants to reset No 10.
"And what we saw [this] week was him following through on that commitment."
He said the government had a "serious agenda" to tackle, with a potential war in Ukraine, the cost of living crisis and bouncing back from the pandemic, "so I don't think focusing on the parties is necessarily the best way for us to get through this".
But with increasing numbers of MPs writing letters of no confidence in the PM - 54 would be needed to trigger a leadership contest - including former schools minister Nick Gibb, has Mr Johnson lost the support of his party to get on with the job?
"I don't think so at all," said Mr Kwarteng. "I have been an MP for 12 years. In all that time, leadership issues have bubbled to the surface.
"A few people have said they are not happy with the direction and gave their views publicly... [but] on balance, if you look at the party in the round, I think people are just very, very focused on delivering on the manifesto."
The business secretary added: "[Mr Johnson] has got a clear mandate and we have got to give him time and space to deliver on that mandate."
One of those unhappy with the PM's leadership is Tory peer Lord Barwell - a former MP who served as Theresa May's chief of staff when she was prime minister.
He told Sky News: "My inclination is that the Conservative Party would be better making a change and I also think, for the good of the country in terms of trust and faith in our politics, there's a strong case for change."
But Lord Barwell said he did not think there was "a chance in hell that the prime minister is going to voluntarily resign".
Sir Iain - who led the Tories in opposition in the early 2000s - said the parties and the aftermath had been "hugely damaging" to the Conservatives and the public were "very angry".
Asked if the PM could recover that reputation, he told the BBC: "None of us know the answer to that question.
"Respect and trust you have to earn, and when you lose it, it's a very difficult task to get it back across the board."
But he said "internecine warfare" should not be the priority of MPs right now, and the government needed to focus on tackling the cost of living crisis.