Home Secretary Priti Patel, a former close ally, has joined a group of rebel ministers who are urging the PM to stand down.
But he has defied the growing calls for him to quit, saying it would not be "responsible" for him to go.
Levelling Up Secretary Michael Gove has been fired from his cabinet, after he urged the PM to resign.
Mr Johnson earlier told senior MPs it would not be right for him to "walk away" amid economic pressures and the war in Ukraine.
Under repeated questioning by the Commons Liaison Committee, he ruled out calling a snap general election, saying the earliest date he can see for one is 2024.
A Downing Street source rejected speculation the PM would announce his resignation in Downing Street later.
"There is no lectern outside No 10 tonight. The PM fights on," the source told BBC News.
The ministers urging him to quit also includes Chief Whip Chris Heaton-Harris, Transport Secretary Grant Shapps, and Welsh Secretary Simon Hart.
Mr Gove told him to go earlier, and Business Secretary Kwasi Kwarteng has told the chief whip the PM should resign.
Mr Johnson rang Mr Gove on Wednesday evening to tell him he was sacked.
A No 10 source said: "You can't have a snake who is not with you on any of the big arguments who then gleefully tells the press the leader has to go."
Mr Hart later resigned from his post, saying he wanted to help Mr Johnson "turn the ship around" but "we have passed the point where this is possible".
BBC political editor Chris Mason said the rebel group was joined in Downing Street by another set of ministers arguing he should stay.
The BBC has also been told Mr Johnson has been stressing that "millions" voted for him, and questioning whether any of his would-be successors would be able to "replicate his electoral success at the next election".
Meanwhile, Tory party bosses on the executive of the backbench 1922 committee have postponed a decision on whether to change the rules governing a vote of confidence.
However, elections to replace the committee have been pulled forward and will now take place next week.
Mr Johnson survived such a vote last month, and under the rules as they currently stand he would be immune from another challenge for a year.
The crisis engulfing Mr Johnson's premiership began on Tuesday, following the dramatic resignations of Chancellor Rishi Sunak and Health Secretary Sajid Javid.
They quit within minutes of each other following a row over Mr Johnson's decision to appoint Chris Pincher deputy chief whip earlier this year.
Their departures have triggered a wave of further resignations, with one in five MPs in government roles quitting their posts.
A senior ally of the prime minister told the BBC: "It's now a question of how he exits," adding the situation was "not sustainable".
At Prime Minister's Questions, Labour leader Sir Keir Starmer said the Conservatives had become a "corrupted party defending the indefensible".
He took aim at ministers who have resigned in recent days, saying their decision not to quit before now showed they lacked a "shred of integrity".
And he rounded on those who have stayed in post, mocking them as a "Z list cast of nodding dogs", keeping the PM in power.
He added they were "only in office because no-one else is prepared to debase themselves any longer", calling them the "charge of the lightweight brigade".
Tory backbencher Gary Sambrook accused the PM of blaming other people for his mistakes and was applauded after calling on him to resign.
But Mr Johnson defied calls for him to go, adding he had a right to stay because of the 80-strong majority he won at the 2019 election.
"The job of a prime minister in difficult circumstances, when he's been handed a colossal mandate, is to keep going - and that's what I'm going to do."
Sometimes politics is subtle. And then there are days like today.
Sitting in the Press Gallery, watching Prime Minister's Questions, you could feel and hear Boris Johnson's authority draining away.
Tribalism is measurable in decibels in Westminster, and the Conservative benches sat in near silence - the noise came from the opposition benches - in what was Boris Johnson's toughest PMQs since the general election.
By early afternoon, the letters of resignation and no confidence were tumbling in, and even Mr Johnson's most loyal supporters privately - and often bluntly and colourfully - acknowledged the game was up, it was over.
Is it curtains? I asked a cabinet minister. "Yes I fear so. It's hours and days," came the reply.
Then, one after another, cabinet ministers texted me saying they were heading to see the prime minister this evening to tell him explicitly he had to go.
Westminster is a postcode defined by power. And tonight it is shifting. The end of Boris Johnson's premiership appears imminent.
In a resignation statement after PMQs, Mr Javid said "treading the tightrope between loyalty and integrity" had become "impossible in recent months".
He added: "At some point we have to conclude that enough is enough. I believe that point is now."
Mr Johnson has admitted it was a "bad mistake" to appoint Mr Pincher, despite being aware of misconduct allegations against him.
The row over his appointment comes after the PM's relations with his backbenchers have been damaged by the Partygate scandal and unhappiness over tax rises.
Several key cabinet ministers have rallied round the prime minister.
However, rebel Tories want to use the upcoming vote to the backbench 1922 committee to scrap the year-long gap between contests, making it possible he could face another challenge later this summer.
If party bosses change the one-year rule on leadership challenges, rebel Tory MPs could try again to oust him later this summer, or in the autumn.
If Mr Johnson lost a vote of no confidence in Parliament, he would have to resign or call an election.
Otherwise, he would have to resign himself - possibly in the face of cabinet pressure, like Margaret Thatcher - or after a fresh wave of ministerial resignations.