Labour has questioned whether Dominic Raab should remain as foreign secretary describing his behaviour as “utterly shameful” after it emerged he delegated a reportedly crucial call to an Afghan counterpart to a junior minister last week while he was on holiday.

After days of increasing pressure on Raab over both his department’s response to the Taliban’s rapid takeover of Afghanistan and his decision to return from a beach holiday only on Sunday, Labour said he “should be ashamed” of his actions.

While not formally calling for Raab to go, Lisa Nandy, Labour’s shadow foreign secretary, said Boris Johnson “has serious questions to answer over why he remains in the job”.

In a tweet, the Labour leader, Keir Starmer, said: “Who wouldn’t make a phone call if they were told it could save somebody’s life?”

Labour’s home affairs spokesperson, Nick Thomas-Symonds, tweeted: “Failing to make a call has put the lives of brave interpreters at risk, after they served so bravely with our military. Utterly shameful.”

The Liberal Democrats called for Raab to go. Layla Moran, the party’s foreign affairs spokesperson, said: “Dominic Raab must resign today. If he does not, the prime minister should finally show some leadership, and sack him.”

The calls for his resignation follow a report that while on holiday in Crete, Raab was advised by departmental officials on Friday to speak by phone to his Afghan counterpart, Hanif Atmar, to request assistance on the removal of translators who had worked with the British military, as the Taliban advanced on Kabul.

According to the Daily Mail, officials were told Raab was not available, and that the junior minister, Zac Goldsmith, a Tory peer, should make the call instead. However, the paper said, as Goldsmith was not Atmar’s direct equivalent, there was a delay of a day in the call happening.

The Foreign Office said in a statement: “The foreign secretary was engaged on a range of other calls and this one was delegated to another minister.”

Nandy said: “What could possibly have been more important than safeguarding the legacy of two decades of sacrifice and hard-won victories in Afghanistan? While the foreign secretary lay on a sun lounger, the Taliban advanced on Kabul and 20 years of progress was allowed to unravel in a matter of hours.”

But in media interviews on Thursday the defence secretary, Ben Wallace – who has been privately critical of the role of Raab’s department over the Afghan crisis – defended his cabinet colleague.

Wallace told BBC Radio 4’s Today programme that by Friday the Afghan government was “melting away quicker than ice”, adding: “A phone call to an Afghan minister at that moment in time would have not made a difference.”

Challenged on whether he could be sure about this, Wallace said: “I do know for sure, because last Friday what we were absolutely worried and unsure about was whether the airport would remain open.

“You can speculate whether the phone call should or shouldn’t have been made, but it wouldn’t have made a blind bit of difference.”

Speaking to Sky, Wallace said: “At that time, last Friday, those were not the problems and the barriers for us getting people out, the problem was about whether we could get the airport open, and whether people could be using the airport as the Taliban advanced. That was the block then.”

The report about Raab follows intense and often furious criticism of the government from Conservative MPs and peers on Wednesday, when the Commons and Lords were recalled from the summer recess to debate Afghanistan.

Across both houses, 11 Tory former cabinet ministers were among those expressing anger and frustration at Britain’s failures in intelligence and preparation. In the Commons, more than 30 Tory MPs spoke against the government, while only a handful voiced support for its actions.

Asked about this, Wallace said the parliamentary session was “quite rightly a real outpouring of anger, frustration, and sadness about what we are seeing on the ground in Afghanistan”.

But he added: “I didn’t hear alternative solutions from anybody, including my colleagues who were criticising us, except one, some of our colleagues suggested that the British go back in again, as an army.

“I don’t think anyone feels sending unilaterally in a single force of tens of thousands, hundreds of thousands of troops would be the right solution now. But apart from that, there was long on criticism and very short on solutions.”