Keir Starmer has called on Boris Johnson to sack his education secretary, Gavin Williamson, for failing children during the pandemic and presiding over a “yawning gap” in attainment between private and state school pupils.

Amid rumours that Williamson could be replaced by either the equalities minister, Kemi Badenoch, or the vaccines minister, Nadhim Zahawi, at the next reshuffle, the Labour leader questioned why the education secretary was still in his job after two years of chaotic exam policies.

Starmer has previously stopped short of demanding that Williamson should be removed from office, saying only that he had “failed over and over again”. In January the shadow education secretary, Kate Green, called on her counterpart to offer his resignation.

Asked by the Guardian whether he thought Johnson should take matters into his own hands and sack Williamson, Starmer said on Wednesday: “Yes, yes and a long time ago. And I don’t think I’m alone.”

Starmer also poured cold water on the idea floated among some in government that letter grades for A-levels should be scrapped in favour of numerical 1-9 grading to provide a reset after the pandemic and address the perceived issue of grade inflation when teachers have been in charge of setting marks.

Nearly 45% of A-level entries in England, Wales and Northern Ireland were awarded top grades this week. For private schools the figure was 70% getting A or A*, compared with 39% at comprehensives and 42% at academies.

The Labour leader told the Guardian: “The problem is not whether we go to a 1-9 system; the problem is the baked-in unfairness. The gap between private schools and state schools has gone up. It was 20%, now it’s 30%. Instead of closing that gap it’s got worse. If you just brand it 1-9 it doesn’t solve that problem. The question the government has to answer is: why was the attainment gap so big before? Why is it even bigger now?”

Asked whether private school teachers had been too generous in their grading, Starmer said the attainment gap appeared to be to do with a “lack of a coherent framework to do the assessment” provided by the government.

“Some were testing very often and some not very often,” he said. “It led to the widening and now yawning gap between private and state schools. The hallmark of this government is wherever there is an inequality they can make it bigger and they are very busy doing that.”

Badenoch, the controversial equalities minister, has been mooted as a future education secretary to replace Williamson, a former chief whip who helped run Johnson’s leadership campaign.

Williamson’s position has always been one of the most precarious in the cabinet, with his stock low among Tory MPs. One said Williamson should have been “put out of his misery” a year ago over the exam results algorithm saga, adding: “No one has properly taken to task shit teaching, shit schools and shit headmasters – it’s utterly boring to hear complaints from middle-class MPs who fail these kids time after fucking time.

“There is a moral urgency to sorting out our schools. If people in the Conservative party are not interested in that, they should get out of politics.”

The fallout from this year’s A-level results, when the gap between private and state school grades grew to the widest in the modern era, appeared to have been seen by Badenoch’s allies as an ideal point to tout her own credentials as a potential successor. However, others believe Zahawi, who is perceived to have performed well as vaccines minister, had a better chance of getting the job given his wider experience.

Badenoch’s “anti-woke” credentials were warmly welcomed by Tory MPs when the Times reported she was in line for a cabinet post on Wednesday. They praised her as “supremely talented” and “very well thought of” in the party. One Downing Street insider said Williamson was widely perceived as “terrible” in the education brief and Badenoch would be a “very good” replacement.

However, some Tory MPs believed her name may have been floated by her opponents to give them an opportunity to point out her weaknesses and controversies, and reduce her chance of getting a promotion.

Badenoch helped pioneer the government-commissioned race report, which was criticised by a UN human rights experts who said it tried to “normalise white supremacy”.

Her appointment would also bring into cabinet someone who is not afraid to be openly hostile with the media. She made headlines this year for publicly branding a journalist “creepy and bizarre” for asking questions about Covid vaccines.

She holds a joint role as exchequer secretary in the Treasury, putting her close to the chancellor, Rishi Sunak – whose stock appears to be on the rise among Tory voters but declining in No 10 after a fallout with the prime minister.