A Conservative police commissioner accused of victim blaming in the wake of the murder of Sarah Everard has refused to resign despite being told there is a “catastrophic lack of confidence” in his position.

Philip Allott, who oversees North Yorkshire police and the region’s fire service, was criticised after he said women “need to be streetwise” about powers of arrest and should “just learn a bit about that legal process” in case they were approached by officers.

His remarks were made in a radio interview following the sentencing of Everard’s killer, the Metropolitan police officer Wayne Couzens, who used his handcuffs and warrant card to abduct the 31-year-old from a south London street.

In an extraordinary meeting of North Yorkshire’s police, fire and crime panel on Thursday, Allott admitted making a “major mistake” in a “car crash” radio interview but he refused repeatedly to resign despite a unanimous vote of no confidence from the 11-member panel.

Allott’s Conservative colleagues lined up to tell him to resign during the 90-minute meeting. Mike Chambers, a Tory councillor, described the remarks as “indefensible”, adding: “This will continually haunt you, Philip, whether you like it or not. For God’s sake, go, and go now.”

Allott has faced a growing chorus of criticism since his comments 13 days ago, including from Boris Johnson, Priti Patel and Keir Starmer. Demands for his resignation grew this week when colleagues accused him of making “sexist and misogynistic” comments to female staff, allegations he denies.

The details of these alleged remarks have not been made public and Allott said they had been leaked “to damage my credibility”.

The majority of his 32-strong team signed a letter urging him to quit, saying they were “shocked” and “dismayed” by his comments. They said his response to the criticism had been “dismissive and completely devoid of emotional intelligence or empathy, approaching disregard, for the human impact his words have had”.

Police, fire and crime commissioners (PFCCs) were introduced in 2011 with the aim of holding forces to account. However, they have faced constant criticism for being too weak, expensive – they are paid GBP70,000 to GBP100,000 a year – and because they can only be sacked if they are disqualified from holding public office.

Carl Les, a Conservative council leader who chairs the oversight panel, said there was a “catastrophic lack of confidence” in Allott and it was “frustrating” that he could not be dismissed from the GBP74,400 a year role. Les has said he will write to the government to recommend that PFCCs are able to be dismissed if they lose a vote of no confidence.

Allott, who previously ran a PR firm and wrote a book about donkeys, said his comments had exposed “shortcomings in my understanding of violence against women and girls” and that he thinks about the comments “every hour of every day”.

Nearly 11,000 people have signed a petition calling for him to resign and almost 1,000 have complained to his office or the North Yorkshire police, fire and crime panel.

However, he also sought to downplay the criticism, accusing BBC Look North of “raking over” his remarks by repeatedly broadcasting them. Appearing via video link at the meeting, he said: “I can see that tensions are running high. We’re still within less than two weeks of that interview. I have to say if everyone resigned who makes a mistake in terms of an interview – and I accept it was a sensitive interview – nothing would ever get done in the country.”