Boris Johnson has been urged to drive out a “dinosaur mentality” from the Conservative party amid warnings that an MP mixing up two ministers of Asian heritage was “not an isolated incident”.

James Gray confused his Tory colleagues Nadhim Zahawi and Sajid Javid at a charity reception in parliament last month, allegedly saying “they all look the same to me,” it emerged on Tuesday.

It was the second time in less than a month that Javid, the health secretary, was mistaken for another minister of Asian heritage. The Guardian has learned that he was confused with Alok Sharma, the Cop26 president, at Conservative party conference in Manchester. A fellow delegate commented on him being driven in a gas-guzzling car despite supposedly being in charge of the climate summit, it is understood.

On Tuesday, Gray, the MP for North Wiltshire, was called in for a meeting with the chief whip and reminded of the standards expected of him. St John Ambulance, which appointed Gray as a commander last year, stripped him of the role after the alleged racist remark made at an event held in honour of its volunteers in September.

Gray, who was hosting its reception, tried to introduce Javid but incorrectly called Zahawi, the education secretary, up to the front. When the mistake was pointed out, the MP was said by a witness quoted in the Daily Mail to have told the audience: “They all look the same to me.” The witness added that Zahawi pulled Gray to one side for a private conversation afterwards.

Gray denied making that specific remark, telling the BBC he thought he might have said “I mixed you up” and adding it was a “very silly non-story”. In a statement to the Daily Mail on Monday night Gray insisted he had said the ministers “look very alike” and that he was “sorry if I got you two mixed up”. He added that the two were close friends of his and that “the notion this is some sort of racist remark is ridiculous”. His office did not respond to requests for further comment.

A spokesperson for St John Ambulance said it “does not tolerate racism in any way, shape or form”, adding: “We spoke with James Gray following the event about our values as an open, inclusive and progressive charity.”

The revelation about Gray’s comment prompted criticism from Sajjad Karim, a Tory former MEP. “He should have apologised immediately,” he said. “It’s certainly demonstrative of the huge journey the Conservative party needs to go on when it comes to the issue of diversity and – dare I say it – levelling up internally.”

Karim told the Guardian it was “not an isolated incident at all” and that he had lost count of the number of times other minority ethnic Conservatives had recounted being confused with someone else or he had experienced it personally, including being mistaken for the innovation secretary, Syed Kamall.

He said it was symptomatic of a “dinosaur mentality” and signalled his concern at the seemingly soft action taken against Gray. Karim said Gray should at the very least apologise and undergo diversity training, adding: “It is a clear signal that such behaviour and flippant reactions by perpetrators are still regarded as the order of the day. Back to the 50s, 60s and 70s, it is.”

A Tory anti-racist activist, who asked not to be named, called Gray’s remark “unpleasant and sloppy” and said comments like his would not have been accepted under past Conservative leaders but “in the post-2016 era, we seem to have to accept mediocrity”. Nick Lowles, chief executive of Hope Not Hate, said it was a “sign of an endemic inability and lack of political will to tackle racism”.

A Conservative party spokesperson said: “These comments were misjudged. We do not tolerate racism or discrimination of any kind.”