Keir Starmer has told Labour that it has to modernise and become “the party of the next 10 or 20 years” if it is to stand a chance of winning the next election, as he vowed to place winning above his popularity across the party.

In an interview with the Observer after a summer tour in which he met voters his party has lost in recent elections, the Labour leader said he would use the forthcoming annual conference to paint “in primary colours” the kind of country a new Labour government would champion.

“In order to win that election, we have to win the future,” he said. “We have to modernise and we have to be the party of the next 10 or 20 years, not the party that simply looks back at the last 10 years and says: ‘You could have had something else if you’d voted Labour.’

“The top priority is a Labour government. That’s the whole purpose of the Labour party. I came into politics to change lives, I didn’t come into politics to be in opposition. However much you may use social media to say what you did in a vote, if you lost the vote, you didn’t change anything. I have burned into my memory the fact that in the first 12 months, as an MP, I voted 172 times and lost 171 times. That is not changing lives. The No 1 priority is winning that election.

“What is the one thing that connects the victories in ’45 with Attlee, in the ’60s with Wilson and with Blair in ’97? It is that the Labour party in those moments glimpsed the future and had a forward-looking programme. That is at the heart of what we are doing at the moment.”

Labour’s conference next month has become a critical moment for Starmer’s leadership, which has come in for severe criticism from all wings of the party for a lack of clarity over its pitch to voters and Starmer’s own vision for the country. However, he is now developing big themes on crime, anti-social behaviour, the stability of jobs, British manufacturing, climate change and the damage done to children’s education during the pandemic.

Starmer said his summer tour had left him believing that some of the outright “hostility” towards his party was subsiding. However, he said it had taught him a personal lesson that voters needed to be convinced that he was listening. “The big takeaways for me were that patience is wearing thin for the prime minister. If there’s one thing people don’t like, it’s when someone like the prime minister says one thing and then does another and they’ve just seen too much of that.

“For Labour, what I was struck by was that people wanted to talk and wanted to engage. They were open to us. Now I’m not suggesting for a minute that people who voted Tory at the last election are already beginning to switch, but they’re open to that discussion … I think the constructive criticism is, can you show that you’ve changed? And probably the biggest takeaway, are you listening to me? Conference, obviously, will be our opportunity to set out in primary colours what post-pandemic Britain needs to look like.”

Starmer signalled he may be preparing to make a more explicit departure from the 10 pledges he signed up to during his leadership campaign, which contained many policies backed by Jeremy Corbyn and earned him significant support from the left of the party. Pledges included abolishing tuition fees, increasing tax on the richest and backing “common ownership” of key utilities. Moving away from them will mean Starmer faces anger within his own ranks.

“The 10 pledges are important statements of value that matter and therefore that is an important starting point for me,” he said. “Obviously, as we come out of the pandemic, the scale of the challenge that we now confront is bigger than even it was back in 2019. If anything, the scale of the answers has to be commensurate with the challenge. That’s what I’ll be setting out in my conference speech.”

Starmer was fiercely critical of the government’s handling of the Afghanistan withdrawal and called for the foreign secretary Dominic Raab to resign, but he said the decision to join the US-led invasion in 2001 had been the right one.

“Al-Qaida was operating from Afghanistan and in the 20 years since we went in, there haven’t been any attacks mounted by al-Qaida from Afghanistan,” he said. “That is a huge difference in the last 20 years. Women have held positions in public office, girls have been educated. The gains of the last 20 years when it comes to particularly women and girls should not be lost.”