Keir Starmer will be packing the laptop as well as the buckets and spades as he heads off to Devon for a family break later this month: aides say he plans to use the holiday to work on a first draft of his party conference speech.

It will be the Labour leader’s first address to the party faithful in person – last year’s was delivered online from an empty arts centre in Doncaster – and both fans and detractors agree it will have to be the speech of his life.

Some believe it will be something of an “autumn relaunch” to convince the public that Labour is serious about winning power, although the party says it is more about Starmer being able to set out his vision properly after the disruption of Covid.

But that phrase may set alarm bells ringing for those in the party who remember the days of Jeremy Corbyn and Ed Miliband attempting to reinvigorate their leaderships – providing an acknowledgement that Starmer’s stewardship is still beset by challenges, both political and practical.

Backbench MPs and some shadow cabinet members continue to gripe that they do not know what Starmer really stands for, or what his political project is. Meanwhile, he is laying off 90 staff in an effort to repair the party’s tattered finances, and has lost or removed almost all of the tight-knit group of longstanding aides who came with him from the backbenches into the leader’s office.

The most recent departure is Chris Ward, Starmer’s deputy chief of staff, who had worked with him since he first became an MP in 2015, wrote his speeches throughout, and prepped him for prime minister’s questions.

Some stalwarts remain, including the policy chief, Claire Ainsley. But the breakup of the “gang of five”, as disgruntled backbenchers called his most powerful aides, has only intensified the sense of mystery among some colleagues about what makes the Labour leader tick.

And as ever in the Labour party of recent years, internecine divisions are never far below the surface.

A slew of announcements on workers’ rights last week have helped to reassure some on the Labour left who feared Starmer planned to ditch Corbyn’s radical 2019 manifesto wholesale, after appearing to endorse much of it during his leadership campaign.

But the Corbynite campaign group Momentum and Labour’s biggest donor, the Unite trade union, have criticised the decision to proscribe four leftwing Labour groups, meaning their members are automatically thrown out of the party.

Starmer’s team say banning small but vocal groups including Labour Against the Witchhunt followed from his determination to root out antisemitism. But Unite called it “political machismo” and Momentum claimed it feared one day also being proscribed.

One party source blamed the clearout on “factionalists” in the leader’s office, rather than Starmer, who has repeatedly underlined his determination to seek party unity.

Meanwhile, without the draw of an impending leadership race or general election, party membership has declined by 120,000 from its peak under Corbyn, to 430,000. “They’re leaving in droves,” claimed one senior party figure from the Corbyn era. “There’s no organising going on in any constituencies. There’s no strategy, or programme, or project.”

Labour insiders will be watching intently when the result of the ballot for Unite general secretary is announced on 26 August. The frontrunner, Steve Turner, has been pressing Starmer to stick to the Corbynite agenda, while the moderate challenger Gerard Coyne has said the union under his leadership would no longer be the “backseat driver of the Labour party”.

Meanwhile, before his family break, Starmer has been continuing his tour around the UK, meeting voters and setting out Labour’s stall. He has sharpened his criticism of Boris Johnson, using a Guardian article to accuse him of being “missing in action” on the climate crisis and risking failure at the vital UN Cop26 talks due to be held in Scotland.

Speaking to the FT, he vowed to “turn the Labour party inside out”, urging activists to embrace Tony Blair’s political legacy to help the UK’s main opposition party win the next election.

As well as workers’ rights, crime is another policy area where Labour hopes to land some punches on Johnson, after years of cuts to public services and the criminal justice system.

A third is the economy, where the shadow chancellor, Rachel Reeves, will become increasingly prominent as the party tries to build up its reputation for competence. Since taking on the post in May’s fraught reshuffle, she has launched a “buy British” campaign that would mean a Labour government using its procurement muscle to support domestic businesses.

Reeves’s promotion was one of the few changes in what multiple party sources claim was initially envisaged as a wider reshuffle, following the local elections in May.

Moves were slated for figures including Lisa Nandy, who at one point threatened to resign altogether, over a rocky weekend, as Angela Rayner believed she was being made the scapegoat for the loss of the Hartlepool byelection.

Starmer then faced a tumultuous eight weeks, with some MPs touting potential leadership challengers, before the parliamentary labour party (PLP) was cheered by the victory of Kim Leadbeater, the sister of the murdered MP Jo Cox, in Batley and Spen.

Starmer loyalists in the shadow cabinet see reason for optimism as they head off on their own summer breaks.

Johnson’s “vaccine bounce” in the polls appears to be fading, and Starmer has beefed up his inner team after the departure of the “Camden set” with a series of experienced new hires, including the chief of staff, Sam White, a former aide to Alistair Darling.

There is also a perception that the new elections coordinator, Shabana Mahmood, and her deputy, Conor McGinn, have sharpened up Labour’s attacks on the Tories.

“It’s about bringing a level of brutality,” said one frontbencher, adding: “I think it’s pretty clear that the shake-up of the back room has made a difference.”

Recent polls have also given Labour cause for hope, with the Conservatives’ lead over the party narrowing from double digits.

Starmer’s team say they are well aware of the electoral mountain they have to climb, but are picking up hints in focus groups and polls of voters starting to have qualms about the Tory government.

“The public are not starting to change their view of Boris Johnson, particularly, but they do worry about the competence of the government, and they do worry about whether he’s up to the job,” said a Labour source.

They believe the charge of hypocrisy – summed up as “one rule for them, another for the rest of us” – is one they can continue to hammer home, helped by Matt Hancock’s lockdown-busting affair, the omnipresence of Dominic Cummings, and the drip, drip of procurement scandals.

Some backbenchers believe Starmer would need not just a solid party conference but a miraculous one to win over waverers.

One soft left Labour MP said colleagues were “just so resigned to the fact that he’s a dud”. They compared the mood in the PLP to the darkest days of Corbyn’s leadership, adding that many MPs were reluctant even to go to Brighton in September.

“It’s going to have to be the speech of his life,” the MP said. “He’s going to have to say to people: ‘I know you don’t think I’m up to it, but I am: and here’s why.'”