The first minister told BBC Scotland News that some of the difficulties he had faced since he succeeded Nicola Sturgeon in March were “outwith my control”.
And he issued a call for unity ahead of the SNP’s annual conference in Aberdeen this weekend.
Mr Yousaf said: “If you want to advance the cause of independence, if that is the reason you are in the SNP, then work with the party.
“Don’t kick the party, don’t damage the party, don’t take shots at the party, because all it is going to do, I’m afraid, is set back the cause of independence.”
The first minister said his door was “genuinely open” for anyone who had any issues.
And he said he was buoyed by polls which suggested that “around 50% of the population believe that Scotland’s future is best as an independent country”.
Mr Yousaf also said he had “every confidence” about overcoming the challenges being faced by the party.
He said that while “we may be slightly down”, the party was “definitely not out”.
Last month former SNP minister Fergus Ewing was suspended from the party for a week after a disciplinary vote by MSPs.
He has been a critic of the party leadership and had voted against the government in a no-confidence motion.
Mr Ewing said said he would not be “hounded out” of the party he loved and is set to appeal the decision.
Last week Labour defeated the SNP to win the Rutherglen and Hamilton West by-election, with Michael Shanks receiving more than double the number of votes polled by the SNP’s Katy Loudon.
Then Lisa Cameron announced her defection to the Conservatives on Thursday.
The East Kilbride, Strathaven and Lesmahagow MP was facing a selection contest to remain as the SNP’s candidate at the next general election.
She said she quit because of a “toxic” culture in the SNP’s Westminster group.
The party also remains under investigation by Police Scotland over its finances, a probe that resulted in the arrests of Ms Sturgeon and her husband, former SNP chief executive Peter Murrell.