Make the vote count required proportional to the number of people in the lobby. If people leave, lower the votes needed.
If some people leave the lobby then it's quite impossible for the rest of the people to meet the required number of votes to start the game, and sometimes the lobby dies because of this.
