Most new users seem to use steam and it is apparent their friends list seems to be synced with steam friends (at least from what I can see with personal tests). This means that the annoyance is only for:
1.) Users who do not use steam.
2.) Users who are using steam but are not steam friends with each other. (I suppose this can happen if one uses steam and the other uses steamless from itch)
Even then, since friends list is one directional, the annoyance from this is rather limited IMO vs the annoyance of having to kick random people who join your room.