Some maps which are quite high on the map ladder are not featured (for example, Random Plateaus v0.84 at #2, DeltaSiegeDuoX at #10 and SandCastles2 at #14) so they do not show up in teams votes. I think this accounts for most of the high-variance maps.
Some maps are featured but not tagged for teams; this is rare, but it happens that the #1 rated Red River Remake v1.2 fits in this category. (Which is probably
why it is #1, since it is rated purely based on its honeymoon.)
(Most of the other maps in this category are FFA maps, and a small handful of strictly-1v1 maps like Anvilwood and Izki Channel, but most of these were never or only briefly tagged for teams so they don't appear on the ladder.)
There might also be some limited interaction between number of players in the lobby and the map size that gets picked, I am not sure.