Hello,

When you set the attribute to be used for variations, you need to select both attributes to change the swatch. Just have a look inside your variations: they are the combination size and color. If you want to change the swatch only by color, do not use the size for variations. Just add it to the product as an attribute.

Best Regards