To achieve this you need to select your specific colors while editing the attribute swatches from the Products >> Attributes>> Your attribute>> Configure Items: https://jmp.sh/V2uCbG7
Edit your swatches from “Edit option” under swatches and select color as in the screenshot: https://jmp.sh/V2eZDFr.
If you want to remove the color filter you can do this in Appearance -> Widgets.