Hello,
The layout you are using is currently assigned to All Products, so it affects every product page.
If you want a layout only for a specific category, you need to create (or duplicate) a Single Product layout and set its condition to Include > Product Categories > Your category, and remove the All Products rule.
This way, only products from that category will use that layout, while the rest keep the default one.
Further, you can read more about the Layout Builder here: https://xtemos.com/docs-topic/woodmart-woocommerce-layout-builder/
Best Regards,