OK I already found the solution.
In case someone else is looking, this worked for me:

function my_woocommerce_catalog_orderby( $orderby ) {
    unset($orderby["menu-order"]);
	unset($orderby["popularity"]);
    unset($orderby["rating"]);
    return $orderby;
}
add_filter( "woocommerce_catalog_orderby", "my_woocommerce_catalog_orderby", 20 );