Hi,
By default, the WooCommerce plugin doesn’t have products next/previous arrows functionality at all. To implement this feature we use standard WordPress PHP get_next_post and get_prev_post functions that can obtain adjacent posts from the database. Unfortunately, it takes products from the whole table and depends only on the shop page order. The system can’t know that you previously visited this product’s category page. Yes, it would be much better to show next/prev product from the same category. We spent a lot of time investigating this issue but, unfortunately, there is no solution for this problem. You can check any other theme for WooCommerce and you will see the same issue.
Regards