Slow loading when many products have been already loaded

Hello,
On the category pages and loop/shop pages I use infinite loading.

When many products have been already loaded (like 100++) the loading becomes slower and slower.

Is there a way to fix or optimize this without using page pagination? Or is there a way to make the infinity scrolling look like infinite scrolling but actually make different pages when you keep scrolling? This way even if a customer clicks on a product and then presses the return button he won’t return at the top of the page but at the area, he was before opening the product page.