Proudct archive inifinite scrolling loading takes few sconds

Hi
On product archive page I use the inifinite scrolling option.
It always takes the products 3-5 seconds to be loaded, regardless of how many products to show and what data.
For example:
If I change the amount of products in page from 24 to 15, it will not affect the loading time.
If I choose to “remove data” to save loading time – such as: color variation or child products (and show only parents) – it will also not affect the loading time.

Does the 3-5 seconds of loading animation is set by code and not affected by the actual content of the page?
How can I make it faster?

Thanks
Hofit