it could be your hosting provider, I had to change three times before I found someone who was able to handle my word press sites… it is called hostpapa.com. I am also using jetpack and jetpack boost. I am using fastest wp cache. Within my basel template settings I do have lazy load and compression of my css files. I have my wp-cron job running via my cpanel versus wordpress.
I also had to install and reinstall and use a few different plugins before I found what works for me.