Done Terrible performance on Sky Factory 2.5

Hey, I just started playing this pack today! Sadly, there's something weird going on with performance.
I'm basically experiencing massive lag spikes. Even though FPS is between 40-60, it definitely feels like half the amount and I'm not sure why. This problem doesn't appear on any other modpack, even Infinity works far better.
Not running any performance-improving mods like Optifine or Fastcraft. My PC should be beefy enough to run it well (Core i5-3350P, 8 GB RAM). Or maybe there's something I don't know?
Screenshot with a lag spike meter:

Spoiler: Screen

Spoiler: Video

Recording did not impact performance at all. This is exactly how my game works, except it's maybe even worse than on this vid.

 

Here's the profile: skyfactory profile.nps - Speedy Share - upload your files here
Installing OptiFine and Fastcraft pretty much solves the issue but given the fact that these mods (especially OF) can cause more and more instability during my progress, I'd like to get the game to work well without those mods

 

Fastcraft itself doesn't seem to cause much. The framerate has gone up just a little bit but those annoying lag spikes are still present.

 

It was worth a try nonetheless Thank you Sir, I guess I'll play with OF+FC for now.[DOUBLEPOST=1466599932,1466599499][/DOUBLEPOST]No texture packs running. Basically no modifications, just downloaded the pack using the MyM launcher.