Part of the mod pack. Have you ever tried growing crops on the moon? Or Mars? Or Venus? Or Any other planet other than Earth?
I played the modpack in singleplayer before I tried the server and things grew much faster in my sp world and I didn't use chunk loaders
That still wouldn't explain much I'm constantly always in it while I play my xbox but I think the tps may be the issue. It drops quite a bit throughout the day..
Hit f9 in came those poles represent chunks every other one is going 20% slower... makes a huge differece.