No IMEI would restrict the phone not the sim.
Congestion may not be happening up until that time but if a speed restriction across the network was in place to keep connections alive it might only be removed at those times.
Hopefully O2's network guys might be able to advise further but if O2 has a traffic management policy in place to maintain network availability the likelihood is they won't tell you what they don't need/have to.
When at the O2 tower nearby are you able to check your LTE Band again to see if its still on Band 20, also do this at work location, if its still showing Band 20 and not showing band 1 or 3 its likely the mast needs upgrading and is suffering congestion.
I had these very problems years ago with O2, specifically on 3G and now I'm remembering at certain times the speed would jump up and then during the day fall again, I left the area and city but then when I started missing important calls daily I left O2 and headed for EE who've been pretty solid.