A VPN hides all traffic from the network. When connected, the network doesn't know if you're downloading something from port 80. Unless the VPN itself works over that port, then you should see faster speeds if the network was slowing down a specific port.
Also, since most traffic is encrypted these days, you're likely seeing reduced speeds on port 443 (https), not port 80 (http) since most services - including some of those mentioned here like Google Drive, Apple TV, iPlayer, etc - encrypt their traffic/use HTTPS. Even this site does.
Could this be a... DNS problem? Speed test services usually do their own detection to find the nearest server (and Fast[.]com probably uses the Netflix caching servers hosted by the network). If the DNS server is returning IPs from CDN servers far away, then speeds will be bad for many sites.
Give it a try with Google DNS (8.8.8.8/8.8.4.4) or Cloudflare (1.1.1.1/1.0.0.1). You probably want to reboot the devices to make sure they're using the updated DNS servers. Also, if using a browser, make sure it's not using DNS-over-HTTPS as it bypasses the DNS from the network/system.
P2P traffic wouldn't be affected by a DNS problem since a torrent client tries to download from all available peers and doesn't rely on DNS.