Sponsored Links

High speed (multi GB) file storage

jon1

ULTIMATE Member
Was just wondering if anyone has any recommendations for high-speed UK (preferably London) based high-speed (ideally 10Gbps bandwidth) file storage servers

Have multiple team members handling 500GB-3TB files and we need to find a proper solution, fastest internet a team member has is 5Gbps, everyone else is either 1Gbps aside from the one very unlucky person with FTTC

Looking at some OVHCloud options but they take 1week+ to deliver, any recommendations?
 
Have multiple team members handling 500GB-3TB files and we need to find a proper solution, fastest internet a team member has is 5Gbps, everyone else is either 1Gbps aside from the one very unlucky person with FTTC
How much total storage would you need?
 
Sponsored Links
500GB - 3TB files and total of 10TB storage is somewhere between three to twenty files for the team.

If the person with 5Gbps connection has the space, wouldn't it be easier to put a fast NAS box there and share it with the other few members of the team?

It reads like there's a handful of users, rather than 100's.
 
500GB - 3TB files and total of 10TB storage is somewhere between three to twenty files for the team.

If the person with 5Gbps connection has the space, wouldn't it be easier to put a fast NAS box there and share it with the other few members of the team?

It reads like there's a handful of users, rather than 100's.
That is a good point. a good symmetric line (even a corporate account from an Altnet) might be perfect.

NVMe TrueNAS with a wireguard setup is easy. I run something similar on a 3gb symmetric.
 
Any particular reason to do it in the "cloud"? I just built a 14TB SSD NAS with 10GE at home. Although of course there are always considerations when it comes to data storage, backups, etc. The cloud is not failsafe. Sometimes datacenters literally burn to the ground.
 
Sponsored Links
Thanks for the answers guys, were looking at a NAS now

One last question, I seem to have incredibly slow upload speeds to a Texas, USA based production server with the following speeds:
7455cbd7-fa97-471f-b320-d774a00e01f7.png


I have a 1Gbps symmetric line, but my upload speed peaks at 150mbps
A team member in London with Community Fibre gets around 4x what I do to the server

Is this just simply because it's so far and theres nothing I can do to improve it?

Edit: I noticed that if I upload 2 files at the same time, they both reach 4.9 MiB/s and no further, 4.9 MiB/s is the max for a single file on its own

Edit 2: The issue goes away on a VPN
 
Thanks for the answers guys, were looking at a NAS now

One last question, I seem to have incredibly slow upload speeds to a Texas, USA based production server with the following speeds:
7455cbd7-fa97-471f-b320-d774a00e01f7.png


I have a 1Gbps symmetric line, but my upload speed peaks at 150mbps
A team member in London with Community Fibre gets around 4x what I do to the server

Is this just simply because it's so far and theres nothing I can do to improve it?

Edit: I noticed that if I upload 2 files at the same time, they both reach 4.9 MiB/s and no further, 4.9 MiB/s is the max for a single file on its own

Edit 2: The issue goes away on a VPN
My guess would be it's simply down to the BGP routing policy your provider is employing. The direct route might be going via a bandwidth constrained or throttled route or policy, and when connected through a VPN tunnel it's taking a less constrained route as all the routers effectively see is your target host address.

Would have to take a BGP view on the two routes to discover a bit more and ultimately it's a case of talking to your ISP and asking for a better route or a soultion.

Distance and speed are not strictly related. It's more distance and latency. Throughput over submarine fiber is incredibly fast, a single wavelength these days is in the order to 100 gigabits, and they can carry hundreds of wavelengths.

This seems most likely down to a Layer 3 issue, as previously described. i.e. the actual routing of the IP traffic.

Just my blabbering and guesswork!

Edit bit:
I'd also say don't be afraid to talk to your provider about it. If they are good they will be receptive to the issue. As an example, I asked my provider if they could peer with Valve Software at the IX (LINX) and they organised it within 24 hours.
 
Edit: I noticed that if I upload 2 files at the same time, they both reach 4.9 MiB/s and no further, 4.9 MiB/s is the max for a single file on its own

TCP adjusts its speed in response to packet loss, which is part of its congestion control algorithm - but packet loss which is *not* due to congestion affects it in the same way. The maximum speed an individual stream can reach is given by the formula in the link above, and this effect gets larger with larger round-trip time (hence why you see this to USA).

But if you run multiple TCP streams to the same destination, each TCP stream gets the same limit separately. This is why multi-threaded speedtests can give higher performance than single-threaded ones.

If you can use a file transfer protocol which opens multiple connections (even Bittorrent or similar) you should be able to improve matters. Ideally you'd want to know where this source of low-level packet loss is, but it could be anywhere, including at the remote site itself.
 
Sponsored Links
Agree, been there, got that t-shirt in the past. I remember having to play around with a 1gig circuit between London and up north, RTT of around 10ms, and this silly "high performance" single threaded application would top out at around 200Mbits with the "network" being blamed by all stakeholders. With some TCP Window size adjustments got the performance closer to 800Mbits if my memory is not failing me.

RTT kills performance for any one TCP stream.
 
Thanks for the answers guys, were looking at a NAS now

One last question, I seem to have incredibly slow upload speeds to a Texas, USA based production server with the following speeds:
7455cbd7-fa97-471f-b320-d774a00e01f7.png


I have a 1Gbps symmetric line, but my upload speed peaks at 150mbps
A team member in London with Community Fibre gets around 4x what I do to the server

Is this just simply because it's so far and theres nothing I can do to improve it?

Edit: I noticed that if I upload 2 files at the same time, they both reach 4.9 MiB/s and no further, 4.9 MiB/s is the max for a single file on its own

Edit 2: The issue goes away on a VPN
build your own, some sort of 9th gen intel or newer, would easily hold out better than all the purpose build "NAS" noide 304 is the perfect case too.
 
Top
Cheapest Superfast ISPs
  • Hyperoptic £17.99
    Speed 33Mbps, Unlimited
    Gift: None
  • UtilityWarehouse £19.99
    Speed 35Mbps, Unlimited (FUP)
    Gift: None
  • NOW £23.00
    Speed 63Mbps, Unlimited
    Gift: None
  • Vodafone £24.00
    Speed 73 - 82Mbps, Unlimited
    Gift: None
  • Shell Energy £24.99
    Speed 38Mbps, Unlimited
    Gift: None
Large Availability | View All
Cheapest Ultrafast ISPs
  • Gigaclear £15.00
    Speed: 150Mbps, Unlimited
    Gift: None
  • Zzoomm £19.95
    Speed: 150Mbps, Unlimited
    Gift: None
  • YouFibre £19.99
    Speed: 150Mbps, Unlimited
    Gift: None
  • Community Fibre £21.00
    Speed: 150Mbps, Unlimited
    Gift: None
  • BeFibre £21.00
    Speed: 150Mbps, Unlimited
    Gift: £25 Love2Shop Card
Large Availability | View All
Helpful ISP Guides and Tips
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
Sponsored Links
The Top 20 Category Tags
  1. FTTP (5411)
  2. BT (3492)
  3. Politics (2500)
  4. Openreach (2279)
  5. Business (2227)
  6. Building Digital UK (2218)
  7. FTTC (2039)
  8. Mobile Broadband (1945)
  9. Statistics (1767)
  10. 4G (1642)
  11. Virgin Media (1596)
  12. Ofcom Regulation (1440)
  13. Wireless Internet (1380)
  14. Fibre Optic (1380)
  15. FTTH (1379)
  16. 5G (1219)
  17. Vodafone (1128)
  18. EE (1113)
  19. TalkTalk (927)
  20. O2 (919)
Sponsored

Copyright © 1999 to Present - ISPreview.co.uk - All Rights Reserved - Terms  ,  Privacy and Cookie Policy  ,  Links  ,  Website Rules