Sponsored Links

Email Platform Update - March 2007

bpullen

ISP Rep
EMail Platform Storage Update

Hi all,

Whilst things have been pretty stable for a while, it's no secret that the last year or so has seen more than its fair share of email problems. It's certainly been a sore point for many customers and despite plans last year to overhaul the platform, a number of circumstantial factors combined with some ill advised decisions saw things take an unfortunate turn for the worse.

The purpose of this post therefore is to provide some detail regarding the problems we encountered, what we did to combat them and what our long term intentions are for the future of the email platform.

Background

Before going into too much detail regarding our plans for the email platform it helps to have a basic understanding as to how the email platform is built. All our email is separated across storage volumes. Each volume contains a certain number of customers' mail directories and is mounted to an area of physical storage that contains all the email for these directories.

These volumes are used for the storage of all customer email. Whilst the mail delivery servers (mxcores) handle and are responsible for getting mail from A to B, they are no good without there being somewhere to store the mail whilst it is awaiting delivery, or once it has been delivered to a customer's mailbox and is awaiting collection.

The StorageTek

Early last year we were using a Network Appliance FAS940 for all of our mail storage requirements. This implementation worked well. We were using kit our engineers were very accustomed to and the platform was reliable, trustworthy and suited its purpose well.

netapp_old.JPG


Despite being satisfied with the implementation, scalability was a concern. Email was on the rise and the volume of spam in circulation had hit new heights and was still rising.

As we began to run out of disk space on the existing storage implementation, it became clear that action was required. The decision was therefore made to rebuild the mail platform in order to future-proof our email service and allow for future expansion in line with both mail volumes and customer numbers.

Last Summer we spent £170,000 purchasing 2 Sun 5310 head units, each with 3 trays of disks.

storagetek.JPG


With this kit in place we started looking at moving email from the NetApp and onto the new StorageTek.

Trouble at the Mill

Those who were customers at the time will undoubtedly remember the problems we had with the migration of mail onto the StorageTek last July which saw the unfortunate loss of some customers email. Sadly this was to mark the start of a number of reoccurring problems we encountered with the StorageTek over the following months.

In August we began the migration of customer email onto the StorageTek and by September had moved all Free-Online customer email and most Force9 email.

It was around this time that the reoccurring mail problems started manifesting themselves in the form of repeated mail delays and mail collection problems.

The reoccurring problems last year were often caused by the same things. Either we'd be hit with excessive volumes of mail (spam bombs) or one of the volumes on the StorageTek would become unwriteable. When the latter happened it caused immediate problems where the mail queues would start increasing. This caused email delays and had a knock on effect on the mail collection servers. This resulted in timeouts and errors when customers tried to pick up their email.

The reoccurring problems threw a spanner in the works and the mail migration was consequently halted. This left us in a position where email was spread across both of our storage volumes. This still allowed for problems and it became clear that further action was required.

Upgrading the NetApp

It was clear at this point that the new platform was not fit for purpose and that we had made a poor decision. After careful consultation and a series of informed discussions we decided that we needed to move away from the StorageTek kit.

Late January we made a positive step in the right direction by scaling the original NetApp FAS940 storage platform.

netapp_storage.png


This additional storage space was beneficial for a couple of reasons. Firstly it provided us with the space to move most of our customer email off the 'New' Sun StorageTek and back onto the tried and tested NetApp platform. Secondly it gave us breathing room to deal with the continuing increase in email volumes.

table1.PNG


Obviously this expansion was a good thing, but was only the start of the plans we had for the mail platform.

Migration Back to the NetApp

With the additional storage in place we were almost in a position to get all customer email back on the original NetApp.

The last time the problems with the StorageTek occurred was towards the end of February as detailed here. This affected the last of the email left on the platform and provided an opportune moment to move away from the problematic system once and for all!

New kit was delivered as a priority to ensure we could facilitate all customer email. Our engineers worked around the clock to install and configure the new equipment and before long we were transferring the last of the email onto the NetApp.

So where does that leave us..?

All email is now on the NetApp FAS940 with the StorageTek being used to quarantine virus emails (although it's possible that this will be moved away at some point too).

This means that customers will never see a repeat of the problems witnessed over recent months and during the latter part of last year. This is fantastic news but there's more yet!

The New Email Platform

Whilst the NetApp FAS940 storage system has been functioning with few problems, available storage space and scope for future expansion is still a concern so the decision was made to engineer a replacement.

I'm pleased to say that we've pulled out all the stops when writing the specification for the platform. We recently parted hands with almost a quarter of a million pounds which has bought us a shiny new NetApp FAS3070 system that landed on our doorsteps last Tuesday.

This equipment has since been fully racked and configured and we're now in an ideal place to start moving customers' data across to the new platform.

netapp_new.JPG


There is of course a risk involved when migrating mail from the old NetApp to the new. This is something we're only too aware of from experience so customers will be pleased to know that we will not be removing any email from the older platform until a month has passed following installation of the new system. This gives us something to fall back on should the unspeakable happen.

Due to the fact that all customers' data is being moved this work is likely to take a number of weeks. During this time each customer will experience short period of time during which they will be unable to access their email. The length of this service interruption is dependent on the size of each customers' mailboxes and will typically last no longer than 10 minutes. Some customers with very large mailboxes may find this service interruption to last slightly longer. Every effort will be made to keep this downtime to a minimum and we will announce details regarding when customers can expect their data to be moved where possible.

There's a Service Status I posted regarding the maintenance work here.

We'll be trying to provide frequent updates regarding the work over the coming weeks via Service Status and the forums. I sure people will see this as a big, big step in the right direction for the provisioning of the email platform. I for one am confident that we'll see a return to the reliable and stable days of old.

We've already overhauled our mxcore mail delivery platform with a £120,000 re-fit and if that's anything to go by then the future of email is looking bright ;)

Feel free to discuss the plans and ask any questions you may have. I'll be more than happy to try and source the answers.

Kind Regards,
 
Last edited:
Thank you for the update maybe now would be a good time to suggest those with mail stored in their mailbox to back this up on their home PCs just incase so they don't lose the improtant data.
 
Top
Cheap BIG ISPs for 100Mbps+
Community Fibre UK ISP Logo
150Mbps
Gift: None
Virgin Media UK ISP Logo
Virgin Media £24.00
132Mbps
Gift: None
Shell Energy UK ISP Logo
Shell Energy £26.99
109Mbps
Gift: None
Plusnet UK ISP Logo
Plusnet £27.99
145Mbps
Gift: None
Zen Internet UK ISP Logo
Zen Internet £28.00 - 35.00
100Mbps
Gift: None
Large Availability | View All
Cheapest ISPs for 100Mbps+
Gigaclear UK ISP Logo
Gigaclear £15.00
150Mbps
Gift: None
YouFibre UK ISP Logo
YouFibre £19.99
150Mbps
Gift: None
Community Fibre UK ISP Logo
150Mbps
Gift: None
BeFibre UK ISP Logo
BeFibre £21.00
150Mbps
Gift: £25 Love2Shop Card
Hey! Broadband UK ISP Logo
150Mbps
Gift: None
Large Availability | View All

Helpful ISP Guides and Tips

Sponsored Links
The Top 15 Category Tags
  1. FTTP (5473)
  2. BT (3505)
  3. Politics (2525)
  4. Openreach (2291)
  5. Business (2251)
  6. Building Digital UK (2234)
  7. FTTC (2041)
  8. Mobile Broadband (1961)
  9. Statistics (1780)
  10. 4G (1654)
  11. Virgin Media (1608)
  12. Ofcom Regulation (1451)
  13. Fibre Optic (1392)
  14. Wireless Internet (1386)
  15. FTTH (1381)
Sponsored

Copyright © 1999 to Present - ISPreview.co.uk - All Rights Reserved - Terms  ,  Privacy and Cookie Policy  ,  Links  ,  Website Rules