Server status

Due to a failure at our hosting provider, our server suffered downtime this morning affecting all sites. Sites were offside from approx 4am, and were all fully restored (without any data loss) by 10am.

Given the hours affected and the typical traffic profiles for most of our websites, luckily the impact was not as significant as it would have been later in the day.

The problem arose due to a server bootloader crash during a routine update. Our hosting provider has promised to investigate and improve processes to ensure this type of event does not reoccur.

Our hosting provider has informed us that a server hardware failure had taken place and all websites are offline.

There is currently no ETA for restoration, will will provide updates as we receive them from the engineering team.

We are very sorry for this unplanned downtime.

Our hosting provider is still unable to provide restoration ETA due to significant hardware fault.

We have initiated our disaster recovery processes for small number of high-profile sites so these can be restored this morning running on our backup server infrastructure.

All target sites for disaster recovery process are restored and running on our backup server infrastructure with new certificates (HTTPS) in place to ensure everyone can access these (required for HSTS use).

Backup database versions are few days old, in practice this has minimal impact on these sites.

Our hosting provider has begun restoring all accounts to a new server. Process is approx 5% complete.

Our account is restored and websites are coming back online, we will be checking for routing and certificate issues across all sites.

All sites and domains are now back online (except two) with new security certificates (HTTPS access) renewed. Two domains have routing issues being resolved.

The last two websites have been restored (at 10:30 and 11:48). All domains, websites and configurations have been manually checked to confirm everything is running as expected.

The backup databases in use have been taken from 01-May, so about 24 hours of data loss has occurred. The realistic impact of this should be minimal, please contact us if you have any concerns and we can help.

We're very sorry for the prolonged downtime over the past couple of days, which has been out of our hands. Our hosting provider has a good track record, and this is by far the longest unplanned outage we have ever suffered.

During this time, any additional domain services (e.g. email hosting) were unaffected and operated normally.

Our own disaster recovery process went well, restoring high-profile sites in about 2 and a half hours of being initiated. This is the first time we've had to test the process in a genuine downtime event, and we're pleased with the result. We will initiate this process earlier in the future if there is a problem when our hosting provider has no ETA for the resolution, and may also look to add further sites to the disaster recovery scope.

Our hosting provider is moving the physical servers we use to a newer infrastructure within the data center.

Down time should be minimised, but may take place in the 4 hour window from 9pm-1am.

We will monitor progress.

We're aware of a server performance issue affecting websites and are investigating the situation. An update will be provided shortly.

We're sorry to report that all sites were unavailable from mid-afternoon today (14 Jan) through until this evening, due to several distributed denial of service (DDoS) attack on various parts of our hosting provider's infrastructure.

To mitigate this continued attack, all sites were moved to a new IP address this evening. This migration is now complete and all sites are online and operating normally. A small number of users may still be awaiting DNS propagation and if this is the case they may not be able to load the site, or might see an error page/certificate error for a short period.

We're very sorry for the prolonged downtime today, which has been out of our hands. Our hosting provider has a good track record, and this is the longest unplanned outage we've had since Jan 2014.

During this time, any additional domain services (e.g. email hosting) were unaffected and operated normally.

A temporary loading issue on shared network infrastructure meant that sites were inaccessible for up to 40 minutes today.

Anyone trying to access websites between around 13:45 and 14:20 today will have not been able to load the page. Full access was restored after this time.

We apologise for the inconvenience this down time may have caused today.

A database fault between 17:00 and 17:30 today meant that some page visits would have received an error. These errors were occasional and intermittent for around 15 minutes, followed by 15 minutes of near complete outage.

In addition, our support service desk was affected by this outage.

The cause of today's problem was a database load balancing problem with our hosting provider. We were in contact with our provider during this period and are pleased that this was resolved quickly.

We apologise for the inconvenience and concern these errors may have caused today.

We will be migrating to a new server with upgraded, faster infrastructure on 08 December 2018. Downtime should be minimal for all sites.

The majority of sites have successfully moved to our new server with minimal downtime. It's possible a small number of users will have a delay seeing your websites due to DNS propagation times. If this is the case, it should be resolved within an hour or so.

Unfortunately, we have a critical issue with 3 websites where data has failed to transfer successfully. This is the responsibility of our hosting provider and we are liaising with them to ensure these are restored rapidly. A further update will be provided before 23:59.

The transfer problem root cause has been identified and a restoration of three affected sites has taking place. We're very sorry for the inconvenience this interruption in service has caused today.

Whilst all new Silhouette Websites customer sites have been deployed using HTTPS as default for some time, some of our older sites still worked via the insecure HTTP protocol. 

To learn why all websites should use HTTPS (even simple sites without e.g. shopping functionality), take a look at this article by Troy Hunt.

Chrome 68 was released yesterday, and started flagging all HTTP websites as "Not secure".  Ahead of this, we've successfully migrated all of our older websites to use HTTPS only. This means:

• All websites have HTTPS enabled and all images/styling loads over HTTPS.
• Users loading the HTTP version of your website will be automatically directed to the HTTPS version

There was no downtime for any sites during this upgrade.

We are aware of a problem where logged in users edit pages on some sites, that hyperlinks in existing page content are being removed. As a precautionary measure we have suspended user editing on affected sites and been in touch with site owners.

Public webpages and other logged-in user tasks are unaffected.

We are working on this issue as a matter of urgency and will send an update tomorrow.

The issue is now resolved, and pages can be edited again as normal without hyperlinks breaking. We are really sorry for the inconvenience caused.

Having looked into the issue to understand the root cause, we can see it was related to a recent upgrade of our server's PHP version. Although considerable testing took place before upgrading, this problem that affected one version of our page editor was missed. We will endeavour to test future PHP upgrades more comprehensively to ensure we do not suffer similar problems in the future.

Some users will have found that they could not login or edit site content for a few minutes at a time between around 13:00 and 13:10. Public webpages were unaffected but may have loaded a little more slowly than usual.

This problem was caused by a temporary disk space problem which restricted database access while this was resolved. We apologise if you suffered any inconvenience today.

We're sorry to report that many sites were unavailable from around 8pm last night (07 Jan) through until this morning, due to a distributed denial of service (DDoS) attack on the nameservers our sites use.

The sites and our server were in fact online and functioning fine throughout the incident, but traffic was unable to reach our server. This is because nameservers link website domain names (e.g. www.phpshadow.co.uk) to the numerical internet addresses (IP addresses) of the servers that hold websites. As the nameserver we were using was attacked this meant internet traffic wasn't given any signposts to our server, so couldn't find it.

During the attack we transferred all the websites whose domains we administer to an alternative nameserver which will have brought them back online instantly for some users, although a nameserver change like this can take several hours to filter through to everyone so for other users the sites would still have been inaccessible.

To prevent against nameserver failure like this, we have now started to use more robust nameservers for domains we administer that are capable of handling much more traffic, so making complete failures due to DDoS attacks much less likely. (Unfortunately we can only use this service for domains we administer, so if you are managing your own domain please still keep using the existing nameservers which will continue to function.)

As ever we must apologise for the downtime suffered today, and please do contact us if you have any further queries.

Thank you for your understanding.

We're writing to let you know that this weekend the server that hosts your website is being upgraded. The reason for the upgrade is to move to a newer more reliable server, as older servers can be prone to crashing and unplanned downtime, which would take your site offline.

The downside is that during the upgrade your website will be unavailable for between a few minutes and several hours. We'll work to ensure that your site is offline for the shortest time possible, but in some cases the changes made to moving to the new server's internet address will take up to 24 hours to filter through to some users.

The upgrade is being performed by our hosting provider and should take place on Sunday for your sites. We will be monitoring the process and let you know if there are any problems that mean the downtime is longer than expected.

Sorry for any problems this causes!

As ever, if you ever notice a problem with your website or have any queries about this weekend's downtime please contact us using the email addresses or phone numbers below, or send us a support ticket here: http://tickets.support.phpshadow.co.uk/new-ticket

Thank you for your understanding

Almost all sites are back online, having each suffered only a few minutes of complete downtime today, followed by about half an hour where the sites were accessible but without all images and graphics loading. A very small number of sites will remain inaccessible for several more hours due to slow DNS updates/propagation from their corresponding domain registrars.

As you may or may not be aware, we suffered a server outage this afternoon that lasted about 90 minutes, for which we must apologise. Unfortunately during this time visitors would not have been able to access your site.

The downtime started at 15:09. We were made aware of the issue at about 15:25 and immediately contacted our hosting provider. Full service for all websites was restored by 16:40.

The cause of the outage appears to be that our server momentarily overloaded which triggered a temporary suspension on all incoming traffic. The source of the overloading is still currently unclear but we, along with our hosting provider, are looking into the issue.

Our server has not suffered any downtime for many months: today's disruption was the longest downtime in several years, and also the first time we have a complete outage over all websites.

We generally have a good record with our server uptime and are very sorry for today's incident and any problems it may have caused. You can be assured that when we have a fuller picture of the events leading to this afternoon's problem, we will take steps to ensure that this does not happen again.

As ever, if you ever notice a problem with your website or have any queries about today's downtime please contact us using the email addresses or phone numbers below, or send us a support ticket here: http://tickets.support.phpshadow.co.uk/new-ticket

Thank you for your understanding

Just to let you know that that when your domain name was renewed recently, the domain registrar accidentally changed some of the domain information (the 'DNS records') which has lead to downtime for some users. We have been in contact with the registrar company this morning and the issue is now resolved, although some affected users may find that the corrected changes take up to 24 hours to propagate through.

The problem meant that about 60% of new visitors over the last 4-5 days will have seen a 'parked domain' of adverts from the domain provider company, although when the page was refreshed a few times the site would generally be found. The issue would have not affected people who had visited the site before unless they had recently cleared their computer's DNS cache.

We would like to apologise for downtime your website experienced today, for about 6 hours from 10.30am. During this time anyone visiting the website would have got an automatic error page from our server.

The error occurred when our server updated part of its configuration for your site (an action that we'd set it to do) at 10.39am. However this update managed to remove some text from a file crucial to the running of the system. We became aware of the problem at about 3:15pm, and all sites were running normally again by 4.42pm.

We've made steps to ensure that this kind of server update does not cause any disruption in the future, and that when any problems occur they are detected more rapidly.