Recent Realm Issues/48 Hour Extension

[Cairenn]

Tyren, 1/18/2005:

In our effort to try to continually improve everyone’s game experience, we recently updated some of our database hardware. This update required extended downtime for certain realms as we transferred some of our data from our launch servers to our new servers. The hardware update and data transfer was intended to split the load among multiple servers, which would in turn improve performance across the servers involved.

Even though our updated hardware represents some of the best technology currently available, configuration of this hardware proved problematic in some cases. We’re working through the challenges diligently, and we feel that resolving the issues that some players are experiencing is just a matter of time now.

We do appreciate your patience and enthusiasm about playing World of Warcraft, and we regret that you faced periods of high traffic this past weekend that caused some of you to have a poor game experience. So, due to the extended nature of the server issues, we will be providing a free 48-hour extension of play time. As with previous extensions, this will show up as free extra time on your current billing period (or on your trial period, for those of you who purchased the game recently). We hope you’ll bear with us as we continue to work hard on implementing changes for long-term stability.

We’ve received many calls and emails from players telling us how much they’re enjoying World of Warcraft and that they would just like to have a more consistent connection experience. In order to achieve this, we will continue our analysis, and we’ll continue to take further measures as needed, such as occasional login queues. However, we will also continue putting every effort into consistently delivering more information about related technical issues. Again, we appreciate your ongoing patience; our ultimate goal is to provide the high-quality gaming experience that you expect from Blizzard.

Blizzard Thread


Ordinn, 1/18/2005:
The overwhelming success of World of Warcraft has brought hundreds of thousands of people together to adventure in Azeroth, and concurrency numbers are well beyond what we expected or even hoped for. Unfortunately, this high concurrency, especially when concentrated on a small number of realms, initially caused issues with our hardware infrastructure. We were able to streamline our code to increase performance in the weeks following launch. However, the holiday season nearly doubled our player base, and it quickly became apparent that in order to handle not only the current player base, but all future players as well, we needed to make some upgrades to our infrastructure.

Last Thursday we made our first such upgrade. 20 of our 88 realms were moved off of the original hardware and placed on a new hardware configuration. These 20 servers initially performed very well, up until we reached our maximum concurrency Friday evening. The high population numbers uncovered an issue in the new backend shared infrastructure. This issue caused some players to experience severe lag and disconnects on a few of the realms, making them virtually unplayable.

In order to stabilize the affected realms and allow as many players as possible the ability to continue playing, we lowered the population caps by 30%. This stabilized the realms to the point where 70% of the players on the realms in question could play, but it also resulted in large queues.

The problems were attributed to high concurrency numbers on individual realms putting extreme stress on the backend infrastructure. We were able to address this problem by implementing additional hardware into the infrastructure this afternoon. This additional hardware has allowed us to stabilize the affected realms, and thus increase the server caps. We will continue to monitor the performance throughout the evening. If we notice any of the performance issues starting up again we will lower the population cap level enough to stabilize performance.

We sincerely apologize for the inconvenience this caused our players this weekend. This process coincides with our constant efforts to improve the current performance of World of Warcraft, and sometimes issues can arise when implementing these improvements. We will do our best to prevent similar situations from happening in the future, and we once again thank you for your patience and understanding.

Blizzard Thread


Caydiem, 1/17/2005:

During the server maintenance last Thursday, we performed a major upgrade to some of our database servers. At first the new servers performed flawlessly, but under heavy load at peak times, the new database servers are experiencing performance issues which have resulted in stability problems on certain realms, minor rollbacks and a problem with character retrieval upon login that some of you have reported.

Naturally we do not consider this an acceptable state of play for our customers. We are working around the clock to diagnose and remedy these problems. As a temporary solution, we have placed lower population restrictions on the affected realms; this results in queues when the population reaches a certain number. This is a short-term solution and, while undesirable, is in place to keep the realms more stable while we work on the main cause of the issues.

During the regular maintenance window that is scheduled for tomorrow, we will be making some additional changes that should allow us to increase the population restrictions on the affected realms. Our team is committed to resolving these issues as quickly and efficiently as they can. In the meantime, we appreciate your patience and understanding.

Blizzard Thread