How one of the biggest games in the world was crippled by a shortage of servers
Final Fantasy XIV fans rejoiced last week when Square Enix opened a data center in the Oceania region intended to take some of the load off the company’s other data centers, meaning users can buy the game again.
In December, the title – which has 25 million registered players – experienced a resurgence in popularity with the launch of its latest expansion, dubbed Endwalker.
Anecdotally, the massively multiplayer role-playing game offered the kind of escapism that proved particularly alluring amid the latest wave of Omicron infections and accompanying lockdowns and social distancing measures.
Excitement for a new Final Fantasy entry isn’t usual, but Square Enix underestimated just how high the demand on its servers was going to be. The launch broke the game, overloading the company’s systems and resulting in hour-long queues for countless customers.
After a week, one of Japan’s biggest and most successful game developers took a highly controversial step: they stopped sales of the game to anyone who wasn’t already a subscriber, until they can solve the problems of its IT infrastructure.
“Players are currently experiencing extremely long wait times due to the high concentration of in-game hours that far exceed our server’s capacity, especially during peak hours, and therefore we have decided to temporarily suspend the sale and delivery of Final Fantasy XIV Starter Edition and Complete Edition,” said Naoki Yoshida, producer and director of the game.
Over the following weeks, the company admitted that it could not find enough servers at short notice to cope with the overload – having long preferred a proprietary, custom approach to its hardware, unlike some of its competitors who have opted for the public cloud.
Last week, Square finally resumed sales of the game, but its infrastructure expansion plans will take months – and it remains to be seen if the same number of users will ever return.
“The Oceania Data Center has been added and can now be selected from the Data Center menu via the title screen,” the company announced in the release notes for version 6.08released on January 25.
This isn’t a brand new facility prompted by the Endwalker debacle; the company announcement he arrived in the region in November, but previously said new servers would be available by February. He beat that two-week deadline.
Technical issue in Eorzea
Final Fantasy XIV’s data centers aren’t the kind we normally write about. Instead, they are logical data centers and are primarily intended to allow users to choose a “server”, which hosts a virtual “world” or neighborhood in which to play. The physical data center in Oceania, with the name logical data center Equipmenthouses five servers: Bismarck, Ravana, Sephirot, Sophiaand Zurvan.
Or exactly that Physical data centers are – you know, the kind we normally write about – Square Enix doesn’t say that. “While we realize that many of you are curious about the actual physical location of the new data center, please understand that for security reasons specific to the operation of an online game, we are unable to release this information to the public“, posted Yoshida in 2017, during the discussion on the American operation. “What we can tell you is that the servers will remain on North American soil, in an area with advanced internet infrastructure.”
Jthere consensus in online forums is that players in North America are served from Sacramento, Californiawith servers having been located in Montreal until 2017. Square-Enix confirmed in patch notes that the Oceania data center was indeed a new physical facility located in the region.
The physical location of data centers is important as ping times can be a critical factor in gaming.
In December, Square Enix revealed that it was having trouble sourcing the hardware it needed for the expansion due to the global shortage of computer chips.
“We need dozens of ‘server machines’ for each world we add,” Yoshida said. “However, due to the COVID-19 countermeasures currently in place, many factories around the world that produce semiconductors have stopped production or faced labor shortages. This eventually led to a decrease in the number of semiconductors being produced and resulted in a global shortage of semiconductors.
“We have made considerable investments, even more than usual, to secure the required hardware, but even so, a long lead time will be required to prepare the server hardware.”
How the New the data center will help
The new data center in Oceania will help with load balancing by moving users in Australia and New Zealand away from corporate facilities in Japan, North America and Europeas well as all other users who wish to discover the new territory.
To encourage this transfer process, Square Enix is not only waiving the usual home world transfer fee, but also rewarding players who have transferred with in-game bonuses. Players creating new characters on machines in Oceania will also receive bonuses.
Square Enix didn’t specify the duration of the incentives, but said it would depend on the balancing process. “These incentives will remain in place until the active player populations of the worlds in question have reached a specified number,” the company said in the patch notes. “Please note that once a world has been removed from the ‘Favorite World’ category, the various bonuses will no longer be applied.”
Free trial registrations are still suspended at this time.
Additional load balancing improvements
When Square Enix announced the availability of Oceania on January 14it also said it plans to expand its physical data centers later in the year, but noted that the global chip supply crisis is expected to continue to affect its operations.
“Currently, the North American Data Center houses three logical data centers and 24 worlds,” Yoshida said. “Due to the global shortage of semiconductors, it has taken a long time to procure server equipment, but the schedule calls for the first phase to be implemented around August 2022, with the second phase to follow in the spring or in the summer of 2023.
“In the first phase of the expansion, scheduled for August 2022, a new logical data center will be established and four new worlds will be added below. In the second phase of the expansion, we plan to add four additional new worlds under the newly established logical DC.
the European data center is also slated for expansion, Yoshida said. “Currently, the European data center consists of two data centers and 12 worlds,” he said. “In order to accommodate the substantial increase in the player population, we are planning a major server expansion in two phases, similar to that planned for the North American data center. Due to the current global shortage of semiconductors , delivering the required server equipment will take time, so the first phase of this expansion is scheduled for July 2022, while the second is scheduled for summer 2023.”
In July, the company also plans to reconfigure its Japanese data centers: “We’re looking to add an additional logical data center and reduce the volume of worlds per logical data center, dramatically increasing the maximum number of possible concurrent connections by several thousand per world,” Yoshida said. “We believe that, given the data center floor space where the servers are physically housed and the current shortage of semiconductors, this will be the fastest and most efficient method for large-scale expansion. .”