Last Sunday, Fortnite reached some rarified air, as a recent blog post on the Epic Games site has revealed that the Battle Royale game hit a new peak of 3.4 million concurrent players, a number that’s ridiculous to even think about it, but even crazier when actually broken down, which is what the folks at Epic Games did.
A technical postmortem of last weekend’s service outage at 3.4M CCU
https://t.co/IgdGxbZauW pic.twitter.com/ff4noWIaqP— Fortnite (@FortniteGame) February 8, 2018
According to the in-depth breakdown, the extreme amount of players caused a ton of incidents, most of which were fixed pretty quickly, but some of which needed some more work. Of course, it’s a great trade off, as 3.4 million concurrent players is certainly something to celebrate. If you want to read the full breakdown on the technical mishaps associated with such a milestone, check out the blog post for all of that. In the meantime, Epic has outlined a ton of new steps and updates focused on improving game stability, some of which you can read below:
- Identify and resolve the root cause of our DB performance issues. We’ve flown Mongo experts on-site to analyze our DB and usage, as well as provide real-time support during heavy load on weekends.
- Optimize, reduce, and eliminate all unnecessary calls to the backend from the client or servers.Some examples are periodically verifying user entitlements when this is already happening implicitly with each game service call. Registering and unregistering individual players on a game play session when these calls can be done more efficiently in bulk, Deferring XMPP connections to avoid thrashing during login/logout scenarios. Social features recovering quickly from ELB or other connectivity issues. When 3.4 million clients are connected at the same time these inefficiencies add up quickly.
- Optimize how we store the matchmaking session data in our DB. Even without a root cause for the current write queue issue we can improve performance by changing how we store this ephemeral data. We’re prototyping in-memory database solutions that may be more suited to this use case, and looking at how we can restructure our current data in order to make it properly shardable.
- Improve our internal operation excellence focus in our production and development process. This includes building new tools to compare API call patterns between builds, setting up focused weekly reviews of performance, expanding our monitoring and alerting systems, and continually improving our post-mortem processes.
Fortnite is available now.
[Source: Epic Games]
