As many of our players experienced, we launched Alpha 3.18 on March 10 of this year, and alongside a bevy of high-profile new features like Salvage, the Vulture patch flyable ship, the surprise Scorpius Antares, new rivers, and new missions, the biggest headliner of the patch was our delivery of Persistent Entity Streaming (PES), which completely rewrote how we save state in the game and ushered in the beginning of the promise of a truly persistent universe where a player’s actions could remain in the game for others to interact with and create a lived-in environment where you could literally leave your mark in the PU. And, just as importantly, PES, was the necessary final stepping stone to delivering Static Server Meshing.
Many who follow Star Citizen appreciated how consequential and important our delivery of Alpha 3.18 was for us and the game itself. Many teams at CIG spent most of the second half of last year finishing out Persistent Entity Streaming, as we deployed it to our Public Test Universe servers for testing in December 2022, and then spent the next 3 months attempting to harden and improve it for launching to our Live Star Citizen service as quickly as possible.
However, when the moment of truth came and Alpha 3.18 shipped to Live, the shock to the system was beyond what we had projected. While it’s true that we expected PES to create a rough experience initially as we ironed out the issues that could only expose themselves at massive scale (and we warned as much), to say that we were surprised by the depth of chaos from the PES launch would be an understatement.
The anticipation for PES and 3.18 was nothing short of unprecedented for us. When the patch first dropped on March 10, we experienced our highest peaks ever for logins per minute and per hour, and had our highest number of attempted logins in a day for the first few days of launch. We say “attempted logins” because as you all know, the service was so overwhelmed by the traffic and teething pains of PES that many players could not get into the game, as various issues stalled users throughout the login funnel. Some were stuck in queues, some couldn’t get their characters to load, some were stuck at an infinite loading screen. As you can read more about in Benoit Beausojour’s (our Chief Technical Officer) account below on PES, we underestimated the multiplicative forces of going to Live and now creating and persisting every entity players created through their actions, creating a load on our service that was beyond our initial forecasts. And it took weeks if not months to expose, diagnose, create fixes, test them, and deploy them to restore the service, all while the game was still running on Live.
We learned a lot from the launch of PES, and while we are still recovering, and regret the compromised service in the first days and weeks of the rollout, it has definitely taught us a valuable lesson to value and preserve the integrity of the service more than we had in the past. That’s why as we begin to roll out the Replication Layer split and crash recovery – two things now enabled by PES – we will do so gradually, and as we begin to deliver Server Meshing, we will create dedicated testing channels to harden those new technologies further and implement standards and thresholds before we “graduate” them to PTU and then Live. You’ll start to see the ramifications of that later in the year and hear more from us later about our new approach to deploying potentially disruptive and game-changing new tech to the game service, but it comes down to us truly committing to preserving the experience for the hundreds of thousands, if not millions, who now play Star Citizen as a live service game, albeit an alpha still in development.