Invictus Launch Week & Alpha 3.9.X Postmortem
INVICTUS LAUNCH WEEK POSTMORTEM
Hello Citizens,
Last month, on June 2, we wrapped up our inaugural Invictus Launch Week event (also known as Fleet Week). Drake Interplanetary crashed the party (of course), and while this event has been around for decades in the ‘verse, in the real world, it was our first ever. By many metrics, it was an incredible success that massively exceeded all our expectations.
We set out to give you a glimpse of the kind of events we plan to run in the future. The Star Citizen universe is meant to be alive and dynamic, more so than any other MMO before it. And while we’re still in alpha and have a fair way to go, we’re already a live service game. As such, we wanted to both demonstrate a sliver of what a live event could be in our game and test out a number of firsts, including the introduction of our massive capital ships, event triggers, and much tighter integration between out-of-game and in-game events.
The combination of Invictus, a Free Fly, and the launch of Alpha 3.9 resulted in not just our best May on record, but our best month ever by every metric we use to measure success. Our funding is public and many of you have already seen that May was our best month ever for revenue. But even more gratifying for us was the unprecedented player engagement and growth we saw in new accounts created, new and old players flocking to the game, and record highs in peak concurrency.
Thanks to the allure of capital ships and the Free Fly, May 2020 broke our record for monthly active unique players by 50%. During Invictus, we achieved our highest totals in unique players, daily actives, and peak concurrency. On the first day of Invictus Launch Week on May 22, we had our highest Daily Active User (DAU) count ever. And every day after that for the entire week was higher still as we recorded more than a week of record DAUs. In fact, so many players logged in to Star Citizen during the week that the number of unique users from the 12-day event alone was greater than every single month up to that point in our entire history of operation.
As with many games and services that see record highs in traffic and concurrency, the crush of users put extreme stress on our services. At the event start, we had server crashes and degraded performance from the overload of users. While our backend teams worked around the clock and through the weekend to address issues and get the game back up and running, it was still disappointing for us to have such service disruption.
Going into Invictus Launch Week, we believed that we could handle the flood of new players. For one, May is typically a low traffic month for us, which was one of the reasons why we used it for our first Invictus event. Secondly, using four years of live service data going back to Alpha 2.0, we estimated a fairly robust surge in players greater than last year’s Alpha 3.5 Free Fly (May 2019) but still slightly lower than our November and December numbers (our busiest months due to holiday traffic and Intergalactic Aerospace Expo event).
But as you now know, Fleet Week smashed all expectations. The thrill of seeing (and hijacking or destroying) Lightnings, Idrises, and Javelins; an IAE-style promotion; and a Free Fly converged to create our best month of engagement to date. Invictus had more than double the total unique users, new users, and DAU of our Alpha 3.5 Free Fly event last May. And during the first weekend of Invictus Launch Week, peak concurrency and daily unique logins were 10-times greater than normal. Instead of being an event that rivaled November’s IAE or the December Alpha 3.8 patch release, Invictus Launch Week blew past both as May 2020 became our biggest month in history.
Based on what we knew going into the week, and some of the work we did on Alpha 3.8, we felt we could handle the near-term load during the event, but we underestimated the sheer number of people logging in to play. The reality is that we’ve known that our first-generation backend tech (that is rife with singletons) and inefficient data formats and usage patterns (on the client/server side) would have scalability challenges beyond a certain number of users. That’s why, for the last couple of years, we’ve had the Backend Team furiously working to remedy these issues, with the first fruits of those labors due to arrive in Q4 2020. The major mistake on our side was that we thought that our first-gen tech could hold out for another few quarters. In addition, we didn’t realize the impact that the first version of long-term persistence, activated in Alpha 3.8.1, would have on the size of the database fetches as we scaled up to unprecedented numbers of players. Since we don’t reset all player accounts with each new patch (unless we have to), and we don’t currently have limits on the number of items people can have, many players had thousands of items versus just a few dozen. The player load and the interim solutions for persistence all resulted in a perfect storm, causing server issues for a number of players.
Although the unfortunate downside of this was degraded performance and stability that we had to quickly correct, there was a real upside to the unprecedented traffic, as painful and disappointing as the issues were. Only an event of this scale, with the massive load from Invictus Launch Week, could have exposed these short-term weaknesses. We are developing systems today, such as iCache and physical inventory, that will be able to handle the increased player traffic; however, they aren’t deployed yet. But because of what we saw during Invictus, we are bringing some elements of our in-development scalable tech online earlier than we planned to. This will allow us to deal with the increased load that our growing engagement and long-term persistence have brought. Solutions to our most egregious backend bottlenecks should start rolling out later this year, but there will still be more work to do in the quarters beyond.
While it may be frustrating to experience these growing pains with us, this is exactly why the live playtesting the community does now is essential; we just don’t see the same loads and player behaviors internally, in Evocati, or even in the PTU. It’s only when the game is played in our live environment at scale that we get the true, real-world testing we need to improve our service. So while we know it’s frustrating to encounter these problems, exposure at this stage gives us the opportunity to make our systems more robust and scalable earlier than would normally be the case.
Our popularity during Invictus Launch Week and the game’s unprecedented growth over the past eight months has shown us that more and more gamers are coming to experience Star Citizen. And it’s turned scalability into an even bigger priority for us than it was before Invictus Launch Week. We want people to be able to test, play, and experience Star Citizen. After all, our vision is to ultimately have not hundreds of thousands but millions of players in the ‘verse. So while the technical issues during Invictus were indeed painful, we emerged from it with renewed effort and immediate actions to prepare us to scale for bigger and better things later in the year and beyond. And that means millions more will be able to enjoy a better Star Citizen experience because of the learnings from Invictus Launch Week.
-CIG
ALPHA 3.9.X POSTMORTEM
Alpha 3.9 introduced a good amount of new content, functionality, and improvements to existing features. Not only did we add the Klescher Rehabilitation Facility and prison escape gameplay, we also unveiled the New Babbage landing zone, microTech’s moons, the Unified Friends List for better initiation of group play, the new “Price of Freedom” Caterpillar mission, and more. For this first postmortem, we’re taking a broader and more high-level look at the most prominent features of Alpha 3.9.X; we aren’t as exhaustive or comprehensive here and may tweak the format while we find the right mix between giving meaningful insight and not getting bogged down in the details. Here, we’ve asked our directors to weigh in on what went well, what didn’t, and what we learned.
Prisons
The first quarter of 2020 saw a major enhancement to Star Citizen’s law system in the form of prisons, which enables criminal behavior to be punished. The initial iteration includes another tier of foundational work but is still pretty barebones versus where we eventually intend to go with it. For example, we knew from the outset that being locked up wasn’t going to be particularly fun; other than some FPS mining, there wasn’t yet enough variety to keep most players entertained for an extended duration. It was nevertheless important to get this feature out the door since it enhances other gameplay. To wit, there’s now a real risk – in the form of time, as opposed to just lost aUEC and a respawn – to serious crimes like murder and smuggling that didn’t previously exist, and that continues the trend of allowing players total freedom but pushing back in logical and appropriate ways. It was also important for us to start gathering information on how such mechanics affect player behavior; whether the risk of jail time dissuades certain types of behavior and to what degree.
For the second quarter, we found and fixed several important bugs, including most notoriously one where upon being released after having served your time, the screen would fade down and never fade up. Kiosk machines will now occasionally break and generate a repair mission, which should draw a fair amount of interest given the number of merits it awards versus what you can achieve mining. We’ll be adding a number of other prison missions in the future.
The third quarter will feature several small but notable enhancements, including the ability to surrender to law enforcement at any time during a battle, additional exploitation of the ship impounding mechanic, and working commissary kiosks that allow prisoners to buy things with the merits that they’ve earned. Locked doors, access cards, and codes will start playing an increasingly important role. The prison escape route will be tweaked so that gaining access to the tunnels requires first finding a security code, and we’ll complement that in a future release by making the extraction process considerably more difficult. A couple of locked rovers outside the prison will reward those players who had the foresight to pilfer the access codes with a fast getaway.
Departing from prison will eventually be more tightly integrated into the game so that, after serving your time, you’ll be ferried by ship to a landing zone to be released rather than relying, as we do now, upon a fade down and teleport. There’s also a lot of work to do on the AI side so that guards and inmates better contribute to making the area look and feel like a prison facility.
To ensure that the system’s initial shortcomings weren’t too pronounced, we temporarily limited the sentence durations and made sure to include a way to prematurely escape your confinement, albeit at the risk of a higher wanted level and – if recaptured – an even longer sentence. This is pretty simplistic at the moment (essentially a maze you have to navigate before you run out of oxygen) and another good example of where we intend to make some serious improvements. One of the major upgrades planned for next year is going to focus on injecting a lot of stealth, timing, and diversionary gameplay into the escape process so that it’s much more fun and challenging.
We’ll also be providing a lot of opportunities for melee combat within prisons next year – including the concept of gangs – and introducing some key mission-giving characters. These will include a shady guard and a powerful criminal boss with whom you can start to forge a relationship, which could prove very valuable on the outside, assuming you haven’t learned from your incarceration and are looking to continue a life of crime.
The ultimate objective of prisons within Star Citizen is actually a bit ironic in that, while they are intended to serve as punishment and something most players will try their best to avoid, we want to ensure that they’re so fun and interesting that players will view them as simply a different set of opportunities to exploit and challenges to solve.
-Tony Zurovec, Persistent Universe Game Director
Actor Status
The Actor Status System (originally called Player Status) has been a goal since very early in the project and is now living and breathing in Alpha 3.9, with the system and a majority of features in place. This is an incredibly involved system that spans the entirety of the game and is intended to link many aspects from characters, animation, environments, items, and more to each other for differing results whether it’s hunger, thirst, heat, cold, wind, vision, or movement. It replaces what was a very simple system of the player suffocating (and being killed) when exposed to space. The player character is notably now affected by the environment in a multitude of ways. And with this system, the environments not only vary dramatically visually, but now have meaningful gameplay elements within them to create specific identities. For example, a planet can be cold, which you can only survive in certain clothing for a given amount of time, but may have geysers to create pockets of warmth. The system itself has been completed; however, there is a robust design with intricately connected aspects still to come to Star Citizen, including weight, drug effects, and more exotic hazards that depend on other features coming along, such as the physical inventory.
This system was quite challenging to design, firstly due to the sheer interconnectedness between what might, on the surface, seem like disparate elements, and then further in approximating real-world behavior such as heat convection in what are extreme environments. Further, future-proofing the system for yet-to-be-added features added to this challenge. Lastly, garnering the support of so many different internal teams was a serious production undertaking, whether it was Graphics and VFX for the effects on the player vision, Audio for muffled hearing during hypothermia, Characters for the armor/clothing for temperature and weather protection (or not!), UI for feedback to the player, FPS Design for stamina and heartrate re-balancing, Animation for the effort set and raising the hand in heavy winds without a helmet, and Props for complex animation and interaction with food and drink items. This was all made even more difficult to develop as environments, planets, and ships needed to have rigidly managed and correctly balanced data, else it would cause issues with all of these systems. We couldn’t have simply left the kill trigger in for “in atmosphere”; Star Citizen demands deeper immersion than this. Now design, through the environment that the player exists in, can create complex effects and expect that players will need to provision themselves appropriately, such as not going out into the desert without water or not diving into an icy cave without an environmental suit. We are, in the end, quite happy with how this feature and system has gone in and look forward to hooking more systems into it.
Unified Friends Service
The unification of our friends service was something treated as the utmost priority for one of our internal PU teams, with the main intention being making it as easy as possible to group-up and join games together within Star Citizen. Though this was always possible, it’s now much easier to see your friends, see where they are playing, and join a party and launch into the game together.
This created much more work than anticipated, as it required a new service to unify the disparate elements and to ensure a real-time link between Spectrum contacts and the game. This also carries through into the game itself, which has updated notification of friend requests and automatic formation of in-game groups, which is a concept within the PU for chat, VOIP (Voice Over IP), and FOIP (Face Over IP). Linking up Spectrum to the front-end and then to the game seems obvious, but it suffered from different systems being completed at very different times, and it was long past time to clean it up. The front-end UI presents its own challenges as technically it’s a mix of our new Building Blocks technology and of the legacy Flash. As this can break the entire flow, our front-end team avoided completely changing out the front-end and opted instead to make new elements from Building Blocks but leave the rest as it was. This was quite restricting and had the effect of leaving us with a functional UI that isn’t quite as slick as you might expect. Though we are happy with the functionality the contacts/friends list gives us, as a very social game we will be continuing work in this area to ensure an interface and functionality set that you’d expect from a modern multiplayer game and to make it as easy as possible to play with your friends.
AC/SM Party and In-game Chat/VOIP/FOIP Support
Arena Commander and Star Marine have never had a fully-fledged chat system. For some time, there were obtuse commands (pressing various different key combinations known to only a few) that triggered a server command to “say” to all, but it wasn’t something that was used by most and it was inaccessible to new players. Though acceptable in the very early state of Star Citizen, as the first gameplay module, it hadn’t kept up with the groups system developed for the Persistent Universe. Now AC and SM share the same friends system and a similar squad system to that of the service mentioned earlier. You can now group together and launch into Arena Commander or Star Marine just as you can into the Persistent Universe, further enabling group and cooperative play. This was encouraged by the needs of the Star Marine game mode and Theaters of War (working title), which encourages in-vehicle groups (gunners and drivers speaking to each other) as well as joining and grouping together for matches. The main challenges are the varying number of players each game mode can support and the different server configurations that are running for session-based modes versus a persistent and perpetual game. Further difficulties are similar to those mentioned for the front-end and an immense amount of Flash having been put in place expecting and designed for a much different lobby concept. In the PU, although there is matchmaking, the requirements are not the same as in AC/SM. An unfortunate victim of the current update is the private match lobby, which could not be updated in time. We would have preferred to not disable it. Though you can still group together and launch matches, these currently cannot be locked off, but we will be addressing this in future updates.
AC/SM Round Cycling
In previous versions of Arena Commander and Star Marine, players would be kicked out to the front-end when matches would end. There were technical challenges and issues for having done this in the past, as when Arena Commander originally launched, this was not the case. Arena Commander and Star Marine would reload their maps on the server, which is totally different from the Persistent Universe, which doesn’t need to ever reload maps. This was incredibly important for Arena Commander and Star Marine, as play session length was cut short and players were arguably encouraged to leave the game when the round ended as they were kicked out and needed to go back through the various menus to return. We have managed to get around some of the technical hurdles and implemented round cycling so that players now get a proper and perpetual play experience. However, the fundamental issues haven’t been addressed, which are clean reloads whilst keeping players grouped with their friends and matchmade into a new round. This has significantly increased play session length and also concurrency of players in both modes. Another change around this was updating our join-in-progress rules, which over time had become unsuitable to most modes and would avoid matchmaking players into already-started rounds.
Personal Inner Thought Radial
The Personal Inner Thought system had technically been in game in a very rudimentary state behind a cvar; some releases had it exposed and others didn’t. It didn’t fully accomplish the intention of being able to map any key to any interaction on the fly, and even more importantly, to be able to perform actions in all various contexts. The programming and design teams did an incredible job on this feature as Star Citizen has hundreds of potential actions depending on the context you are in and default key mappings are becoming increasingly complex and far less accessible.
This feature even introduced some developers to actions they weren’t aware were possible or aren’t obviously shared on the default key mappings, from as small as toggling a light on a weapon attachment to as large as deploying cargo ramps and doors. It also exposes emotes and other player activities whilst on foot. This should not, however, be confused with the Player Interaction System, which will also be getting some treatment in the future that has to do with specific, and contextual, interactions with items. Again, this was a huge challenge, not only to update and replace the bare bones implementation of old but also to implement UI and real-time key mapping onto the hundreds of actions, but it was very much worth the effort.
-Sean Tracy, Technical Director, Content
-Richard Tyrer, Combat Pillar Director
New Babbage v2
This was easily one of the largest and most complex Landing Zones we’ve done to date. Compared to previous Landing Zones, we tried to give more freedom to the player to explore the location and improve the accessibility the player has to the planet through the introduction of surface entrances. Moving forwards, we are keen to keep improving this experience to be as seamless as possible.
New Babbage features an expansion of the Hi-Tech architectural style, which is the first time this has been showcased since we did the outpost designs. The team had a great time developing the architectural language and how it fit the narrative of the location and climate.
New Babbage also utilized our new lighting features, which means we can design lighting scenarios based on the time of day. During the development of The Promenade, we had a lot of fun with this feature as it enables us to create multiple moods and experiences within the same space. Right now we are implementing day and night light designs into some existing locations, so keep your eyes peeled.
Inside the domes of New Babbage we have various exotic and engineered botanicals, with display screens giving some backstory. This is the first step towards weaving the rich and complex lore of a location into the game environment for players to discover.
Low Orbital Stations Around Planets
The experience of being able to stand on a viewing deck in a spacestation and look out over a planetary sunrise is something we’ve wanted to put into the game for some time. Also, being able to see the same station from the ground is what the scale in Star Citizen is all about.
Klescher Automated Prison
As the perfect contrast to the warm interior climate, we introduced the first prison location into the game. It was the first release where we delivered two major ground locations at the same time.
We wanted to maximize on the cave development work we’ve already done, so we designed the central hub to feed off into a sprawling cave network. This gave the perfect space to try and work off your sentence.
The escape route featured a variety of advanced traversal along with a visual theme of feeling like you’re working through the various layers of the infrastructure. In the future, we would like to expand on this experience and introduce more complexity to challenge the escapee.
Moons of microTech
Calliope, Clio, and Euterpe were an exercise in seeing how flexible the new Planetary v4 tech would be for us. The idea was to see how quickly we could define a unique visual identity for these moons utilizing the existing biome asset packs we had already created. Also with the introduction of the player status, we were able to define some extremely cold and hostile climates to complement the barren landscapes.
-Ian Leyland, Star Citizen Art Director
Price of Freedom Mission
We got this built fairly quickly, since it leveraged a lot of what we learned in creating the 890 Jump mission (URGENT: Boarding Action in Progress). The goal was to provide an illicit variant of the 890 Jump mission that would be part of a Mission Giver experience. This means that it is partially bespoke, more in-depth in the design, and has an overall gameplay loop for the mission type.
When concepting this mission, we knew that we wanted to build off of the core navigational mixture of flying, EVA, and ground traversal. Then, we sprinkled in a unique combat scenario in a spaceship that the player has multiple different ways to solve.
Whenever we build a new mission, we always run into some challenges either on the development side or bugs in our systems. Here are just a few examples:
- The designer was dealing with learning the new UI Building Blocks Tech
- The initial collision pass on the Caterpillar wasn’t setup for FPS gunplay
- Originally the laptop was supposed to be a datapad that the player would have to scroll through to find the right contacts. However, because of time constraints, we had to use the laptop
- There were tech hurdles that the team faced with the Vis Areas in the Caterpillar
Overall we are extremely happy with the result of the mission and hope everyone gets a chance to play it.
-Todd Papy, Star Citizen Live Director
Moving forward, expect to read similar postmortems on Alpha 3.10 and future patches as we invite you to hear from our directors on the successes, challenges, and learnings in the aftermath of each update. Alongside our goal to revamp our roadmap to be more reflective of progress (stay tuned for more news on that soon), this is one of our initiatives to provide a more transparent look into our open development process.
Loading Additional Feedback