Respawn Lead Engineer Addresses Apex Legends Game Server Issues

Samy "@ricklesauceur" Duc, Lead Engineer at Respawn Entertainment, posted a long explanation regarding the inner-workings of Apex Legends and how Respawn tracks issues.

In a blog post published via the Apex Legends news tab on Thursday, April 29, Duc went over how the company is improving performance, explained the mechanisms behind common problems, and answered some frequently asked questions. Duc even goes into "comprehensive notes on our server tickrate"—making this a long and rather complicated post.

So, we broke it down for fans who want the information without the headache.

Respawn Lead Engineer Addresses Apex Legends Game Server Issues

Duc begins by explaining the implementation of the "performance display" as part of the player's UI in Season 6. Giving players this information is immensely helpful as it allows the technicians to have precise information about the problem being experienced.

Additionally, the team can pull the entire match's log and look at all the data together to determine the direct source of the problem and whether or not it was a player issue or server issue.

According to Duc, this has already paid off, writing, "We found out that one piece of network equipment in our datacenter was faulty, causing every game hosted on a handful of servers to have horrible network performances. The server themselves were fine, but the hardware that connected players to the servers in question was causing massive packet loss."

This isn't enough for Respawn, however. In the future, the server engineering team will be receiving real-time alerts to problems occurring at that moment so they can identify and solve the problem much quicker. Additionally, they will be adding a "unique server ID"—denoted as "SID"—to the player performance display so they can immediately find the problem child in the mix.

The most common problems reported by players have a lot to do with lag or latency. Players finding themselves "rolled back" to a previous spot" or seeing the images running in "slow motion" mean the server is having trouble keeping up. Issues like "I have a lot of packet loss / packet choke" and "My latency is going up and down" have more to do with the way the connection is handling the data rather than how the server is rendering it.

Finally, Duc lifts the veil of server tickrate, explaining, "The tickrate of a server is the number of simulations that the server runs per second." This number is fixed, and essentially all it's simply the number of renders and distributions to individual players it can achieve per second. The "ticks" include anything and everything about the entire map, every player, weapons, loot, and more.

This is the reason why players can't compare Apex Legends' servers to those of other games. Other games don't render all that information in every tick and Respawn's engineers have no plans to change that.

"We want the world state to be as accurate as possible, which is why our servers save the full world state on each tick. If we didn’t do this, it would probably save some of the CPU costs on our servers, but we would lose accuracy in our simulations, and that isn’t worth the risk." Duc wrote.

While upgrading to a 60Hz server instead of their 20Hz would increase the amount of ticks, it would also increase the amount of bandwidth needed to flawlessly perform each tick. This directly translates to the possibility of dropping data and skipping renders—which nobody wants.

Based on the company's findings, making the jump to a 60Hz server would only cut down on a potential frame delay by two at the cost of triple the bandwidth and CPU.

Duc closed the post by acknowledging player frustration over server performance issues, writing, "When you have to deal with lag, or no-regs, or slow-mo servers, itsucks. It takes you out of the game, and can feel very demotivating for you."

The engineering team only wants to make the game better for its current player base and future users. They're working with several entities on a "server-level" and "[Internet Service Provider]-level" to provide the best online experience players can get.

"Our hope is that if we start to communicate more with you about the issues that concern us, we’ll begin to share more of a common language to talk about the root causes of the issues we’re dealing with." Duc wrote, "We hope it’s the start of more conversations to come."