Syncing World State in a Multiplayer Co-Op Mod

For the Ratchet & Clank (2002) multiplayer mod project, I've always wanted to somehow fully sync the world state between the clients, but it's hard to do in a way that's stable and reliable. Adapting single-player games for multiplayer is always difficult because of core assumptions made during development that are hard to unravel when adding extra players. Enemies are hardcoded to look for Ratchet's current coordinates, mobys (Insomniac's term for "game objects") only update when they're close to the camera, triggers are hardcoded with only 1 player in mind, and more. There are some different ways to potentially deal with this to sync events, triggers and mobys between players.

I've played around with some options and partially implemented others already. One option I tried early on was to hand over control of a moby to the nearest player, which resulted in a very unstable and unreliable experience. Nearest player control would be convenient for things like enemy targeting, because it would naturally target the nearest player, but it's a double-edged sword because targeting the nearest player might not always be the best solution for all enemies. It also required syncing moby properties and variables with the server to distribute to clients and ensuring those stay properly synced when enabling and disabling the moby on the respective client. Switching the controlling player of a moby would either be stuttery to ensure a proper sync took place, or would cause crashes due to inconsistencies in moby state and its variables.

Another alternative could be to only selectively sync properties and variables on mobys between clients with a little glue logic on the server. This requires server-side code for every single type of moby class we want to sync, and is much more limited in functionality. There's no way to fully control how enemy targeting works, and it's unreasonable to expect to fully visually sync most moby types between clients using this method. But it's still useful functionality the multiplayer mod, and I've used it already to sync some world state. On Eudora, I've used this method to sync the bolt cranks and associated bridges.

Server-side emulation?

My latest attempt involves trying to emulate the mobys on the server by using a custom PowerPC emulator. The mobys' code are mostly limited to single update functions that run in a single pass every game tick. We don't need to emulate the entire PS3, and not even the entire game to emulate a moby's update function, we just need to load the specific things the mobys need like paths, level collision, moby presets, and similar. We can provide replacement functions for things that are hard to emulate, or entirely skip functions like those related to setting animations. We can control which player an enemy is targeting by writing the coordinates of the player we want to the emulated memory, and similarly control moby behavior.

As a simple proof-of-concept I started out trying to emulate one of the simplest mobys in the game; the big rotating robot on top of Al's Roboshack. It's a simple moby that basically only rotates its Z axis by a little every tick:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

void UpdateMoby_825(Moby *moby) {
  void *pvVar1;
  float fVar2;
  vec4 local_30;
  vec4 local_20;

  local_30.x = 0.0;
  local_30.y = 0.0;
  local_30.z = 0.0;
  local_30.w = 0.0;
  local_20.x = (moby->rotation).x;
  local_20.y = (moby->rotation).y;
  moby->draw_distance = 0xff;
  local_20.z = (moby->rotation).z;
  moby->update_distance = 0xff;
  local_20.w = (moby->rotation).w;
  pvVar1 = moby->pVars;
  fVar2 = FUN_00502578(local_20.z,seconds_per_frame * 0.47996554);
  (moby->rotation).z = fVar2;

  get_moby_orientation_info((void *)((int)pvVar1 + 0x20),&local_30,&local_20,&moby->rotation);
  return;
}

get_moby_orientation_info is a function we can skip in this case that are used to deal with setting velocity information for moving objects that I have more information about in this blog post.

This proved to work pretty well, but it's an extremely simple example so it's not particularly surprising nor impressive. So I moved on to a slightly more complex moby that has animations, state transitions and targets Ratchet.

In the video you can see the mine properly activate when one of the players gets too close, and it changes direction when another player becomes closer to it. This is a much better proof-of-concept and really shows that there's some promise to this method of world syncing. However, this is still a relatively simple example and doesn't yet touch any functionality that requires loading level data like level collision.

I've put in some work to load splines (paths) from the level files into emulator memory so that I can emulate mobys that rely on splines for their functionality. But trying to emulate any moby causes heavy glitching and sometimes crashes when trying to follow a spline, and I'm not yet sure if it's caused by inaccurate PowerPC instruction emulation or buggy spline loading code.

While it's clearly not working correctly, it is somehow following splines, just not the correct splines. But I'm excited about emulation as a method to achieve synced world and enemies between players in co-op.

Performance

I've been mildly worried about the performance aspect of emulating the game code on the server for each moby we want synced, but it's also worth noting that we skip all code related to animations and rendering. This was also code originally meant to run on a PlayStation 2, so even a slow, crappy emulator should be able to run the code we need with plenty of headroom. I'm currently running co-op servers on a low tier shared hosting server, and I'm more worried that this emulation might be too much for that, but I'll burn that bridge when I get to it.