Effective Execution Statistics

As designer of a highly scalable system I'm always trying to get good execution statistics data. Even from the frequent short test runs I'm doing during daily development. For instance, how long time have the simulation threads spent executing each simulation tick? How much time is spent persisting/loading data? What is the client IO volume?

I'm not talking about actual code profiling. What I'm after is rather a summary of how the essential processes and tasks have performed after each run. Was something unexpectedly slow or fast? Did some data set grow or shrink?

The behavior and performance can change when something apparently unrelated is changed. Indirect caching and memory consumption changes are two potential culprits. But it's also easy to make mistakes with more direct dependencies, for instance being sloppy when adding some attribute to some game entity type which happens to cause twice the data amount to be sent to clients! An appropriate execution statistics summary after each run enables a simple sanity check to catch such things.

Execution stats are not about fixed time/data limits - in this system there are no "hard" such requirements. If a simulation tick takes 3 ms that's good - but if it takes 50 ms for 1000 simultaneous players that can be ok too. Often I'm not looking at the measurement values by themselves as much as how much they change depending on code I modify. How long does it take on average? Is that average less or greater after I attempted to cache something? (This is very hard to see from series of individual time samples/time stamps.)

And it's important to not only examine an average value, nor the maximum (or minimum) value. For any given metric, I find that all these values are important:

• count (number of executions/packets/whatnot)
• average / median
• standard deviation
• minimum and maximum.

The count of course tells how many times something has occurred. This info must be there to put the other measurements in context. If a DB load has happened only once but took 900 ms, it's probably slow because nothing had been cached and the JVM hadn't optimized the code yet.

The average and median indicate the general performance to expect over time. For different metrics the average or the median can be a more relevant measure. It's best to produce both! Combined with standard deviation these values provide a good indication of the "typical span" of a metric, for example that it usually takes 10-40 ms to execute.

The extreme cases must be examined as well however. If a simulation tick has a good average performance but occasionally takes 1000 ms (when, say 10 or 100 is expected) this has a serious effect on game play and must be investigated.

Collecting and Printing Statistics

For a long time I was manually adding statistics measurements to the code in the way lots of people do - calling System.currentTimeMillis() or nanoTime() at some execution points and logging the time spent. But this was cumbersome, unclean, ad-hoc and the results (lots of logged values) were difficult to interpret.

I finally ended up writing a relatively simple but effective and efficient statistics collector and printer. My objective was to emulate the logger in simplicity: Put the measurements to the collector in a single line just as you would put a string to the logger:

public class IOPacker {
  public static final Logger LOG = Logger.getLogger(IOPacker.class);
  public static final StatCompiler STAT = StatCompiler.getStatCompiler(IOPacker.class);
  ...
  public void pack() {

    long startTime = System.nanoTime();

    ...
    STAT.putSample("Pack time [ms]", (System.nanoTime()-startTime)/1e6);
  }
}

To output the compiled statistics, I execute this line before the program exits:
    StatCompiler.logAllStats(LOG, Level.INFO);

Efficient Representation

Just like for a logger, the implementation needs to be as fast as possible. This is not terribly hard. But then there is the concern for memory: in a long application run, the amount of held data shouldn't grow indefinitely, even though we want the information to encompass the full run!

The solution lies in managing the precision with which all the collected data samples are represented. The implementation is really a bucket map where each bucket represents a certain value interval. Within each decimal order of magnitude (e.g. between 1.0 and 9.9, between 10 and 99 and so on) up to 90 intervals (buckets) are stored. A bucket is simply a counter that records the number of samples that has hit its interval.

This means that the memory size grows with the order of magnitude between the smallest and the largest data point, but not with the number of data points. As long as there is a lower boundary on the smallest data point precision we care about, and since we only create buckets when a sample actually hits them, there rarely needs to be more than a few dozen buckets. We will still safely reflect any unusually small or large values (far outside the "dense bucket range" in the middle), and we still get good results with sufficient precision:

Pack time [ms]: sum: 3 326; avg: 11,9 (sd +- 48,6); count: 279; min<med<max: 5,5 < 8,2 < 823,2

Implementation

The code is in a public Gist, feel free to use it! (without any warranty of course ;-) )

https://gist.github.com/2854260

Computing Space Travel - Implementation

This is the Java code implementing the space travel solution I described in the previous post.

(The vector and point classes it uses are very similar to Java 3D's vecmath and it would be straight-forward to modify this code to use vecmath instead.)

Computing Space Travel

How can a 3D-autopilot be implemented in a space opera game? How do you compute the optimal trajectory from A to B in a solar system using classical mechanics?

This has been one of the tougher problems I've encountered in the MMO game project I'm working on. No other space game I've played has had a solution for this. The problem is usually avoided by either inventing other physics or hiding or skipping the actual travel. But I aim higher than that. I want it to feel like piloting a ship in space.

Since it's a space opera game, the simulation primarily involves classical mechanics, i.e. Newtonian physics. (Einsteinian relativity I consider non-practical from an implementation point-of-view. I don't say this only because it's harder physics... and the light-speed limit would definitely limit game design!) In the game, celestial orbits, space ship movement, thrust engine power, fuel consumption etc are all derived with Newtonian physics.

A lot of such behavior can be computed using high school level physics. But planning an optimal (or at least fairly efficient) space trajectory path to reach a given destination proved to be a considerably more difficult problem. It took much longer to solve it than I had expected. In this post I thought I'd describe what I came up with. It is an interesting problem and the solution can perhaps be useful to others.

Defining the Problem

The objective is to plan the space trajectory needed to rendezvous with a given destination within the solar system. It is needed to implement the game's autopilot which is used by both computer-controlled ships and players. The trajectory must be computed in advance since simply going in the direction of the destination would result in arriving with a speed very different from that of the destination object. We need to know beforehand when to 'gas' and when to 'break', and in what precise directions to do this.

The planned trajectory should be fairly efficient (primarily regarding travel time, but also regarding fuel consumption) and preferably not cheat by using different physics than the player experiences with the manual controls. It must also be fairly fast to compute - it's an MMO game and there can be many thousands of travelling ships.

The formal problem is: The travelling spaceship, at current position P in space and with current velocity V, is to travel and rendezvous with a destination object (a planet, another ship etc) which is at current position Q and with current velocity W. Since it's a rendezvous (e.g. docking or landing) the travelling spaceship must have the same velocity W as the destination upon arrival.

The above diagram is in absolute frame coordinates (sometimes called the inertial frame). However since we're adhering to Newtonian physics we can simplify the problem by defining it relative to the travelling ship's starting position and velocity instead. The next diagram shows the same case but in relative frame coordinates.

                                           P₀ = 0                Q₀ = Q - P                ΔV = W - V

A game space ship travels using its thrust engine, which accelerates the vessel in whatever direction it is pointing. The planned trajectory should be expressed as a sequence of thrust applications, each describing a trajectory leg:

1. From time 0 to t1 apply thrust X
2. From time t1 to t2 apply thrust Y
3. ...

The technical problem is: Find the trajectory legs (each expressed as a thrust vector and a length of time) that as soon as possible will move the ship to the same position and velocity as the destination, so that:

P(t) = Q(t);     velocity = ΔV;      t as small as possible