Tuesday, May 20, 2014

Efficiency of instanceof operator (revisited)

For the previous test, all conclusions had to be prefaced with a disclaimer because the test suite's configuration  made it difficult to accurately determine instanceof's efficiency relative to the alternatives (i.e. .getClass().equals()). The test suite's inability to generate average values for each test turned out to be the biggest stumbling block. This flaw in the test's design meant that the values gotten from each test varied, sometimes drastically, making it difficult to pin down an accurate relationship. With the new test suite it is possible to specify how many times each test should run. For example, to test every method fifty times at a hundred iterations, the following command can be used:

java PerformanceTest 50 100

Hence, the general command is:

java PerformanceTest [# of repetitions] [# of iterations]

Once the repetition value is supplied, the test suite repeats each test for the specified amount of time and then automatically generates the average values. Unsurprisingly, the average times for each test are more consistent than the individual values (which could have a 10-20 ms margin of error).

On my aging core i3 machine, testing each operation fifty times for one million iterations yielded the following times:

.getClass.getName().equals("A"): 48ms
.getClass().getName() == "A": 65ms
a.equals(Type.A): 48ms
a == Type.A: 58ms
a instanceof A: 58ms

Testing each operation 50 times for ten million iterations yielded the following times:

.getClass.getName().equals("A"): 441ms
.getClass().getName() == "A": 624ms
a.equals(Type.A): 451ms
a == Type.A: 536ms
a instanceof A: 539ms

(Note how increasing the # of iterations by ten increased the running times by ten as well)

With reliable data in hand, it's now reasonable to conclude that:
1. Trying to elucidate a class' identity using .getClass().getName() == will far and away take the longest amount of time.
2. Conversely, using .getClass.getName().equals("A") for the same purpose will take the least amount of time.
3. Using the equals method associated with the enum (i.e. a.equals(Type.A)), will result in a relatively short running time.
4. instanceof and a == Type.A are neither the slowest nor the fastest ways to determine class identity.

Representing the relative efficiency of all five options visually you get (from least efficient to most efficient):
.getClass().getName() == "A" > a instanceof A > a == Type.A > a.equals(Type.A) > .getClass.getName().equals("A")

Thus, the data favours a class hierarchy design in which every entity (i.e. operation, integer, function) is represented by its own class (Category A in the previous entry). From an efficiency standpoint, the current system is not the worst. However, letting a full fledged math library handle all of the ASTNode operations will result in a marginal increase in efficiency, which is the end-goal of this project.

2 comments:

  1. Here are the results I am getting on my desktop with 50 repetitions and 10 000 000 iterations:

    .getClass.getName().equals("A"): 281ms
    .getClass().getName() == "A": 281ms
    a.equals(Type.A): 280ms
    a == Type.A: 278ms
    a instanceof A: 278ms

    Using java version "1.7.0_51"
    Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
    Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)
    on Centos 6.5.

    I think the differences are too little, even with your results Victor, to influence too much
    the design decisions

    ReplyDelete
  2. Hello Nico,

    Thanks for taking a look at the code.

    I played with the test a few times after posting this, and I agree with you that the differences are not significant enough to influence design decisions. Instead of trying to tease out inefficiencies from the 'bottom-up' like I've been trying to do thus far, I have decided to start from the absolute top aka the Simulation Core Library. I was able to do the profiling just fine. However, I wasn't sure what JSBML's exact role was viz a viz the Simulation Core Library.

    This is something I am currently working on, and I will have an entry related to the subject shortly.

    ReplyDelete