Performance tuning Java with secret vm flags
I've just spent about an hour and half with Jared Jenson and Adam Leventhal performance tuning DWR with dtrace. Here's what I learnt:
There are a stack of options to java.exe that Sun don't document, but can help with tuning Java in a Tomcat environment. Particularly -XX:NewSize=200m which increases the size of the heap allocated to short lived objects. With tomcat you typically have a very large number of short lived objects so this can make a difference. I found docs written by BEA and IBM on the subject.
It is sometimes worth running tomcat with the client vm. (java -client) this is because tomcat ends up causing a large number of JIT compiles. This surprised me a bit because it wasn't as though we were testing code that was just being run for the first time where you'd expect JIT compiles, but the results were fairly clear - we got the number of calls that DWR could make per second up from 42 to 46. (This is a fairly worthless number because the client was on the same machine as the server). The point is the increase.
We might be able to wring some more performance out of DWR by doing less logging, but the % increase would not be great.
But here is the really good news - the performance overhead that DWR causes is close to zero compared to that caused by tomcat, and I expect the tomcat overhead is small compared to the network overhead. So there no point in tuning DWR.