Just for record. Java is absolutely unstable OutOfMemoryError. In multi threaded app you can't even do a System.exit(whatever). Because it tries to do various thing like run finalize and whatever it does it can hang up application.
If that's a server like app that can be restated by some process monitor one can use following jvm parameters:
-XX:OnOutOfMemoryError="kill -9 %p"on unix or
"-XX:OnOutOfMemoryError=taskkill /F /PID %p"on win32 to kill your process forcefully in case OOME.
One useful option is also -XX:+HeapDumpOnOutOfMemoryError which dumps whole heap into file before it croaks on OOME. You can analyze this dump with jhat tool that can be found in jdk1.6.0.
Related links:
