You should look at the ILE RPG Programmer's Guide, SC09-2507. At V6R1 see Chapter 11. RPG and the eBusiness World; the there's a fairly comprehensive section on RPG and Java.

As I understand it, JVM won't do garbage collection on things created from RPG using JNI. freelocalref() frees up a single object (RPG type O). endObjGroup() will do the equivalent free on all objects back to the last strObjGroup(). See the manual for example.

To find if JVM is active, you could try to get the JNI environment pointer, but the problem with this is it will create the JVM if it's not already started. Maybe someone else knows a way that queries it without that side effect?