Detecting Java Threads in Deadlock with Groovy and JMX

Unfortunately, Java applications taking advantage of multiple threads can at times run into the dreaded deadlock condition. Fortunately, the Java Platform makes deadlock detection relatively easy. In fact, the built-in (since J2SE 5) ThreadMXBean (a PlatformManagedObject exposed via JMX) makes this information available to any client that "speaks JMX" via the findDeadlockedThreads() and findMonitorDeadlockThreads() methods. General "JMX clients" such as JConsole and VisualVM use this to provide information on detected deadlocks, but custom tools and scripts can be written to provide the same details. In this post, I look at using Groovy in conjunction with the Attach API to detect deadlocked threads for a locally running JVM process.

The Java Tutorials provides a simple and fairly interesting example of Java code that will typically result in deadlock on the "Deadlock" page in the "Concurrency" lesson of the "Essential Classes" trail. I provided that example here in only the slightest adapted form as the code I will run JMX clients against to detect deadlock. (As a side note, I keep seeing posts on reddit/Java and other online forums asking for the best free online introductory resources for learning Java; I cannot think of a better answer to this than the Java Tutorials.)

Deadlock.java (Adapted from Java Tutorials)

package dustin.examples.jmx.threading;

import static java.lang.System.out;

/**
 * Example of a class that often will lead to deadlock adapted from the 
 * Java Tutorials "Concurrency" section on "Deadlock":
 * http://docs.oracle.com/javase/tutorial/essential/concurrency/deadlock.html.
 */
public class Deadlock
{
   static class Friend
   {
      /** Friend's name. */
      private final String name;

      /**
       * Parameterized constructor accepting name for Friend instance.
       * @param newName Name of new Friend instance.
       */
      public Friend(final String newName)
      {
         this.name = newName;
      }
      
      /**
       * Provide this instance's name.
       * 
       * @return Friend's name.
       */
      public String getName()
      {
         return this.name;
      }

      /**
       * Bow from friend. Synchronized for thread-safe access.
       * 
       * @param bower Friend that is bowing.
       */
      public synchronized void bow(final Friend bower)
      {
         out.format("%s: %s has bowed to me!%n", 
            this.name, bower.getName());
         bower.bowBack(this);
      }

      /**
       * Bow back to friend who bowed to me. Synchronized for thread-safe access.
       * 
       * @param bower Friend who has bowed back to me.
       */
      public synchronized void bowBack(final Friend bower)
      {
         out.format("%s: %s  has bowed back to me!%n",
            this.name, bower.getName());
      }
   }

   /**
    * Simple executable function that demonstrates deadlock when two friends
    * are waiting on each other to bow to finish bowing.
    * 
    * @param arguments Command-line arguments: none expected.
    */
   public static void main(final String[] arguments)
   {
      final Friend alphonse = new Friend("Alphonse");
      final Friend gaston = new Friend("Gaston");
      new Thread(new Runnable()
      {
         public void run() { alphonse.bow(gaston); }
      }, "Gaston Bowing").start();
      new Thread(new Runnable()
      {
         public void run() { gaston.bow(alphonse); }
      }, "Alphonse Bowing").start();
   }
}

The next screen snapshot shows that this simple application becomes deadlocked between the two threads.

Even if we had not known this application was intended to demonstrate deadlock, the fact that it never ends or changes state is a good hint that it has run into deadlock. We can use JMX to determine that for certain. Whether we use a JMX client, jstack, or some other mechanism for determining deadlock, we'll likely need the Java process's pid (process ID), so the next screen snapshot indicates doing that (pid is 3792 in this case) with both jps and jcmd (the latter is only available since JDK 7).

Knowing that the pid is 3792, I can run "jconsole 3794" at the command line to have JConsole come up loaded with information about that deadlocked Java process. In this case, I'm interested in the threads and the next image shows the "Threads" tab in JConsole for this deadlock demonstration application.

The immediately previous screen snapshot has the two threads of interest circled. In my adapted code example, I named these "Gaston Bowing" and "Alphonse Bowing" to make them easier to find in JConsole and other tools. Had I not provided these explicit names in the example (which the original example in the Java Tutorials did not do), they would have been referenced by the more generic names "Thread-0" and "Thread-1." With JConsole loaded and the "Threads" tab available, I can click on the "Detect Deadlocks" button to see the information in the next two screen snapshots (one each for the two threads being selected).

By looking at the JConsole output for each of the identified deadlocked threads, we can see that each thread is BLOCKED and waiting on the other thread. The stack trace provided with each thread also provides pretty clear documentation of where this deadlock occurs in the code.

Java VisualVM can be started with the command jvisualvm and then the Java process in question can be selected as demonstrated in the next screen snapshot. Note that VisualVM automatically detects the deadlocks on the "Threads" tab of VisualVM and provides a message in red font indicating that.

When I click the "Thread Dump" button as indicated to see which threads are deadlocked, a thread dump is provided. Toward the bottom of that thread dump, as indicated in the next screen snapshot, are the details surrounding the deadlocked threads.

As JConsole did, VisualVM also indicates in the provided thread dump that the two bowing threads are waiting upon one another and "Java stack information" is provided for each of the threads identified as participating in the deadly embrace.

JConsole and Java VisualVM make it easy to identify the threads involved in deadlock. However, when a command line tool is preferred over a GUI-based tool, jstack is an obvious choice. Again using the pid identified earlier for this particular process (3792 in my case), one can simply type jstack 3792 to see a thread dump for that application. The next two screen snapshots show a piece of this with the first snapshot showing the command being run and the beginning of the output and the second snapshot showing the portion related to deadlock.

One doesn't have to look too closely to realize that jstack's output is the same as that provided in the "thread dump" in VisualVM.

JConsole and VisualVM provide a nice GUI approach for identifying deadlock and jstack does the same from the command-line. Given these tools, one might wonder why we'd care about being able to build our own tools and scripts to do the same. Often, these tools are enough, but there are times I may want to do specific things with the information once I have it or I want the answer to be even more direct than these tools provide. If I need to programmatically do something with the information or simply want the identification of threads involved in deadlock without the other threads' details, a custom tool might be in order. The remainder of this post focuses on that.

The following code listing contains Groovy code for printing threading information related to deadlocked threads as provided by the ThreadMXBean. This code makes use of a piece of Groovy code not shown here (JmxServer using Attach API) that can be found in my last blog post.

displayDetectedDeadlock.groovy

#!/usr/bin/env groovy
def pid = args[0]
def javaThreads = new javax.management.ObjectName("java.lang:type=Threading")
def server = JmxServer.retrieveServerConnection(pid)
long[] deadlockedThreadIds = server.invoke(javaThreads, "findDeadlockedThreads", null, null)
deadlockedThreadIds.each
{ threadId ->
   Object[] parameters = [threadId, 10]
   String[] signature = [Long.TYPE, Integer.TYPE]
   def threadInfo = server.invoke(javaThreads, "getThreadInfo", parameters, signature)
   print "\nThread '${threadInfo.threadName}' [${threadInfo.threadId}] is "
   print "${threadInfo.threadState} on thread '${threadInfo.lockOwnerName}' ["
   println "${threadInfo.lockOwnerId}]:\n"
   println "Java stack information for the threads listed above:"
   println "==================================================="
   println "'${threadInfo.threadName}':"
   threadInfo.stackTrace.each
   { compositeData ->
      print "\tat ${compositeData.className}.${compositeData.methodName}("
      println "${compositeData.fileName}:${compositeData.lineNumber})"
   }
   println "\n\n"
}

The Groovy script above specifies 10 as the maximum depth for the stack trace information to be provided as it provides details similar to those provided in JConsole, VisualVM, and jstack. Of course, the advantage of this approach is that any of this data that is printed to standard output here can also be used in runtime decisions and acted upon programatically from Java or Groovy code. The next screen snapshot shows the output from running this script against the deadlocked application used previously in this post.

Java makes it easy to detect threads locked in the deadly embrace known as deadlock. Although general tools like JConsole, VisualVM, and jstack are often sufficient to identify these cases, it is nice to be able to write custom scripts and tools for doing the same thing. This allows developers the flexibility to include deadlock detecting directly in their scripts and tools rather than needing to parse jstack output or resort to other data scraping approaches.

Monitoring Key JVM Characteristics with Groovy, JMX, and RuntimeMXBean

Since J2SE 5, Platform MBeans have been available that allow some key characteristics regarding the JVM to be monitored and (even managed in some cases) via JMX. In addition, many JVM-based applications add their own JMX-enabled features for monitoring and management. In the blog post Groovy, JMX, and the Attach API, I looked at how to display many of the platform-provided MBeans using Groovy, JMX, and the Attach API. In this post, I look at more specific scripts that access a narrowly focused subset of these platform-exposed values available in the RuntimeMXBean.

My previous post demonstrated ability to view a wide spectrum of JVM details with Groovy and JMX. In most cases, I don't need all of these details all at once, but simply need one or two of the items. Although I could use a tool such as JConsole, VisualVM, or even a modern Java IDE to see these details, I sometimes wish to run a simple script to provide the exact results I'm looking for rather than using a general JMX client that I need to start up and navigate to that information. This is where simple scripts such as shown in this post are particularly handy.

All of my examples in this post assume the person running the Java process to be monitored/managed is the same person (same username) running the scripts and that they are being used to monitor Java processes running on the same local machine. This assumption allows the Attach API to be used. If a different user was running the script than ran the Java processes or if the scripts were to be run on remote Java processes, techniques for remote JMX would be used instead. Because the Attach API will be used in these examples, I have placed the code that each example uses in a Groovy script file that each example invokes. That file to be used by all of my scripts is called JmxServer.groovy and is shown next.

JmxServer.groovy

/*
 * JmxServer.groovy
 *
 * Functions meant to be called by other scripts or tools that need to use
 * the Attach API to access an MBeanServerConnection to use to manage and
 * monitor a particular JVM (and possibly the application hosted in that JVM)
 * identified by the provided Process ID (pid)
 */

import javax.management.MBeanServerConnection
import javax.management.ObjectName
import javax.management.remote.JMXConnector
import javax.management.remote.JMXConnectorFactory
import javax.management.remote.JMXServiceURL

import com.sun.tools.attach.VirtualMachine

/**
 * Provide an MBeanServerConnection based on the provided process ID (pid).
 *
 * @param pid Process ID of Java process for which MBeanServerConnection is
 *    desired.
 * @return MBeanServerConnection connecting to Java process identified by pid.
 */
def static MBeanServerConnection retrieveServerConnection(String pid)
{
   def connectorAddressStr = "com.sun.management.jmxremote.localConnectorAddress"
   def jmxUrl = retrieveUrlForPid(pid, connectorAddressStr)
   def jmxConnector = JMXConnectorFactory.connect(jmxUrl)
   return jmxConnector.getMBeanServerConnection()
}

/**
 * Provide JMX URL for attaching to the provided process ID (pid).
 *
 * @param @pid Process ID for which JMX URL is needed to connect.
 * @param @connectorAddressStr String for connecting.
 * @return JMX URL to communicating with Java process identified by pid.
 */
def static JMXServiceURL retrieveUrlForPid(String pid, String connectorAddressStr)
{
   // Attach to the target application's virtual machine
   def vm = VirtualMachine.attach(pid)

   // Obtain Connector Address
   def connectorAddress =
      vm.getAgentProperties().getProperty(connectorAddressStr)

   // Load Agent if no connector address is available
   if (connectorAddress == null)
   {
      def agent = vm.getSystemProperties().getProperty("java.home") +
          File.separator + "lib" + File.separator + "management-agent.jar"
      vm.loadAgent(agent)

      // agent is started, get the connector address
      connectorAddress =
         vm.getAgentProperties().getProperty(connectorAddressStr)
   }

   return new JMXServiceURL(connectorAddress);
}

The most important method defined in JmxServer.groovy, from a script client perspective, is the retrieveServerConnection(String) method. That methods excepts a pid as a process identifier and uses that and a call to the other method, retrieveUrlForPid(String,String) to attach to the Java process identified by that pid and provide an associated MBeanServerConnection instance. The easiest approaches for finding the appropriate Java pid are use of jps (pre-Java 7 and Java 7) or jcmd (Java 7+ only) as I briefly described in my recent post Booting AMX in GlassFish 3 with Groovy.

Having encapsulated the logic for acquiring a particular JVM (Java process's) MBeanServerConnection instance based on a provided pid, it is easy to start constructing simple Groovy scripts that use this provided MBeanServerConnection to provide specific desirable details about the JVM (Java process) in question. Note that all of these example scripts are in the same directory as the JmxServer.groovy file and so do not need to explicitly specify package or scoping details.

It is common to want to know what is on the classpath of a particular JVM for various reasons including detection of causes of ClassNotFoundException and NoClassDefFoundError. The classpath of a particular JVM is exposed by one of its platform MBean ("ClassPath" attribute of RuntimeMXBean) and can be easily discovered with the next Groovy script:

displayClasspath.groovy

#!/usr/bin/env groovy
def pid = args[0]
def javaRuntime = new javax.management.ObjectName("java.lang:type=Runtime")
def classpath = JmxServer.retrieveServerConnection(pid).getAttribute(javaRuntime, "ClassPath")
println "\nClasspath for Java pid (${pid}):\n${classpath}\n"

That same RuntimeMXBean also provides the boot classpath and the Java library path. The next two code listings show Groovy scripts for accessing both of those.

displayBootClasspath.groovy

#!/usr/bin/env groovy
def pid = args[0]
def javaRuntime = new javax.management.ObjectName("java.lang:type=Runtime")
def bootClasspath = JmxServer.retrieveServerConnection(pid).getAttribute(javaRuntime, "BootClassPath")
println "\nBoot Classpath for Java pid (${pid}):\n${bootClasspath}\n"

displayLibraryPath.groovy

#!/usr/bin/env groovy
def pid = args[0]
def javaRuntime = new javax.management.ObjectName("java.lang:type=Runtime")
def libraryPath = JmxServer.retrieveServerConnection(pid).getAttribute(javaRuntime, "LibraryPath")
println "\nLibrary Path for Java pid (${pid}):\n${libraryPath}\n"

The Groovy scripts shown so far for displaying details related finding classes (the classpath, boot classpath, and library path) are all essentially the same script, but with a different attribute looked up on the RuntimeMXBean in each case. All three scripts are executed against a running instance of GlassFish in the following screen snapshot. The snapshot includes commands to jps and jcmd to determine the pid (5352) for the GlassFish instance.

The RuntimeMXBean has more to offer than just information on where classes were loaded from. It also includes attributes related to the JVM vendor such as name and version, but the other three methods that I like to have scripts to use are input arguments, system properties, and JVM uptime (and start time). The next three Groovy code listings show three scripts for displaying these details and each code snippet is followed by a screen snapshot demonstrating that script in action against the same GlassFish instance used in the last examples.

displayInputArguments.groovy

#!/usr/bin/env groovy
def pid = args[0]
def javaRuntime = new javax.management.ObjectName("java.lang:type=Runtime")
def inputArguments = JmxServer.retrieveServerConnection(pid).getAttribute(javaRuntime, "InputArguments")
println "\nInput Arguments for Java pid (${pid}):\n${inputArguments}\n"

displaySystemProperties.groovy

#!/usr/bin/env groovy
def pid = args[0]
def javaRuntime = new javax.management.ObjectName("java.lang:type=Runtime")
def systemProperties = JmxServer.retrieveServerConnection(pid).getAttribute(javaRuntime, "SystemProperties")
println "\nSystem Properties for Java pid (${pid}):\n${systemProperties}\n"

displayJvmUptime.groovy

#!/usr/bin/env groovy
def pid = args[0]
def javaRuntime = new javax.management.ObjectName("java.lang:type=Runtime")
def server = JmxServer.retrieveServerConnection(pid)
def startTime = server.getAttribute(javaRuntime, "StartTime")
def uptime = server.getAttribute(javaRuntime, "Uptime")
println "\nJava process pid (${pid}) was started at ${new Date(startTime)} and has been up for ${uptime} ms.\n"

I have used this post to demonstrate simple Groovy scripts that acquire information from a JVM's Platform RuntimeMXBean such as the JVM's start time and uptime, the system properties and input arguments for the JVM process, and path information such as classpath, boot classpath, and library path. Although this information is available via tools such as JConsole, VisualVM, and Java IDEs (they all get it from the same source as these scripts!), the scripts can be advantageous at times (quicker to run and can be included within other scripts, for example).

"Booting AMX" in GlassFish 3 with Groovy

In my previous blog post, I looked at using JMX as one of multiple methods supported by GlassFish 3 for its administration, monitoring, and management. In this blog post, I look in more detail at monitoring and managing GlassFish 3 via JMX and Groovy. I focus on local connection to GlassFish using the Attach API in this post, but I have covered remote JMX access of GlassFish in a previous post (see also Remote Glassfish V3.1 and the mystical JMX settings). The MBeans used for administration are generally the same in either case.

Because I'm using the Attach API in this post's examples to connect to local Java processes that are started by me, I don't need to specify the JMX remote connection properties (java.rmi.server.hostname,com.sun.management.jmxremote.port, and com.sun.management.jmxremote.ssl=false, and com.sun.management.jmxremote.authenticate) for remote access. The easiest way to find Java processes meeting the standard of being on the local machine and being processes that I started is via use of jps (pre-JDK 7 and JDK 7) or jcmd (JDK 7 only), which I show in the next two screen snapshots.

As the above images indicate, GlassFish is running locally with PID (AKA Process ID or pid) 1584.

To begin, I'm going to use Groovy to access the relevant MBeanServerConnection via the Attach API. This is done as shown in the following Groovy code (next two code listings) in which the method retrieveServerConnection(String) accepts the PID and returns the MBeanServerConnection for that Java process. That method uses another method in the code listing, retrieveUrlForPid(String, String), which uses the Attach API to provide a JMXServiceURL for the Java process.

retrieveServerConnection(String)

/**
 * Provide an MBeanServerConnection based on the provided process ID (pid).
 *
 * @param pid Process ID of Java process for which MBeanServerConnection is
 *    desired.
 * @return MBeanServerConnection connecting to Java process identified by pid.
 */
def MBeanServerConnection retrieveServerConnection(String pid)
{
   def connectorAddressStr = "com.sun.management.jmxremote.localConnectorAddress"
   def jmxUrl = retrieveUrlForPid(pid, connectorAddressStr)
   def jmxConnector = JMXConnectorFactory.connect(jmxUrl)
   return jmxConnector.getMBeanServerConnection()
}

retrieveUrlForPid(String, String)

/**
 * Provide JMX URL for attaching to the provided process ID (pid).
 *
 * @param @pid Process ID for which JMX URL is needed to connect.
 * @param @connectorAddressStr String for connecting.
 * @return JMX URL to communicating with Java process identified by pid.
 */
def JMXServiceURL retrieveUrlForPid(String pid, String connectorAddressStr)
{
   // Attach to the target application's virtual machine
   def vm = VirtualMachine.attach(pid)

   // Obtain Connector Address
   def connectorAddress =
      vm.getAgentProperties().getProperty(connectorAddressStr)

   // Load Agent if no connector address is available
   if (connectorAddress == null)
   {
      def agent = vm.getSystemProperties().getProperty("java.home") +
          File.separator + "lib" + File.separator + "management-agent.jar"
      vm.loadAgent(agent)

      // agent is started, get the connector address
      connectorAddress =
         vm.getAgentProperties().getProperty(connectorAddressStr)
   }

   return new JMXServiceURL(connectorAddress);
}

With access to the MBeanServerConnection, I can start to do all types of useful things with Groovy and JMX to manage and monitor the a Java process. For example, the next code listing demonstrates how easy it is to now list the MBeans exposed by the Java process identified by a provided PID.

displayHostedMBeans(MBeanServerConnection)

/**
 * Display MBeans hosted on the provided MBeanServerConnection.
 *
 * @param mbeanServer MBeanServerConnection for which hosted MBeans are to be
 *    provided.
 */
def displayHostedMBeans(MBeanServerConnection mbeanServer)
{
   mbeanServer.queryNames(null, null).each
   {
      println it
   }
}

Running the above Groovy method against my running GlassFish instance with PID 1584 is demonstrated in the next screen snapshot.

The next code listing and associated screen snapshot demonstrate using Groovy and JMX to find not only the exposed MBeans, but to also provide the attributes and operations available on each of those exposed MBeans.

displayHostedMBeansAttributesAndOperations(MBeanServerConnection)

/**
 * Display MBeans hosted on provided MBean Server along with the attributes and
 * operations available on each MBean.
 */
def displayHostedMBeansAttributesAndOperations(MBeanServerConnection mbeanServer)
{
   mbeanServer.queryNames(null, null).each
   { mbeanObjectName ->
      def mbeanInfo = mbeanServer.getMBeanInfo(mbeanObjectName)
      println mbeanObjectName
      println "\tAttributes:"
      mbeanInfo.attributes.each
      { attribute ->
         println "\t\t${attribute.type} ${attribute.name}"
      }
      println "\tOperations:"
      mbeanInfo.operations.each
      { operation ->
         def operationStr = new StringBuilder();
         operationStr << "\t\t" << operation.name << "("
         operation.signature.each
         { parameter ->
            operationStr << parameter.type << " " << parameter.name << ", "
         }
         def operationString = operationStr.contains(",") ? operationStr.substring(0, operationStr.length()-2) : operationStr
         println "${operationString})"
      }
      println ""
   }
}

The previously displayed screen snapshot shows the bootAMX operation available on the amx-support:type=boot-amx MBean. This needs to be run to make other GlassFish-provided AMX MBean available. The next snippet of Groovy code, a method I've named bootAmx(MBeanServerConnection), invokes this bootAMX operation to instruct GlassFish to expose significantly more details via its JMX interface.

bootAmx(MBeanServerConnection)

/**
 * "Boot AMX" on GlassFish.
 *
 * @param mbeanServer MBeanServerConnection to be used to invoke bootAMX
 *    operation.
 */
def bootAmx(MBeanServerConnection mbeanServer)
{
   def amxObjectName = new ObjectName("amx-support:type=boot-amx")
   mbeanServer.invoke(amxObjectName, "bootAMX", null, null)
}

When the simple Groovy just shown is run, GlassFish exposes significant more functionality via JMX. Now that I've "booted AMX," I can rerun displayHostedMBeans(MBeanServerConnection) to see the new MBeans that are available. A portion of this is shown next in the screen snapshot and the small-font text below it contains the entire output.

amx:pp=/domain/configs/config[server-config]/network-config,type=transports
amx:pp=/domain/configs/config[default-config]/iiop-service,type=iiop-listener,name=orb-listener-1
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=property,name=encryption.key.alias
amx:pp=/domain/resources/jdbc-connection-pool[DerbyPool],type=property,name=serverName
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[http-listener-1]/http,type=file-cache
amx:pp=/domain/configs/config[server-config],type=mdb-container
amx:pp=/domain/configs/config[default-config],type=system-property,name=HTTP_LISTENER_PORT
amx:pp=/domain/configs/config[server-config]/web-container/session-config/session-manager,type=store-properties
amx:pp=/ext,type=config-tools
amx:pp=/domain/nodes,type=node,name=localhost-domain1
amx:pp=/domain/servers/server[server],type=application-ref,name=__admingui
amx:pp=/domain/configs/config[default-config]/network-config/protocols,type=protocol,name=sec-admin-listener
amx:pp=/J2EEDomain/J2EEServer[server]/WebModule[__admingui],type=Servlet,name=DownloadServlet,j2eeType=Servlet,J2EEServer=server,WebModule=__admingui,J2EEApplication=null
amx:pp=/domain/configs/config[server-config]/network-config/transports,type=transport,name=tcp
amx:pp=/domain/configs/config[default-config],type=system-property,name=OSGI_SHELL_TELNET_PORT
amx:pp=/domain/configs/config[server-config],type=thread-pools
amx:pp=/domain/configs/config[server-config]/security-service,type=audit-module,name=default
amx:pp=/domain/secure-admin,type=secure-admin-principal,name="CN=localhost-instance,OU=GlassFish,O=Oracle Corporation,L=Santa Clara,ST=California,C=US"
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ServerProvider],type=property,name=signature.key.alias
amx:pp=/,type=logging
java.lang:type=MemoryPool,name=Code Cache
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=property,name=encryption.key.alias
amx:pp=/domain/configs/config[default-config]/web-container,type=session-config
amx:pp=/domain/configs/config[server-config]/security-service/auth-realm[file],type=property,name=file
amx:pp=/domain/configs/config[server-config]/iiop-service,type=iiop-listener,name=SSL
amx:pp=/domain/configs/config[server-config]/security-service,type=message-security-config,name=SOAP
amx:pp=/domain/configs/config[server-config]/iiop-service/iiop-listener[SSL_MUTUALAUTH],type=ssl
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[http-listener-1],type=http
amx:pp=/domain/configs/config[default-config]/network-config,type=transports
amx:pp=/domain/configs/config[server-config]/http-service,type=virtual-server,name=server
amx:pp=/domain/configs/config[default-config]/web-container/session-config/session-manager,type=store-properties
amx:pp=/domain/configs/config[default-config],type=ejb-container
amx:pp=/ext,type=system-status
amx:pp=/domain/configs/config[server-config]/network-config/protocols/protocol[admin-listener],type=http
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=response-policy
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[HttpServlet]/provider-config[GFConsoleAuthModule],type=property,name=loginPage
amx:pp=/domain/servers/server[server],type=resource-ref,name=jdbc/__TimerPool
java.lang:type=Memory
amx:pp=/mon/server-mon[server],type=compilation-system-mon,name=jvm/compilation-system
amx:pp=/domain/configs/config[server-config]/security-service,type=auth-realm,name=admin-realm
amx:pp=/domain/configs/config[default-config]/log-service,type=module-log-levels
amx:pp=/mon/server-mon[server],type=server-runtime-mon
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP],type=provider-config,name=XWS_ClientProvider
amx:pp=/domain/configs/config[default-config]/web-container/session-config,type=session-manager
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP],type=provider-config,name=XWS_ServerProvider
amx:pp=/domain/configs/config[server-config]/thread-pools,type=thread-pool,name=http-thread-pool
amx:pp=/domain/configs/config[default-config]/admin-service,type=property,name=adminConsoleDownloadLocation
amx:pp=/domain/resources/jdbc-connection-pool[DerbyPool],type=property,name=connectionAttributes
amx:pp=/domain/resources/jdbc-connection-pool[__TimerPool],type=property,name=connectionAttributes
amx:pp=/domain/configs/config[default-config]/network-config/protocols,type=protocol,name=pu-protocol
amx:pp=/domain/configs/config[server-config]/network-config,type=protocols
amx:pp=/ext,type=connector-runtime-api-provider
amx:pp=/domain,type=resources
jmxremote:type=jmx-connector-server,protocol=rmi_jrmp,name=system
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[http-listener-2],type=ssl
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[http-listener-2],type=http
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=property,name=signature.key.alias
amx:pp=/,type=ext
amx:pp=/domain/configs/config[default-config]/monitoring-service,type=module-monitoring-levels
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP],type=provider-config,name=ClientProvider
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[HttpServlet]/provider-config[GFConsoleAuthModule],type=property,name=restAuthURL
amx:pp=/domain/configs/config[server-config]/network-config/protocols/protocol[http-listener-1],type=http
amx:pp=/domain/configs/config[server-config]/jms-service,type=jms-host,name=default_JMS_host
amx:pp=/domain/configs/config[server-config]/web-container/session-config/session-manager,type=manager-properties
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ServerProvider],type=property,name=debug
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=property,name=debug
amx:pp=/domain/configs/config[default-config]/network-config/network-listeners,type=network-listener,name=admin-listener
amx:pp=/domain/configs/config[default-config]/security-service/jacc-provider[default],type=property,name=repository
amx:pp=/mon/server-mon[server],type=operating-system-mon,name=jvm/operating-system
amx:pp=/domain/configs/config[default-config]/security-service,type=audit-module,name=default
java.lang:type=MemoryPool,name=Survivor Space
amx:pp=/domain/configs/config[default-config]/network-config,type=protocols
amx:pp=/domain/configs/config[server-config]/network-config/protocols/protocol[http-listener-2],type=ssl
amx:pp=/domain/configs/config[default-config]/thread-pools,type=thread-pool,name=http-thread-pool
amx:pp=/J2EEDomain/J2EEServer[server]/WebModule[__admingui],type=Servlet,name=jsp,j2eeType=Servlet,J2EEServer=server,WebModule=__admingui,J2EEApplication=null
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[pu-protocol],type=port-unification
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=property,name=security.config
amx:pp=/domain/configs/config[default-config],type=system-property,name=HTTP_SSL_LISTENER_PORT
amx:pp=/domain/configs/config[default-config]/http-service,type=virtual-server,name=server
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=request-policy
amx:pp=/domain/configs/config[server-config],type=iiop-service
amx:pp=/domain/configs/config[default-config]/security-service,type=jacc-provider,name=default
amx:pp=/domain/configs/config[default-config]/web-container/session-config/session-manager,type=manager-properties
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=request-policy
amx:pp=/domain/configs/config[default-config]/security-service,type=auth-realm,name=certificate
java.nio:type=BufferPool,name=mapped
java.lang:type=Compilation
amx:pp=/,type=runtime
amx:pp=/domain/configs/config[server-config]/monitoring-service,type=module-monitoring-levels
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=property,name=dynamic.username.password
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=property,name=signature.key.alias
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=property,name=encryption.key.alias
amx-support:type=amx-loader,name=config
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=property,name=dynamic.username.password
amx:pp=/runtime,type=server-runtime,name=server
amx:pp=/domain/configs/config[server-config]/network-config/protocols,type=protocol,name=http-listener-1
amx:pp=/domain,type=system-applications
amx:pp=/domain/configs/config[server-config]/security-service,type=jacc-provider,name=default
amx:pp=/J2EEDomain/J2EEServer[server]/JDBCResource[jdbc/__default]/JDBCDataSource[jdbc/__default],type=JDBCDriver,name=jdbc/__default,j2eeType=JDBCDriver,J2EEServer=server,JDBCResource=jdbc/__default,JDBCDataSource=jdbc/__default
amx:pp=/domain/configs/config[server-config]/security-service/auth-realm[file],type=property,name=jaas-context
amx:pp=/domain/configs/config[server-config]/ejb-container,type=ejb-timer-service
amx:pp=/domain/configs/config[default-config]/security-service,type=auth-realm,name=file
java.lang:type=Runtime
amx:pp=/J2EEDomain/J2EEServer[server]/WebModule[__admingui],type=Servlet,name=default,j2eeType=Servlet,J2EEServer=server,WebModule=__admingui,J2EEApplication=null
amx:pp=/domain/configs/config[default-config],type=web-container
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=property,name=debug
amx:pp=/domain/system-applications/application[__admingui]/module[__admingui],type=engine,name=security
amx:pp=/domain/configs/config[server-config]/admin-service,type=jmx-connector,name=system
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ServerProvider],type=response-policy
amx:pp=/ext,type=realms
amx:pp=/,type=query
amx:pp=/domain/configs/config[default-config],type=thread-pools
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[HttpServlet]/provider-config[GFConsoleAuthModule],type=property,name=loginErrorPage
amx:pp=/domain/configs/config[server-config],type=connector-service
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=property,name=signature.key.alias
amx:pp=/domain/configs/config[default-config]/network-config,type=network-listeners
amx:pp=/J2EEDomain/J2EEServer[server]/JDBCResource[jdbc/__TimerPool]/JDBCDataSource[jdbc/__TimerPool],type=JDBCDriver,name=jdbc/__TimerPool,j2eeType=JDBCDriver,J2EEServer=server,JDBCResource=jdbc/__TimerPool,JDBCDataSource=jdbc/__TimerPool
amx:pp=/domain/configs/config[default-config],type=system-property,name=JMS_PROVIDER_PORT
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ServerProvider],type=request-policy
amx:pp=/domain/configs/config[default-config],type=availability-service
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[admin-http-redirect],type=http-redirect
amx:pp=/,type=J2EEDomain,j2eeType=J2EEDomain,name=amx
amx:pp=/domain/configs/config[default-config],type=system-property,name=IIOP_SSL_LISTENER_PORT
amx:pp=/domain/configs/config[default-config]/iiop-service,type=iiop-listener,name=SSL
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=property,name=encryption.key.alias
amx:pp=/domain/configs/config[default-config]/network-config/transports,type=transport,name=tcp
amx:pp=/domain/configs/config[default-config],type=system-property,name=JMX_SYSTEM_CONNECTOR_PORT
amx:pp=/mon/server-mon[server],type=garbage-collector-mon,name=jvm/garbage-collectors/Copy
amx:pp=/domain/configs/config[default-config]/http-service,type=virtual-server,name=__asadmin
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=request-policy
amx:pp=/,type=system-info
amx:pp=/domain/configs/config[default-config],type=group-management-service
amx:pp=/domain/configs/config[default-config]/availability-service,type=web-container-availability
amx:pp=/domain/configs/config[server-config]/group-management-service,type=failure-detection
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=response-policy
java.lang:type=GarbageCollector,name=Copy
amx:pp=/domain/configs/config[server-config],type=java-config
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP],type=provider-config,name=ServerProvider
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[http-listener-2]/http,type=file-cache
amx:pp=/domain/configs/config[default-config],type=system-property,name=ASADMIN_LISTENER_PORT
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=request-policy
amx:pp=/domain/configs/config[server-config]/network-config/protocols,type=protocol,name=http-listener-2
amx:pp=/domain/configs/config[default-config]/jms-service,type=jms-host,name=default_JMS_host
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=property,name=debug
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP],type=provider-config,name=ServerProvider
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=property,name=signature.key.alias
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=response-policy
amx:pp=/domain/configs/config[server-config],type=web-container
amx:pp=/domain/configs/config[default-config]/security-service/auth-realm[admin-realm],type=property,name=jaas-context
java.util.logging:type=Logging
amx:pp=/,type=tools
java.lang:type=MemoryPool,name=Eden Space
amx:pp=/domain/configs/config[default-config]/network-config/protocols,type=protocol,name=http-listener-2
com.sun.management:type=HotSpotDiagnostic
amx:pp=/domain/configs/config[default-config]/security-service,type=jacc-provider,name=simple
amx:pp=/domain/configs/config[default-config]/security-service,type=auth-realm,name=admin-realm
amx:pp=/J2EEDomain/J2EEServer[server]/WebModule[__admingui],type=Servlet,name=FacesServlet,j2eeType=Servlet,J2EEServer=server,WebModule=__admingui,J2EEApplication=null
amx:pp=/domain/configs/config[server-config]/admin-service,type=property,name=adminConsoleContextRoot
amx:pp=/domain/configs/config[default-config],type=monitoring-service
amx:pp=/domain/configs/config[server-config]/security-service,type=auth-realm,name=file
amx:pp=/domain/configs/config[server-config]/security-service/auth-realm[admin-realm],type=property,name=jaas-context
amx:pp=/domain/configs/config[server-config]/security-service/audit-module[default],type=property,name=auditOn
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=response-policy
amx:pp=/domain/configs/config[default-config]/security-service/auth-realm[file],type=property,name=jaas-context
amx:pp=/domain/system-applications,type=application,name=__admingui
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=property,name=security.config
amx:pp=/domain/configs/config[default-config]/iiop-service/iiop-listener[SSL],type=ssl
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=property,name=signature.key.alias
amx:pp=/domain/configs/config[default-config],type=connector-service
amx:pp=/domain/configs/config[server-config],type=ejb-container
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=request-policy
amx:pp=/mon/server-mon[server],type=garbage-collector-mon,name=jvm/garbage-collectors/MarkSweepCompact
amx:pp=/domain/configs/config[default-config]/security-service/audit-module[default],type=property,name=auditOn
amx:pp=/domain,type=secure-admin
amx:pp=/domain/configs/config[default-config]/network-config/protocols,type=protocol,name=http-listener-1
amx:pp=/,type=sample
amx:pp=/domain/configs/config[default-config]/group-management-service,type=failure-detection
amx:pp=/domain/configs/config[default-config]/network-config/network-listeners,type=network-listener,name=http-listener-1
amx:pp=/domain/system-applications/application[__admingui],type=module,name=__admingui
amx:pp=/J2EEDomain/J2EEServer[server]/WebModule[__admingui],type=Servlet,name=ThemeServlet,j2eeType=Servlet,J2EEServer=server,WebModule=__admingui,J2EEApplication=null
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ServerProvider],type=request-policy
amx:pp=/domain/configs,type=config,name=default-config
amx:pp=/domain/system-applications/application[__admingui]/module[__admingui],type=engine,name=web
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=property,name=dynamic.username.password
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=property,name=security.config
java.nio:type=BufferPool,name=direct
amx:pp=/domain/configs/config[server-config]/network-config/network-listeners,type=network-listener,name=http-listener-2
amx:pp=/domain/configs/config[server-config]/iiop-service,type=orb
amx:pp=/domain/configs/config[default-config],type=java-config
amx:pp=/domain,type=property,name=administrative.domain.name
amx:pp=/domain,type=load-balancers
amx:pp=/domain/resources/jdbc-connection-pool[__TimerPool],type=property,name=databaseName
amx:pp=/domain/resources,type=jdbc-connection-pool,name=DerbyPool
amx:pp=/domain/configs/config[default-config]/thread-pools,type=thread-pool,name=thread-pool-1
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ServerProvider],type=property,name=encryption.key.alias
amx:pp=/domain/configs/config[default-config],type=diagnostic-service
amx:pp=/domain/configs/config[default-config],type=security-service
amx:pp=/domain/configs/config[default-config],type=system-property,name=IIOP_LISTENER_PORT
amx:pp=/J2EEDomain/J2EEServer[server]/JDBCResource[jdbc/__TimerPool],type=JDBCDataSource,name=jdbc/__TimerPool,j2eeType=JDBCDataSource,J2EEServer=server,JDBCResource=jdbc/__TimerPool
amx:pp=/domain/resources/jdbc-connection-pool[DerbyPool],type=property,name=PortNumber
amx:pp=/domain/configs/config[server-config],type=admin-service
amx:pp=/domain/resources/jdbc-connection-pool[DerbyPool],type=property,name=DatabaseName
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[admin-listener]/http,type=file-cache
amx:pp=/domain/configs/config[server-config]/web-container/session-config,type=session-manager
java.lang:type=MemoryPool,name=Tenured Gen
amx:pp=/domain/configs/config[server-config]/thread-pools,type=thread-pool,name=admin-thread-pool
amx:pp=/domain/configs/config[server-config]/security-service,type=jacc-provider,name=simple
amx:pp=/domain/configs/config[server-config],type=monitoring-service
amx:pp=/domain/configs/config[server-config]/security-service,type=property,name=default-digest-algorithm
amx:pp=/domain/configs/config[default-config],type=network-config
amx:pp=/,type=mon
amx:pp=/domain/configs/config[default-config],type=mdb-container
amx-support:type=boot-amx
amx:pp=/domain/configs/config[default-config],type=transaction-service
amx:pp=/J2EEDomain/J2EEServer[server],type=JDBCResource,name=jdbc/__default,j2eeType=JDBCResource,J2EEServer=server
amx:pp=/domain/configs/config[server-config]/network-config/protocols/protocol[admin-listener]/http,type=file-cache
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[admin-listener],type=http
amx:pp=/domain,type=servers
amx:pp=/domain/configs/config[default-config]/network-config/protocols,type=protocol,name=admin-http-redirect
amx:pp=/domain/servers,type=server,name=server
amx:pp=/domain/configs/config[default-config]/availability-service,type=ejb-container-availability
amx:pp=/domain/configs/config[server-config]/admin-service,type=property,name=ipsRoot
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ServerProvider],type=response-policy
amx:pp=/domain/configs/config[server-config]/iiop-service,type=iiop-listener,name=orb-listener-1
java.lang:type=GarbageCollector,name=MarkSweepCompact
amx:pp=/domain/configs/config[server-config]/admin-service,type=das-config
amx:pp=/mon/server-mon[server],type=thread-system-mon,name=jvm/thread-system
amx:pp=/domain/resources,type=jdbc-resource,name=jdbc/__TimerPool
amx:pp=/J2EEDomain/J2EEServer[server]/JDBCResource[jdbc/__default],type=JDBCDataSource,name=jdbc/__default,j2eeType=JDBCDataSource,J2EEServer=server,JDBCResource=jdbc/__default
amx:pp=/domain/configs/config[server-config]/security-service/jacc-provider[default],type=property,name=repository
amx:pp=/domain/configs/config[server-config]/http-service,type=access-log
java.lang:type=ClassLoading
amx:pp=/domain/configs/config[server-config]/network-config/protocols,type=protocol,name=admin-listener
java.lang:type=Threading
amx-support:type=amx-loader,name=j2ee
amx:pp=/domain/configs/config[server-config]/security-service/auth-realm[admin-realm],type=property,name=file
amx:pp=/mon/server-mon[server],type=memory-mon,name=jvm/memory
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=property,name=debug
amx:pp=/domain/configs/config[default-config]/network-config/protocols,type=protocol,name=admin-listener
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=property,name=debug
amx:pp=/domain/configs/config[server-config]/security-service,type=message-security-config,name=HttpServlet
amx:pp=/domain/configs/config[server-config]/web-container,type=session-config
amx:pp=/domain/resources/jdbc-connection-pool[DerbyPool],type=property,name=User
amx:pp=/domain/configs/config[server-config]/thread-pools,type=thread-pool,name=thread-pool-1
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=property,name=encryption.key.alias
amx:pp=/domain/configs/config[default-config]/http-service,type=access-log
amx:pp=/mon,type=server-mon,name=server
amx:pp=/domain/servers/server[server],type=resource-ref,name=jdbc/__default
amx:pp=/domain/configs/config[default-config],type=http-service
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=property,name=signature.key.alias
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=property,name=dynamic.username.password
amx:pp=/domain/resources/jdbc-connection-pool[DerbyPool],type=property,name=Password
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[pu-protocol]/port-unification,type=protocol-finder,name=admin-http-redirect
amx:pp=/domain/configs/config[default-config]/security-service/auth-realm[file],type=property,name=file
amx:pp=/domain/configs/config[default-config]/security-service/auth-realm[admin-realm],type=property,name=file
amx:pp=/domain/configs/config[default-config]/iiop-service,type=orb
amx:pp=/domain/configs/config[default-config]/admin-service,type=das-config
amx:pp=/domain/configs,type=config,name=server-config
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[HttpServlet]/provider-config[GFConsoleAuthModule],type=request-policy
amx:pp=/domain/configs/config[default-config],type=log-service
amx:pp=/domain/configs/config[default-config]/iiop-service,type=iiop-listener,name=SSL_MUTUALAUTH
amx:pp=/domain/configs/config[default-config],type=iiop-service
amx:pp=/domain/configs/config[server-config]/iiop-service,type=iiop-listener,name=SSL_MUTUALAUTH
amx:pp=/domain/configs/config[server-config]/security-service,type=auth-realm,name=certificate
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP],type=provider-config,name=ClientProvider
amx:pp=/domain/configs/config[server-config],type=security-service
amx:pp=/domain/configs/config[default-config]/security-service,type=message-security-config,name=SOAP
amx:pp=/mon/server-mon[server],type=runtime-mon,name=jvm/runtime
amx:pp=/domain,type=lb-configs
amx:pp=/J2EEDomain/J2EEServer[server],type=WebModule,name=__admingui,j2eeType=WebModule,J2EEServer=server,J2EEApplication=null
java.lang:type=MemoryPool,name=Perm Gen
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[HttpServlet],type=provider-config,name=GFConsoleAuthModule
amx:pp=/mon/server-mon[server],type=class-loading-system-mon,name=jvm/class-loading-system
java.lang:type=MemoryManager,name=CodeCacheManager
amx:pp=/domain/resources,type=jdbc-resource,name=jdbc/__default
amx-support:type=mbean-tracker
amx:pp=/domain/configs/config[server-config],type=network-config
amx:pp=/domain/configs/config[server-config]/network-config/protocols/protocol[http-listener-2],type=http
amx:pp=/domain,type=configs
amx:pp=/domain/configs/config[server-config]/http-service,type=virtual-server,name=__asadmin
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP],type=provider-config,name=XWS_ClientProvider
amx:pp=/domain/configs/config[default-config]/web-container/session-config,type=session-properties
amx-support:type=amx-loader,name=startup
amx:pp=/domain/configs/config[default-config],type=admin-service
amx:pp=/domain/configs/config[default-config]/network-config/network-listeners,type=network-listener,name=http-listener-2
amx:pp=/domain/configs/config[server-config],type=jms-service
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[sec-admin-listener]/http,type=file-cache
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[sec-admin-listener],type=ssl
amx:pp=/domain,type=clusters
amx:pp=/domain,type=nodes
amx:pp=/domain/configs/config[default-config]/thread-pools,type=thread-pool,name=admin-thread-pool
amx-support:type=AMXConfigLoader
amx:pp=/domain/configs/config[default-config]/ejb-container,type=ejb-timer-service
amx:pp=/domain/configs/config[default-config]/iiop-service/iiop-listener[SSL_MUTUALAUTH],type=ssl
amx:pp=/domain/configs/config[server-config]/network-config/protocols/protocol[http-listener-2]/http,type=file-cache
amx:pp=/domain/resources,type=jdbc-connection-pool,name=__TimerPool
amx:pp=/domain/configs/config[default-config],type=jms-service
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=request-policy
amx:pp=/domain/configs/config[server-config]/network-config/network-listeners,type=network-listener,name=admin-listener
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ServerProvider],type=property,name=debug
amx:pp=/domain/configs/config[default-config],type=system-property,name=JAVA_DEBUGGER_PORT
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ServerProvider],type=property,name=encryption.key.alias
amx:pp=/domain/configs/config[server-config]/network-config,type=network-listeners
amx:pp=/domain/secure-admin,type=secure-admin-principal,name="CN=localhost,OU=GlassFish,O=Oracle Corporation,L=Santa Clara,ST=California,C=US"
amx:pp=/domain/configs/config[server-config]/network-config/network-listeners,type=network-listener,name=http-listener-1
amx:pp=/domain/configs/config[server-config]/web-container/session-config,type=session-properties
amx:pp=/,type=bulk-access
amx:pp=/domain/configs/config[server-config],type=group-management-service
amx:pp=/domain/configs/config[default-config],type=management-rules
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ServerProvider],type=property,name=signature.key.alias
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=response-policy
amx:pp=/domain/configs/config[default-config]/availability-service,type=jms-availability
amx:pp=/domain/configs/config[default-config]/admin-service,type=jmx-connector,name=system
amx:pp=/domain/configs/config[default-config]/http-service/virtual-server[server],type=property,name=default-web-xml
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=response-policy
amx:pp=/domain/configs/config[server-config]/network-config/protocols/protocol[http-listener-1]/http,type=file-cache
amx:pp=/domain/configs/config[server-config]/admin-service,type=property,name=adminConsoleDownloadLocation
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[sec-admin-listener],type=http
amx:pp=/domain/configs/config[server-config],type=transaction-service
java.lang:type=OperatingSystem
amx:pp=/J2EEDomain/J2EEServer[server],type=JVM,j2eeType=JVM,J2EEServer=server
amx:pp=/,type=pathnames
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=property,name=security.config
amx-support:type=amx-loader,name=ext
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP],type=provider-config,name=XWS_ServerProvider
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[HttpServlet]/provider-config[GFConsoleAuthModule],type=response-policy
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=property,name=debug
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[pu-protocol]/port-unification,type=protocol-finder,name=http-finder
amx:pp=/J2EEDomain/J2EEServer[server],type=JDBCResource,name=jdbc/__TimerPool,j2eeType=JDBCResource,J2EEServer=server
amx:pp=/domain/configs/config[server-config],type=http-service
amx:pp=/,type=domain
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=property,name=encryption.key.alias
amx:pp=/domain,type=applications
amx:pp=/domain/configs/config[default-config],type=system-property,name=IIOP_SSL_MUTUALAUTH_PORT
amx:pp=/domain/configs/config[server-config]/iiop-service/iiop-listener[SSL],type=ssl
amx:pp=/J2EEDomain,type=J2EEServer,name=server,j2eeType=J2EEServer
amx:pp=,type=domain-root
JMImplementation:type=MBeanServerDelegate

As this image and the small-font text demonstrate, "booting AMX" has led to GlassFish exposing significant monitoring and management capabilities via JMX.

Conclusion

This post has demonstrated using Groovy to explore the standard JMX MBeans provided by the Java Platform as well as the AMX beans that GlassFish provides. The last Groovy code snippet invokes GlassFish's "bootAMX" operation to unleash significantly more GlassFish JMX-based monitoring and managing capabilities. Although all of this could be accomplished with a GUI-based JMX client such as JConsole or JVisualVM, there are advantages at times to running things in scripts. Using Groovy as demonstrated in this post allows scripts to easily interact with GlassFish for administration.

Jolokia, Java, JMX, and Groovy

The Bhut Jolokia is not just a chili pepper; it was deemed for several years as the hottest chili pepper in the world. In this blog post, I talk about something presumably named after this chili pepper (the logo and subtitle "JMX on Capsaicin" are strong hints): the "remote JMX with JSON over HTTP" project called Jolokia.

The main Jolokia page describes this project:

Jolokia is remote JMX with JSON over HTTP. It is fast, simple, polyglot and has unique features. It's JMX on Capsaicin. Jolokia is a JMX-HTTP bridge giving an alternative to JSR-160 connectors. It is an agent based approach with support for many platforms. In addition to basic JMX operations it enhances JMX remoting with unique features like bulk requests or fine grained security policies.

Java Management Extensions can be extremely useful for managing and monitoring Java-based applications. For the most part, JMX remote clients are typically written in Java or a JVM-based language like Groovy or JRuby because JMX is most friendly to JVM-based clients.

Although there are non-JVM approaches to JMX web clients such as the experimental JMX Web Services Connector, these are the exception rather than the rule. Perhaps the most significant advantage of Jolokia is that it is based on HTTP (rather than traditional RMI used for JMX remote access) and its format is language-independent JSON (which I recently discussed in Groovy 1.8 Introduces Groovy to JSON). Having HTTP as the transport mechanism and JSON as the data format allows Jolokia-based clients to be written in nearly any conceivable language. At the time of this writing, out-of-the-box Jolkia agents are WAR (Java EE web application) Agent, OSGi Agent, JVM JDK6 Agent, and the Mule Agent. I focus on the JVM JDK6 Agent in this post.

The next code listing contains a simple nonsensical class that is intended solely to provide a long-running Java application whose automatically-provided MXBeans can be accessed via Jolokia.

Main.java

package dustin.examples;

import static java.lang.System.out;

/**
 * Main class for demonstrating Jolokia JVM Agent. This class doesn't really do
 * anything special except "live" in the JVM so that its built-in JMX MXBeans
 * can be accessed by JMX clients.
 */
public class Main
{
   /**
    * Main executable function.
    *
    * @param arguments Command-line arguments: one numeric argument expected.
    */
   public static void main(final String[] arguments)
   {
      final long numberLoops = arguments.length > 0 ? Long.parseLong(arguments[0]) : 1000000L;
      for (long count = 0; count < numberLoops; ++count)
      {
         if (count % 100 == 0)
         {
            try
            {
               Thread.sleep(10000);
            }
            catch (InterruptedException interruptedEx)
            {
               out.append("Thread's beauty sleep was interrupted.");
            }
         }
      }
   }
}

The above class, once compiled, can be run with the Java launcher. However, to employ Jolokia's JVM JDK6 Agent, the JVM Tool Interface (JVMTI) must be used. The JVMTI documentation warns that "JVM TI may not be available in all implementations of the Java^TM virtual machine." The Jolokia documentation is more specific in saying that Oracle's HotSpot JVM is the one for which their particular Agent is supported (because it leverages the HttpServer embedded in HotSpot).

Assuming that the Main.java class shown above is compiled and then packaged into a JAR file called JolokiaDemo.jar, the Java launcher can be run against this JAR file and associate the Jolokia JVM JDK6 Agent with the following command:

java -javaagent:C:\jolokia-0.90\jolokia-jvm-jdk6-0.90-agent.jar=port=8778,host=localhost -cp dist\JolokiaDemo.jar dustin.examples.Main 1000000000

The command just shown specifies the Jolokia agent JAR (jolokia-jvm-jdk6-0.90-agent.jar) and specifies two properties with values (host and port). Note that the specified host and port happen to be the defaults, but I included them here explicitly to indicate how they are supplied to the agent. The portion "-javaagent:C:\jolokia-0.90\jolokia-jvm-jdk6-0.90-agent.jar=port=8778,host=localhost" is the JVMTI portion and everything else is "traditional" Java launcher syntax.

Running the java launcher as indicated above leads to output like that shown in the next screen snapshot.

I like that Jolokia reports the URL by which its HTTP-exposed JMX information is available. This makes it easier to ensure that the client connects properly. In "normal" mode, this is about all that is seen as the client interacts with the application. However, I find it useful to enable Jolokia's debugging when getting the client to connect to this exposed application and this is easily done by adding debug=true onto the end of the JVMTI agent specification. An example of this is shown next.

java -javaagent:C:\jolokia-0.90\jolokia-jvm-jdk6-0.90-agent.jar=port=8778,host=localhost,debug=true -cp dist\JolokiaDemo.jar dustin.examples.Main 1000000000 

Running with debug=true leads to output like the following:

I haven't shown the client yet, but this debug output gives pretty good hints as to what the client is asking for. I found this invaluable as I experimented with the Jolokia requests. Speaking of the client, the next code listing contains a Groovy script for accessing some built-in MXBeans information about this simple thread sleeping class.

runJolokiaJmxClient.groovy

#!/usr/bin/env groovy
// runJolokiaJmxClient.groovy

import org.jolokia.client.J4pClient
import org.jolokia.client.exception.J4pConnectException
import org.jolokia.client.exception.J4pRemoteException
import org.jolokia.client.request.J4pReadRequest
import org.jolokia.client.request.J4pReadResponse

def localHost = InetAddress.getLoopbackAddress()
def urlStr = "http://${localHost.hostAddress}:8778/jolokia/"
def j4pClient = new J4pClient(urlStr)
println "Attempting to connect to ${urlStr}..."
def heapMemoryRequest = new J4pReadRequest("java.lang:type=Memory",
                                           "HeapMemoryUsage")
def threadingRequest = new J4pReadRequest("java.lang:type=Threading",
                                          "ThreadCount")
def operatingSystemRequest = new J4pReadRequest("java.lang:type=OperatingSystem",
                                                "Arch")

J4pReadResponse heapMemoryResponse = null
J4pReadResponse threadingResponse = null
J4pReadResponse osResponse = null
try
{
   heapMemoryResponse = j4pClient.execute(heapMemoryRequest)
   threadingResponse = j4pClient.execute(threadingRequest)
   osResponse = j4pClient.execute(operatingSystemRequest)
}
catch (J4pConnectException connEx)
{
   println "ERROR: Cannot connect to ${urlStr}\n${connEx.message}"
}
catch (J4pRemoteException remoteEx)
{
   println "ERROR encountered while trying to access Jolokia-exposed JVM\n${remoteEx.message}"
   println "Status: ${remoteEx.status}"
   println "Error Code: ${remoteEx.errorType}"
   println "Remote Stack Trace: ${remoteEx.remoteStackTrace}"
}
println "Heap Memory: ${heapMemoryResponse?.value}"
println "Thread Count: ${threadingResponse?.value}"
println "Operating System Architecture: ${osResponse?.value}"

The simple script above uses basic Jolokia classes and exceptions to communicate with the Jolokia instrumented Java application. The output from running the above script is shown in the next screen snapshot.

The output proves that the client was able to talk to the simple Java application and acquire information about its memory usage, its thread count, and its operating system architecture. For this exercise, I threw all the required dependencies on the classpath explicitly provided to the Groovy launcher, but there are other ways to handle classpath dependencies in Groovy.

Accessing JMX information via Groovy clients is pretty straightforward thanks to Groovy being a JVM-based language and thanks to its JMX syntactic goodness. In the case of Groovy clients, I probably would not be likely to employ Jolokia. However, the fact that the remote access is via HTTP and that the data is passed back in JSON format means there are numerous possibilities for non-JVM-based clients to access JVM information. Tomasz Nurkiewicz's post Jolokia + Highcharts = JMX for human beings provides a thorough example of doing just this. Nurkiewicz not only covers using JavaScript on the client side, but also provides examples via curl of the JSON format returned by Jolokia.

Jolokia's current version is 0.90 and the main page suggests that 1.0 will be released in late summer of 2011. Jolokia can be downloaded as binary or source code (it has an Apache 2 license). Individual JARs such as the JVM-Agent JAR and the Java Client JAR used in my example can be downloaded as well. The main Jolokia site provides simple examples of using the Jolokia Java Client Library, the JavaScript Client Library, and the Perl Client Library. There are also simple examples of code using the various agents such as the JVM Agent. The Jolokia Reference Documentation is still a work in progress and is also available in PDF format (though the HTML format seems more complete to me).


Conclusion

Jolokia holds significant promise as a mechanism for exposing valuable information held within Java applications and their hosting JVMs to non-Java/non-JVM clients. In this post, I've looked at one very simple example of accessing a Java application's JVM information from a Groovy client using Jolokia.

Posts of Particular Interest: Java and JavaOne, PHP, PowerShell,

In this post, I reference and briefly some posts of particular interest to me in recent weeks. Posts referenced here cover topics such as Java (JavaOne, JMX, Java Memory), PHP, and PowerShell.


Top Java Memory Problems

Michael Kopp's The Top Java Memory Problems – Part 1 provides a nice overview of common memory issues in Java applications. Specifically, he covers memory leaks related to thread local variables, mutable static fields and collections, circular/bi-directional references, JNI, improper equals/hashCode implementations, and classloaders.


Three Free E-books on Beginning Java (and Five on PHP)

Klint Finley's post 3 Free E-Books on Java for Beginners references three freely available e-books about Java. I was already aware of Bruce Eckel's Thinking in Java (as usual, he offers the second newest edition free of charge), but he also mentions How to Think Like a Computer Scientist: Java Edition and Introduction to Programming in Java.

In a related post called 5 Free E-Books and Tutorials for Learning PHP, Finley outlines five freely available PHP resources. As I recently posted, I have plans to start learning and working with PHP, so this post interests me.


Best JVM Replacement Language is Java? (and the Play Framework)

In the post And the best JVM replacement language for Java is: Java?, Mark Watson talks about some of the strengths of Java and discusses things he does "to make Java more agile." He also mentions early satisfaction with the Play framework. In one of the feedback comments on this post, Steven states, "I had to leave Java for the Rails-type frameworks because of the productivity gains. I have to say that with Play! you get the best of both worlds."


Accessing JMX on the Web with Jolokia

In the post Jolokia + Highcharts = JMX for human beings, Tomasz Nurkiewicz writes about using Jolokia ("HTTP/JSON bridge for remote JMX access") in conjunction with Highcharts ("free for noncommercial") to display JMX-provided JVM metrics in a web browser application. There are generic tools such as VisualVM and JConsole that can do some of this, but the flexibility and customizability of an approach like that demonstrated by Nurkiewicz can provide significant advantages in certain situations.


Actors and GPars

Vaclav Pech's Dr. Dobb's Journal article JVM Concurrency and Actors with GPars discusses how GPars (Groovy Parallel Systems) allows actors to be used on the JVM to simulate continuations. As Pech demonstrates this, he also covers highly-related topics such as basics of the actor model, an introduction to GPars for use in Groovy or Java or any JVM language, and lists other features provided by GPars.


PowerShell Language Licensed Under the Community Promise

The Windows PowerShell Blog post PowerShell Language now licensed under the Community Promise states:

The PowerShell team is excited to announce that starting today we are licensing the language specification for Windows PowerShell 2.0 under the Microsoft Community Promise. This means that now anyone can implement PowerShell on any platform they want to.

The specification is available for free download as a docx file.


JavaOne 2011 Formally Announced

There were multiple posts on JavaOne 2011's formal announcement. The JavaOne 2011 post in The Java Source blog and the JavaOne Conference Blog post JavaOne 2011 - October 2nd to 6th in San Francisco confirm that JavaOne 2011 will be held October 2-6, 2011, in San Francisco. This announcement also headlines the DZone Daily Dose and is featured in Java.net's "Spotlights" section.


Conclusion

There's much going on in the world of software development in general and the world of JVM development in particular. The posts I cited above are evidence of this and it is particularly exciting to see fresh new posts on subjects such as JMX, JVM actors with GPars, in-depth coverage of Java memory issues, and so forth.