Stream-JMX

A Lightweight framework to stream and monitor JMX metrics. (Formerly known as PingJMX)

Stream JMX metrics to:

These metrics can be used to monitor health, performance and availability of your JVMs and applications. Use Stream-JMX to embed a monitoring agent within your application and monitor memory, GC activity, CPU as well as user defined MBeans.

Here is what you can do with Stream-JMX:

Why Stream-JMX

Stream-JMX provides and easy, lightweight and secure way to stream and monitor JMX metrics from within java runtime containers.

NOTE: JESL provides a way to stream events generated by Stream-JMX to jKool Cloud. For more information on JESL visit: http://nastel.github.io/JESL/

Using Stream-JMX

It is simple: (1) run Stream-JMX as a -javaagent or (2) imbed Stream-JMX code into your application.

Running Stream-JMX as javaagent:

java -javaagent:tnt4j-stream-jmx.jar="*:*!30000" -Dtnt4j.config=tnt4j.properties -classpath "tnt4j-stream-jmx.jar;lib/tnt4j-api-final-all.jar" your.class.name your-args

Imbed Stream-JMX code into your application:

// obtain SamplerFactory instance
SamplerFactory factory = DefaultSamplerFactory.getInstance();
// create an instance of the sampler that will sample mbeans
Sampler sampler = factory.newInstance();
// schedule collection (ping) for given MBean filter and 30000 ms sampling period
sampler.setSchedule(Sampler.JMX_FILTER_ALL, 30000).run();

NOTE: setSchedule(..).run() sequence must be called to run the schedule. setSchedule(..) just sets the scheduling parameters, run() executes the schedule.

To schedule metric collection for a specific MBean server:

// obtain SamplerFactory instance
SamplerFactory factory = DefaultSamplerFactory.getInstance();
// create an instance of the sampler that will sample mbeans
Sampler sampler = factory.newInstance(ManagementFactory.getPlatformMBeanServer());
// schedule collection (ping) for given MBean filter and 30000 ms sampling period
sampler.setSchedule(Sampler.JMX_FILTER_ALL, 30000).run();

Stream-JMX supports inclusion and exclusion filters. To schedule metric collection for a specific MBean server and exclude certain MBeans: (Exclusion filters are applied after inclusion filters)

// obtain SamplerFactory instance
SamplerFactory factory = DefaultSamplerFactory.getInstance();
// create an instance of the sampler that will sample mbeans
Sampler sampler = factory.newInstance(ManagementFactory.getPlatformMBeanServer());
String excludeMBeanFilter = "mydomain:*";
// schedule collection (ping) for given MBean filter and 30000 ms sampling period
sampler.setSchedule(Sampler.JMX_FILTER_ALL, excludeMBeanFilter, 30000).run();

Below is an example of how to sample all registered mbean servers:

// obtain SamplerFactory instance
SamplerFactory factory = DefaultSamplerFactory.getInstance();
// find other registered mbean servers
ArrayList<MBeanServer> mlist = MBeanServerFactory.findMBeanServer(null);
for (MBeanServer server: mlist) {
	Sampler jmxp = factory.newInstance(server);
	jmxp.setSchedule(Sampler.JMX_FILTER_ALL, 30000).run();
}

Alternatively, Stream-JMX provides a helper class SamplingAgent that lets you schedule sampling for all registered MBeanServer instances.

SamplingAgent.sample(Sampler.JMX_FILTER_ALL, Sampler.JMX_FILTER_NONE, 60000, TimeUnit.MILLISECONDS);

NOTE: Sampled MBean attributes and associated values are stored in a collection of Snapshot objects stored within Activity instance. Current Activity instance can be obtained via AttributeSample passed when calling listeners such as AttributeCondition, SampleListener. Snapshots can be accessed using Activity.getSnapshots() method call.

NOTE: Sampled output is written to underlying tnt4j event sink configured in tnt4j.properties file. Sink destinations could be a file, socket, log4j, user defined event sink implementations.

For more information on TNT4J and tnt4j.properties see (https://github.com/Nastel/TNT4J/wiki/Getting-Started).

Running Stream-JMX as standalone app

Example below runs SamplingAgent helper class as a standalone java application with a given MBean filter "*:*", sampling period in milliseconds (10000), and time to run in milliseconds (60000):

java -Dorg.tnt4j.stream.jmx.agent.trace=true -classpath "tnt4j-stream-jmx.jar;lib/tnt4j-api-final-all.jar" org.tnt4j.stream.jmx.SamplingAgent "*:*" none 10000 60000

Running Stream-JMX as -javaagent

Stream-JMX can be invoked as -javaagent without changing your application code:

java -javaagent:tnt4j-stream-jmx.jar="*:*!30000" -Dtnt4j.config=tnt4j.properties -classpath "tnt4j-stream-jmx.jar;lib/tnt4j-api-final-all.jar" your.class.name your-args

The options are -javaagent:tnt4j-stream-jmx.jar="mbean-filter!sample-time-ms", classpath must include pingjmx jar files as well as locations of log4j and tnt4j configuration files.

Where do the streams go?

Stream-JMX streams all collected metrics based on a scheduled interval via TNT4J event streaming framework. All streams are written into TNT4J event sinks defined in tnt4j.properties file which is defined by -Dtnt4j.config=tnt4j.properties property.

To stream Stream-JMX to jkool cloud (https://www.jkoolcloud.com): (Requires JESL libraries (see https://github.com/Nastel/JESL))

;Stanza used for Stream-JMX sources
{
	source: org.tnt4j.stream.jmx
	source.factory: com.nastel.jkool.tnt4j.source.SourceFactoryImpl
	source.factory.GEOADDR: New York
	source.factory.DATACENTER: YourDC
	source.factory.RootFQN: SERVER=?#DATACENTER=?#GEOADDR=?	
	source.factory.RootSSN: tnt4j-stream-jmx	
	
	tracker.factory: com.nastel.jkool.tnt4j.tracker.DefaultTrackerFactory
	dump.sink.factory: com.nastel.jkool.tnt4j.dump.DefaultDumpSinkFactory

	; Event sink definition where all streams are recorded
	event.sink.factory: com.nastel.jkool.tnt4j.sink.BufferedEventSinkFactory

	; Event Sink configuration for streaming to jKool Cloud
	; event.sink.factory.EventSinkFactory.Filename: jkoocloud.json
	event.sink.factory.EventSinkFactory.Url: https://data.jkoolcloud.com:6585
	event.sink.factory.EventSinkFactory.Token: ACCESS-TOKEN
	event.formatter: com.nastel.jkool.tnt4j.format.JSONFormatter

	; Configure default sink filter 
	event.sink.factory.Filter: com.nastel.jkool.tnt4j.filters.EventLevelTimeFilter
	event.sink.factory.Filter.Level: TRACE
	
	tracking.selector: com.nastel.jkool.tnt4j.selector.DefaultTrackingSelector
	tracking.selector.Repository: com.nastel.jkool.tnt4j.repository.FileTokenRepository
}

Below is an example of TNT4J stream definition where all Stream-JMX streams are written into a socket event sink com.nastel.jkool.tnt4j.sink.SocketEventSinkFactory, formatted by org.tnt4j.stream.jmx.format.FactNameValueFormatter :

;Stanza used for Stream-JMX sources
{
	source: org.tnt4j.stream.jmx
	source.factory: com.nastel.jkool.tnt4j.source.SourceFactoryImpl
	source.factory.GEOADDR: New York
	source.factory.DATACENTER: YourDC
	source.factory.RootFQN: SERVER=?#DATACENTER=?#GEOADDR=?	
	source.factory.RootSSN: tnt4j-stream-jmx	
	
	tracker.factory: com.nastel.jkool.tnt4j.tracker.DefaultTrackerFactory
	dump.sink.factory: com.nastel.jkool.tnt4j.dump.DefaultDumpSinkFactory

	; Event sink definition where all streams are recorded

	event.sink.factory: com.nastel.jkool.tnt4j.sink.BufferedEventSinkFactory
	event.sink.factory.EventSinkFactory: com.nastel.jkool.tnt4j.sink.SocketEventSinkFactory
	event.sink.factory.EventSinkFactory.eventSinkFactory: com.nastel.jkool.tnt4j.sink.NullEventSinkFactory
	event.sink.factory.EventSinkFactory.Host: localhost
	event.sink.factory.EventSinkFactory.Port: 6060

	; Configure default sink filter 
	event.sink.factory.Filter: com.nastel.jkool.tnt4j.filters.EventLevelTimeFilter
	event.sink.factory.Filter.Level: TRACE
	
	event.formatter: org.tnt4j.stream.jmx.format.FactNameValueFormatter
	tracking.selector: com.nastel.jkool.tnt4j.selector.DefaultTrackingSelector
	tracking.selector.Repository: com.nastel.jkool.tnt4j.repository.FileTokenRepository
}

To stream Stream-JMX into a log file MyStream.log:

;Stanza used for Stream-JMX sources
{
	source: org.tnt4j.stream.jmx
	source.factory: com.nastel.jkool.tnt4j.source.SourceFactoryImpl
	source.factory.GEOADDR: New York
	source.factory.DATACENTER: YourDC
	source.factory.RootFQN: SERVER=?#DATACENTER=?#GEOADDR=?	
	source.factory.RootSSN: tnt4j-stream-jmx	
	
	tracker.factory: com.nastel.jkool.tnt4j.tracker.DefaultTrackerFactory
	dump.sink.factory: com.nastel.jkool.tnt4j.dump.DefaultDumpSinkFactory

	; Event sink definition where all streams are recorded
	event.sink.factory: com.nastel.jkool.tnt4j.sink.BufferedEventSinkFactory
	event.sink.factory.EventSinkFactory: com.nastel.jkool.tnt4j.sink.FileEventSinkFactory
	event.sink.factory.EventSinkFactory.FileName: MyStream.log

	; Configure default sink filter 
	event.sink.factory.Filter: com.nastel.jkool.tnt4j.filters.EventLevelTimeFilter
	event.sink.factory.Filter.Level: TRACE
	
	event.formatter: org.tnt4j.stream.jmx.format.FactNameValueFormatter
	tracking.selector: com.nastel.jkool.tnt4j.selector.DefaultTrackingSelector
	tracking.selector.Repository: com.nastel.jkool.tnt4j.repository.FileTokenRepository
}

You can write your own custom event sinks (HTTPS, HTTP, etc) and your own stream formatters without having to change Stream-JMX code or your application. TNT4J comes with a set of built-in event sink implementations such as:

Auto-generating application state dump

Stream-JMX is utilizing TNT4J state dump capability to generate application state dumps

(1) Dump on VM shut-down:

java -Dtnt4j.dump.on.vm.shutdown=true -Dtnt4j.dump.provider.default=true -Dtnt4j.dump.folder=./ ...

(2) Dump on uncaught thread exceptions:

java -Dtnt4j.dump.on.exceptionn=true -Dtnt4j.dump.provider.default=true -Dtnt4j.dump.folder=./ ...

-Dtnt4j.dump.folder=./ specifies the destination folder where dump (.dump) files will be created (default is current working directory).

By default Stream-JMX will generate dumps with the following info:

You may create your own dump providers and handlers (https://github.com/Nastel/TNT4J/wiki/Getting-Started#application-state-dumps)

Overriding default SamplerFactory

SamplerFactory instances are used to generate Sampler implementation for a specific runtime environment. Stream-JMX supplies sampler and ping factories for standard JVMs, JBoss, WebSphere Application Server. You may want to override default SamplerFactory with your own or an altenative by specifying:

java -Dorg.tnt4j.stream.jmx.factory=org.tnt4j.stream.jmx.PlatformSamplerFactory ...

SamplerFactory is used to generate instances of the underlying sampler implementatons (objects that provide sampling of underlying mbeans).

// return default or user defined SamplerFactory implementation
SamplerFactory factory = DefaultSamplerFactory.getInstance();
...

Managing Sample Behavior

Stream-JMX provides a way to intercept sampling events such as pre, during an post for each sample run and control sample behavior. See SampleListener interface for more details. Applications may register more than one listener per Sampler. Each listener is called in registration order.

In addition to intercepting sample events, applications may want to control how one ore more attributes are sampled and whether each sample is reported/logged. See example below:

// return default or user defined SamplerFactory implementation
SamplerFactory factory = DefaultSamplerFactory.getInstance();
// create an instance of the sampler that will sample mbeans
Sampler sampler = factory.newInstance();
sampler.setSchedule(Sampler.JMX_FILTER_ALL, 30000).addListener(new MySampleListener())).run();

Below is a sample of what MySampleListener may look like:

class MySampleListener implements SampleListener {
	@Override
    public void getStats(SampleContext context, Map<String, Object> stats) {
		// add your own stats to the map
	}
	
	@Override
    public void register(SampleContext context, ObjectName oname) {
		System.out.println("Register mbean: " + oname + ", mbean.server=" + context.getMBeanServer());
	}

	@Override
    public void unregister(SampleContext context, ObjectName oname) {
		System.out.println("Unregister mbean: " + oname + ", mbean.server=" + context.getMBeanServer());
    }

	@Override
	public void pre(SampleContext context, Activity activity) {
		// called once per sample, beginning of each sample
		// set activity to NOOP to disable further sampling
		// no other attribute will be sampled during current sample
		if (some-condition) {
			activity.setType(OpType.NOOP);
		}
	}

	@Override
	public void pre(SampleContext context, AttributeSample sample) {
		// called once before attribute is sampled
		// set exclude to true to skip sampling this attribute
		sample.excludeNext(sample.getAttributeInfo().isReadable());
	}

	@Override
	public void post(SampleContext context, AttributeSample sample) {
		// called once after attribute is sampled
		Object value = sample.get();
	}

	@Override
	public void post(SampleContext context, Activity activity) {
		// called once per sample, end of each sample
		// set activity to NOOP to disable sampling reporting
		if (some-condition) {
			activity.setType(OpType.NOOP);
		}
	}
	
	@Override
	public void error(SampleContext context, Throwable ex) {
		// called once for every exception that occurs not associated with a sample
		ex.printStackTrace();
	}
	
	@Override
	public void error(SampleContext context, AttributeSample sample) {
		// called once for every exception that occurs during each sample
		Throwable ex = sample.getError();
		ex.printStackTrace();
	}	
}

Conditions and Actions

Stream-JMX allows you to associate conditions with user defined actions based on values of MBean attributes on each sampling interval. For example, what if you wanted to setup an action when a specific MBean attribute exceeds a certain threshold?

Stream-JMX AttributeCondition and AttributeAction interfaces allow you to call your action at runtime every time a condition is evaluated to true. See example below:

// return default or user defined SamplerFactory implementation
SamplerFactory factory = DefaultSamplerFactory.getInstance();
// create an instance of the sampler that will sample mbeans
Sampler sampler = factory.newInstance();
// create a condition when ThreadCount > 100
AttributeCondition myCondition = new SimpleCondition("java.lang:type=Threading", "ThreadCount", 100, ">");
// schedule collection (ping) for given MBean filter and 30000 ms sampling period
sampler.setSchedule(Sampler.JMX_FILTER_ALL, 30000).register(myCondition, new MyAttributeAction()).run();

Below is a sample of what MyAttributeAction may look like:

public class MyAttributeAction implements AttributeAction {
	@Override
	public Object action(SampleContext context, AttributeCondition cond, AttributeSample sample) {
		Activity activity = sample.getActivity();
		// obtain a collection of all sampled metrics
		Collection<Snapshot> metrics = activity.getSnapshots();
		System.out.println("Myaction called with value=" + sample.get()
			+ ", age.usec=" + sample.ageUsec()
			+ ", count=" + metrics.size());
		return null;
	}
}

Project Dependencies

Stream-JMX requires the following:

Related Projects

Available Integrations