Stream-JMX
A Lightweight framework to stream and monitor JMX metrics. (Formerly known as PingJMX)
Stream JMX metrics to:
- Central monitoring server
- File, socket, log4j
- User defined destination
These metrics can be used to monitor health, performance and availability of your JVMs and applications. Use Stream-JMX to embed a monitoring agent within your application and monitor memory, GC activity, CPU as well as user defined MBeans.
Here is what you can do with Stream-JMX:
- Periodic JVM heartbeat
- Monitor memory utilization, GC activity, memory leaks
- High/Low, normal vs. abnormal CPU usage
- Monitor threading, runtime and other JVM performance metrics
- Monitor standard and custom MBean attributes
- Conditional actions based on MBean attribute values
- Conditional streaming based on custom filters
- Application state dumps on VM shut-down for diagnostics
Why Stream-JMX
Stream-JMX provides and easy, lightweight and secure way to stream and monitor JMX metrics from within java runtime containers.
- Stream JMX metrics out of the JVM container (vs. polling from outside/remote)
- Makes it easy to monitor farms of JMVs, application servers
- Reduce cyber security risk: No need to enable remote JMX, SSL, security, ports, firewalls
- Integration with monitoring tools for alerting, pro-active monitoring (AutoPilot M6)
- Integration with cloud analytics tools (https://www.jkoolcloud.com via JESL)
- Integration with log4j, slf4j, jkoocloud (via TNT4J event sinks)
- Embedded application state dump framework for diagnostics
- Easily build your own extensions, monitors
NOTE: JESL provides a way to stream events generated by Stream-JMX to jKool Cloud. For more information on JESL visit: http://nastel.github.io/JESL/
Using Stream-JMX
It is simple: (1) run Stream-JMX as a -javaagent
or (2) imbed Stream-JMX code into your application.
Running Stream-JMX as javaagent:
java -javaagent:tnt4j-stream-jmx.jar="*:*!30000" -Dtnt4j.config=tnt4j.properties -classpath "tnt4j-stream-jmx.jar;lib/tnt4j-api-final-all.jar" your.class.name your-args
Imbed Stream-JMX code into your application:
// obtain SamplerFactory instance
SamplerFactory factory = DefaultSamplerFactory.getInstance();
// create an instance of the sampler that will sample mbeans
Sampler sampler = factory.newInstance();
// schedule collection (ping) for given MBean filter and 30000 ms sampling period
sampler.setSchedule(Sampler.JMX_FILTER_ALL, 30000).run();
NOTE: setSchedule(..).run()
sequence must be called to run the schedule. setSchedule(..)
just sets the
scheduling parameters, run()
executes the schedule.
To schedule metric collection for a specific MBean server:
// obtain SamplerFactory instance
SamplerFactory factory = DefaultSamplerFactory.getInstance();
// create an instance of the sampler that will sample mbeans
Sampler sampler = factory.newInstance(ManagementFactory.getPlatformMBeanServer());
// schedule collection (ping) for given MBean filter and 30000 ms sampling period
sampler.setSchedule(Sampler.JMX_FILTER_ALL, 30000).run();
Stream-JMX supports inclusion and exclusion filters. To schedule metric collection for a specific MBean server and exclude certain MBeans: (Exclusion filters are applied after inclusion filters)
// obtain SamplerFactory instance
SamplerFactory factory = DefaultSamplerFactory.getInstance();
// create an instance of the sampler that will sample mbeans
Sampler sampler = factory.newInstance(ManagementFactory.getPlatformMBeanServer());
String excludeMBeanFilter = "mydomain:*";
// schedule collection (ping) for given MBean filter and 30000 ms sampling period
sampler.setSchedule(Sampler.JMX_FILTER_ALL, excludeMBeanFilter, 30000).run();
Below is an example of how to sample all registered mbean servers:
// obtain SamplerFactory instance
SamplerFactory factory = DefaultSamplerFactory.getInstance();
// find other registered mbean servers
ArrayList<MBeanServer> mlist = MBeanServerFactory.findMBeanServer(null);
for (MBeanServer server: mlist) {
Sampler jmxp = factory.newInstance(server);
jmxp.setSchedule(Sampler.JMX_FILTER_ALL, 30000).run();
}
Alternatively, Stream-JMX provides a helper class SamplingAgent
that lets you schedule sampling for all registered MBeanServer
instances.
SamplingAgent.sample(Sampler.JMX_FILTER_ALL, Sampler.JMX_FILTER_NONE, 60000, TimeUnit.MILLISECONDS);
NOTE: Sampled MBean attributes and associated values are stored in a collection of Snapshot
objects stored within Activity
instance. Current Activity
instance can be obtained via AttributeSample
passed when calling listeners such as AttributeCondition
, SampleListener
. Snapshots can be accessed using Activity.getSnapshots()
method call.
NOTE: Sampled output is written to underlying tnt4j event sink configured in tnt4j.properties
file. Sink destinations could be a file, socket, log4j, user defined event sink implementations.
For more information on TNT4J and tnt4j.properties
see (https://github.com/Nastel/TNT4J/wiki/Getting-Started).
Running Stream-JMX as standalone app
Example below runs SamplingAgent
helper class as a standalone java application with a given MBean filter "*:*"
, sampling period in milliseconds (10000
), and time to run in milliseconds (60000
):
java -Dorg.tnt4j.stream.jmx.agent.trace=true -classpath "tnt4j-stream-jmx.jar;lib/tnt4j-api-final-all.jar" org.tnt4j.stream.jmx.SamplingAgent "*:*" none 10000 60000
Running Stream-JMX as -javaagent
Stream-JMX can be invoked as -javaagent
without changing your application code:
java -javaagent:tnt4j-stream-jmx.jar="*:*!30000" -Dtnt4j.config=tnt4j.properties -classpath "tnt4j-stream-jmx.jar;lib/tnt4j-api-final-all.jar" your.class.name your-args
The options are -javaagent:tnt4j-stream-jmx.jar="mbean-filter!sample-time-ms"
, classpath must include pingjmx jar files as well as locations of log4j and tnt4j configuration files.
Where do the streams go?
Stream-JMX streams all collected metrics based on a scheduled interval via TNT4J event streaming framework.
All streams are written into TNT4J event sinks defined in tnt4j.properties
file which is defined by -Dtnt4j.config=tnt4j.properties
property.
To stream Stream-JMX to jkool cloud (https://www.jkoolcloud.com): (Requires JESL libraries (see https://github.com/Nastel/JESL))
;Stanza used for Stream-JMX sources
{
source: org.tnt4j.stream.jmx
source.factory: com.nastel.jkool.tnt4j.source.SourceFactoryImpl
source.factory.GEOADDR: New York
source.factory.DATACENTER: YourDC
source.factory.RootFQN: SERVER=?#DATACENTER=?#GEOADDR=?
source.factory.RootSSN: tnt4j-stream-jmx
tracker.factory: com.nastel.jkool.tnt4j.tracker.DefaultTrackerFactory
dump.sink.factory: com.nastel.jkool.tnt4j.dump.DefaultDumpSinkFactory
; Event sink definition where all streams are recorded
event.sink.factory: com.nastel.jkool.tnt4j.sink.BufferedEventSinkFactory
; Event Sink configuration for streaming to jKool Cloud
; event.sink.factory.EventSinkFactory.Filename: jkoocloud.json
event.sink.factory.EventSinkFactory.Url: https://data.jkoolcloud.com:6585
event.sink.factory.EventSinkFactory.Token: ACCESS-TOKEN
event.formatter: com.nastel.jkool.tnt4j.format.JSONFormatter
; Configure default sink filter
event.sink.factory.Filter: com.nastel.jkool.tnt4j.filters.EventLevelTimeFilter
event.sink.factory.Filter.Level: TRACE
tracking.selector: com.nastel.jkool.tnt4j.selector.DefaultTrackingSelector
tracking.selector.Repository: com.nastel.jkool.tnt4j.repository.FileTokenRepository
}
Below is an example of TNT4J stream definition where all Stream-JMX streams are written into a socket event sink
com.nastel.jkool.tnt4j.sink.SocketEventSinkFactory
, formatted by org.tnt4j.stream.jmx.format.FactNameValueFormatter
:
;Stanza used for Stream-JMX sources
{
source: org.tnt4j.stream.jmx
source.factory: com.nastel.jkool.tnt4j.source.SourceFactoryImpl
source.factory.GEOADDR: New York
source.factory.DATACENTER: YourDC
source.factory.RootFQN: SERVER=?#DATACENTER=?#GEOADDR=?
source.factory.RootSSN: tnt4j-stream-jmx
tracker.factory: com.nastel.jkool.tnt4j.tracker.DefaultTrackerFactory
dump.sink.factory: com.nastel.jkool.tnt4j.dump.DefaultDumpSinkFactory
; Event sink definition where all streams are recorded
event.sink.factory: com.nastel.jkool.tnt4j.sink.BufferedEventSinkFactory
event.sink.factory.EventSinkFactory: com.nastel.jkool.tnt4j.sink.SocketEventSinkFactory
event.sink.factory.EventSinkFactory.eventSinkFactory: com.nastel.jkool.tnt4j.sink.NullEventSinkFactory
event.sink.factory.EventSinkFactory.Host: localhost
event.sink.factory.EventSinkFactory.Port: 6060
; Configure default sink filter
event.sink.factory.Filter: com.nastel.jkool.tnt4j.filters.EventLevelTimeFilter
event.sink.factory.Filter.Level: TRACE
event.formatter: org.tnt4j.stream.jmx.format.FactNameValueFormatter
tracking.selector: com.nastel.jkool.tnt4j.selector.DefaultTrackingSelector
tracking.selector.Repository: com.nastel.jkool.tnt4j.repository.FileTokenRepository
}
To stream Stream-JMX into a log file MyStream.log
:
;Stanza used for Stream-JMX sources
{
source: org.tnt4j.stream.jmx
source.factory: com.nastel.jkool.tnt4j.source.SourceFactoryImpl
source.factory.GEOADDR: New York
source.factory.DATACENTER: YourDC
source.factory.RootFQN: SERVER=?#DATACENTER=?#GEOADDR=?
source.factory.RootSSN: tnt4j-stream-jmx
tracker.factory: com.nastel.jkool.tnt4j.tracker.DefaultTrackerFactory
dump.sink.factory: com.nastel.jkool.tnt4j.dump.DefaultDumpSinkFactory
; Event sink definition where all streams are recorded
event.sink.factory: com.nastel.jkool.tnt4j.sink.BufferedEventSinkFactory
event.sink.factory.EventSinkFactory: com.nastel.jkool.tnt4j.sink.FileEventSinkFactory
event.sink.factory.EventSinkFactory.FileName: MyStream.log
; Configure default sink filter
event.sink.factory.Filter: com.nastel.jkool.tnt4j.filters.EventLevelTimeFilter
event.sink.factory.Filter.Level: TRACE
event.formatter: org.tnt4j.stream.jmx.format.FactNameValueFormatter
tracking.selector: com.nastel.jkool.tnt4j.selector.DefaultTrackingSelector
tracking.selector.Repository: com.nastel.jkool.tnt4j.repository.FileTokenRepository
}
You can write your own custom event sinks (HTTPS, HTTP, etc) and your own stream formatters without having to change Stream-JMX code or your application. TNT4J comes with a set of built-in event sink implementations such as:
-
com.nastel.jkool.tnt4j.logger.Log4JEventSinkFactory
– log4j -
com.nastel.jkool.tnt4j.sink.BufferedEventSinkFactory
– buffered sink -
com.nastel.jkool.tnt4j.sink.FileEventSinkFactory
- standard log file -
com.nastel.jkool.tnt4j.sink.SocketEventSinkFactory
– socket (tcp/ip) -
com.nastel.jkool.tnt4j.sink.NullEventSinkFactory
– null (empty)
Auto-generating application state dump
Stream-JMX is utilizing TNT4J state dump capability to generate application state dumps
(1) Dump on VM shut-down:
java -Dtnt4j.dump.on.vm.shutdown=true -Dtnt4j.dump.provider.default=true -Dtnt4j.dump.folder=./ ...
(2) Dump on uncaught thread exceptions:
java -Dtnt4j.dump.on.exceptionn=true -Dtnt4j.dump.provider.default=true -Dtnt4j.dump.folder=./ ...
-Dtnt4j.dump.folder=./
specifies the destination folder where dump (.dump) files will be created (default is current working directory).
By default Stream-JMX will generate dumps with the following info:
- Java Properties Dump –
PropertiesDumpProvider
- Java Runtime Dump –
MXBeanDumpProvider
- Thread Stack Dump –
ThreadDumpProvider
- Thread Deadlock Dump –
ThreadDumpProvider
- Logging Statistics Dump –
LoggerDumpProvider
You may create your own dump providers and handlers (https://github.com/Nastel/TNT4J/wiki/Getting-Started#application-state-dumps)
Overriding default SamplerFactory
SamplerFactory
instances are used to generate Sampler
implementation for a specific runtime environment. Stream-JMX supplies sampler and ping factories for standard JVMs, JBoss,
WebSphere Application Server. You may want to override default SamplerFactory
with your own or an altenative by specifying:
java -Dorg.tnt4j.stream.jmx.factory=org.tnt4j.stream.jmx.PlatformSamplerFactory ...
SamplerFactory
is used to generate instances of the underlying sampler implementatons (objects that provide sampling of underlying mbeans).
// return default or user defined SamplerFactory implementation
SamplerFactory factory = DefaultSamplerFactory.getInstance();
...
Managing Sample Behavior
Stream-JMX provides a way to intercept sampling events such as pre, during an post for each sample run and control sample behavior. See SampleListener
interface for more details. Applications may register more than one listener per Sampler
. Each listener is called in registration order.
In addition to intercepting sample events, applications may want to control how one ore more attributes are sampled and whether each sample is reported/logged. See example below:
// return default or user defined SamplerFactory implementation
SamplerFactory factory = DefaultSamplerFactory.getInstance();
// create an instance of the sampler that will sample mbeans
Sampler sampler = factory.newInstance();
sampler.setSchedule(Sampler.JMX_FILTER_ALL, 30000).addListener(new MySampleListener())).run();
Below is a sample of what MySampleListener
may look like:
class MySampleListener implements SampleListener {
@Override
public void getStats(SampleContext context, Map<String, Object> stats) {
// add your own stats to the map
}
@Override
public void register(SampleContext context, ObjectName oname) {
System.out.println("Register mbean: " + oname + ", mbean.server=" + context.getMBeanServer());
}
@Override
public void unregister(SampleContext context, ObjectName oname) {
System.out.println("Unregister mbean: " + oname + ", mbean.server=" + context.getMBeanServer());
}
@Override
public void pre(SampleContext context, Activity activity) {
// called once per sample, beginning of each sample
// set activity to NOOP to disable further sampling
// no other attribute will be sampled during current sample
if (some-condition) {
activity.setType(OpType.NOOP);
}
}
@Override
public void pre(SampleContext context, AttributeSample sample) {
// called once before attribute is sampled
// set exclude to true to skip sampling this attribute
sample.excludeNext(sample.getAttributeInfo().isReadable());
}
@Override
public void post(SampleContext context, AttributeSample sample) {
// called once after attribute is sampled
Object value = sample.get();
}
@Override
public void post(SampleContext context, Activity activity) {
// called once per sample, end of each sample
// set activity to NOOP to disable sampling reporting
if (some-condition) {
activity.setType(OpType.NOOP);
}
}
@Override
public void error(SampleContext context, Throwable ex) {
// called once for every exception that occurs not associated with a sample
ex.printStackTrace();
}
@Override
public void error(SampleContext context, AttributeSample sample) {
// called once for every exception that occurs during each sample
Throwable ex = sample.getError();
ex.printStackTrace();
}
}
Conditions and Actions
Stream-JMX allows you to associate conditions with user defined actions based on values of MBean attributes on each sampling interval. For example, what if you wanted to setup an action when a specific MBean attribute exceeds a certain threshold?
Stream-JMX AttributeCondition
and AttributeAction
interfaces allow you to call your action at runtime every time a condition is evaluated to true. See example below:
// return default or user defined SamplerFactory implementation
SamplerFactory factory = DefaultSamplerFactory.getInstance();
// create an instance of the sampler that will sample mbeans
Sampler sampler = factory.newInstance();
// create a condition when ThreadCount > 100
AttributeCondition myCondition = new SimpleCondition("java.lang:type=Threading", "ThreadCount", 100, ">");
// schedule collection (ping) for given MBean filter and 30000 ms sampling period
sampler.setSchedule(Sampler.JMX_FILTER_ALL, 30000).register(myCondition, new MyAttributeAction()).run();
Below is a sample of what MyAttributeAction
may look like:
public class MyAttributeAction implements AttributeAction {
@Override
public Object action(SampleContext context, AttributeCondition cond, AttributeSample sample) {
Activity activity = sample.getActivity();
// obtain a collection of all sampled metrics
Collection<Snapshot> metrics = activity.getSnapshots();
System.out.println("Myaction called with value=" + sample.get()
+ ", age.usec=" + sample.ageUsec()
+ ", count=" + metrics.size());
return null;
}
}
Project Dependencies
Stream-JMX requires the following:
- JDK 1.6+
- TNT4J (https://github.com/Nastel/TNT4J)
Related Projects
- TrackingFilter (http://nastel.github.io/TrackingFilter/)
- JESL (http://nastel.github.io/JESL/)
Available Integrations
- TNT4J (https://github.com/Nastel/TNT4J)
- jkoolcloud.com (https://www.jkoolcloud.com)
- AutoPilot M6 (http://www.nastel.com/products/autopilot-m6.html)