Thursday, February 09, 2012

Remember to put a timeout on your heartbeat pings

I'm working a distributed piece of software which needs to be disabled when a tier is not reachable. To do that, I developed a Spring bean (a java service class) and I scheduled a method on it (with a @scheduled annotation). This method makes some heartbeat pings on each tier. The java service class contains a flag which indicates if all is OK. If an heartbeat fails, this boolean status is set to false. Of course, thanks to Spring, I easily injected this class into my web pages. If a user tries to access to the application and the health status is set to false, he is automatically redirected to a page which indicates that the service is closed. But, let's go back to pings.

I scheduled my pings to be executed each minute. That was for me a good value. But, a problem appeared during my first real tests. I realized that pings durations could be longer than one minute because the network connect() method blocked my thread. Of course, I forgot to manage short timeouts!

After a few googlings, I found that : http://blog.smartkey.co.uk/2011/09/adding-a-thread-timeout-to-methods-in-java/

This is a quite simple code. I copied it here :

public void doWork() {
//perform some long running task here...
}

public void doWorkWithTimeout(int timeoutSecs) {

//set the executor thread working
final Future future = executor.submit(new Runnable() {
public void run() {
try {
doWork();
} catch (Exception e) {
throw new RuntimeException(e);
}
}
});

//check the outcome of the executor thread and limit the time allowed for it to complete
try {
future.get(timeoutSecs, TimeUnit.SECONDS);
} catch (Exception e) {
//ExecutionException: deliverer threw exception
//TimeoutException: didn't complete within downloadTimeoutSecs
//InterruptedException: the executor thread was interrupted

//interrupts the worker thread if necessary
future.cancel(true);

log.warn("encountered problem while doing some work", e);
}
}

Now, all is good for me and my heartbeat pings.

Download Oracle JDK from command line

Found on the web. You can adapt the url with one grabbed from Oracle JDK download page. What I love here is the fact is send Oracle licence ...