Remember to put a timeout on your heartbeat pings

I'm working a distributed piece of software which needs to be disabled when a tier is not reachable. To do that, I developed a Spring bean (a java service class) and I scheduled a method on it (with a @scheduled annotation). This method makes some heartbeat pings on each tier. The java service class contains a flag which indicates if all is OK. If an heartbeat fails, this boolean status is set to false. Of course, thanks to Spring, I easily injected this class into my web pages. If a user tries to access to the application and the health status is set to false, he is automatically redirected to a page which indicates that the service is closed. But, let's go back to pings.

I scheduled my pings to be executed each minute. That was for me a good value. But, a problem appeared during my first real tests. I realized that pings durations could be longer than one minute because the network connect() method blocked my thread. Of course, I forgot to manage short timeouts!

After a few googlings, I found that : http://blog.smartkey.co.uk/2011/09/adding-a-thread-timeout-to-methods-in-java/

This is a quite simple code. I copied it here :

public void doWork() {
        //perform some long running task here...
    }

    public void doWorkWithTimeout(int timeoutSecs) {

        //set the executor thread working
        final Future future = executor.submit(new Runnable() {
            public void run() {
                try {
                    doWork();
                } catch (Exception e) {
                    throw new RuntimeException(e);
                }
            }
        });

        //check the outcome of the executor thread and limit the time allowed for it to complete
        try {
            future.get(timeoutSecs, TimeUnit.SECONDS);
        } catch (Exception e) {
            //ExecutionException: deliverer threw exception
            //TimeoutException: didn't complete within downloadTimeoutSecs
            //InterruptedException: the executor thread was interrupted

            //interrupts the worker thread if necessary
            future.cancel(true);

            log.warn("encountered problem while doing some work", e);
        }
    }

Now, all is good for me and my heartbeat pings.