How to remotly debug Cloud Foundry applications

When using Cloud Foundry for deploying apps, one might expect that all the test suites will run painlessly and what works in development/testing/staging will work in production. However, what if it won’t? What if it is impossible to replicate those conditions in a non-production environment? Here, remote debugging comes to the rescue.
In this blog post, I provide guidelines on how to remotely debug Ruby and Java applications deployed with Cloud Foundry.

General considerations

Remote debugging implies connecting to the application server from your local environment and setting up breakpoints, inspecting variables and diagnosing the issue that is hurting the application. This way, you don’t need to SSH into the server itself, just connect your remote debugger and start working on it.

The debugging processes for Java and Ruby apps differ a bit. I’ll start with Java and explain how to get a suitable IP/port combination (to execute a remote debugger), set up an IDE, and get the app server ready. For a Ruby app, I’ll demonstrate how to set up an IDE, modify a Gemfile and manifest.yml, and use the Pry gem as an alternative to the classical IDE debugger.

Still, there are some things you need to check on:

Make sure that you have exactly the same code version in your IDE/local box and in your remote server.

Open the necessary ports in your computer and/or router (see the next section).

If the remote debug fails, you can always inspect the logs with cf logs –recent to see what went wrong.

Make sure that you set the number of instances to 1 in your manifest.yml. If you set it to more than 1, many instances will try to connect to your debugger, which is not possible.

Install OpenSSL in your local box if you still don’t have it.

Install Cloud Foundry’s CLI tool.

Getting a suitable IP/port combination

In order to execute some remote debuggers, you need to send data from your application server to your computer. This can be a little tricky. Why? It is most likely that you are behind a router and/or firewall.

So firstly, you need to get your public IP address by going to Google and typing in the search box “public ip address.” This will return your public IP address that the application server’s JVM will be connecting to for remote debugging.

Next, you will have to open and forward an external (public) port to your computer LAN IP address and port. It is done by tapping into the configuration of your router or asking the support guys to do it for you.

Let’s say, your LAN IP address is 192.168.1.11 and the port that will be listening to connections in your computer is 9571. Assume that your public IP address is 186.201.35.89 and the public port is 29571. First thing to do is port forwarding that will move the packages that the remote application server sends to 186.201.35.89:29571 right to your computer (192.168.1.11:9571). So, the IP/port combination we are going to use for the configuration of the remote application server debugger is: 186.201.35.89:29571.

Remotely debugging Java applications

In order to remotely debug Java applications, we need to tell the JVM to establish the connection to the remote debugger, and then set up your IDE’s debugger to listen to the connections from the application.

Setting up our IDE

Since one of the most used IDEs out there for Java development is Eclipse, we will use it as a reference. The instructions are very translatable to others, such as NetBeans or JetBrain’s IDEA.

Open your project in Eclipse.

Right-click on your project, go to Debug as and pick Debug Configurations.

Create a new Remote Java Application:

Make sure your project is selected.

Choose Standard (Socket Listen) from the Connection Type drop down.

Now, set a port for the remote debugger to connect to. In our example, it is 9571.

Click Debug. 

Getting our application server ready

In CF, we do this by modifying the manifest.yml file of the application, adding the env section that will hold the JAVA_OPTS variable with the appropriate instructions.

Let’s say, this is our (very simple) manifest.yml:


applications:

  • name: my-java-application

  memory: 768M

  instances: 1

  path: my-java-application-1.0.0.BUILD-SNAPSHOT.jar

  buildpack: https://github.com/cloudfoundry/java-buildpack.git

We add the env section and the JAVA_OPTS lines:


applications:

  • name: my-java-application

  memory: 768M

  instances: 1

  path: my-java-application-1.0.0.BUILD-SNAPSHOT.jar

  buildpack: https://github.com/cloudfoundry/java-buildpack.git

env:

  JAVA_OPTS: -agentlib:jdwp=transport=dt_socket,address=<your-ip>:<your-port>

Surely, you need to change the <your-ip>:<your-port> to the actual values. So, that JAVA_OPTS line will be as follows:

JAVA_OPTS: -agentlib:jdwp=transport=dt_socket,address=186.201.35.89:29571

To complete this part, just push the changes with cf push. You need the Cloud Foundry CLI to do this.

Now, you will be able to set breakpoints in your code and watch it execute, inspect variables and all the debug features as if they were running in your local box.

Remotely debugging Ruby applications

Remote debugging Ruby applications involves adding a custom start command to the manifest.yml (or running that custom start from the CF CLI), adding a couple of gems to your application, and setting up your IDE. 

How to set up an IDE

Since the Eclipse configuration is exactly the same both for Java and Ruby applications (in case you work with Aptana), we are going to set up JetBrain’s RubyMine IDE, one of the most popular Ruby IDEs.

Open your project in RubyMine.

Go to the Run Configurations drop down in the toolbar and click on Edit Configurations.

In the Run/Debug configurations dialog, click on Add New Configuration and select Ruby remote debug:

Set the name accordingly.

In Remote Host, put the URL or IP address where the remote debugging is going to happen.

In Remote Port, type the port of the remote machine where the remote debugger will listen. Make sure this port is open in the server. In our example, let’s set it to 8080.

In Remote Root Folder, specify the root folder of your application.

In Local Port, specify the local port that the remote debugger will connect to. In our case, it is 29571.

In Local Root Folder, click “browse” and find the root folder of your application in the dialog box.

Copy the line in the “Server command” read-only field in your clipboard.

Apply the changes and click on the Debug icon in the toolbar. 

Modify your Gemfile

Simply add this line to your Gemfile to allow your application to debug remotely to an IDE:

gem ‘ruby-debug-ide’ 

Modify your manifest.yml or start the remote debugger

You have two options here. The first one is to modify the manifest.yml and add the command attribute under the applications section.

Just use the line you copied to your clipboard. For example:

rdebug-ide -d –host 0.0.0.0 –port 8080 –dispatcher-port 29571 – bin/rails s

So, your (very simple) manifest.yml will look like this:


applications:

  • name: my-rails-app

  instances: 1

  command: “rdebug-ide -d –host 0.0.0.0 –port 8080 –dispatcher-port 29571 – bin/rails s”

Do the cf push and you are ready to start remote debugging.

The other option is to simply run the custom start command from the CLI:

cf push my-rais-app -c “rdebug-ide -d –host 0.0.0.0 –port 8080 –dispatcher-port 29571 – bin/rails s”

That is all for debugging Ruby applications. In this case, a Rails one. However, if you have a pure and simple Ruby app, replace the “bin/rails s” part of the command with the filename of your Ruby script. 

Using Pry with Cloud Foundry

An alternative to the classical IDE debugger is to use the much beloved Pry gem.

First, you need to include the following line in your application’s Gemfile:

gem ‘pry-remote’

Then, in your code, you need to tell Pry to start debugging and connect to the host and port that will listen remotely:

def my_function

  binding.remote_pry(186.201.35.89, 29571)

end

After that, do the cf push to deploy and restart the application.

When the application hits the binding.remote_pry line, it will start the DRb server in port 9876 (by default, make sure it is open) and wait for client connections.

Next, go to your application root directory and do:

pry-remote -s <host> -c

where <host> is the URL or IP address of the remote application server. And there you are—remote debugging your Ruby application with Pry!

 
Conclusion
Remote debugging can be very useful in some circumstances and, as shown here, it is quite simple to remotely debug applications deployed with Cloud Foundry. Only a couple of changes in some files, cf push, and you are done.

Circuit Breaker design pattern

“Protect your software with the Circuit Breaker design pattern” http://prsm.tc/B7Fixu

Protect your software with the Circuit Breaker design pattern

Life as a software developer is definitely lived in the fast lane. After weeks and hours of cranking out the code to get the features developed, and after the builds and tests are green and QA stamps its seal of approval, the code is deployed to Production. And then the most dreaded thing happens, the deployed software fails in production in a bad sort of way. In a Murphy’s law sort of way. As the saying goes, “Anything that can go wrong, will go wrong”. But, just what if the code we write took this sort of thing into consideration? 

So, how do we take a bad thing and turn it around into something good?

Electronics to the rescue

I still remember the day when my brother and I had to change the fuse in our house due to a surge, granted I didn’t know the gravity of the situation at the time, he’s the one with the electrical savvy. The fuse was completely burnt. But it saved our TV. In electrical engineering, fuses and circuit breakers were invented for exactly this sort of thing. An overload of power can cause serious damage, from ruining electrical equipment to even setting the house on fire! A fuse contains a small filament of wire that will melt during an electrical overload, similar to a light bulb burning out, stopping the dangerous flow of current and keeping other electrical equipment and the house safe.

Fuses evolved into circuit breakers, which commonly use an electromagnet to break the circuit instead of burning it up, allowing a circuit breaker to be reset and used over and over. However, the basic premise is the same. It detects the over usage, then fails fast, without destroying everything else. 

Thinking back about it, that’s a pretty powerful concept. By actually killing the component (the fuse was literally dead), you can save serious damage. Our TV was alive thanks to the dead fuse after that surge episode. And rightfully, why can’t the same be done in software? Something bad happens and you have this component in your software that will help you fail fast. Mimic that real life behavior and we have the Circuit Breaker design pattern.  

In dealing with distributed scenarios, some failures are transient, where quick successive retries will fix the problem. But there may be some scenarios where connectivity to a critical dependency is lost and may not be restored for a while. For example, an application may lose its connection to a persistent store hosted on the cloud. In these scenarios, by shutting down your service, you can prevent further damage to the rest of your system by avoiding bad processing of data, or even worse, data loss or cascading failures.

By failing fast, it also becomes easier for Ops resources to monitor and respond. As far as they are concerned, services attempting in vain to re-establish connectivity may still appear healthy, hence not triggering alarms when in fact it should’ve. Now if you cause the service to fail completely when appropriate, the warning lights go off and Ops is now aware of the problem and can respond right away. 

The Circuit Breaker Design Pattern

It’s easy to create reusable infrastructure to enable the circuit breaker design pattern within your own systems. This is how it works:

1. Define a reusable CircuitBreaker class with Trip and Reset methods, and provide it an action to call when the circuit breaker is tripped.

2. Use the CircuitBreaker to monitor the dependency upon which your system depends. For every single failure, trip the circuit breaker, which sets it in an armed state. This is like the beginning of an electrical surge. 

3. If a subsequent attempt succeeds within a specified time window (the current has subsided) then reset the breaker, and all is well.

4. If the circuit breaker is not reset within the specified time, and exceptions continue to occur (the current continues to rise to unsafe levels) then the breaker will invoke the action you’ve provided. You can choose to fail fast (terminate the process) when the circuit breaker is tripped, or whatever other action you choose.

Sample Usage

In this example, ExternalServiceAdapter is a class that helps connects to some external dependency. There could be a web program that makes requests executing the DoStuff operation constantly. While executing, if the GetConnection method fails, it will trip the circuit breaker when an exception occurs. It resets the circuit breaker when the connection is re-established. But if the connection exceptions continue to occur, the circuit breaker will be tripped and the specified Trip action will be executed, which in this case will fail fast.

public class ExternalServiceAdapter

    private CircuitBreaker circuitBreaker;

    public ExternalServiceAdapter()

    {

        circuitBreaker = new CircuitBreaker(“CheckConnection”, /*name of your circuit breaker */

            exception => /* The action to take when the circuit breaker is tripped */

            {

                Console.WriteLine(“Circuit breaker tripped! Fail fast!”);

                // Terminate the process, skipping any pending try/finally blocks or finalizers

                Environment.FailFast(exception.Message);

            },

        3, /* Max threshold before tripping the circuit breaker */

        TimeSpan.FromSeconds(2)); /* Time to wait between each try before attempting to trip the circuit breaker */

    }

    public void DoStuff()

    {

        var externalService = GetConnection();

        externalService.DoStuff();

    }

    ConnectionDependency GetConnection()

    {

        try

        {

            var newConnection = new ConnectionDependency();

            circuitBreaker.Reset();

            return newConnection;

        }

        catch (Exception exception)

        {

            circuitBreaker.Trip(exception);

            throw;

        }

    }

Simple Implementation of the Circuit Breaker pattern.

using System;

using System.Threading;

public class CircuitBreaker 

    public CircuitBreaker(string name, /*name of the operation */

        Action<Exception> tripAction, /* action to invoke when the circuit breaker is tripped */

        int maxTimesToRetry, /* number of times to retry before tripping the circuit breaker */

        TimeSpan delayBetweenRetries /* time to wait between each retry*/) 

    {

        this.name = name;

        this.tripAction = tripAction;

        this.maxTimesToRetry = maxTimesToRetry;

        this.delayBetweenRetries = delayBetweenRetries;

        // Don’t start the timer, just as yet. Start it when the user trips the circuit breaker. 

        timer = new Timer(CircuitBreakerTripped, null, Timeout.Infinite, (int)delayBetweenRetries.TotalMilliseconds);

    }

    public void Reset()

    {

        var oldValue = Interlocked.Exchange(ref failureCount, 0);

        timer.Change(Timeout.Infinite, Timeout.Infinite);

        Console.WriteLine(“The circuit breaker for {0} is now disarmed”, name);

    }

    public void Trip(Exception ex)

    {

        lastException = ex;

        var newValue = Interlocked.Increment(ref failureCount);

        if (newValue == 1)

        {

            // Start the retry timer. 

            timer.Change(delayBetweenRetries, TimeSpan.FromMilliseconds(-1));

            // Log that the circuit breaker is triggered.

            Console.WriteLine(“The circuit breaker for {0} is now in the armed state”, name);

        }

    }

    void CircuitBreakerTripped(object state)

    {

        Console.WriteLine(“Check to see if we need to trip the circuit breaker. Retry:{0}”, failureCount);

        if (Interlocked.Increment(ref failureCount) > maxTimesToRetry)

        {

            Console.WriteLine(“The circuit breaker for {0} is now tripped. Calling specified action”, name);

            tripAction(lastException);

            return;

        }

        timer.Change(delayBetweenRetries, TimeSpan.FromMilliseconds(-1));        

    }

    readonly string name;

    readonly int maxTimesToRetry;

    long failureCount;

    readonly Action<Exception> tripAction;

    Exception lastException;

    readonly TimeSpan delayBetweenRetries;

    readonly Timer timer;

Unit Tests for the CircuitBreaker

[TestFixture]

public class CircuitBreakerTests

    [Test]

    public void When_the_circuit_breaker_is_tripped_the_trip_action_is_called_after_reaching_max_threshold()

    {

        bool circuitBreakerTripActionCalled = false;

        var connectionException = new Exception(“Something bad happened.”);

        var circuitBreaker = new CircuitBreaker(“CheckServiceConnection”, exception =>

        {

            Console.WriteLine(“Circuit breaker tripped – fail fast”);

            circuitBreakerTripActionCalled = true;

            // You would normally fail fast here in the action to faciliate the process shutdown by calling:

            // Environment.FailFast(connectionException.Message);

        }, 3, TimeSpan.FromSeconds(1));

        circuitBreaker.Trip(connectionException);

        System.Threading.Thread.Sleep(5000); 

        Assert.IsTrue(circuitBreakerTripActionCalled); 

    }

    [Test]

    public void When_the_circuit_breaker_is_reset_the_trip_action_is_not_called()

    {

        bool circuitBreakerTripActionCalled = false;

        var connectionException = new Exception(“Something bad happened.”);

        var circuitBreaker = new CircuitBreaker(“CheckServiceConnection”, exception =>

        {

            Console.WriteLine(“Circuit breaker tripped – fail fast”);

            circuitBreakerTripActionCalled = true;

            // You would normally fail fast here in the action to faciliate the process shutdown by calling:

            // Environment.FailFast(connectionException.Message);

        }, 3, TimeSpan.FromSeconds(2));

        circuitBreaker.Trip(connectionException);

        System.Threading.Thread.Sleep(1000); 

        circuitBreaker.Reset();

        Assert.False(circuitBreakerTripActionCalled);

    }

The code example above uses Console.WriteLine. Replace it with your favorite logger. 

Closing Thoughts

Circuit breakers are an essential part of the modern world, and arguably one of the most important safety devices ever invented. There’s always a good reason behind a blown fuse or a tripped circuit breaker.

Monitor your critical resources, fail fast when they don’t respond. Facilitate your Ops team to take corrective actions. 

If you’re further interested in these types of patterns, Michael T. Nygard’s Release It is a fantastic read. 

Be safe. Don’t get zapped.

About the author: Indu Alagarsamy has been programming for more than fifteen years. She is most passionate about Event Driven Architecture and Messaging. She is currently practicing this passion as a developer at Particular Software.

Cloud Foundry Summit 2015 Videos

In case you missed the conference, check these videos out:

Bosh slideshare

What is Bosh and how to configure services ?