Circuit Breaker Pattern

Comments

A wrapper around services to check on their state i.e. running/ not running

Code

interface ICircuitBreakerStateStore
{
  CircuitBreakerStateEnum State { get; }

  Exception LastException { get; }

  DateTime LastStateChangedDateUtc { get; }

  void Trip(Exception ex);

  void Reset();

  void HalfOpen();

  bool IsClosed { get; }
}

The State property indicates the current state of the circuit breaker, and will be one of the values Open, HalfOpen, or Closed as defined by theCircuitBreakerStateEnum enumeration. The IsClosed property should be true if the circuit breaker is closed, but false if it is open or half-open. TheTrip method switches the state of the circuit breaker to the open state and records the exception that caused the change in state, together with the date and time that the exception occurred. The LastException and the LastStateChangedDateUtc properties return this information. The Resetmethod closes the circuit breaker, and the HalfOpen method sets the circuit breaker to half-open.

The InMemoryCircuitBreakerStateStore class in the example contains an implementation of the ICircuitBreakerStateStore interface. TheCircuitBreaker class creates an instance of this class to hold the state of the circuit breaker.
The ExecuteAction method in the CircuitBreaker class wraps an operation (in the form of an Action delegate) that could fail. When this method runs, it first checks the state of the circuit breaker. If it is closed (the local IsOpen property, which returns true if the circuit breaker is open or half-open, is false) the ExecuteAction method attempts to invoke the Action delegate. If this operation fails, an exception handler executes the TrackExceptionmethod, which sets the state of the circuit breaker to open by calling the Trip method of the InMemoryCircuitBreakerStateStore object. The following code example highlights this flow.

public class CircuitBreaker
{
  private readonly ICircuitBreakerStateStore stateStore =
    CircuitBreakerStateStoreFactory.GetCircuitBreakerStateStore();

  private readonly object halfOpenSyncObject = new object ();
  ...
  public bool IsClosed { get { return stateStore.IsClosed; } }

  public bool IsOpen { get { return !IsClosed; } }

  public void ExecuteAction(Action action)
  {
    ...
    if (IsOpen)
    {
      // The circuit breaker is Open.
      ... (see code sample below for details)
    }

    // The circuit breaker is Closed, execute the action.
    try
    {
      action();
    }
    catch (Exception ex)
    {
      // If an exception still occurs here, simply 
      // re-trip the breaker immediately.
      this.TrackException(ex);

      // Throw the exception so that the caller can tell
      // the type of exception that was thrown.
      throw;
    }
  }

  private void TrackException(Exception ex)
  {
    // For simplicity in this example, open the circuit breaker on the first exception.
    // In reality this would be more complex. A certain type of exception, such as one
    // that indicates a service is offline, might trip the circuit breaker immediately. 
    // Alternatively it may count exceptions locally or across multiple instances and
    // use this value over time, or the exception/success ratio based on the exception
    // types, to open the circuit breaker.
    this.stateStore.Trip(ex);
  }
}

The following example shows the code (omitted from the previous example) that is executed if the circuit breaker is not closed. It first checks if the circuit breaker has been open for a period longer than the time specified by the local OpenToHalfOpenWaitTime field in the CircuitBreaker class. If this is the case, the ExecuteAction method sets the circuit breaker to half-open, then attempts to perform the operation specified by the Actiondelegate.

If the operation is successful, the circuit breaker is reset to the closed state. If the operation fails, it is tripped back to the open state and the time at which the exception occurred is updated so that the circuit breaker will wait for a further period before attempting to perform the operation again.

If the circuit breaker has only been open for a short time, less than the OpenToHalfOpenWaitTime value, the ExecuteAction method simply throws aCircuitBreakerOpenException exception and returns the error that caused the circuit breaker to transition to the open state.