Go Circuit Breakers with Hystrix: Protecting Services from Failure

Modern software systems often depend on third-party services. As those dependencies grow, handling service failures becomes essential. This guide introduces the Hystrix circuit-breaker pattern in Go, from its basic behavior to an example configuration.

What is a Hystrix circuit breaker?

A circuit breaker is a design pattern that controls calls to an external service. Like an electrical breaker, it can stop requests temporarily when failures or abnormal behavior cross a configured threshold. This helps prevent one failing dependency from disrupting the wider system.

Where Hystrix came from

Netflix developed Hystrix to manage failures in services on which its systems depended. It was designed for complex environments with many service calls and was intended to help maintain system stability when some services fail.

What Hystrix does

Hystrix is a library for handling service failures. Its settings can include retries, timeouts, and thresholds for deciding when a service is failing. It also provides monitoring features for observing circuit-breaker state and service performance.

The three circuit-breaker states

• Closed: Requests pass through normally. Successful calls leave the breaker closed.
• Open: When failures exceed the configured threshold, the breaker stops requests to the service temporarily to limit further impact.
• Half-open: After a waiting period, the breaker allows a small number of test requests. If they succeed, it returns to closed; if they fail, it opens again.

Why use Hystrix?

Hystrix is intended to help limit the effects of dependency failures, improve resilience, and make circuit-breaker state and service behavior easier to monitor. These benefits depend on choosing settings that fit the system.

Using a circuit breaker in Go

Using Hystrix in Go requires installing the library and configuring the command that represents the service call.

Install the Hystrix package

Install the package with this Go command:

go get github.com/afex/hystrix-go/hystrix

Separate the server and client

The example uses two parts: a server that provides the service to protect, and a client that calls that service through Hystrix.

• Server: Create the service that needs protection.
• Client: Make requests to the server through Hystrix.

Configure a Hystrix command

The configuration example sets a timeout, a concurrency limit, a minimum request volume, a sleep window, and an error-percentage threshold:

hystrix.ConfigureCommand("my_service", hystrix.CommandConfig{
    Timeout:                1000, // milliseconds
    MaxConcurrentRequests:  10,
    RequestVolumeThreshold: 10,
    SleepWindow:            5000, // milliseconds
    ErrorPercentThreshold:  50,
})

Run and test a request

After configuration, call the server from the client and inspect the result. If the server fails, Hystrix can open the circuit and pause calls temporarily.

err := hystrix.Do("my_service", func() error {
    _, err := http.Get("http://localhost:8080/api/data")
    if err != nil { return err }
    return nil
}, nil)
if err != nil { fmt.Println("Error:", err) }

The example endpoint is http://localhost:8080/api/data.

Monitor the circuit breaker

The Hystrix Dashboard can display circuit-breaker state, request volume, and error frequency. This can help teams investigate service behavior and find problems sooner.

Adjust settings for the system

• Timeout: How long to wait for a response.
• MaxConcurrentRequests: The maximum number of simultaneous requests.
• RequestVolumeThreshold: The minimum number of requests before calculating the error percentage.
• SleepWindow: How long the breaker stays open before moving to half-open.
• ErrorPercentThreshold: The error percentage that triggers the breaker to open.

Summary and practical considerations

Hystrix monitors service calls and can open a circuit when failures exceed a threshold, then use a half-open state to test whether the service has recovered. This can help a Go application contain dependency failures, but configuration and monitoring need to match the application’s needs.

• Consider the small runtime overhead described in the source when evaluating performance.
• Configure timeouts, retries, and thresholds for the service and workload.
• Monitor circuit state and service behavior so failures can be investigated.
• Hystrix is presented here for external services such as APIs, databases, and other dependencies that may fail.
• Further study of resilience patterns can help when designing systems for failure.

For a free consultation, contact the team on LINE, or scan the QR code below.