Simplifying REST API Testing with GoReplay

GoReplay is an open-source tool that simplifies REST API testing by capturing and replaying HTTP traffic. It helps replicate real-world user interactions in test environments, enabling teams to identify bugs, performance issues, and edge cases that traditional testing methods might miss. Here’s a quick overview of what you can do with GoReplay:

• Record Traffic: Capture HTTP/HTTPS traffic from production environments.
• Replay Traffic: Test APIs by replaying the recorded traffic in staging or testing environments.
• Filter and Modify Requests: Focus on specific endpoints, methods, or headers and tweak requests as needed.
• Control Replay Speed: Adjust traffic volume and speed to simulate realistic conditions.
• Send Traffic to Multiple Endpoints: Useful for A/B testing or comparing API versions.
• Ensure Data Security: Mask sensitive information like headers during recording.

Quick Setup Example:

1. Install GoReplay:

   wget https://github.com/buger/goreplay/releases/latest/download/gor_latest_x64.tar.gz
   tar -xzf gor_latest_x64.tar.gz
   sudo mv gor /usr/local/bin/
   

2. Record Traffic:

   gor --input-raw :80 --output-file="traffic.gor"
   

3. Replay Traffic:

   gor --input-file="traffic.gor" --output-http="http://staging.example.com"
   

GoReplay is especially useful for performance testing, shadow testing, and debugging API issues, all while maintaining session boundaries and handling TLS traffic effectively. It’s a practical tool for teams aiming to improve API reliability and security.

Related video from YouTube

Setting Up GoReplay

Getting started with GoReplay involves ensuring your system is prepared and configured correctly to capture and replay traffic without issues. Let’s break down the setup process.

What to Check Before Installing

Before diving into the installation, make sure your system is ready. You’ll need root or sudo access, enough storage space, and properly configured network interfaces to capture traffic. For the best results, your test environment should closely match your production setup. This will help ensure accurate traffic replay results. Once these basics are in place, you’re ready to install and configure GoReplay.

Installing and Configuring GoReplay

Installing GoReplay is quick and can be done via the command line. Use the following commands:

wget https://github.com/buger/goreplay/releases/latest/download/gor_latest_x64.tar.gz
tar -xzf gor_latest_x64.tar.gz
sudo mv gor /usr/local/bin/

After installation, you’ll need to configure the tool by setting up input and output endpoints. Input endpoints capture traffic from a source, while output endpoints specify where the replayed traffic will go. Here’s an example:

gor --input-raw :8080 --output-http="http://staging.example.com"

This command captures traffic on port 8080 and sends it to a staging server. The straightforward setup reflects the tool’s focus on ease of use, as emphasized in its documentation. Once configured, ensure GoReplay has the required permissions to function properly.

Running GoReplay with the Right Permissions

When running GoReplay, you have two main options:

• Use sudo for full access.
• Adjust capabilities to allow non-root usage.

“Exercise caution when using sudo with GoReplay, as it grants elevated privileges that may pose security risks. Start with a basic setup and gradually add more complex configurations as needed.”

For production environments, it’s better to assign specific user permissions instead of relying on full sudo access. This approach balances security with functionality.

GoReplay is particularly effective for API testing, especially when working with REST APIs. Its ability to handle TLS traffic and maintain session boundaries makes it a powerful tool for ensuring accurate and reliable tests.

Using GoReplay to Replay Traffic

With GoReplay set up, let’s dive into how you can record and replay traffic for testing REST APIs.

How to Record Traffic

To capture HTTP requests from your production environment, use the --input-raw flag with the appropriate port. For HTTPS traffic, you’ll also need to include your TLS certificates using --input-raw-certificate and --input-raw-key. Here’s how it looks:

gor --input-raw :80 --output-file="recorded_traffic.gor"

gor --input-raw :443 --input-raw-certificate cert.pem --input-raw-key key.pem --output-file="https_traffic.gor"

Once you’ve successfully recorded the traffic, you’re ready to replay it in your testing setup.

Replaying Traffic in Test Environments

To replay the recorded traffic in your test environment, use the --input-file flag to specify the file and --output-http to direct the traffic to your test server:

gor --input-file="recorded_traffic.gor" --output-http="http://staging-api.example.com"

To simulate realistic traffic without overwhelming your servers, you can tweak the replay speed using options like --output-http-stats and --output-http-worker-timeout. This helps maintain a steady flow of requests, mimicking actual usage patterns.

Saving and Reusing Traffic Data

For better organization, save your recordings in a dedicated directory and filter traffic to focus on specific API endpoints. Here’s an example:

mkdir -p /var/log/gor
gor --input-raw :80 --output-file="/var/log/gor/%Y-%m-%d.gor" --http-allow-url="/api/v1/*"

To protect sensitive information, GoReplay lets you mask headers like Authorization or Cookie during the recording process:

gor --input-raw :80 --output-file="traffic.gor" --http-hide-header "Authorization|Cookie"

These features make it easier to store and reuse traffic data securely. GoReplay also provides advanced options to fine-tune how traffic is replayed - stay tuned for more on that!

sbb-itb-6130b03

Advanced Features in GoReplay

Now that you’ve got the basics of recording and replaying traffic down, let’s dive into some of GoReplay’s more advanced capabilities. These features can help you fine-tune your testing process and make it more effective.

Filtering Traffic

With GoReplay, you can zero in on the traffic that matters most. By filtering requests, you can focus on specific data and reduce unnecessary noise. Here are a few examples:

# Capture traffic matching a specific URL pattern
gor --input-raw :80 --output-file="api_traffic.gor" --http-allow-url "/api/v2/*"

# Filter traffic by HTTP method or header
gor --input-raw :80 --output-file="post_requests.gor" --http-allow-method POST
gor --input-raw :80 --output-file="mobile_traffic.gor" --http-allow-header "User-Agent: Mobile*"

Modifying Requests

Sometimes, your test environment requires tweaks to incoming requests. GoReplay makes it easy to adjust headers or rewrite paths during replay:

# Change the Host header
gor --input-file="traffic.gor" --output-http="http://staging.example.com" --http-set-header "Host: staging.example.com"

# Rewrite request paths to match a test API
gor --input-file="traffic.gor" --output-http="http://test-api.com" --http-rewrite-url "api.production.com/v1 -> test-api.com/v1"

Controlling Traffic Volume

Need to mimic real-world traffic conditions or specific scenarios? GoReplay lets you control the speed and volume of replayed traffic:

# Replay traffic at half the original speed
gor --input-file="traffic.gor" --output-http="http://staging.example.com" --output-http-throttle 50%

# Limit the number of concurrent connections
gor --input-file="traffic.gor" --output-http="http://staging.example.com" --output-http-workers 10

Sending Traffic to Multiple Endpoints

GoReplay also supports sending the same traffic to multiple destinations. This is especially useful for A/B testing or comparing different API versions:

gor --input-file="traffic.gor" \
    --output-http="http://staging1.example.com" \
    --output-http="http://staging2.example.com" \
    --output-http="http://qa.example.com"

“At Videology, we leveraged GoReplay’s multi-endpoint capability to stream production traffic to multiple QA environments simultaneously. This allowed us to perform comprehensive soak testing and compare metrics across different configurations to identify potential issues before they reached production.”

Examples of Using GoReplay for API Testing

Here’s how you can use GoReplay to test your APIs effectively.

Testing API Performance

GoReplay lets you simulate real traffic to evaluate how well your API performs. By capturing live traffic and replaying it at different speeds, you can identify bottlenecks and test how your API handles stress.

# Capture production traffic for performance testing
gor --input-raw :8080 --output-file="peak_traffic.gor" --http-allow-url "/api/*"

# Replay traffic at various speeds to stress test the API
gor --input-file="peak_traffic.gor" --output-http="http://staging.example.com" --output-http-workers 50

This method is great for testing how your API handles peak traffic and sudden demand spikes. After stress testing, you can validate new updates in a safe environment using shadow testing.

Shadow Testing in Staging

Shadow testing involves replaying live traffic in a staging setup, allowing you to test new API versions without risking production stability. It’s a great way to catch issues early in development.

# Capture production traffic for shadow testing
gor --input-raw :80 --output-file="shadow_test.gor" --output-http="http://staging.example.com"

By mirroring production traffic, you can confidently test changes and ensure they don’t introduce new problems.

Finding and Fixing Issues

GoReplay’s ability to capture and replay specific traffic makes it an excellent tool for debugging and resolving API problems:

• Capture traffic related to the issue
• Replay it in a test environment
• Debug and fix the problem without disrupting live users

# Capture traffic related to the problematic endpoint
gor --input-raw :80 --output-file="debug.gor" --http-allow-url "/api/problematic-endpoint/*"

# Replay in debug environment with detailed logging
gor --input-file="debug.gor" --output-http="http://debug.example.com" --verbose

Replaying captured traffic lets you pinpoint and resolve issues, then verify fixes before going live. Keeping a library of these traffic patterns helps with regression testing and prevents similar problems down the road. This approach ensures your API is dependable and ready for production demands.

Tips for Using GoReplay

GoReplay makes API testing easier, but following some best practices can help you get better results and keep your data secure. Here’s how to use this tool effectively while protecting sensitive information.

Keeping Data Safe

When capturing or replaying traffic, it’s crucial to sanitize sensitive data to protect user privacy and meet security standards. GoReplay’s middleware allows you to dynamically clean or replace sensitive data patterns without affecting the original traffic flow. This approach ensures compliance with data protection rules while still providing realistic test scenarios.

Checking Replayed Traffic

To ensure your replayed requests match the original traffic, use GoReplay’s detailed logging features. This helps you confirm that your tests are reliable and accurate:

# Enable verbose logging for detailed traffic checks
gor --input-file="traffic.gor" --output-http="http://staging.example.com" --verbose --stats

“At Videology, we implemented continuous traffic verification during replay sessions, which helped us identify and fix discrepancies between production and QA environments. This approach was crucial for maintaining reliable soak testing across multiple service versions”, shares their engineering team.

Traffic Aspect	What to Monitor	Why It Matters
Data Security	Sensitive headers, user data	Prevents exposure of confidential information
Request Accuracy	Response codes, timing	Ensures tests are valid and reliable
IP Preservation	Original source IPs	Maintains realistic traffic patterns for testing

Maintaining Original IPs

After verifying traffic accuracy, ensure replayed traffic retains original attributes like source IP addresses. This helps simulate realistic conditions and diagnose region-specific issues. By default, GoReplay preserves IP addresses, but you can configure headers for added accuracy:

# Use the X-Forwarded-For header to preserve original IPs
gor --input-raw :8080 --output-http="http://staging.example.com" --http-set-header "X-Forwarded-For: {request_ip}"

Keeping original IPs intact allows you to analyze geographic traffic patterns and debug location-based problems effectively.

Conclusion

What You Learned

GoReplay is an open-source tool that gives developers a practical way to test REST APIs. It offers features like modifying requests, filtering traffic, and distributing it across multiple endpoints. These functions help tackle common testing issues while mimicking production-like environments. GoReplay also ensures session boundaries are preserved and handles TLS efficiently. Plus, its ability to replicate traffic patterns while protecting sensitive data makes it a great choice for teams focused on realistic testing and meeting security standards.

Now that you’re familiar with its features, you’re ready to start integrating GoReplay into your testing processes.

What to Do Next

Kick off your API testing journey with GoReplay’s free plan, which works well for individual developers or small teams. Start by capturing traffic and then try shadow testing and filtering to narrow your focus.

“GoReplay’s approach to using real-world traffic for testing provides more accurate and reliable results compared to synthetic testing methods”, says the GoReplay development team.

Here’s how to get started:

• Set up basic traffic capture in your development environment.
• Use shadow testing in your staging environment.
• Apply traffic filters to target specific API endpoints.
• Share the results with your team to refine your testing strategy.