GoReplay Setup for Testing Environments
Want to test your app with real production traffic without any risks? GoReplay is your answer.
Here’s what GoReplay does in simple terms:
- Captures live HTTP traffic from your production servers
- Replays that traffic to your test environments
- Lets you test new features with actual user behavior
- Doesn’t impact your production system at all
Quick Setup Guide:
- Download GoReplay binary
- Make it executable
- Start capturing traffic with:
sudo ./gor --input-raw :8000 --output-http http://staging.env
Key features you’ll get:
- Shadow testing: Test new code with real traffic
- Load testing: See if your app can handle production loads
- Traffic filtering: Focus on specific requests
- Security controls: Remove sensitive data automatically
Real Example: Videology used GoReplay to stream production traffic to QA environments for soak testing across different service versions.
Feature | Free Version | Pro Version |
---|---|---|
HTTP Traffic Capture | ✓ | ✓ |
Multiple Endpoints | ✓ | ✓ |
Binary Protocols | ✗ | ✓ |
Cloud Storage | ✗ | ✓ |
Enterprise Support | ✗ | ✓ |
This guide shows you how to set up GoReplay, capture traffic, replay it safely, and integrate it into your testing workflow. You’ll learn practical tips for performance tuning and security best practices.
Related video from YouTube
Setup and Installation
Getting GoReplay up and running is a simple process that delivers strong traffic capture tools without unnecessary hassle. Its lightweight build makes it easy to integrate without major infrastructure adjustments.
Dependencies
Before you begin, ensure your system is ready with these basic needs:
- Root or sudo access to capture network traffic
- Enough storage for saving traffic data
- Network access to both source and target environments
- A supported Linux distribution or macOS (for network capture features)
If you’re working in a Kubernetes setup, make sure you have the following:
Component | Purpose | Requirement Level |
---|---|---|
Container Runtime | Running GoReplay pods | Required |
Persistent Storage | Storing traffic data | Recommended |
Network Policies | Allowing traffic mirroring | Required |
Service Account | Pod permissions | Required |
How to Install GoReplay
Installing GoReplay is a quick process. The easiest way is to download the latest binary from its official GitHub repository.
For Linux users, here’s how to get started:
- Download the binary: Head to the official repository and grab the latest version.
- Set permissions: Make the binary executable by adjusting its permissions.
- Add to PATH: Move the binary to
/usr/local/bin
for system-wide use.
“GoReplay’s installation process is designed to be minimally invasive, requiring no changes to your production infrastructure beyond running the daemon on the same machine as your service,” according to the official documentation.
To ensure better security and performance, follow these tips:
- Use dedicated storage volumes for traffic data to avoid running out of disk space.
- Pre-configure network interfaces before starting the traffic capture.
- Set up proper permissions for the GoReplay binary to control access.
- Test the setup with a command like
--input-raw :8000 --output-stdout
to confirm everything works as expected.
GoReplay’s design makes it an easy fit for existing systems, catering to teams looking to enhance their testing workflows. It avoids unnecessary complexity, delivering a balance of straightforward setup and effective traffic capture and replay.
Using GoReplay to Record Traffic
GoReplay simplifies the process of capturing real-world user interactions directly from production systems, making it an effective tool for testing. Think of it as a more HTTP-focused version of tcpdump, with added optimizations tailored for traffic analysis.
Commands for Recording Traffic
At its core, GoReplay uses a straightforward input-output system to define where traffic is captured and where it’s sent. This makes it easy to start recording live data.
For a basic setup that captures traffic and prints it to your terminal:
sudo ./gor --input-raw :8000 --output-stdout
Want to send that captured traffic to a staging environment instead? Use:
sudo ./gor --input-raw :8000 --output-http http://staging.env
For example, Videology - a digital advertising platform - used this method to stream traffic from their production load balancers to various QA environments. This enabled them to run soak tests on real data while comparing performance metrics across environments.
Filtering and Limiting Traffic
Not all captured traffic is useful for testing, so GoReplay comes with plenty of options to filter and limit traffic. These features let you focus on the requests that matter most, ensuring your testing environment stays efficient.
Filter Type | Purpose | Example Usage |
---|---|---|
URL Path | Target specific endpoints | --input-raw-track-params api_version |
HTTP Method | Filter by request type | --http-allow-method GET |
Request Size | Control payload size | --http-max-size 1mb |
Rate Limiting | Manage traffic volume | --input-raw-limit 10% |
“GoReplay’s filtering capabilities allow us to stream a small slice of traffic from production load balancers, making it an invaluable tool for realistic testing scenarios”, says the engineering team at Videology.
Need to store your captured traffic for later processing or analysis? GoReplay has you covered with several output options:
- Stream directly to ElasticSearch for data analysis.
- Use Kafka for scalable, distributed traffic processing.
- Save traffic to Amazon S3 for long-term storage.
When handling sensitive data, it’s critical to exclude confidential information like headers or request parameters. GoReplay makes this easy by letting you disable such fields:
sudo ./gor --input-raw :8000 --output-http "http://staging.env" --http-disable-headers "Authorization,X-API-Key"
If you’re dealing with advanced setups, such as microservices that use binary protocols like Thrift or ProtocolBuffers, the Pro version extends support for these protocols, increasing its flexibility in complex architectures.
Replaying Traffic in Testing Environments
Once you’ve recorded production traffic, GoReplay makes it simple to replay that data in your testing environments. This allows teams to test new features and infrastructure changes using realistic usage patterns.
Setting Up Replay Endpoints
Configuring replay endpoints is crucial to ensuring traffic is accurately reproduced. Here’s a basic example to replay traffic to a staging environment:
sudo gor --input-file ./requests.gor --output-http http://staging.example.com
If you’re working across multiple environments, GoReplay supports replaying to multiple endpoints at once. This is especially useful for comparing how applications perform in different setups:
Replay Scenario | Command Example | Use Case |
---|---|---|
Single Endpoint | --output-http http://staging.env | Testing a single setup |
Multiple Endpoints | --output-http "http://staging1.env|http://staging2.env" | Comparing versions |
Rate-Limited Replay | --output-http "http://staging.env --rate-limit 10" | Simulating controlled load |
Advanced Replay Options
GoReplay also offers powerful customization options for advanced replay needs. You can adjust traffic to better match your test requirements, especially for environments with specific data handling rules. For example, to rewrite headers or filter URLs:
sudo gor --input-file requests.gor \
--output-http "http://staging.env" \
--http-rewrite-header "Host: staging.example.com" \
--http-allow-url ^/api/.*
“GoReplay’s filtering capabilities allow us to stream a small slice of traffic from production load balancers. It’s been a game-changer for catching performance issues early by comparing metrics across environments”, shares the engineering team at Videology.
For secure and accurate replay, GoReplay allows you to:
- Remove sensitive headers or cookies.
- Update authentication tokens to match staging or testing setups.
- Retain original client IPs to simulate actual user patterns.
- Modify request bodies to align with test environment needs.
When combined with CI/CD pipelines, traffic replay can be fully automated, ensuring every deployment is rigorously tested with real traffic before reaching production.
Adding GoReplay to Testing Processes
Incorporating GoReplay into your testing workflows can significantly improve deployment reliability by using actual production traffic patterns. Unlike synthetic tests, these patterns reflect real user behavior, offering better validation.
Shadow Testing
With GoReplay, shadow testing becomes straightforward. It lets teams duplicate live traffic and send it to staging environments without affecting production. Since it listens directly on the network interface, there’s no need to make changes to your production setup.
gor --input-tcp :80 --output-http "http://staging.example.com" --http-allow-url ^/api/.* --http-disallow-url ^/admin/.*
This configuration enables:
- Testing new features against realistic user behavior
- Validating infrastructure updates in a staging environment
- Spotting potential issues before they make it to production
Testing Aspect | Without GoReplay | With GoReplay Shadow Testing |
---|---|---|
Traffic Pattern | Simulated data | Real production patterns |
Data Coverage | Limited scenarios | Diverse user flows |
Issue Detection | Basic validations | Production-like testing |
Load Testing
GoReplay shifts the way load testing is done by replacing artificial loads with real recorded traffic. You can play back these user patterns at different scales to find performance bottlenecks.
gor --input-file recorded_traffic.gor --output-http "http://staging.example.com" --rate 200%
“By implementing GoReplay in our testing pipeline, we identified performance bottlenecks that synthetic load testing missed. The ability to replay real user patterns helped us optimize our infrastructure more effectively”, says the engineering team at Speedscale, demonstrating just how impactful GoReplay can be.
For effective load testing, consider these tips:
- Start by running recorded traffic at its original rate (100%)
- Slowly increase the load to pinpoint the system’s breaking point
- Keep an eye on system metrics during replay sessions
- Use filtering to test specific API endpoints or user paths in detail
When combined with CI/CD pipelines, GoReplay allows you to automate both shadow and load tests. This ensures all deployments are tested under realistic conditions, making it easier to catch issues early and meet performance expectations.
Improving and Fixing GoReplay Setup
Getting the best out of GoReplay means nailing down its setup and tweaking key settings. Over time, teams using it have identified practical ways to fine-tune performance and build more efficient testing environments.
Tuning Performance
A key step in boosting performance is adjusting the worker configuration. Instead of relying on just one worker thread, spreading tasks across multiple workers can prevent bottlenecks and improve how much traffic the system can handle. To enable this, you can use the following example:
gor --input-raw :8080 --output-http "http://staging.example.com" --workers 4
Configuration Area | Effect | Suggested Setting |
---|---|---|
Worker Count | Balances Load | 2–4 workers for medium traffic, 4–8 for heavy loads |
Request Timeout | Improves Response Handling | 5–10 seconds for standard APIs |
Buffer Size | Manages Memory Usage | 64MB for typical use, 128MB for heavy traffic |
If your setup needs to deal with redirects, make sure you set timeouts to avoid piling up requests:
gor --input-raw :8080 --output-http "http://staging.example.com" --http-timeout 5
Security Tips
When it comes to security, protecting sensitive data during traffic replay is crucial. Thankfully, GoReplay comes with built-in tools for handling such challenges.
“After implementing GoReplay’s TLS configuration and data filtering, Videology successfully streamed production traffic to QA environments while maintaining strict security protocols. This enabled comprehensive soak testing without compromising sensitive user data”, shared the Videology engineering team.
To securely handle traffic, use TLS for encrypted input, as shown below:
gor --input-raw :443 --output-http "https://staging.example.com" --input-raw-tls-cert "/path/to/cert.pem" --input-raw-tls-key "/path/to/key.pem"
You can further safeguard sensitive data by setting up filters to block or replace sensitive information:
gor --input-raw :8080 --output-http "http://staging.example.com" --http-set-header "Authorization: FILTERED" --http-disallow-url "/admin/.*"
For teams needing even more advanced security options, GoReplay Pro adds premium features such as cloud storage integration and TCP session replication. These tools bring better testing capabilities while adhering to strict security standards.
Conclusion
GoReplay stands out as a practical tool for building test environments by capturing and replaying traffic. Its real-world value has been highlighted through successful use cases, such as Videology’s implementation for extensive soak testing across multiple QA setups.
The ability to manage HTTP traffic coupled with features for security and performance optimization makes it an effective choice for teams that need solid testing solutions. By fine-tuning worker configurations, timeout settings, and security approaches, teams can replicate production traffic while safeguarding data and maintaining system performance.
“GoReplay enabled us to stream production traffic to QA environments while maintaining strict security protocols, which was crucial for our comprehensive soak testing approach”, shared the Videology engineering team, demonstrating the tool’s utility in enterprise-level testing.
Testing Capability | Primary Benefit | Real-World Impact |
---|---|---|
Traffic Shadowing | No Production Risks | Safely tests live traffic patterns |
Load Testing | Validates Performance | Detects bottlenecks before deployment |
Security Features | Protects Data | Ensures TLS support and sensitive data filtering |
For teams working on HTTP-based applications, GoReplay provides an easy way to set up end-to-end testing environments. While alternatives like Speedscale may better suit Kubernetes-specific needs, GoReplay’s ease of use and effectiveness for general HTTP testing makes it a valuable tool. With both open-source and Pro versions available, organizations can choose the right functionality to match their testing goals.