add traffic mirroring as endpoint middleware

[sedkis]

User description

Description

Related Issue

Motivation and Context

How This Has Been Tested

Screenshots (if appropriate)

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Refactoring or add test (improvements in base code or adds test coverage to functionality)

Checklist

• I ensured that the documentation is up to date
• I explained why this PR updates go.mod in detail with reasoning why it's required
• I would like a code coverage CI quality gate exception and have explained why

---

PR Type

Enhancement, Tests

---

Description

• Add traffic mirroring endpoint middleware

• Define mirror config in API spec

• Compile and route mirror path specs

• Include unit tests for middleware

---

Diagram Walkthrough

flowchart LR
  Spec["API Spec: ExtendedPaths.TrafficMirror"] -- "compiled to" --> URLSpec["URLSpec.TrafficMirror"]
  URLSpec -- "status" --> Status["TrafficMirrored / StatusTrafficMirrored"]
  Loader["api_loader: register middleware"] -- "append" --> MW["TrafficMirrorMiddleware"]
  MW -- "match & mirror" --> Dest["Configured Destinations"]

Loading

File Walkthrough

	Relevant files
Enhancement	api_definitions.go Define traffic mirroring configuration schema                        apidef/api_definitions.go Add TrafficMirror to ExtendedPathsSet. Introduce TrafficMirrorMeta configuration struct. Add TrafficMirrorDestination with headers/timeout/samplerate. +19/-0    api_definition.go Compile and register traffic mirror path status                    gateway/api_definition.go Add TrafficMirrored URLStatus and StatusTrafficMirrored. Compile traffic mirror paths into URL specs. Include mirror paths in combined path list. Map status to human-readable string. +24/-0    api_loader.go Wire traffic mirror middleware into pipeline                          gateway/api_loader.go Register TrafficMirrorMiddleware in middleware chain. +1/-0      model_urlspec.go Extend URLSpec to carry mirror metadata                                    gateway/model_urlspec.go Add TrafficMirror field to URLSpec. Support mode-specific spec retrieval for mirroring. Add method matching for TrafficMirrored. +7/-0      mw_traffic_mirror.go Implement traffic mirroring middleware                                      gateway/mw_traffic_mirror.go Implement TrafficMirrorMiddleware with async/sync send. Clone requests and forward to destinations. Support sample rates, headers, timeouts. Resource cleanup and structured logging. +244/-0
Tests	mw_traffic_mirror_test.go Unit tests for traffic mirroring middleware                            gateway/mw_traffic_mirror_test.go Test enablement based on spec configuration. Test basic processing and request mirroring. Verify middleware name. +168/-0

---

[buger]

💔 The detected issue is not in one of the allowed statuses 💔

Detected Status	Open	❌
Allowed Statuses	In Dev,In Code Review,Ready for Testing,In Test,In Progress,In Review	✔️

Please ensure your jira story is in one of the allowed statuses

[github-actions]

API Changes

--- prev.txt	2025-09-11 16:39:43.682393463 +0000
+++ current.txt	2025-09-11 16:39:34.142345459 +0000
@@ -520,6 +520,7 @@
 	ValidateJSON            []ValidatePathMeta    `bson:"validate_json" json:"validate_json,omitempty"`
 	ValidateRequest         []ValidateRequestMeta `bson:"validate_request" json:"validate_request,omitempty"`
 	Internal                []InternalMeta        `bson:"internal" json:"internal,omitempty"`
+	TrafficMirror           []TrafficMirrorMeta   `bson:"traffic_mirror" json:"traffic_mirror,omitempty"`
 	GoPlugin                []GoPluginMeta        `bson:"go_plugin" json:"go_plugin,omitempty"`
 	PersistGraphQL          []PersistGraphQLMeta  `bson:"persist_graphql" json:"persist_graphql"`
 	RateLimit               []RateLimitMeta       `bson:"rate_limit" json:"rate_limit"`
@@ -1332,6 +1333,24 @@
 	Method   string `bson:"method" json:"method"`
 }
 
+type TrafficMirrorDestination struct {
+	URL        string            `bson:"url" json:"url"`
+	Headers    map[string]string `bson:"headers,omitempty" json:"headers,omitempty"`
+	Timeout    int               `bson:"timeout,omitempty" json:"timeout,omitempty"` // seconds
+	SampleRate float64           `bson:"sample_rate,omitempty" json:"sample_rate,omitempty"`
+}
+
+type TrafficMirrorMeta struct {
+	Disabled     bool                       `bson:"disabled" json:"disabled"`
+	Path         string                     `bson:"path" json:"path"`
+	Method       string                     `bson:"method" json:"method"`
+	IgnoreCase   bool                       `bson:"ignore_case" json:"ignore_case"`
+	Destinations []TrafficMirrorDestination `bson:"destinations" json:"destinations"`
+	SampleRate   float64                    `bson:"sample_rate,omitempty" json:"sample_rate,omitempty"`
+	Async        bool                       `bson:"async" json:"async"`
+	Headers      map[string]string          `bson:"headers,omitempty" json:"headers,omitempty"`
+}
+
 type TransformJQMeta struct {
 	Filter string `bson:"filter" json:"filter"`
 	Path   string `bson:"path" json:"path"`
@@ -9789,11 +9808,10 @@
     CreateIntrospectionClient creates an HTTP client for OAuth introspection
     requests.
 
-func (f *ExternalHTTPClientFactory) CreateJWKClient() (*http.Client, error)
-    CreateJWKClient creates an HTTP client specifically configured for
-    JWK endpoint requests. This method requires external services OAuth
-    configuration to be set up and will fail if not configured. Callers should
-    fall back to getJWK() function if this method returns an error.
+func (f *ExternalHTTPClientFactory) CreateJWKClient(insecureSkipVerify bool) (*http.Client, error)
+    CreateJWKClient creates an HTTP client specifically configured for JWK
+    endpoint requests. This method preserves existing SSL skip verify behavior
+    while adding proxy support.
 
 func (f *ExternalHTTPClientFactory) CreateWebhookClient() (*http.Client, error)
     CreateWebhookClient creates an HTTP client for webhook requests.
@@ -11401,6 +11419,7 @@
 	StatusGoPlugin                 RequestStatus = "Go plugin"
 	StatusPersistGraphQL           RequestStatus = "Persist GraphQL"
 	StatusRateLimit                RequestStatus = "Rate Limited"
+	StatusTrafficMirrored          RequestStatus = "Traffic Mirrored"
 )
     Statuses of the request, all are false-y except StatusOk and
     StatusOkAndIgnore
@@ -11830,6 +11849,23 @@
     ProcessRequest will run any checks on the request on the way through the
     system, return an error to have the chain fail
 
+type TrafficMirrorMiddleware struct {
+	*BaseMiddleware
+	// Has unexported fields.
+}
+    TrafficMirrorMiddleware mirrors incoming requests to configured destinations
+
+func (t *TrafficMirrorMiddleware) EnabledForSpec() bool
+
+func (t *TrafficMirrorMiddleware) Init()
+
+func (t *TrafficMirrorMiddleware) Name() string
+
+func (t *TrafficMirrorMiddleware) ProcessRequest(w http.ResponseWriter, r *http.Request, _ interface{}) (error, int)
+
+func (t *TrafficMirrorMiddleware) Unload()
+    Unload cleans up resources
+
 type TransformHeaders struct {
 	*BaseMiddleware
 }
@@ -12012,6 +12048,7 @@
 	GoPluginMeta              GoPluginMiddleware
 	PersistGraphQL            apidef.PersistGraphQLMeta
 	RateLimit                 apidef.RateLimitMeta
+	TrafficMirror             apidef.TrafficMirrorMeta
 
 	IgnoreCase bool
 	// Has unexported fields.
@@ -12048,6 +12085,7 @@
 	GoPlugin
 	PersistGraphQL
 	RateLimit
+	TrafficMirrored
 )
     Enums representing the various statuses for a VersionInfo Path match during
     a proxy request

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Request Body Reuse cloneRequest reads and resets the body but subsequent sendSingleMirror calls clone again per destination; for large bodies this duplicates memory and CPU. Consider buffering once and reusing an io.ReadSeeker or TeeReader pattern to avoid multiple full copies. func (t *TrafficMirrorMiddleware) cloneRequest(r *http.Request) (*http.Request, error) { // Read the body var bodyBytes []byte if r.Body != nil { var err error bodyBytes, err = io.ReadAll(r.Body) if err != nil { return nil, fmt.Errorf("failed to read request body: %w", err) } // Restore the original request body r.Body = io.NopCloser(bytes.NewBuffer(bodyBytes)) } // Create new request clonedReq := &http.Request{ Method: r.Method, URL: &url.URL{}, Proto: r.Proto, ProtoMajor: r.ProtoMajor, ProtoMinor: r.ProtoMinor, Header: make(http.Header), ContentLength: r.ContentLength, Host: r.Host, RemoteAddr: r.RemoteAddr, RequestURI: r.RequestURI, } // Deep copy URL *clonedReq.URL = *r.URL if r.URL.User != nil { clonedReq.URL.User = &url.Userinfo{} *clonedReq.URL.User = *r.URL.User } // Deep copy headers for k, v := range r.Header { clonedReq.Header[k] = make([]string, len(v)) copy(clonedReq.Header[k], v) } // Set body if exists if len(bodyBytes) > 0 { clonedReq.Body = io.NopCloser(bytes.NewBuffer(bodyBytes)) } return clonedReq, nil } Context Propagation Mirrored requests are sent with context.Background and a timeout; original request cancellation is not propagated. Consider deriving from the incoming request context with timeout so mirrors cancel when the client disconnects. // Create context with timeout ctx, cancel := context.WithTimeout(context.Background(), client.Timeout) defer cancel() mirrorReq = mirrorReq.WithContext(ctx) // Send the mirrored request resp, err := client.Do(mirrorReq) if err != nil { Host Header Override mirrorReq.Host is set to destination host but existing Host header from original request may remain in headers; ensure header alignment or explicitly manage the Host header to avoid mismatches in upstreams relying on it. // Set destination host mirrorReq.Host = destURL.Host // Add global headers for k, v := range spec.Headers { mirrorReq.Header.Set(k, v) } // Add destination-specific headers for k, v := range dest.Headers { mirrorReq.Header.Set(k, v) }

[github-actions]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	Fix invalid Userinfo copying Remove attempts to copy url.User because url.Userinfo is an unexported struct with no addressable fields; this code will not compile on older Go versions and is unnecessary since User can be shared. Simply assign the URL value copy which already copies User by pointer. gateway/mw_traffic_mirror.go [118-122] -// Deep copy URL +// Deep copy URL (value copy is sufficient; Userinfo pointer is copied safely) *clonedReq.URL = *r.URL -if r.URL.User != nil { - clonedReq.URL.User = &url.Userinfo{} - *clonedReq.URL.User = *r.URL.User -} Suggestion importance[1-10]: 8 __ Why: The code attempts to allocate and assign url.Userinfo, which is not constructible and will not compile; removing this fixes a real bug. This is a high-impact correctness fix directly tied to the shown lines.	Medium
	Reuse client; move timeout to context Avoid creating a new http.Client per request which leaks connections and disables connection pooling. Derive a per-request context timeout while reusing the same client and transport. gateway/mw_traffic_mirror.go [204-210] -// Create client with destination-specific timeout -client := t.client +// Reuse shared client; control deadline via context only +timeout := t.client.Timeout if dest.Timeout > 0 { - client = &http.Client{ - Timeout: time.Duration(dest.Timeout) * time.Second, - Transport: t.client.Transport, - } + timeout = time.Duration(dest.Timeout) * time.Second } +ctx, cancel := context.WithTimeout(context.Background(), timeout) +defer cancel() +mirrorReq = mirrorReq.WithContext(ctx) +resp, err := t.client.Do(mirrorReq) Suggestion importance[1-10]: 7 __ Why: Avoiding per-request http.Client creation preserves connection pooling and reduces overhead; using context timeouts maintains functionality. Correctly derived from the code and offers a meaningful efficiency and robustness gain.	Medium
	Make sampling thread-safe and seeded Use a concurrency-safe, seeded random source to avoid predictable sampling across processes and data races in high-concurrency contexts. Initialize a rand.Rand with a mutex-protected source in the middleware and use it for all sampling decisions. gateway/mw_traffic_mirror.go [68-70] -if mirrorSpec.SampleRate > 0 && rand.Float64() > mirrorSpec.SampleRate { +// In the struct: +// rnd *rand.Rand +// rndMu sync.Mutex +// +// In Init(): +// t.rnd = rand.New(rand.NewSource(time.Now().UnixNano())) +// +// Usage here: +t.rndMu.Lock() +sampled := t.rnd.Float64() +t.rndMu.Unlock() +if mirrorSpec.SampleRate > 0 && sampled > mirrorSpec.SampleRate { return nil, http.StatusOK } Suggestion importance[1-10]: 6 __ Why: Using package-level rand.Float64 in concurrent middleware can be racy and non-deterministically seeded; proposing a mutex-guarded rand.Rand improves thread safety and reproducibility. It's a moderate reliability improvement without changing behavior.	Low

[github-actions]

📦 Impact Review Snapshot

Effort	Downstream Updates	Compatibility	Docs	TL;DR
Medium	⚠️	🟢	⚠️	Traffic mirroring middleware requires schema updates in tyk-operator and documentation

## Impact Assessment

This PR adds a new traffic mirroring middleware to Tyk Gateway that allows mirroring API requests to configured destinations. The implementation is well-contained and follows existing patterns for middleware integration, but requires schema updates in downstream repositories.

The changes introduce TrafficMirrorMeta and TrafficMirrorDestination structs to the API definition schema, add URL status types, and implement the middleware functionality. The feature is additive and doesn't modify existing behavior, making it backward compatible with existing APIs.

## Required Updates

tyk-operator:

• Update CRD schema to include new trafficMirror field in extendedPaths
• Add support for the new traffic mirroring configuration in API definition conversion logic
• Update validation logic to handle the new fields
• Add examples of traffic mirroring configuration in documentation

portal:

• Update API Designer UI to expose traffic mirroring configuration options
• Add documentation for the new feature in the portal UI

tyk-charts:

• No immediate updates required as this doesn't affect configuration parameters

tyk-sink (MDCB):

• No immediate updates required as this doesn't affect the RPC protocol or data model
• Ensure that traffic mirroring configuration is properly synchronized between gateways

## Compatibility Concerns

The changes are additive and don't modify existing behavior, making them backward compatible. Existing APIs will continue to function as before, and only APIs that explicitly configure traffic mirroring will use the new functionality.

However, there's a potential compatibility issue if older versions of tyk-operator try to manage APIs with traffic mirroring configured, as they won't understand the new fields. This could lead to the traffic mirroring configuration being lost during API updates through tyk-operator.

The middleware implementation includes proper error handling and fallbacks, ensuring that if mirroring fails, it won't affect the primary request processing.

## Summary & Recommendations

• Update tyk-operator to understand and manage the new traffic mirroring configuration fields
• Add documentation for the new feature in both Tyk Gateway and tyk-operator repositories
• Consider adding examples of traffic mirroring configuration to help users understand how to use the feature
• Ensure portal UI is updated to expose the new configuration options
• Add tests in tyk-operator to verify that traffic mirroring configuration is properly preserved during API updates

---

Tip: Mention me again using /dependency <request>.
Powered by Probe AI
Tyk Gateway Dependency Impact Reviewer

[github-actions]

Error during chat processing: Error: Failed to get response from AI model during iteration 5. terminated

---

Tip: Mention me again using /connectivity <request>.
Powered by Probe AI
Connectivity Issues Reviewer Prompt for Tyk Gateway

[github-actions]

🚀 Performance Snapshot

Effort	Perf Risk	Hot Paths	Benchmarks	TL;DR
Medium	🟡	⚠️	❔	Traffic mirroring adds memory overhead and potential latency impact when not using async mode

## Performance Impact Analysis

The new traffic mirroring middleware introduces a mechanism to duplicate and forward requests to configured destinations. While the implementation includes optimizations like sampling and async processing, it has several performance implications:

1. Request body buffering: The middleware reads the entire request body into memory to clone it, which could be problematic for large payloads.

2. Deep copying overhead: Each mirrored request requires a complete deep copy of the original request, including URL, headers, and body.

3. Synchronous vs. asynchronous operation: When configured for synchronous operation, the middleware blocks the main request until all mirrors are sent, potentially increasing latency.

4. Connection pooling: The middleware properly configures connection pooling with reasonable defaults, but creates new clients for custom timeouts.

5. Goroutine usage: Each destination gets its own goroutine, which could lead to resource contention with many destinations.

## Critical Areas

• Request body buffering: The cloneRequest function reads the entire request body into memory, which could cause memory pressure with large payloads or high traffic volumes.

• Synchronous mirroring: When Async: false, the main request is blocked until all mirrors complete, directly impacting API response times.

• Multiple cloning operations: The implementation clones the request twice - once initially and again for each destination, doubling the memory overhead.

• HTTP client creation: New HTTP clients are created for destinations with custom timeouts rather than reusing a client pool with different timeout contexts.

• Goroutine management: Each destination spawns a goroutine, even in synchronous mode, which could lead to excessive goroutine creation under high load.

## Optimization Recommendations

1. Stream request bodies: Consider using a TeeReader or similar approach to avoid buffering the entire request body in memory.

2. Optimize cloning: Implement a more efficient request cloning mechanism that avoids duplicate deep copies for each destination.

3. Client pooling: Use a single HTTP client with context timeouts instead of creating new clients for different timeout values.

4. Goroutine limiting: Add a worker pool or semaphore to limit concurrent goroutines when many destinations are configured.

5. Batch processing: Consider implementing batch sending for multiple destinations to reduce overhead.

6. Monitoring: Add metrics to track mirroring performance impact, especially memory usage and latency in synchronous mode.

7. Configurable limits: Add configuration options to limit maximum request size for mirroring or maximum number of destinations.

## Summary

• The traffic mirroring middleware is well-designed with sampling and async options, but has potential performance impacts.
• Memory usage is the primary concern due to request body buffering and multiple deep copies.
• Synchronous mirroring mode can directly impact API response times and should be used cautiously.
• The implementation would benefit from optimizations around request cloning, connection pooling, and goroutine management.
• Proper monitoring should be implemented to track the performance impact in production environments.

---

Tip: Mention me again using /performance <request>.
Powered by Probe AI
Performance Impact Reviewer Prompt