---
name: resilient-api-clients
description: Resilient API integration patterns with exponential backoff, circuit breakers, idempotency, and webhook verification.
---

# Resilient API Clients & Webhook Integrations

Architectural patterns and production-ready implementations for robust external API communication, rate-limit handling, idempotency, and webhook security.

## When to Use

- Calling third-party REST/GraphQL services (weather, routing, payment, SMS, notification APIs) that may experience intermittent downtime or rate limiting.
- Preventing duplicate side-effects (double charges, duplicate emails) using idempotency keys.
- Building resilient webhook receivers (authenticating signatures, protecting against replay attacks).
- Implementing circuit breaker patterns to prevent cascading failures across microservices.

---

## 1. Retry with Exponential Backoff & Jitter

Never retry immediately in a tight loop. Always back off exponentially and add random jitter to prevent "thundering herd" spikes:

### In Python (with `httpx` or `requests`)

```python
import time
import random
import httpx

def fetch_with_retry(url: str, headers: dict = None, max_retries: int = 4, base_delay: float = 0.5):
    attempt = 0
    while True:
        try:
            with httpx.Client(timeout=10.0) as client:
                response = client.get(url, headers=headers)
                
                # Check for rate-limiting
                if response.status_code == 429:
                    retry_after = response.headers.get("Retry-After")
                    sleep_time = float(retry_after) if retry_after else (base_delay * (2 ** attempt)) + random.uniform(0.1, 0.5)
                elif response.status_code >= 500:
                    response.raise_for_status()
                else:
                    return response

        except (httpx.RequestError, httpx.HTTPStatusError) as exc:
            attempt += 1
            if attempt > max_retries:
                raise RuntimeError(f"Max retries exceeded for {url}: {exc}") from exc
            sleep_time = (base_delay * (2 ** (attempt - 1))) + random.uniform(0.1, 0.5)
            time.sleep(sleep_time)
```

---

## 2. Idempotency Keys Pattern

For any POST/PUT operation that modifies state or triggers money transactions:

```python
import uuid
import httpx

def submit_booking(booking_payload: dict, client_token: str):
    # Derive deterministic idempotency key or generate unique UUID
    idempotency_key = f"book_{client_token}_{hash(frozenset(booking_payload.items()))}"
    
    headers = {
        "Idempotency-Key": idempotency_key,
        "Content-Type": "application/json"
    }
    
    # Safe to retry multiple times without creating duplicate bookings
    return fetch_with_retry(
        "https://api.example.com/v1/bookings",
        headers=headers
    )
```

---

## 3. Webhook Signature Verification (HMAC SHA-256)

Always verify cryptographic signatures before processing webhook payloads:

```python
import hmac
import hashlib
import time

def verify_webhook_signature(payload_bytes: bytes, signature_header: str, secret: str, tolerance_seconds: int = 300) -> bool:
    # Example format: t=1730446800,v1=abcdef123456...
    parts = dict(item.split("=") for item in signature_header.split(","))
    timestamp = int(parts.get("t", 0))
    expected_sig = parts.get("v1", "")
    
    # Replay attack protection
    if abs(time.time() - timestamp) > tolerance_seconds:
        return False
        
    signed_payload = f"{timestamp}.".encode("utf-8") + payload_bytes
    computed_sig = hmac.new(secret.encode("utf-8"), signed_payload, hashlib.sha256).hexdigest()
    
    # Constant-time comparison prevents timing attacks
    return hmac.compare_digest(computed_sig, expected_sig)
```

---

## 4. Circuit Breaker Pattern

When a remote service goes completely down, the circuit breaker stops sending requests and fails immediately, giving the service time to recover:

```
[CLOSED: Normal Traffic] 
    --> (On 5 Consecutive Failures) --> [OPEN: Fail Fast, No Remote Requests]
    --> (After Cooldown 30s) --> [HALF-OPEN: Test Single Request]
    --> (Success) --> [CLOSED]
```

### Best Practices:
- Always define clear fallback behavior (e.g. return cached data or a graceful empty list).
- Set explicit connect and read timeouts (e.g., connect: 3s, read: 10s).
- Never log authorization tokens or webhook secrets in error messages.