An inventory script works against a few devices, then receives HTTP 429 when scheduled across the estate. Increasing worker count may make the problem worse. The useful question is how much work the integration asks the controller to perform, and how it handles incomplete responses.
Lab status: This is a proposed exercise, not a completed device test or client case study. Code examples require validation against your image and dependency versions. Controller/fabric platform access is not confirmed; offline samples demonstrate report logic only. My documented background is NOC/data center operations, multi-vendor Netmiko workflows and Python/Flask integration.
Offline example checks: October 20, 2026: the added helper/demo block executed with local fixtures, and related negative cases were checked offline. No live device, controller, fabric or service behavior was validated.
What does HTTP 429 tell an API client?
HTTP 429 indicates request throttling. Catalyst Center API limits vary by endpoint and software release; do not apply a limit copied from an unrelated operation. SDK behavior also varies by version and configuration. Check both the API documentation and whether your client already waits or retries.
A 429 response asks the client to slow down; it does not establish a universal requests-per-minute value. Use the endpoint/release documentation and observed response headers. Keep authentication failures and controller/server errors outside the rate-limit category.
Why can scheduled collectors create request bursts?
Several individually modest jobs can synchronize at the same minute. Count the total worker budget per controller, include retries and stagger schedules. A per-thread sleep does not cap aggregate load across all replicas.
Offline exercise: calculate a bounded retry delay
The read-only retry wrapper below handles Retry-After seconds and HTTP-date, caps attempts and rejects a required wait that exceeds the remaining budget. Tests inject a clock and sleep, so no deliberate controller overload is needed. For production, add controlled jitter to fallback delay and one shared request budget.
import random
def retry_delay(attempt, retry_after=None, budget=60):
if retry_after is not None:
try:
delay = float(retry_after)
except (TypeError, ValueError):
raise ValueError("Parse HTTP-date Retry-After separately")
if not 0 <= delay <= budget:
raise TimeoutError("Retry delay exceeds local budget")
return delay
return min(budget, 2 ** min(attempt, 5) + random.uniform(0, 1))
print(round(retry_delay(2), 2)) # Illustrative delay, not a measurement
Which failed requests are eligible for retry?
Only the injected GET is retried here. The adapter receives a finite timeout and owns TLS/auth/redirect policy. Requests timeouts are not hard process deadlines, so the application budget is approximate if an adapter blocks or DNS stalls. Never reuse this wrapper for a write without an idempotency and reconciliation design.
Worked Python example and input contract
import math
import time
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
def retry_after(value, now):
if value is None:
return None
if str(value).isdigit():
return float(value)
try:
parsed = parsedate_to_datetime(value)
if parsed.tzinfo is None:
return None
return max(0, (parsed-now).total_seconds())
except (ValueError, TypeError, OverflowError):
return None
def bounded_get(get, max_attempts=3, budget=20, clock=time.monotonic, sleep=time.sleep):
deadline = clock()+budget
for attempt in range(max_attempts):
remaining = deadline-clock()
if remaining <= 0:
raise TimeoutError('retry_budget_exhausted')
response = get(min(5, remaining)) # GET adapter sets TLS and redirect policy.
if response.status_code != 429:
response.raise_for_status()
return response
wait = retry_after(response.headers.get('Retry-After'), datetime.now(timezone.utc))
if wait is None:
wait = min(2**attempt, 8)
if not math.isfinite(wait) or wait >= deadline-clock():
raise TimeoutError('retry_after_exceeds_budget')
if attempt+1 < max_attempts:
sleep(wait)
raise TimeoutError('rate_limit_attempts_exhausted')
# Supply a callable that performs one approved Session.get(...), with
# allow_redirects=False, validated TLS, auth and the supplied timeout.
# response = bounded_get(lambda timeout: session.get(approved_url,
# timeout=timeout, allow_redirects=False))
print(retry_after('5', datetime.now(timezone.utc))) # Offline header parsing
How can caching and scheduling reduce controller requests?
Cache permitted inventory by an explicit freshness rule, use filtering/pagination supported by the API, and collapse duplicate requests for the same target. Do not return cached data as current after collection fails; expose its age and source.
Dashboard reporting: rate limits, retries and incomplete collection
Report latest attempt, last success, response category, retry count and exhaustion separately. A failed collection must remain in the denominator. Agree with the client which stale-data window is tolerable and when a human should investigate rather than retry more aggressively.
Related troubleshooting and implementation
Explore the network automation troubleshooting collection to connect this issue to adjacent workflows. The Netmiko, NETCONF and RESTCONF comparison helps when you are choosing an interface rather than fixing an existing implementation.