Scaling APIs for Millions of AI-Driven Calls

AI agents are becoming a new class of API consumers. Unlike human users, agents can create bursty traffic, retry aggressively, call multiple tools in parallel, and accidentally amplify downstream failures. A single user request can become a large chain of API calls, model calls, vector searches, database lookups, and workflow events.

This talk explains how to design APIs for this new reality.

We will cover agent-aware rate limiting, budget-aware throttling, backpressure, load shedding, idempotency, deduplication, deterministic caching, async workflows, event-driven APIs, tail-latency SLOs, and cost observability.

Participants will learn how to tag and trace agent traffic, control runaway tool calls, prevent retry amplification, design graceful degradation, and build runbooks for cache storms, retry storms, dependency brownouts, and cost spikes.

The core message:

APIs exposed to AI agents must be contract-safe, retry-safe, cost-aware, observable, and degradation-ready.

Classic API scaling assumed relatively predictable traffic.

AI-driven API traffic is different because:

  • One prompt can create many downstream API calls.
  • Agents can retry, loop, and fan out.
  • Tool-calling creates bursty and non-human traffic patterns.
  • Cost grows with requests, retries, context size, model calls, and downstream work.
  • Failures can amplify quickly across gateways, SDKs, queues, databases, and model APIs.

Agenda

  1. Why AI Changes API Scaling
    Human traffic versus agent traffic, tool chains, fan-out, retries, and burst patterns.
  2. New Failure Modes
    Retry storms, cache-miss storms, malformed tool calls, version drift, DB saturation, and cost spikes.
  3. Traffic Control for AI Agents
    Agent-aware rate limits, per-tenant budgets, per-tool quotas, fair queuing, and adaptive backpressure.
  4. Resilience Patterns
    Idempotency keys, deduplication, bounded retries, circuit breakers, bulkheads, timeouts, and load shedding.
  5. Caching for AI Workloads
    Deterministic-result caching, semantic-aware caching, stale-while-revalidate, negative caching, and cache warming.
  6. Async and Event-Driven APIs
    Queue-first design, workflows, webhooks, streaming responses, outbox patterns, and dead-letter handling.
  7. Observability and Cost Governance
    Chain IDs, tool IDs, agent IDs, tail-latency SLOs, per-agent cost attribution, anomaly detection, and loop detection.
  8. Runbooks and Readiness
    Playbooks for retry storms, cache storms, provider brownouts, cost spikes, and safe degradation.

About Rohit Bhardwaj

Rohit Bhardwaj is Director of AI & Data Architecture at Salesforce, an O’Reilly instructor, and founder of AI Architect Circle. With 30 years of experience in software and enterprise architecture, he helps engineers, architects, and technology leaders make architecture decisions they can explain and defend.

His work brings together enterprise AI, agentic systems, data architecture, and distributed systems. He focuses on the decisions that determine whether a system remains reliable, secure, accountable, and economically sustainable when it encounters real-world demands.

Rohit’s talks and workshops explore practical questions: What happens when an AI agent acts on conflicting information? Can a failed payment be retried safely? Who can stop an autonomous action, and what evidence explains the decision afterward? His teaching connects these questions to architecture, customer trust, and business outcomes.

He is the author of System Design with AI Interview Guide: Designing Scalable, Agentic, and Defensible Systems, published by Apress. He also developed ARCHVIEW™, ROCS Loops™, and LLM-SHIELD™ to help teams structure architectural thinking and address the challenges of building trustworthy AI systems.

Through AI Architect Circle, Rohit extends this learning through live discussions, book walkthroughs, videos, and interactive architecture challenges. These challenges explore incidents such as payment timeouts and conflicting bookings, helping participants practice the judgment behind their design decisions.

Continue learning with Rohit

Join Rohit live on Saturdays at 9:00 a.m. Eastern
System Design with AI — Live with the Author is a free live podcast and Q&A series featuring practical system-design scenarios, AI architecture discussions, and questions from the community.
See the next session and register

Try an interactive architecture challenge
Put your design decisions under pressure and explore the consequences.
Explore the challenges

Explore AI Architect Circle
Find architecture lessons, videos, resources, and ways to continue learning.
Visit AIArchitectCircle.com

Read the book: Amazon · O’Reilly

Watch and connect: YouTube · Instagram: @aiarchitectcircle · LinkedIn · X / Twitter: @rbhardwaj1

More About Rohit »