---
title: "How Spotify runs multi-agent systems at scale with deterministic guardrails"
canonical: https://www.aa-labs.co/blog/how-spotify-runs-multi-agent-systems-at-scale-with-deterministic-guardrails
author: A & A Labs
published: 2026-10-14
updated: 2026-10-14
category: Engineering
publisher: A & A Labs (Private) Limited
reading_minutes: 6
---

# How Spotify runs multi-agent systems at scale with deterministic guardrails

> Spotify runs production multi-agent systems for Ads Manager using Google ADK Java. Each agent is a single Bazel package with an explicit owner, and cross-agent dependencies are enforced at compile time through Bazel visibility rules. Evidence collection (intent extraction) is separated from LLM inference, making the deterministic parts testable and auditable.

## Key takeaways

- Each agent is a single Bazel package with explicit ownership; cross-agent dependencies enforced at compile time via visibility rules.
- Evidence collection (intent extraction) is separated from non-deterministic LLM inference, enabling unit testing and auditability.
- Guardrails run as independent agents with separate ownership, schemas, and deployment cycles — not as prompt instructions.
- Tracing is the primary evaluation strategy: traces capture intent, tool calls, reasoning, outputs, and guardrail verdicts for automated pipelines.
- Framework choice (Google ADK Java) follows build system and ownership model alignment, not community popularity.

Spotify's Ads Manager runs production multi-agent systems using Google ADK Java, with domain ownership models and deterministic guardrails that keep agents auditable. The architecture separates evidence collection from inference, enforces agent boundaries at compile time, and uses traceability as the primary evaluation strategy. These patterns map directly to the decisions teams face when building their first multi-tenant agent platform: framework choice, observability requirements, and where to enforce policy versus where to allow emergence.

## One agent, one package, one owner — enforced at compile time

Spotify structures every agent as a single Bazel package with an explicit owner. The package contains the agent instructions, its LLM configuration stored as front matter, dependency-injection scaffolding (AgentFactory, Adapter, Service, AgentModule), a `lib-info` file that resolves ownership and PR reviewers, and a `monitoring-info` YAML that wires dashboards and PagerDuty alerts before any code is written [^1].

Cross-agent dependencies are controlled by Bazel visibility rules. The Ad Script Generation Agent cannot import the Audience Resolver unless the Audience Resolver explicitly exports it. Violations produce a build failure, not a runtime surprise. This makes agent boundaries a compile-time contract rather than a documentation convention.

For a multi-tenant platform, the same principle applies: tenant isolation begins at the module boundary. When every agent is a deployable unit with a declared owner, onboarding the hundredth tenant does not require re-architecting ownership or observability — it requires adding a package. Our [multi-tenant SaaS platform engineering](https://aa-labs.co/capabilities) work applies this pattern to billing, usage metering and admin tooling so that tenant isolation is designed in from the outset.

In our client work building multi-agent systems, the most common trap teams fall into is treating agent boundaries like microservices before their communication contracts are mature. Spotify’s Bazel approach works because the interfaces are compile-time rigid; when teams attempt this with loose JSON over HTTP or ad-hoc Python imports, boundary lines blur within three sprints. We routinely advise clients to enforce strict boundary packages from day one—even if the initial implementation feels overly structured—because untangling runtime agent bleed across multiple tenants later is exponentially more expensive than paying the compilation tax early.

## Evidence collection separated from inference

The Ads AI orchestration layer extracts intent from natural language — audience, geo-targeting, goal, CTA — then feeds that structured intent to parallel agents: Ad Script Generation, Ad Guardrail, Audience Recommendation (in pilot). Each agent produces a discrete output object: an audience recommendation and a creative. The orchestration layer combines them [^1].

This separation matters. The intent extraction step is evidence collection: deterministic, testable, auditable. The agent inference step is non-deterministic by nature. By keeping them in separate stages, Spotify can unit-test the extraction logic, replay the same evidence through different model versions, and audit why a particular creative was generated without re-running the LLM.

Teams building multi-tenant agent platforms should treat evidence collection as platform infrastructure — shared, versioned, observable — while allowing agent inference to vary per tenant or use case. The [Cloudflare Clef decision models](https://aa-labs.co/blog/cloudflare-clef-decision-models-shift-ai-agent-routing-out-of-prompt-engineering) pattern follows the same logic: routing decisions are extracted from prompts into configurable models that can be tested and audited independently.

## Deterministic guardrails as a separate agent, not a prompt instruction

Spotify runs an Ad Guardrail Agent in parallel with the creative agents. It evaluates the generated output against policy — brand safety, compliance, format constraints — and returns a pass/fail with evidence. The guardrail agent is owned by a different team, versioned independently, and deployed as its own package [^1].

This is a deliberate architectural choice. Prompt-level guardrails ("do not generate harmful content") are unreliable and untestable. A dedicated guardrail agent with its own schema, tests, and ownership becomes a platform capability that any tenant can inherit. Policy changes become a deployment, not a prompt rewrite.

For multi-tenant platforms, this pattern means guardrails are part of the shared control plane. Tenants declare their policies; the platform runs the guardrail agents. The [Google Cloud API Gateway exposing REST endpoints as MCP tools](https://aa-labs.co/blog/google-cloud-api-gateway-exposes-rest-endpoints-as-mcp-tools-without-custom-middleware) pattern shows a related approach: policy enforcement (auth, rate limits, quotas) lives in the gateway layer, not in each tool implementation.

## Traceability as the primary evaluation strategy

Spotify uses tracing not just for debugging but as the evaluation backbone. Every agent execution produces a trace that captures: the input intent, the tool calls made, the intermediate reasoning, the final output, and the guardrail verdict. These traces feed automated evaluation pipelines that measure quality, latency, cost, and policy compliance across thousands of campaigns [^1].

The platform treats traces as first-class data — queryable, versioned, and joined with business metrics (creative performance, advertiser spend). This turns observability into a product requirement rather than an afterthought.

Multi-tenant platforms need the same: a trace schema that is stable across tenants, storage that scales with tenant count, and query APIs that let platform teams and tenant admins answer "why did this agent do that?" without instrumenting every agent individually. The [Cloudflare AI Gateway adding managed web search for RAG](https://aa-labs.co/blog/cloudflare-ai-gateway-adds-managed-web-search-for-retrieval-augmented-applications) illustrates how managed infrastructure can provide this tracing layer without each team building their own.

## Framework choice: Google ADK Java, not Python

Spotify chose Google ADK Java over the more common Python variant. The talk notes this explicitly: "We are using Google ADK Java" [^1]. The reasons are implicit in the architecture: Bazel integration, compile-time dependency enforcement, type-safe tool schemas, and alignment with Spotify's existing JVM platform.

For teams choosing a framework, the lesson is not "use Java" but "choose the framework that integrates with your build, deployment, and ownership model." If your platform is Bazel-and-JVM, ADK Java gives you compile-time agent boundaries. If your platform is Python-and-pip, the Python variant may fit better. The framework should make your ownership model enforceable, not just expressible.

## Cost management through parallel agents and discrete outputs

Running three agents in parallel (script, guardrail, audience) costs more than a single monolithic agent — but Spotify accepts this because discrete outputs enable independent evaluation, independent rollback, and independent ownership. The guardrail agent can be swapped without touching the script agent. The audience agent can be piloted without blocking creative generation [^1].

This is a classic platform trade-off: higher inference cost for lower coordination cost. In a multi-tenant setting, the same logic applies. Tenants may need different agent compositions; the platform should support composing agents from a shared catalogue rather than forcing every tenant into the same pipeline. The [DoorDash move from vendor-first to open-weight models](https://aa-labs.co/blog/how-doordash-moved-from-vendor-first-to-open-weight-models-in-its-genai-platform) shows a similar shift: platform teams provide composable primitives (models, tools, guardrails) rather than monolithic pipelines.

## What to do next

- **Model agent boundaries as deployable packages with explicit owners.** Enforce cross-agent dependencies at build time, not runtime.
- **Separate evidence collection (intent extraction, schema validation) from inference.** Make the evidence layer shared, testable, and auditable.
- **Implement guardrails as independent agents with their own schemas, tests, and deployment cycles.** Keep policy out of prompts.
- **Design tracing as a first-class platform capability.** Define a stable trace schema, store traces centrally, and build evaluation pipelines on top.
- **Choose your agent framework based on how it fits your build system, ownership model, and type-safety requirements** — not on community popularity alone.

A & A Labs helps teams build multi-tenant platforms where these patterns are designed in from the start, so the hundredth tenant onboards without re-architecture.

## Frequently asked questions

### How does Spotify enforce agent boundaries in its multi-agent system?

Spotify structures each agent as a single Bazel package with an explicit owner. Cross-agent dependencies are controlled by Bazel visibility rules — an agent cannot import another unless it is explicitly exported. Violations produce a build failure, making boundaries a compile-time contract.

### Why does Spotify separate evidence collection from inference?

Evidence collection (intent extraction from natural language) is deterministic and testable. Inference is non-deterministic. Separating them allows unit testing of extraction logic, replaying evidence through different model versions, and auditing outputs without re-running the LLM.

### How are guardrails implemented in Spotify's architecture?

Guardrails run as a dedicated Ad Guardrail Agent in parallel with creative agents. It evaluates outputs against policy (brand safety, compliance, format constraints) and returns a pass/fail with evidence. The guardrail agent is owned by a different team, versioned independently, and deployed as its own package.

### What role does tracing play in Spotify's evaluation strategy?

Tracing is the evaluation backbone. Every execution produces a trace capturing input intent, tool calls, intermediate reasoning, final output, and guardrail verdict. Traces feed automated pipelines measuring quality, latency, cost, and policy compliance, and are joined with business metrics like creative performance and advertiser spend.

### Why did Spotify choose Google ADK Java over the Python variant?

The choice aligns with Spotify's existing JVM platform and Bazel build system. ADK Java provides compile-time dependency enforcement, type-safe tool schemas, and Bazel integration, making the ownership model enforceable at build time rather than just expressible.

## Sources

- Presentation: Multi-Agent Patterns from Spotify’s AI Powered Advertising Platform — https://www.infoq.com/presentations/spotify-multi-agent-ai-architecture/?utm_campaign=infoq_content&utm_source=infoq&utm_medium=feed&utm_term=global

---

Published by A & A Labs (Private) Limited · https://www.aa-labs.co/blog/how-spotify-runs-multi-agent-systems-at-scale-with-deterministic-guardrails