Centralize the AI Platform, Not Every Decision

An organization with five AI teams can easily end up paying for five versions of the same foundation. Each team creates its own model gateway, prompt store, evaluation scripts, secrets, access rules, logging, and vendor contracts. The duplication looks like autonomy until an incident reveals that nobody has a complete inventory and every team records different evidence.

The obvious response is to centralize. Build one platform, create one backlog, set one standard, and ask every business unit to use it.

That response solves one class of problem and can create another. The shared platform team becomes a queue. Domain teams wait for small changes. Important context is compressed into tickets. A common workflow grows so many options that nobody finds it simple. People route around the platform, and unofficial tools return under new names.

The useful choice is not centralization versus decentralization. It is deciding where each kind of decision belongs.

For modern AI and data work, the strongest operating model is usually a deliberate split: centralize capabilities that become safer, cheaper, or easier to reuse at scale; keep decisions that require local knowledge close to the domain; and explicitly share the responsibilities that cannot survive a handoff. This article offers a framework for drawing those boundaries.

Start With the Boundary Matrix

Before changing an org chart, classify the work. The matrix below is a starting point, not a universal template.

This matrix prevents a common category error: treating ownership of infrastructure as ownership of the business problem. A platform can provide approved model access, but it should not decide what a claims assistant may approve. A governance group can define minimum testing, but it should not write every evaluation case. A domain team can own an agent’s workflow, but it should not create an invisible identity system to make the agent run.

The boundary is the design.

Centralize Repetition, Risk, and Interoperability

Some capabilities improve when one team builds them for many teams. Identity and access management are obvious examples. The same is true of secrets management, network boundaries, encryption defaults, audit records, incident hooks, cost allocation, and approved paths to model providers.

AI adds more candidates for a shared layer:

• a model gateway with authentication, quotas, routing, and usage records;
• standard telemetry for latency, errors, token use, tool calls, and model versions;
• an evaluation harness that teams can extend with domain-specific cases;
• registries for prompts, models, datasets, agents, and owners;
• reusable controls for sensitive data, retention, and human approval;
• deployment templates with rollback and change history;
• a supported route for retrieval, vector storage, and tool connections.

These are not valuable merely because they are central. They are valuable when their reuse removes work from product teams and their consistency makes the organization easier to operate.

AWS describes a related pattern in its data mesh guidance: domain ownership is paired with a self-service platform and federated governance. The point is not to remove the center. It is to make the center provide leverage rather than absorb every decision.

Interoperability also deserves central attention. If teams use incompatible identity conventions, metadata, event formats, evaluation records, or lineage rules, their local systems become expensive to connect. A small number of enforced interfaces can preserve far more autonomy than a single giant application. Teams remain free to solve different problems because they share enough language for systems to work together.

Keep Context and Consequence Near the Domain

Central teams are usually weakest where meaning changes by business context.

Consider a customer-service agent, an engineering assistant, and a finance document reviewer. All three might use the same model gateway, observability stack, and deployment path. Yet their acceptable errors are different. Their users interpret confidence differently. Their source data has different owners. Their escalation paths, response-time targets, and review obligations are not interchangeable.

Those differences belong with domain teams. They should own the user outcome, the workflow, the meaning of the data, and the evidence that the system is fit for its intended use. That includes decisions such as:

• which task should be assisted rather than automated;
• what counts as a correct or harmful output;
• which sources are authoritative;
• who may use the system and for what purpose;
• when a person must review or approve an action;
• how exceptions enter the workflow;
• what metric shows improvement for users or the business;
• when the system should be paused or retired.

This is also why centralizing all AI development into a center of excellence rarely settles the operating-model question. A central group can seed expertise, create standards, and help early projects. If it remains the permanent builder of every use case, however, it must continually reconstruct context that already exists elsewhere. Its backlog becomes a map of organizational distance.

Domain ownership is not permission to ignore shared rules. It is accountability for applying those rules to a real use case and producing evidence that central policy alone cannot create.

AI Agents Make the Split More Important

A chatbot that drafts text has limited authority. An agent that can query customer records, create a refund, change a schedule, or trigger a deployment has a different risk profile. Tool access converts an output-quality question into an operational-control question.

Google Cloud’s current multi-tenant agentic AI reference architecture makes the tension concrete. It combines a central governance and security hub with isolated tenants for business units that need specialized tools, rules, and sensitive data. The architecture is centralized and decentralized at the same time because the problem demands both.

For agentic systems, the platform team should usually own how identity is propagated, credentials are protected, calls are logged, tenant boundaries are enforced, and emergency controls work. The domain team should own which tools the agent needs, the actions it may take, the preconditions for those actions, and the cases that require human approval.

Several decisions must be joint. An action limit, for example, has technical and business meaning. The platform can enforce a maximum number of tool calls, but the domain must decide whether three attempted payments are three retries or three separate financial actions. Central telemetry can record a trace, but the domain must say which trace pattern signals harm. Security can define privileged access, but process owners must identify which apparently ordinary action creates a consequential commitment.

Central controls without domain semantics are blunt. Domain freedom without central controls is fragile.

A Platform Must Behave Like a Product

Centralization often fails through service design rather than architecture. Leaders fund a platform, mandate its use, and assume adoption will follow. The platform team then optimizes for its own roadmap while domain teams experience slow support, missing capabilities, and unclear documentation.

A shared platform needs customers, discovery, service levels, documentation, usage measures, and a feedback loop. It should offer a paved road: the easiest supported path for common work, with deliberate escape routes for cases that genuinely do not fit.

That changes the platform team’s questions. Instead of asking only, “Which standard should everyone follow?” it asks:

• How long does a team need to deploy a compliant first version?
• Which steps still require a ticket or meeting?
• Where do users leave the paved road, and why?
• Which platform failures stop multiple products at once?
• What does adoption save compared with each team building independently?
• Which exceptions reveal a missing shared capability rather than resistance?

This connects directly to treating internal AI systems like products. A mandate can force nominal adoption, but it cannot create a usable product. If the shared platform is consistently harder than an unofficial alternative, policy will spend its time fighting the evidence of poor service design.

There is an economic discipline here too. Shared platforms have fixed costs. The organization should not build a sophisticated model-routing service, evaluation portal, and multi-tenant agent runtime because such components look mature on an architecture diagram. Centralize when repeated demand exists or when the control is necessary from the first use. Otherwise, choosing what not to build is part of platform strategy.

Federated Governance Is More Than a Committee

Federation is sometimes used as a polite word for ambiguity: a central committee writes guidance, local teams interpret it, and nobody knows who can stop a launch. That is coordination theater, not an operating model.

Effective federated governance separates four things:

1. Organization-wide policy. Leadership defines risk tolerance, prohibited uses, minimum controls, and escalation authority.
2. Platform enforcement. Shared services turn suitable policies into defaults, permissions, logs, quotas, and deployment checks.
3. Domain evidence. Product teams map the use context, test likely failures, document data and workflow decisions, and monitor outcomes.
4. Joint accountability. Named people decide whether evidence is sufficient for higher-impact systems and who responds when conditions change.

The NIST AI Risk Management Framework supports this division without prescribing an org chart. It calls for clear roles and communication, executive responsibility for AI risk decisions, multidisciplinary perspectives, and risk work across the system lifecycle. That is a useful warning against delegating “AI governance” to either a distant central office or an isolated engineering team.

Controls should also scale with consequence. A private drafting assistant does not need the same approval route as an agent that changes customer records. The platform can offer several control tiers, while domain teams classify use cases and justify their choice. This preserves speed for low-impact experiments without weakening scrutiny where failure matters.

Move Capabilities in Stages, Not With a Big-Bang Reorganization

Reassigning ownership changes more than reporting lines. It changes queues, relationships, access, budgets, and the informal help people depend on. A clean target diagram can hide an unsafe transition.

A staged move is easier to inspect:

Inventory the current system. Record active AI applications, models, data sources, vendors, owners, costs, permissions, evaluation methods, and dependencies. Do not assume the official portfolio is the real portfolio.

Select one shared capability. Model access, audit logging, or deployment templates may be a better first move than a complete enterprise AI platform. Choose something multiple teams need and can evaluate quickly.

Run with representative domains. Include teams with different risk, data, and latency needs. A platform proven only on the central team’s demo is not proven.

Measure the transition cost. Track migration time, delivery delay, incidents, support load, and duplicated work removed. Savings that appear in one budget while domain teams absorb more manual work are not genuine savings.

Keep local expertise connected. People who understand existing workflows should help design the new path, evaluation cases, and exception process. Centralization should not discard the knowledge required to make the platform useful.

Expand only after the service works. A mandate issued before reliability and support are ready converts platform defects into organization-wide defects.

Platform choices should remain revisable. Shared infrastructure can quietly create dependency on one provider, model family, or proprietary control plane. The goal is not theoretical portability everywhere, but explicit awareness of switching costs. The note on avoiding AI platform lock-in offers a deeper way to assess those dependencies.

Test the Operating Model With Five Questions

An org chart cannot tell you whether the boundary works. These questions can:

1. Can a domain team launch a low-risk use case through the supported path without joining a long central queue?
2. Can the organization identify every production AI system, its owner, its data, and its current model or provider?
3. Can a domain owner explain the system’s success metric, failure modes, and human-review rule?
4. Can central teams enforce critical controls without understanding every detail of every workflow?
5. When an incident occurs, is it clear who can contain the platform, who judges domain impact, and who communicates the decision?

If the first answer is no, the center is probably a bottleneck. If the second is no, local autonomy has become invisibility. If the third is no, ownership is too far from the problem. If the fourth is no, the controls may be too manual or too vague. If the fifth is no, shared responsibility has become unowned responsibility.

These answers will change as the organization grows. A small company may need one enabling team and lightweight standards. A larger regulated enterprise may need separate platform, security, governance, and domain product groups. The right design depends on scale, consequence, and repeated demand—not on copying another company’s org chart.

The Goal Is Coordinated Autonomy

Centralization is useful when it turns repeated work into a dependable shared capability. Decentralization is useful when it keeps judgment near users, data, and consequences. Either can fail when treated as an ideology.

For AI systems, the practical division is increasingly clear. Share the foundations that need consistent security, observability, interoperability, and cost control. Keep product choices and domain meaning with the teams that can understand them. Join the two through enforceable guardrails, self-service paths, explicit evidence, and named decision rights.

The result is not perfect uniformity. It is coordinated autonomy: teams can move without rebuilding the foundation, while the organization can govern the systems without pretending that one central group understands every problem.