Quick answer (TL;DR): Most of the friction ops teams hit with AI comes from assuming a capability that isn’t there, or not knowing about one that is. The fix is a capability audit: deliberately test what your assistant can and can’t do in your environment (with your HubSpot or Marketo portal, your Salesforce org, your shared drive and your permissions), not what the product page claims, and write the answers into a file. Then design the team’s workflow around the real constraints. An hour spent auditing capabilities saves ten spent assuming them.
If your team is building an AI operating system for ops, this is the unglamorous step that decides whether it feels smooth or maddening.
Why you can’t trust the product page
Every AI tool describes the happy path: “connects to your CRM”, “edits your documents”, “works with your data”. What it rarely tells you is how those capabilities behave in practice:
- It “connects to HubSpot”, but can it read workflow definitions, or only contact records? Can it write, or only read?
- It “works with files”, up to what size? Will it handle a 200,000-row contact export, or choke on it?
- It “integrates with Salesforce”, but is that connection authorized for your profile and permission set, or quietly failing at the permissions layer?
These distinctions stay invisible until you test them, and each can derail a workflow built on the happy path. The most common source of AI frustration isn’t a weak model. It’s a mismatch between assumed and actual capability.
What a capability audit is
You deliberately probe what the assistant can do in your real environment and write down the answers. Not in theory: you ask it to do the thing and watch what happens.
The output is a capabilities file in the team’s shared memory: a living document listing, for each tool and integration, what works, what doesn’t and what the workaround is. It turns every painful surprise into a known constraint the whole team can design around.
How to run one
Go through each capability the team might rely on and test it. A practical checklist:
Files and documents:
- Can it read our exports and docs? Which formats (CSV, XLSX, PDF)?
- Can it create new files, and where do they land?
- Can it edit existing files in place, or only create new versions?
- What’s the size limit, and what happens with a large contact or activity export?
CRM and marketing automation connectors:
- Is the connector actually authorized in our account, with the permissions we think it has?
- Can it read records? Properties and field metadata? Workflow or flow definitions?
- Can it write? If so, does every write require an approval step?
- Are there parts that look connected but fail silently, such as custom objects, or fields hidden by field-level security?
Other integrations (Slack, email, ticketing, BI):
- Connected and authorized? Read, write or both?
- Does it see private channels or only public ones?
The assistant’s own behaviour:
- Can it run scheduled tasks, or only act while someone is present?
- What does it do when it hits a wall: say so clearly, or produce a plausible guess?
Write the real answer beside each item, specifically. “Can read HubSpot contacts and properties; cannot read workflow definitions; exports over a certain size must be split” is useful. “HubSpot: yes” is not.
What our own audit turned up
When we audited our setup, we found a mix of pleasant surprises and hard limits:
- Some tools could read and create files but not edit them in place. That one finding reshaped our file convention: we adopted “recreate with a version stamp” because in-place editing wasn’t available.
- Some files were too large for certain channels. We learned which route to use for large exports and stopped wasting time on the one that would fail.
- One whole integration was silently broken at the permissions layer. It looked connected but couldn’t do anything. Once we knew, we routed around it.
None of this was visible until we tested it. Once written down, each became something to design around rather than trip over again.
Design around limits, don’t fight them
The constraints are usually fixed; you rarely control what a tool can do. So a good ops method treats them as fixed and builds accordingly:
- If the assistant can read HubSpot but not workflow logic, export the workflow summaries into the shared memory and let it reason over those.
- If exports are too big for one channel, split them or use another route, and note which.
- If an integration is broken, route around it and record the workaround so the next session doesn’t rediscover the wall.
The goal isn’t a perfect toolchain. It’s a reliable one whose edges you’ve mapped.
Re-audit periodically
Capabilities change. Vendors ship connector updates, permissions get tightened after a security review, limits move. Re-run a light version of the audit every few months and after any change to your CRM permissions model. A constraint you designed around may be gone, and a new one may have appeared.
Why the hour is worth it
It’s tempting to skip the audit and start working. But every untested assumption is a future failure, and some of those failures are silent: the assistant reports on a pipeline it could only partly see, and nobody knows why the numbers look off.
An hour of deliberate testing, written down, turns “why isn’t this working?” into a set of known facts the team builds on.
Write down what’s real. Build on that. Stop assuming the happy path.
Frequently asked questions
What is an AI capability audit for an ops team? It’s deliberately testing what your AI assistant can and can’t do in your actual environment (CRM and marketing automation connectors, file handling, permissions, size limits, scheduled tasks) and recording the real answers so the team designs around true constraints rather than assumed ones.
Why does our AI give incomplete answers about CRM data? Usually because of a gap between assumed and actual access. The connector may lack permission to certain objects or fields, or may not be able to read workflow logic at all. A capability audit surfaces these gaps before they produce misleading output.
How often should we re-run it? Every few months, and after any change to CRM permissions, connectors or AI tooling. Vendors update integrations and limits change.
What should we do when the AI can’t do something we need? Design around it. Export what it can’t read into the shared memory, split files that are too large, route around broken integrations, and record each workaround so the team doesn’t rediscover the limit.
This is Part 4 of The Ops Operating System. Next: the reconcile, and how to catch ops requests that fall through the cracks between Slack, email, tickets and trackers.
Not sure what your connectors are really doing either? Our GTM Systems Audit tests the syncs and integrations between HubSpot, Marketo and Salesforce, not just the AI.