What AI Changes for Engineering Managers - Part 2: Governance in the Age of AI

When I wrote earlier about how the work of Engineering Management changes once AI is doing a lot of the coding, I said that governance was large enough to need its own piece, and this is that piece. My earlier observations were about the need to master the skills to write specifications, documentation and overall to develop good judgment. On the other side, once we are able to produce software products with the desired quality and a higher speed, how do we manage the whole process? Are there aspects of software production we need to pay more attention to than before, and most of all, if you work in regulated environments, how do we explain to regulators this new era of software development?

What I uncovered writing the two parts is that they are not equally advanced: producing software is an activity that gives you quick feedback, so you can relatively quickly improve your ability in it, while governance failures are rare and severe and the lesson arrives long after the decision that caused them. That is why the production side is mostly understood while the governance side is not.

In this article I talk about the three points that I am considering right now as the most critical.

Security

We still have a serious supply chain security problem [1], and AI is not making the situation any better. While we connect agents to more tools and more external contents, the cost of building a basic attack falls, and what once needed a skilled developer now needs a single carefully built web page; as our defenses are not evolving at the same pace as our enthusiasm for AI tools, I expect that within the next six months to a year a share of the public Web will consist of pages whose only purpose is to perform prompt injection on whatever agent reads them. Agents will treat those instructions with the same trust given to our own instructions, making a basic Web access become immediately both the hacking system and the data exfiltration channel.

A more ordinary version of the same problem arrives before any agent is connected to anything. The code we write, the customer data we process and the secrets in our systems now flow into third party model providers as part of normal use, and for a company in a regulated environment this will be one of the first questions to answer. Where that information goes, how long it is retained, and whether it is used to train a model we do not control, are questions that need a documented answer.

The only reasonable response right now is to segregate as much as is reasonably possible agents who have access to potential injection/leakage vectors from agents that handle critical data, or perform critical operations. Separate agents by reach: agents who handle confidential information will not access the Web, agents reaching the Web can’t handle confidential information, and lots of guardrails and checks in between them [2]. It is an awkward separation by our usual standards and will slow operations down, but it seems the only alternative to having a human-in-the-loop in every operation, an approach unfeasible at scale.

Denial of Service and Sponge Attacks [5] will be on the rise to target companies and institutions, trying to cause maximum financial harm by abusing publicly exposed chatbots connected to expensive online AI services. This is where the security problem and the cost problem meet, because an attacker no longer needs to take a service offline to hurt you; making it answer enough expensive requests is sufficient.

Cost

For most of the last two years we have been spending on AI as an investment, where the cost was justified by what we were learning; this phase will have to end soon as AI platforms keep pushing how much they can charge customers, and how opaque their pricing can be. Once it ends, we will be expected to look at our software production and show that the money produced the return we said it would. But what are the key metrics revealing the “AI gains”? Number of Pull Request / Month? Feature cycle time? And what if the coding time reduces, but not the end-to-end delivery time (hint: see my previous article [3])? Many companies use token consumption as a proxy measure [4], but is it the correct metric in every industry, including mature, lower growth industries?

My own position is that token consumption measures activity, not value, and that the only defensible metric is the end-to-end one: the time and cost to take a unit of work from request to something running in front of a customer, compared with what it was before. It is harder to measure and slower to move, but it is the number a finance function will eventually ask for, and it is the one that cannot be inflated simply by spending more.

Allocating capital across models is also a topic that is objectively difficult because the options are increasing, including not only the well-known major vendors but also self hosted open-weights models. At low volumes this might not be such a dilemma but when annual costs reach 6-7 digits comparing infrastructures, prices and performances becomes a hard problem, in the absence of independent and reliable benchmarks. In the absence of those benchmarks, the practical answer is to build a small internal evaluation on your own representative tasks and re-run it as prices and models change, accepting that the result is specific to your case, and has to be re-evaluated very frequently as the technology develops.

Audit defensibility

The third area, the one most CTOs must pay attention to and where getting it right is a must rather than an advantage, is audit defensibility. When a regulator asks who decided that a certain customer facing operation was compliant with internal policies and external regulations, an answer that is not completely rule based and deterministic will lead to more uncomfortable questions, if not rejected completely. Regulators want a documented chain of approvals, defined checklists, measured evaluation criteria, clear authorisations that remain defensible when someone inspects them months later. Fintech CTOs who cannot produce those records will find it impossible to ship AI-influenced work into regulated products at all.

A lot of the regulations we are currently subjected to were designed (and are still described) as human operations based on written specifications and checklists; they require that a named person reviewed or approved a significant action at a defined point, and that the review was recorded and can be inspected later. A model that is not deterministic, that can answer the same question differently on two occasions, does not fit this framework, and pointing at the model as the decision maker will not be an option. And while some regulators are more forward-looking, and others definitely less so, it will only take a few visible incidents to see regulated companies hit the brakes on AI adoption in consumer facing functionalities. How do you ensure AI (LLM) model compliance? As for AI testing, and possibly more, we are missing a framework to formally prove model compliance, or at least to be able to estimate what the level of compliance is.

Where we are right now

As I read again the first article I wrote on “What AI Changes for Engineering Managers” [3] and I compare it to this one, it strikes me that the way to get better at producing AI-aided software is mostly known (although not necessarily simple), while the way to properly govern it is frankly unknown or very roughly defined at this point.

I suppose that’s not unusual for new technologies; cryptocurrencies are another recent example of technology that has seen fast adoption well before governance was able to keep up.

Engineering Managers and CTOs will have to ride this transition phase with extreme caution, trying to rely on all the current best practices, research daily the latest development and apply lots of creativity, to find the missing answers.

Notes

[1] Recent supply chain attacks: Shalai-Hulud (2025) https://www.blackduck.com/blog/npm-malware-attack-shai-hulud-threat.html; mini Shalai-Hulud attack (2026) https://thehackernews.com/2026/05/mini-shai-hulud-worm-compromises.html; Miasma worm (2026) https://thehackernews.com/2026/06/ironworm-and-new-miasma-worm-variant.html

[2] Example of risks of having Web connected agents handling confidential information: https://krebsonsecurity.com/2026/06/hackers-used-metas-ai-support-bot-to-seize-instagram-accounts/

[3] What AI Changes for Engineering Managers https://www.diotalevi.com/essays/ai-changes-em-1/

[4] Jensen Huang: “If that $500,000 engineer did not consume at least $250,000 worth of tokens, I am going to be deeply alarmed,” https://www.businessinsider.com/jensen-huang-500k-engineers-250k-ai-tokens-nvidia-compute-2026-3

[5] Understanding Model Denial of Service: The Rise of Sponge Attacks on LLMs https://stayrelevant.globant.com/en/technology/cybersecurity/increase-denial-service-attacks/