Security leaders are asking the wrong question about artificial intelligence, according to a security officer at a firm that specializes in resilience and chaos engineering. While many organizations want to know what data a model was trained on, that question only addresses confidentiality. The more important issue is how to maintain authority and integrity over the data itself—who can access it, under what conditions, and with what limits. That shift in framing has major implications for how AI agents are deployed, how attackers are evolving, and what skills security teams need to hire for.

Key Facts

• Leaders frequently ask what data an AI model was trained on, but the better question is how to maintain authority and integrity of that data.
• Least privilege, authentication, and access controls remain essential for AI agents.
• Session-based RBAC via OIDC/OBO and context-appropriate permission scoping help prevent AI from exceeding expected capabilities.
• Agent-to-agent communication increases the need for strict human-in-the-loop architectures.
• AI has not just made attacks faster; it has eroded the asymmetry of ability between attackers and defenders.
• Vulnerability reporting has surged, leaving many reports unclassified.
• One security team handled just over nine times the vulnerabilities with the same staffing, reducing time to remediate by just over 5% using LLMs and supporting harnesses.
• Hiring now favors experienced operators who can catch confident but wrong AI output and understand compliance and governance.
• AI-generated inputs raise concerns about model collapse and the need for human-curated diversity.
• AI is moving application security earlier in the software development life cycle, enabling real-time guidance and reducing review bottlenecks.

The Question Behind the Question

When security leaders gather, one AI-related question dominates: “What data was this model trained on?” The concern is confidentiality. Companies fear intellectual property will become part of an AI’s general knowledge. That concern is valid. But stopping there leaves gaps in a layered security approach. A far less frequently asked question is: “How do you maintain authority and integrity of the data in question?” The answer lies in principles that predate generative AI: least privilege, authentication, and access controls.

For decades, least privilege has been the bedrock of enterprise security. It ensures users and systems have only the access needed to perform their tasks. The same principle applies to AI agents. If an AI agent can read, write, or execute beyond its intended scope, the model’s training data becomes less relevant. The damage is done through excessive permissions. Technologies such as session-based role-based access control (RBAC) via OpenID Connect (OIDC) and on-behalf-of (OBO) flows, combined with appropriate permission scoping for the context, help ensure AI does not exceed the capabilities expected of it.

As agent-to-agent communications become more prevalent, these strategies provide stricter human-in-the-loop architectures. Operators can recognize and correct issues before damage occurs. The goal is not to block AI but to bound it. Integrity and authority over data are maintained through identity, session context, and scoped permissions—not just through model provenance.

How the Attacker Toolkit Has Changed

The attacker toolkit looks different than it did three years ago. Speed is part of the story. State actors and criminal groups race to find and exploit flaws before others discover them. Vulnerability reporting has increased sharply. The National Vulnerability Database (NVD) has had to reprioritize classifications, leaving many reported vulnerabilities unclassified. The volume is simply too high for manual triage.

But the deeper change is the erosion of the asymmetry of ability that the security industry has historically enjoyed. Finding the conditions for an exploit, designing and testing it, and deploying it once required significant expertise. AI has brought that asymmetry closer to even. Exploitable contexts can be found more quickly, and the method to leverage them can be devised and implemented by people who otherwise lack the ability. Increasingly, they can do it faster than those with the ability.

As the contextual data AIs receive grows and models become better at recognizing patterns, this factor will become more significant. Discounting external factors such as government controls, it is possible that symmetry is achieved. At that point, the question becomes one of human nature: how large is the red team compared to the blue team? The side with more automated capability and more persistent exploration may gain the advantage. Defenders must assume that the barrier to entry for sophisticated attacks will continue to fall.

Staffing, AI, and the Vulnerability Flood

Security teams are famously understaffed. Has AI closed the gap or changed its shape? According to the security officer, headcount is not the answer to AI-driven risks. In one organization, the team processed just over nine times the number of vulnerabilities over the last year compared with the previous year, while maintaining the same staffing levels. That created a real gap in process. Using AI tools—largely large language models and supporting harnesses—they handled the increase without additional headcount and simultaneously reduced time to remediate (TTR) by just over 5%.

Achieving that throughput was not trivial. The volume increase was large, and the team had to react in a relatively short time. Experienced human engineers designed the system to leverage what LLMs provide and to stay current as those systems evolve. By embracing the same tools adversaries were using and investing in infrastructure around those tools, the team saw success. The lesson is not that AI replaces staff. It is that AI can augment experienced staff when workflows are redesigned around it.

Still, the pressure remains. Vulnerability management is only one part of security. Incident response, threat hunting, architecture review, and compliance all compete for attention. AI can help automate triage, summarize findings, and draft remediation guidance. But it cannot take responsibility. The human role shifts from doing every task manually to designing, supervising, and validating AI-assisted processes.

Hiring for Judgment, Not Just Output

AI is changing what organizations look for when hiring security talent. Current AI systems are quite capable of producing work on par with many entry-level analysts and engineers. More often than not, they need an experienced operator to recognize confidently wrong outputs and correct them. That takes a broad range of knowledge and experience—not just with diverse technical systems, but with the compliance and governance requirements that drive many security programs.

This raises the bar for all talent. It also creates a gap: it is increasingly difficult for new talent to gain the experience needed to operate these AI systems effectively. If AI handles the entry-level tasks that once served as training grounds, where do junior professionals learn to spot subtle errors? Organizations may need to rethink apprenticeship, mentorship, and review practices. The ability to question AI output, verify sources, and understand business context becomes more valuable than the ability to produce routine work quickly.

Five years ago, hiring might have emphasized tool-specific skills or the ability to execute known procedures. Today, judgment matters more. Security professionals must understand why a control exists, how it can fail, and what an attacker might do to bypass it. They must also understand how AI systems can be manipulated, over-permissioned, or fed misleading data. The best candidates combine technical depth with skepticism and governance awareness.

The Model Collapse Concern

There is an overarching concern about generational processing and the potential for model collapse when the majority—or all—inputs into AI systems are themselves generated by AI systems. Similar to the need for genetic diversity in a population, LLMs need input diversity that today requires human curation. Currently, many organizations think of this as two separate problems: a labor problem and a data problem. But historically, labor was how fresh inputs were provided to the systems. As organizations minimize human involvement, they may be reducing the very diversity that keeps models useful.

For security teams, this has practical implications. If AI-generated content floods training pipelines, models may become less accurate, more biased, or more prone to confident errors. Security controls that rely on AI for anomaly detection or vulnerability triage could degrade. Human curation, red teaming, and diverse data sources become not just nice-to-have but necessary for maintaining model integrity. The same principle of least privilege applies to data: not every data source should be trusted equally, and not every AI output should be fed back without review.

Application Security Moves Left

One of the most exciting developments in AI security is its impact on application-level security. AI allows security concerns to move much earlier in the software development life cycle (SDLC). By evaluating code against a curated list of criteria and policies, AI can surface concerns and give guidance in real time, rather than at a bottlenecked review period. This allows developers to propose and implement solutions with review from specialists, often avoiding the need for additional architecture and design requirements to filter back down the pipeline.

Security teams no longer feel like they are playing a catch-up game to stay current with changes. Many basic tasks that previously depended on external development teams are now internal side projects because, with explicit review, teams know the LLM can do the work. That makes a significant difference in cadence. The shift does not eliminate the need for security expertise. It changes where that expertise is applied: from manual review of every change to designing guardrails, curating policy, and validating AI-generated recommendations.

As AI agents become more autonomous, the principles of least privilege, session-based RBAC, and human-in-the-loop review will remain essential. The question is not only what data trained the model, but who or what can act on that data, under what authority, and with what integrity. Leaders who ask that question will be better positioned to adopt AI without outsourcing their security judgment to it.

Source:Help Net Security News