Adversaries have never been shy about abusing new technology. While ethical developers spent 2024 and 2025 building guardrails, threat actors spent that same time figuring out how to route around them. The democratization of powerful LLMs has lowered the barrier to entry for complex attacks and turned script kiddies into far more capable operators.

1. Malicious LLMs: From WormGPT to DarkBART-Style Models

The emergence of models like WormGPT and its successors marked a clear shift. These are effectively “jailbroken” LLMs trained on malware code, exploit writeups, and phishing templates—without the ethical constraints of mainstream tools.

Instead of simply helping an attacker write a generic phishing email, malicious LLMs now generate hyper‑personalized, context‑aware lures tuned to a target’s role, region, and internal jargon. They strip out classic red flags (poor grammar, odd phrasing), making it far easier for these messages to bypass traditional secure email gateways (SEGs) and trick even savvy users.

2. AI-Amplified Social Engineering and Deepfakes

We’ve entered the era of technology‑enhanced social engineering. The now‑well‑known Hong Kong “deepfake CFO” case where an employee was tricked into authorizing a large wire transfer during a video call where the “colleagues” were AI‑generated fakes is no longer an outlier. It’s a preview.

Today’s social engineering blends:

• Real-time voice cloning to impersonate executives or vendors on calls.

• Video synthesis and face swapping to fake presence in meetings.

• LLM‑written scripts and emails that sustain long‑running scams with consistent tone and believable detail.

These multi‑modal attacks directly weaponize human trust. For a deeper dive into how deepfakes and GenAI are changing social engineering tradecraft and how verification processes are being inverted. See ourSocial Engineering Guide and our Tradecraft Tuesday recap onAI: Friend or Faux in Cybersecurity?.

3. Polymorphic Malware and Automated Vulnerability Research

Attackers are also using GenAI to automate some of the most labor‑intensive parts of the kill chain.

• Polymorphic malware:
  As we’ve covered in our guide topolymorphic viruses, polymorphic malware mutates its code or appearance on each execution while keeping its malicious behavior intact. GenAI can now dynamically:

• Rewrite code structure and encryption routines

• Rotate keys and obfuscation strategies

• Generate many slightly different variants on demand

• This makes simple, signature‑based defenses effectively useless. Detection has to pivot to behavior, telemetry, and human‑guided analysis rather than static hashes.

• Automated vulnerability research (“vibe coding” for exploits):
  Adversaries are feeding large proprietary codebases into LLMs to identify potential weaknesses at a scale humans can’t match. In some cases, models can propose exploit patterns or proof‑of‑concepts directly, dramatically shortening the time from bug discovery to weaponization. That’s “vibe coding” applied to offensive security.

GenAI isn’t just an offensive tool. It’s also becoming essential for defenders who need to process billions of signals per day across endpoints, identities, SaaS, and cloud. At that scale, humans alone can’t keep up, but AI alone can’t be trusted to make every decision.

1. Self-Healing Code and Adaptive Honeypots

Defenders are using GenAI to close the gap between detection and response:

• Self-healing systems: Modern security platforms can flag misconfigurations and vulnerabilities in near real time, then suggest or automatically apply remediations tailored to the specific environment. Think of it as AI‑augmented patching and configuration management that shrinks your exposure window.

• Adaptive honeypots and deception: AI can generate convincing decoy environments on the fly—fake file shares, identity graphs, and application instances that closely mirror your real infrastructure. These honeypots adapt as attackers probe them, keeping adversaries engaged longer while your team collects high‑fidelity intelligence on their tactics, techniques, and procedures (TTPs).

2. The AI-centric SOC reality

The real value of AI in the SOC isn’t replacement; it’s triage at scale.

AI is well‑suited to:

• Correlate noisy alerts from endpoints, identities, email, and network telemetry

• Enrich raw signals with context (asset criticality, recent changes, user behavior)

• Surface a smaller, higher‑quality set of investigations for analysts

Where AI struggles is binary judgment: deciding with certainty whether activity is malicious in a messy, real‑world environment. As we’ve noted elsewhere, current models are strong at inference but weak at saying “I don’t know” and they’re vulnerable to prompt injection, hallucinations, and blind spots.

That’s where human analysts remain non‑negotiable.

While the AI vs. AI battle plays out in the headlines, IT admins and security leads are left to manage real risk inside their own environments. Three concerns are now top of mind.

1. Data Leakage and Shadow AI

Data leakage is the quiet killer of corporate IP.

Employees, eager for productivity gains, routinely paste:

• Customer lists and deal data

• Proprietary source code or scripts

• Financial models and roadmap details into public LLMs and unvetted browser extensions.

This fuels Shadow AI: unsanctioned, invisible AI usage that routes around your existing controls and governance. Without strong policies and monitoring, your perimeter may look locked down while sensitive data steadily bleeds out through AI tools you don’t control.

2. Prompt Injection: The New SQLi

Just as security teams had to learn to sanitize database inputs to prevent SQL injection, we now have to secure prompts and model inputs.

Prompt injection attacks aim to:

• Override the system’s original instructions

• Exfiltrate sensitive internal knowledge

• Abuse the model’s integrations (e.g., file access, ticketing, or admin APIs)

• Induce unauthorized actions, from changing configurations to leaking secrets

Any AI agent that can “take actions” on your behalf must be treated as a powerful, high‑risk component and defended with the same rigor as a production web app.

3. The “AI-only SOC” myth

There’s a growing marketing push for the fully “autonomous SOC”—the promise that you can hand the keys to an AI agent and walk away.

At Huntress, we view that as a liability, not a strategy.

AI lacks situational awareness and a sense of consequence. It can see anomalies but doesn’t understand:

• That a rare login might be an engineer on a last‑minute trip, not an intruder

• That a noisy tool on a legacy system is business‑critical and can’t just be killed

• That an attacker “living off the land” with legitimate tools may look similar to a power user doing advanced troubleshooting

An AI-only SOC risks both false positives with real business impact (unnecessary shutdowns, blocked processes) and false negatives where subtle, human‑driven intrusions slip through.

The push toward a fully AI‑driven SOC runs straight into a core requirement of security programs: accountability.

When a breach occurs:

• An algorithm can’t brief your board, regulators, or customers.

• A model can’t be questioned about judgment calls it made or context it missed.

• “The AI did it” is not a defensible control explanation.

Human analysts provide the sanity check that AI cannot. They’re responsible for interpreting signals, weighing business context, and making risk‑informed decisions.

The most effective posture we see is AI‑assisted, human‑led:

• AI handles scale: normalizing telemetry, ranking alerts, and enriching context.

• Humans handle judgment: deciding what’s truly malicious, what to contain, and how to communicate and remediate.

To navigate this new landscape, organizations need more than tools they need governance. Use this checklist as a starting point for managing GenAI and reducing Shadow AI risk.

This framework won’t eliminate AI risk, but it transforms it from an uncontrolled experiment into a managed, auditable program.

Generative AI has radically reshaped the security landscape, but it hasn’t changed the core mission: protecting people and their data.

AI is a double‑edged sword:

• In an attacker’s hands, it accelerates phishing, deepfakes, reconnaissance, and exploit development.

• In a defender’s hands, it accelerates detection, investigation, and remediation—if guided and governed correctly.

Organizations that chase a pure “AI‑only” solution are betting their risk posture on systems that lack context, accountability, and true understanding. Organizations that adopt an AI‑assisted, human‑led model keep people in the loop for the decisions that matter most.

That’s the future of security: a “centaur” model where the speed and scale of machines are directed by the judgment, experience, and accountability of humans.