Adversaries have never been shy about abusing new technology. While ethical developers spent 2024 and 2025 building guardrails, threat actors spent that same time figuring out how to route around them. The democratization of powerful LLMs has lowered the barrier to entry for complex attacks and turned script kiddies into far more capable operators.

1. Malicious LLMs: From WormGPT to DarkBART-Style Models

The emergence of models like WormGPT and its successors marked a clear shift. These are effectively “jailbroken” LLMs trained on malware code, exploit writeups, and phishing templates—without the ethical constraints of mainstream tools.

Instead of simply helping an attacker write a generic phishing email, malicious LLMs now generate hyper‑personalized, context‑aware lures tuned to a target’s role, region, and internal jargon. They strip out classic red flags (poor grammar, odd phrasing), making it far easier for these messages to bypass traditional secure email gateways (SEGs) and trick even savvy users.

2. AI-Amplified Social Engineering and Deepfakes

We’ve entered the era of technology‑enhanced social engineering. The now‑well‑known Hong Kong “deepfake CFO” case where an employee was tricked into authorizing a large wire transfer during a video call where the “colleagues” were AI‑generated fakes is no longer an outlier. It’s a preview.

Today’s social engineering blends:

• Real-time voice cloning to impersonate executives or vendors on calls.

• Video synthesis and face swapping to fake presence in meetings.

• LLM‑written scripts and emails that sustain long‑running scams with consistent tone and believable detail.

These multi‑modal attacks directly weaponize human trust. For a deeper dive into how deepfakes and GenAI are changing social engineering tradecraft and how verification processes are being inverted. See ourSocial Engineering Guide and our Tradecraft Tuesday recap onAI: Friend or Faux in Cybersecurity?.

3. Polymorphic Malware and Automated Vulnerability Research

Attackers are also using GenAI to automate some of the most labor‑intensive parts of the kill chain.

• Polymorphic malware:
  As we’ve covered in our guide topolymorphic viruses, polymorphic malware mutates its code or appearance on each execution while keeping its malicious behavior intact. GenAI can now dynamically:

  • Rewrite code structure and encryption routines

  • Rotate keys and obfuscation strategies

  • Generate many slightly different variants on demand

• This makes simple, signature‑based defenses effectively useless. Detection has to pivot to behavior, telemetry, and human‑guided analysis rather than static hashes.

• Automated vulnerability research (“vibe coding” for exploits):
  Adversaries are feeding large proprietary codebases into LLMs to identify potential weaknesses at a scale humans can’t match. In some cases, models can propose exploit patterns or proof‑of‑concepts directly, dramatically shortening the time from bug discovery to weaponization. That’s “vibe coding” applied to offensive security.

4. AI Changes the Economics of Attack: The EvilTokens Case Study

One of the clearest signs that Generative AI has changed the economics of attack is not a research paper—it is a product sold on Telegram. In early 2026, Huntress researchers traced a wave of device-code-phishing incidents to EvilTokens, a phishing-as-a-service platform advertised for $1,500 plus a $500 maintenance fee.

During a 16-day surge, EvilTokens affected 344 organizations across five countries by abusing a legitimate Microsoft device-code authentication flow. There was no password to steal, malware to deploy, or MFA mechanism to break. Instead, victims were persuaded to authorize the attacker’s session through a process that looked normal to them.

AI helped make that workflow faster and more scalable. EvilTokens generated role-specific, contextually relevant lures at machine speed, dynamically generated device codes when users reached the phishing page, and used inbox, calendar, and document context to draft convincing follow-on wire-fraud emails in the victim’s voice within minutes of token capture. What previously required hours of work from a skilled social engineer could now be compressed into minutes.

That is the real shift. EvilTokens did not need to invent a new authentication technique. It packaged an existing one with AI-assisted automation, sold it as a service, and lowered the entry cost for running a targeted phishing campaign to roughly $2,000 before other operating expenses. The barrier is no longer only technical skill; it is access to a ready-made platform and the budget to use it.

5. Trust Is Now an Attack Surface

As AI tools become part of daily workflows, attackers are targeting the trust people place in those tools directly—using familiar brands, legitimate domains, and trusted platform features to make malicious actions look routine.

Between July 21 and July 22, 2026, Huntress documented FakeAgent, a malvertising campaign that used sponsored Bing results to direct users to a malicious public Claude Artifact hosted on the legitimate Claude.ai domain. The Artifact redirected users to a fake Claude Desktop download, and the resulting payload was attributed to SectopRAT. At least 29 organizations fell victim to the campaign, and the Artifact received 7,100 page views before it was removed.

In a separate July investigation, Huntress reverse-engineered MacSync, a six-stage macOS stealer and remote access trojan. The victim searched Google for how to install Claude on a Mac, clicked a sponsored result, and reached a weaponized claude.ai/share conversation styled as an Apple Support installation guide. The page instructed the victim to paste a single Terminal command, which launched a chain designed to steal browser data, Keychain secrets, account credentials, cloud and SSH keys, and cryptocurrency-wallet information.

Neither campaign required attackers to break the underlying AI service. Instead, both abused the trust signals surrounding AI tools: legitimate domains, public shares, sponsored search results, and familiar installation flows. As AI platforms become as routine as search engines, attackers will keep counterfeiting that familiarity rather than trying to break the technology underneath it.

GenAI isn’t just an offensive tool. It’s also becoming essential for defenders who need to process billions of signals per day across endpoints, identities, SaaS, and cloud. At that scale, humans alone can’t keep up, but AI alone can’t be trusted to make every decision.

1. Self-Healing Code and Adaptive Honeypots

Defenders are using GenAI to close the gap between detection and response:

• Self-healing systems: Modern security platforms can flag misconfigurations and vulnerabilities in near real time, then suggest or automatically apply remediations tailored to the specific environment. Think of it as AI‑augmented patching and configuration management that shrinks your exposure window.

• Adaptive honeypots and deception: AI can generate convincing decoy environments on the fly—fake file shares, identity graphs, and application instances that closely mirror your real infrastructure. These honeypots adapt as attackers probe them, keeping adversaries engaged longer while your team collects high‑fidelity intelligence on their tactics, techniques, and procedures (TTPs).

2. The AI-centric SOC reality

The real value of AI in the SOC isn’t replacement; it’s triage at scale.

AI is well‑suited to:

• Correlate noisy alerts from endpoints, identities, email, and network telemetry

• Enrich raw signals with context (asset criticality, recent changes, user behavior)

• Surface a smaller, higher‑quality set of investigations for analysts

Where AI struggles is binary judgment: deciding with certainty whether activity is malicious in a messy, real‑world environment. As we’ve noted elsewhere, current models are strong at inference but weak at saying “I don’t know” and they’re vulnerable to prompt injection, hallucinations, and blind spots.

That’s where human analysts remain non‑negotiable.

While the AI vs. AI battle plays out in the headlines, IT admins and security leads are left to manage real risk inside their own environments. Three concerns are now top of mind.

1. Data Leakage and Shadow AI

Data leakage is the quiet killer of corporate IP.

Employees, eager for productivity gains, routinely paste:

• Customer lists and deal data

• Proprietary source code or scripts

• Financial models and roadmap details into public LLMs and unvetted browser extensions.

This fuels Shadow AI: unsanctioned, invisible AI usage that routes around your existing controls and governance. Without strong policies and monitoring, your perimeter may look locked down while sensitive data steadily bleeds out through AI tools you don’t control.

2. Prompt Injection: The New SQLi

Just as security teams had to learn to sanitize database inputs to prevent SQL injection, we now have to secure prompts and model inputs.

Prompt injection attacks aim to:

• Override the system’s original instructions

• Exfiltrate sensitive internal knowledge

• Abuse the model’s integrations (e.g., file access, ticketing, or admin APIs)

• Induce unauthorized actions, from changing configurations to leaking secrets

Any AI agent that can “take actions” on your behalf must be treated as a powerful, high‑risk component and defended with the same rigor as a production web app.

3. The “AI-only SOC” myth

There’s a growing marketing push for the fully “autonomous SOC”—the promise that you can hand the keys to an AI agent and walk away.

At Huntress, we view that as a liability, not a strategy.

AI lacks situational awareness and a sense of consequence. It can see anomalies but doesn’t understand:

• That a rare login might be an engineer on a last‑minute trip, not an intruder

• That a noisy tool on a legacy system is business‑critical and can’t just be killed

• That an attacker “living off the land” with legitimate tools may look similar to a power user doing advanced troubleshooting

An AI-only SOC risks both false positives with real business impact (unnecessary shutdowns, blocked processes) and false negatives where subtle, human‑driven intrusions slip through.

The push toward a fully AI‑driven SOC runs straight into a core requirement of security programs: accountability.

When a breach occurs:

• An algorithm can’t brief your board, regulators, or customers.

• A model can’t be questioned about judgment calls it made or context it missed.

• “The AI did it” is not a defensible control explanation.

Human analysts provide the sanity check that AI cannot. They’re responsible for interpreting signals, weighing business context, and making risk‑informed decisions.

The most effective posture we see is AI‑assisted, human‑led:

• AI handles scale: normalizing telemetry, ranking alerts, and enriching context.

• Humans handle judgment: deciding what’s truly malicious, what to contain, and how to communicate and remediate.

To navigate this new landscape, organizations need more than tools they need governance. Use this checklist as a starting point for managing GenAI and reducing Shadow AI risk.

This framework won’t eliminate AI risk, but it transforms it from an uncontrolled experiment into a managed, auditable program.

Generative AI has radically reshaped the security landscape, but it hasn’t changed the core mission: protecting people and their data.

AI is a double‑edged sword:

• In an attacker’s hands, it accelerates phishing, deepfakes, reconnaissance, and exploit development.

• In a defender’s hands, it accelerates detection, investigation, and remediation—if guided and governed correctly.

Organizations that chase a pure “AI‑only” solution are betting their risk posture on systems that lack context, accountability, and true understanding. Organizations that adopt an AI‑assisted, human‑led model keep people in the loop for the decisions that matter most.

That’s the future of security: a “centaur” model where the speed and scale of machines are directed by the judgment, experience, and accountability of humans.