Guide Cybersecurity AI

Polymorphic and AI-Assisted Malware: How It Works and How to Detect It

Polymorphic and metamorphic malware change their structure to get past some detection methods, and they have done so for decades. AI models give attackers and researchers new ways to generate, modify and adapt code. It matters to separate what has been shown in the lab from what has been seen in real campaigns. This guide explains the mechanisms, their limits and the controls that help organisations prevent, detect and contain these threats.

9 June 2026
Updated 18 November 2026
7 min read
Key takeaways
  • Polymorphism is not new and does not need AI: malware has changed its encrypted form or its code structure since the early 1990s
  • AI can help produce code variants that look different while aiming to keep the same function, but functional equivalence is not guaranteed
  • Research proofs of concept such as BlackMamba (2023) show what is possible; they are not evidence of widespread autonomous malware
  • A changed file defeats exact-hash lookups and some static signatures, but modern endpoint protection also uses behaviour, memory, emulation and reputation
  • Different binaries still show recognisable behaviour: unusual child processes, credential access, new outbound connections and persistence
  • Prevention matters as much as detection: application control, attack surface reduction, phishing-resistant MFA and patching by known exploitation

From mutation engines to AI-assisted code

Polymorphic malware appeared in the early 1990s, when the Dark Avenger Mutation Engine (MtE) showed that one piece of malware could produce thousands of different binary forms. The core technique stayed the same for years: encrypt the malicious payload and vary the decryption stub, so each infection looks different to a scanner that matches exact patterns.

Metamorphic malware, which appeared in the late 1990s and early 2000s, went further. Instead of encrypting a fixed payload, it rewrites its own instructions on each run (reordering them, swapping in equivalent operations, adding instructions with no real function) so the structure changes while the behaviour stays the same. Toolkits such as NGVCK generated metamorphic variants automatically.

TechniqueWhat changesDoes it need AI?
Polymorphic malwareThe encrypted or encoded form of the malicious codeNo
Metamorphic malwareThe structure of the executable instructionsNo
AI-assisted malware developmentCode generated or modified with a language model, before or during executionYes
Autonomous malicious agentAttack tasks planned or carried out with varying independenceUsually AI-based; capability depends on the implementation

Large language models can generate syntactically different versions of a piece of code at a higher level than reordering assembly: whole logic blocks, different languages, variable names, control flow and API call sequences. The aim is to keep the function while changing the form. That is not guaranteed. Automatically rewritten code can break, behave differently, introduce compatibility problems or carry patterns of its own that defenders can detect.

Proof of concept: BlackMamba

In 2023, researchers at HYAS presented BlackMamba, a proof of concept that called a language model at runtime to generate keylogging code and ran it in memory. The experiment showed how dynamic generation can make traditional detection and some endpoint controls harder. Read the results within the conditions of the test: a lab demonstration against selected controls does not show that the technique defeats properly configured modern EDR in general, or that it works reliably at scale.

What AI changes, and what is still research

It helps to separate four things that are often mixed together: conventional malware that uses polymorphism or metamorphism; attackers using language models to help with scripts, phishing or obfuscation; research demonstrations that generate executable code on the fly; and autonomous agents that plan and carry out multi-stage intrusions. These differ in capability, reliability and how much the attacker stays in control. The first two are well documented. The last two are an active research area, with limited public evidence of operational use.

Code variants written with AI help
A language model rewrites functional logic into versions that look different, before or during execution. This defeats indicators based on exact file hashes and some static signatures. It does not hide what the code does when it runs, which is where behavioural detection looks.
Environment-aware evasion
Malware has long checked for sandboxes and analysis tools before running, looking for virtual machine artefacts, specific registry keys or unusual mouse inactivity. AI could make these checks more adaptive. The checks themselves are also behaviour that defenders can detect.
Automated reconnaissance and lateral movement
Automation, sometimes with AI, can speed up mapping a network and finding high-value targets such as domain controllers, finance systems and credential stores. Researchers have shown agents that carry out parts of this chain with little human direction. Treat fully autonomous intrusions as an emerging risk rather than an everyday one, and plan for faster attacks either way.
AI-written phishing and initial access
Language models make convincing, personalised phishing cheaper to produce at scale, using material from LinkedIn, company websites and public filings. Poor grammar and generic wording are no longer reliable warning signs, so training should focus on the request itself and on reporting.

AI tools sold on criminal forums

Services advertised on criminal forums under names such as WormGPT (mid-2023) and FraudGPT have been marketed as AI tools without restrictions for phishing, fraud and malicious code. Their advertised capabilities are rarely verified independently, and some offerings are resold, exaggerated or scams aimed at other criminals. Their existence does not show that sophisticated autonomous malware is widely available.

What these tools and general-purpose models do lower is the effort needed for phishing content, simple scripts and social engineering. The more advanced capabilities, such as reliable runtime code generation or adaptive evasion, still need real technical skill.

The democratisation of AI tools does not primarily give unsophisticated attackers sophisticated capabilities. It gives sophisticated attackers speed, scale and the ability to personalise attacks far beyond what was previously economical.

What signature detection misses, and what still works

A change to a file's structure can defeat indicators based on an exact hash and some static signatures. But an antivirus signature is not just a file hash. Modern protection also inspects byte patterns in decoded content, emulates code, checks reputation, watches behaviour and looks at memory. How well evasion works depends on the technique and on the controls in the environment. MITRE ATT&CK lists polymorphic code as a defence evasion technique and also describes how to detect it.

Behavioural detection has limits too. Some malware delays its malicious actions until it judges that it is not being analysed: it stays dormant, runs only on certain dates or waits for user action, so an automated sandbox can miss it. Machine learning detection models are strong against new samples but not immune, because adversarial inputs designed to look normal to a model are a documented attack category.

The key point for defenders: different binaries can still show recognisable behaviour. A sample whose file changes on every infection may still launch unusual child processes from an Office application, read credential stores, open connections to new external hosts or create persistence. Behavioural blocking and containment and detection engineering focus on exactly these signals.

What to do

ObjectiveControls
Prevent executionApplication control, exploit protection, attack surface reduction rules, modern endpoint protection
Reduce initial accessPhishing-resistant MFA, vulnerability management, email security, least privilege
Detect executionEndpoint behaviour telemetry, memory signals, suspicious process activity
Detect lateral movementIdentity telemetry, network monitoring, unusual authentication and remote execution
ContainEndpoint isolation, credential revocation, segmentation
RecoverTested backups, incident response procedures, validated restoration

Stop unapproved code from running

Detecting malicious code after it runs is only half the job. Application control (allowlisting which executables and scripts may run), script restrictions and attack surface reduction rules block many techniques whatever the binary looks like. Roll these out in audit mode first, review what would have been blocked, then enforce in stages so business applications keep working.

Use modern endpoint protection, matched to your capacity

Use endpoint protection with prevention, behavioural analysis, telemetry and response capabilities. Choose between EPP, EDR, XDR and managed detection and response (MDR) based on your team's capacity to act on alerts, integration with your other controls, device coverage and results in independent tests. The label "AI-powered" says little on its own; configuration, coverage and the people responding matter more.

Protect initial access

Require MFA on all accounts with external access, phishing-resistant authentication (passkeys or security keys) for privileged accounts, and an email security gateway that inspects links and attachments. Which controls matter most depends on your real attack paths: malware also arrives through exploited vulnerabilities, compromised software updates, drive-by downloads and abused administration tools.

Patch by known exploitation

Automation and AI tools can help with parts of vulnerability research and exploitation. Whatever their role, prioritise vulnerabilities known to be exploited, internet-facing systems and business-critical assets. The CISA Known Exploited Vulnerabilities catalog is a good operational reference.

Segment your network

Lateral movement only helps an attacker if the infected endpoint can reach other systems. Separating production systems, administrative interfaces, backup infrastructure and user workstations into zones with controlled traffic paths limits what one compromised device can reach. A device on the finance network should not be able to open connections to manufacturing control systems or the backup server.

Add network and identity detection

Network detection and response (NDR) tools can spot anomalies that endpoint agents miss, especially when attackers use legitimate tools already on the system ("living off the land"): unexpected internal scanning, unusual data volumes moved to staging locations or connections to new external hosts at odd hours. What they see depends on where sensors sit and what traffic they can observe, so treat them as a complement to endpoint and identity telemetry, not a guarantee.

Hunt for threats and test your detections

Threat hunting is the proactive search for attacker activity that has not triggered alerts. For most SMEs this is best delivered through a managed provider, but even periodic manual review of endpoint telemetry against threat intelligence adds value. Test your detections against the behaviours above, so you know they fire.

Our cybersecurity team helps organisations assess their current detection capability, select and deploy endpoint protection and managed detection, and design network segmentation that reflects their actual risk profile. Speak to us if you want a practical assessment of your exposure to this class of threat.

Sources

Frequently asked questions

What is the difference between polymorphic and metamorphic malware?

Polymorphic malware encrypts or encodes its payload and changes the decryption stub on each infection, so its binary form differs. Metamorphic malware rewrites its own instructions, producing structurally different code that aims to behave the same, without needing encryption. Neither needs AI; both have existed since the 1990s. AI adds new ways to generate and modify code.

Can AI generate working malware from scratch?

Language models can help write malicious code, and research such as BlackMamba (2023) has shown runtime code generation in lab conditions. The generated code is not always correct or reliable, and the capabilities advertised by criminal AI services are rarely verified independently. AI mainly makes skilled attackers faster and lowers the effort needed for phishing and simple scripts.

Can polymorphic malware evade any antivirus?

No. Changing a file's form defeats exact-hash lookups and some static signatures, but modern endpoint protection also uses behaviour monitoring, emulation, memory inspection, reputation and machine learning. The malicious actions themselves, such as credential access or persistence, remain detectable.

Is traditional antivirus still effective against AI-assisted malware?

Signature-only antivirus has limited effect against code that changes its form. Endpoint protection that combines prevention with behavioural detection and response is far stronger, especially alongside application control, attack surface reduction and network and identity monitoring.

What should an SME prioritise?

Stop unapproved code from running with application control and attack surface reduction, use modern endpoint protection that someone actually monitors (in-house or through MDR), require MFA on all accounts with phishing-resistant methods for administrators, patch known exploited vulnerabilities first, segment the network and keep tested backups.

Ryland Deakin
About the author
Lead Consultant, Cyvra · CISM · CompTIA Security+ · MCP

Ryland has delivered cybersecurity, compliance, and IT management programmes for regulated organisations across the UK and the Netherlands for over 20 years. View full profile

Endpoint Security

Want to know how well you'd detect these threats?

We assess your current detection capability and help you deploy EDR, MDR, and network detection tools that keep pace with modern threats.