Pillar Security researchers have uncovered a dangerous new supply chain attack vector targeting the AI inference pipeline. This novel technique, termed "Poisoned GGUF Templates," allows attackers to embed malicious instructions that execute during model inference, compromising AI outputs. While developers and AI security vendors focus on validating user inputs and filtering model outputs, our research reveals the critical blind spot between them: the chat template layer.
The attack exploits the GGUF (GPT-Generated Unified Format) model distribution standard to manipulate AI responses during normal conversations. By embedding persistent, malicious instructions directly within these chat templates, attackers can bypass all existing security controls. This targets a massive ecosystem, with hundreds of thousands of GGUF files currently distributed across platforms like Hugging Face. Attackers can smuggle malicious instructions into model components and even manipulate repositories to display clean templates online, while the actual downloaded file contains the poisoned version.
This vector achieves a persistent compromise that affects every user interaction while remaining completely invisible to users and security systems. Because the attack is positioned between input validation and model output, it bypasses most existing AI guardrails, system prompts, and runtime monitoring. This attack remains undetected by current security scanners focused on infrastructure threats, creating a previously unknown supply chain compromise that fundamentally undermines user trust in AI-generated content.
This new attack vector, alongside our earlier Rules File Backdoor discovery, underscores the growing accountability gap in the AI ecosystem.
.png)
To understand this vulnerability, we first need to understand GGUF (GPT-Generated Unified Format) - the file format that has quietly become the backbone of local AI deployment. Think of GGUF as the ".exe" of the AI world: a single file containing everything needed to run a language model locally.
GGUF files aren't just data containers. They include:
What makes GGUF particularly dangerous from a security perspective is that these templates aren't just configuration - they contain active logic processed in every user interaction. Unlike traditional model formats that separate weights from runtime behavior, GGUF bundles everything together, creating a single point of failure that attackers can exploit. This architectural decision, while convenient, means that a compromised GGUF file can silently manipulate AI behavior without touching the model weights themselves.
The explosive growth of GGUF adoption isn't accidental - it solves real problems that developers face when deploying AI:
Understanding how GGUF models propagate through the ecosystem is crucial to grasping the scale of this threat:
Unlike traditional software with established signing and verification processes, the GGUF ecosystem operates on community trust. A model with thousands of downloads is assumed safe simply because others are using it. There's no sufficient security scanning, and minimal documentation of modifications.
For example, the screenshot below shows the existing security scans of a malicious GGUF file on Hugging Face. The existing scanners, like those from ProtectAI and JFrog only look for malicious code embedded within the models, not for AI security vulnerabilities.
This combination of widespread adoption, minimal security controls, and trust-based distribution creates the perfect conditions for this kind of supply chain attack vector.
Our research reveals a novel attack vector in GGUF models that fundamentally changes how we think about AI security threats.
The attack exploits a critical oversight in how GGUF templates process user interactions. Every prompt sent to an AI model passes through a template layer that formats it for the model to understand. We discovered that attackers can embed malicious instructions within these templates.
This creates an entirely new threat model:
The model appears to function normally, passing all security checks, while manipulating only specific responses based on attackers’ goals. Every chatbot conversation, every code generation request, and every AI-assisted analysis becomes a potential backdoor via this attack technique.
In 2024 Patrick Peng discovered how malicious Jinja templates can execute arbitrary code during model loading. His research identified risks during model initialization that compromise the systems running AI models. Our discovery targets the same layer (chat templates), but during inference time.
Our proof of concept demonstrates how a seemingly legitimate model can harbor malicious code within its chat template. While the model weights remain untouched and the HuggingFace repository shows clean code, the actual GGUF file contains a modified template that injects malicious content during specific interactions.
We demonstrated this attack using Microsoft's Phi-4-mini model, chosen because it requires custom templates for advanced features like function calling. Here's what makes this attack particularly insidious:
What Users See:
What's Happening:
The malicious script injection only activates for HTML-related requests, making detection nearly impossible through casual testing.
.png)
The attack exploits a critical gap in the GGUF distribution pipeline. The attackers can download a model with a standard template, modify the template and repackage the model into a new GGUF file. Then they upload to Hugging Face and wait for the malicious instructions to get activated.
Crafting the payload is as simple as editing a word document. To increase the stealth of this supply chain attack, the attackers can even upload a clean version of the quantized model to Hugging Face and trick the platform into displaying the clean version of the chat template as the default.
%20(1).png)
As of today, hundreds of thousands of GGUF files are distributed across platforms like Hugging Face, representing a massive attack surface for this novel supply chain threat. The vulnerability landscape is particularly concerning because of how chat templates are handled across different model categories.
Standard Models: Low Risk Profile
Many foundational models can use simple chat templates that are automatically detected from their metadata or provided as defaults by inference frameworks such as llama.cpp, Ollama, vLLM, or SGLang. These models present a lower attack surface because users typically rely on the framework's built-in templates rather than custom ones embedded in GGUF files.
Advanced Capability Models: Higher-Risk Targets
Models with specialized capabilities - including tool calling, reasoning, image processing, or audio processing - represent the highest-risk attack vectors. These models often require custom chat template formats to function properly, creating the perfect cover for poisoned templates. For instance, a recent model from Microsoft's Phi family, Phi-4-mini, supports function calling capabilities through specific template formatting, making template customization not just acceptable but necessary in order to actually enable tool calling.
This vulnerability exposes a critical gap, as many organizations build their entire AI security posture around prompt engineering and input/output guardrails, incorrectly assuming what is sent to the model is what is processed by the model.
Even sophisticated AI security platforms are blind to this vector because they only monitor prompts and outputs at the API level, without inspecting the GGUF file's internal structure. Without comprehensive template auditing and model provenance, these security layers become ineffective. Organizations must recognize that model files are not passive data containers but active code requiring the same scrutiny as any other part of the software supply chain.
This vulnerability exposes critical gaps in standard defense-in-depth architectures, but there are practical steps that can be taken at every level to mitigate the risk. Defending against poisoned GGUF templates requires a combination of immediate actions, architectural changes, and a coordinated industry response.
Immediate Actions for Practitioners
The primary defense against this attack vector is the direct inspection of GGUF files to identify chat templates containing uncommon or non-standard instructions. Security teams should immediately:
Hugging Face
LM Studio
The responses above, which place these new kinds of attacks outside the AI platform vendors' responsibility, underscore the importance of public awareness regarding the security implications of AI model distribution and deployment. This research reveals an expanded attack surface that can affect the AI ecosystem, especially as given the growing reliance on AI-generated outputs across industries and the widespread adoption of GGUF models in production environments, the lack of vendor accountability for template security creates a significant blind spot that organizations must address through their own security practices.
The discovery of poisoned GGUF templates reveals a critical vulnerability in the AI supply chain. This research demonstrates that GGUF files are not just data containers; they can be weaponized to bypass an organization's most trusted security controls, including system prompts and input filters.
Both the loading-time and the newly discovered inference-time attack vectors show that the current trust-based model for sharing GGUF files is insufficient. Attackers can manipulate templates to compromise either the model operator during loading or, more insidiously, every end-user during inference by making the model itself a malicious actor.
Addressing this threat requires a coordinated industry response analogous to traditional software supply chain security. As AI models become more deeply integrated into our digital infrastructure, practices like model provenance verification and comprehensive template auditing must become standard, non-negotiable components of any secure AI deployment.
The Pillar platform discovers and flags malicious GGUF files and other types of risks in the template layer.
Subscribe and get the latest security updates
Back to blog
.png){kind=small}
%20(1).jpg)
%20(1).png)
.jpg)
%20(1).png)
%20(1)%20(1).jpg)
.png)