In a significant development, over three dozen security weaknesses have been unearthed within various open-source artificial intelligence (AI) and machine learning (ML) frameworks, exposing these platforms to risks including remote code execution and potential data breaches.
The vulnerabilities, highlighted in projects like ChuanhuChatGPT, Lunary, and LocalAI, were disclosed under Protect AI’s Huntr bug bounty initiative.
Among the most severe are two critical weaknesses impacting Lunary, a production toolkit for large language models (LLMs):
- CVE-2024-7474 (CVSS score: 9.1) – An Insecure Direct Object Reference (IDOR) vulnerability that could grant authenticated users the ability to view or delete external user information, enabling unauthorized data access and risking data integrity.
- CVE-2024-7475 (CVSS score: 9.1) – A vulnerability rooted in improper access controls, allowing attackers to alter the Security Assertion Markup Language (SAML) configuration. This loophole enables unauthorized users to access restricted data by logging in with elevated permissions.
A further IDOR flaw was discovered within Lunary as well, identified as CVE-2024-7473 (CVSS score: 7.5), which permits an attacker to alter prompts of other users by tampering with a user-controlled parameter.
In an advisory, Protect AI elaborated on this exploit: “An attacker logs in as User A and intercepts the request for prompt updates. By modifying the ‘id’ parameter to reference a prompt belonging to User B, the attacker can alter User B’s prompt without permission.”
Another pressing issue emerged with ChuanhuChatGPT, where a path traversal vulnerability in the user upload function (CVE-2024-5982, CVSS score: 9.1) could lead to unauthorized code execution, arbitrary directory creation, and exposure of sensitive data.
The LocalAI project, an open-source initiative that supports self-hosted LLMs, also harbors two vulnerabilities. The first (CVE-2024-6983, CVSS score: 8.8) allows an attacker to execute arbitrary code by submitting a malicious configuration file, while the second (CVE-2024-7010, CVSS score: 7.5) enables attackers to infer valid API keys through server response timing analysis.
Protect AI noted, “This flaw permits timing attacks, a form of side-channel attack. By observing the response time for different API key entries, an attacker can deduce the valid API key one character at a time.”
A further vulnerability was disclosed in the Deep Java Library (DJL), where an arbitrary file overwrite issue within the untar function (CVE-2024-8396, CVSS score: 7.8) paves the way for remote code execution.
Additionally, NVIDIA has issued patches to resolve a path traversal flaw in its NeMo generative AI framework (CVE-2024-0129, CVSS score: 6.3) to counter risks related to code execution and data tampering.
Mitigation and Defense Strategies
Users are strongly advised to upgrade their installations to the latest versions to bolster the security of their AI/ML ecosystems and prevent potential exploitation.
This disclosure arrives alongside Protect AI’s release of Vulnhuntr, an open-source Python static code analyzer that employs LLMs to detect zero-day vulnerabilities in Python codebases.
Vulnhuntr operates by segmenting code into smaller, digestible parts, thus preventing the LLM from being overwhelmed by large inputs. The tool systematically scans project files to identify files likely to handle user inputs, then dives into each function call chain, following user inputs through to server outputs to map potential vulnerabilities.
“This methodical approach ensures that Vulnhuntr captures the complete function chain from user input to server output for each possible vulnerability, examining each component in detail,” explained Protect AI’s Dan McInerney and Marcello Salvati.
Emerging Exploitation Techniques
Beyond structural weaknesses in AI frameworks, researchers at Mozilla’s 0Day Investigative Network (0Din) have unveiled a novel “jailbreak” method. This technique, by encoding malicious prompts in hexadecimal or emoji formats, can circumvent OpenAI ChatGPT’s security filters, potentially enabling exploitation of known vulnerabilities.
Security expert Marco Figueroa commented, “This jailbreak method leverages a semantic blind spot by directing the model to perform an ostensibly benign task, such as hex conversion. The model’s focus on natural language processing means it follows instructions sequentially without recognizing the potential threat in converting hex data that may yield malicious results.”
In essence, this loophole exists because language models, designed to follow instructions linearly, may lack the contextual awareness needed to identify each step’s risks within the broader, potentially harmful sequence.
