Attackers are using less stringent security checks to post malicious projects to Hugging Face and other repositories for open source artificial intelligence ( AI ) models. The growing issue emphasizes the need for businesses pursuing inside AI projects to possess powerful tools to identify security flaws and malicious code in their supply chains.
Hugging Face’s automated checks, for example, just failed to detect malicious code in two AI models hosted on the store, according to a Feb. 3 study published by program provide network security firm ReversingLabs. The threat actor used a common vector — data files using the Pickle format — , with a new technique, dubbed” NullifAI”, to evade detection.
While the assaults appeared to be proofs-of-concept, their achievement in being hosted with a” No problem” label shows that companies should not rely on Hugging Face’s and other repositories ‘ health checks for their own protection, says Tomislav Pericin, chief software architect at ReversingLabs.
” You have this common collection where any designer or device learning expert you host their own thing, and obviously harmful actors abuse that,” he claims. ” Depending on the ecosystem, the matrix is going to be somewhat different, but the concept is the same: One’s going to host a destructive version of a thing and hope for you to unwittingly place it”.
Hugging Face, TensorFlow Hub, and PyTorch Hub are just a few examples of companies using open source AI types from libraries. Overall, 61 % of companies are using models from the open source ecosystem to create their own AI tools, according to a of 2, 400 IT decision-makers sponsored by IBM.
Yet many of the components can contain executable code, leading to a variety of safety risks, such as script murder, backdoors, fast injections, and line issues — the latter being how well an AI design matches the intent of the developers and users.
In an Insecure Pickle
One of the most important issues is that a pickle file, a widely used data format, is unsecure and can be used to execute arbitrary code. Despite vocal warnings from security researchers, the continues to be used by many data scientists, says Tom Bonner, vice president of research at HiddenLayer, an AI-focused detection and response firm.
” I really hoped that we’d make enough noise about it that Pickle would’ve gone by now, but it’s not”, he says. ” I’ve seen organizations compromised through machine learning models — multiple]organizations ] at this point. So yeah, whilst it’s not an everyday occurrence such as ransomware or phishing campaigns, it does happen”.
Hugging Face performs explicit checks for Pickle files, but ReversingLabs ‘ malicious code was bypassed by using a different file compression for the data. Application security firm Checkmarx conducted additional research to discover novel ways to detect dangerous Pickle files, including PickleScan used by Hugging Face.
Despite having offensive features, this model passes Hugging Face security checks. Source: ReversingLabs
” PickleScan uses a blocklist which was successfully bypassed using both built-in Python dependencies”, Dor Tumarkin, director of application security research at Checkmarx, . It is obviously vulnerable, but it would still be vulnerable with highly popular imports in its scope if it were to use third-party dependencies like Pandas to bypass it, even if it were to take into account all cases baked into Python.
Rather than Pickle files, data science and AI teams should move to Safetensors — a library for a new data format managed by Hugging Face, EleutherAI, and Stability AI — , which . The Pickle format is regarded as much safer than the Safetensors format.
Deep-Seated AI Vulnerabilities
Executable data files are not the only threats, however. Another problem is that pre-trained AI models, which are frequently referred to as “open source AI,” typically do not include all the necessary data, such as training data and code. Instead, they provide the training’s weights, and they are protected by licenses that aren’t always open source compatible.
According to Andrew Stiefel, a senior product manager at Endor Labs, using such models to create commercial products or services could lead to license violations.
” There’s a lot of complexity in the licenses for models”, he says. ” You have the actual model binary itself, the weights, the training data, all of those could have different licenses, and you need to understand what that means for your business”.
The final wildcard is model alignment, which measures how well its output matches the values of the developers and users. DeepSeek, for example, allows users to create malware and viruses, <a class="ContentText-BodyTextChunk ContentText-BodyTextChunk_link" target="_self" href="https://www.darkreading.com/cyber-risk/deepseek-fails-multiple-security-tests-business-use”>researchers found. Other models — such as OpenAI’s o3-mini model, which boasts more stringent alignment — has <a class="ContentText-BodyTextChunk ContentText-BodyTextChunk_link" target="_self" href="https://www.darkreading.com/application-security/researcher-jailbreaks-openai-o3-mini”>already been jail broken by researchers.
Researchers are still finding it difficult to determine how to test for such weaknesses because these issues are unique to AI systems, according to Pericin from ReversingLabs.
There is already research on what kind of prompts might cause the model to act in an unanticipated manner, divulge sensitive information, or instruct actions that might be harmful, he says. ” That’s a whole other discipline of machine learning model safety that people are, in all honesty, mostly worried about today”.
Companies should be sure to understand any permits that apply to the AI models they employ. In addition, they should pay attention to common signals of software safety, including the source of the model, development activity around the model, its popularity, and the operational and security risks, Endor’s Stiefel says.
” You kind of need to manage AI models like you would any other open source dependencies”, Stiefel says. You are bringing them in because they were created by people outside your organization, which requires that you take a similar holistic approach to risk assessment.