Study Compares ModelScan, ModelAudit, Fickling, Reveals They Miss Malicious ML Payloads
Researchers Qianlong Lan, Vinothini Pandurangan, and Anuj Kaul evaluated three static scanners—ModelScan, ModelAudit, and Fickling—for detecting unsafe content in machine-learning artifacts. Using a controlled dataset, they found that conventional metrics only measure cases where scanners produce a usable security judgment, ignoring failures where scanners return no verdict or crash. The study reveals significant coverage gaps: scanners often miss malicious payloads embedded in PyTorch or TensorFlow files, and recovery from malformed inputs is inconsistent across tools. This is the first systematic comparison of failure modes in ML artifact scanners, highlighting that current tools are not yet reliable for production security pipelines.
breakthroughs8/28/2026