Software Security
Software-side security of AI covers adversarial examples, poisoning, model extraction, privacy leakage, and prompt-level attacks in modern AI services.
Overview
Software security is the layer where most people first encounter AI security, but it is already broad: data pipelines, training loops, model access interfaces, prompts, agents, and retrieval components all create opportunities for misuse or adversarial influence.
Threat model
Typical software-side attackers try to change inputs, poison data, manipulate prompts, steal model behavior through queries, or infer private information from outputs. In modern systems, the software attack surface also includes orchestration logic, plugins, and retrieval layers.
Countermeasures
Defenses include robust training, better data provenance, access control, rate limiting, auditing, prompt isolation, output filtering, and continuous red-team style testing. The practical challenge is combining these without making the system unusably rigid.
Open challenges
A major open problem is how to connect software robustness claims with what actually happens in deployed products. Prompt-layer attacks, retrieval corruption, and complex pipelines mean security must be evaluated at the system level, not only at the model level.