AI cybersecurity: what the AI Act requires

AI cybersecurity means protecting artificial intelligence systems against attacks targeting their training data, their model, the prompts they receive and the actions they trigger. The AI Act makes it a legal requirement for high-risk systems: its Article 15 requires accuracy, robustness and cybersecurity, and names the attacks to counter (data or model poisoning, adversarial examples, confidentiality attacks). These requirements will apply in December 2027, but the attacks already exist. This article, which complements our guide to the EU AI Act, covers the threats, the legal framework and the measures recommended by ANSSI, France's national cybersecurity agency.

An AI system is still an information system: servers, accounts, application programming interfaces (APIs), software dependencies. All the usual measures apply. But it has three particularities. Its behaviour depends on data as much as on code: tampering with the training set amounts to changing the program. It interprets natural language inputs, where instructions and data are mixed. And, more and more, it acts: sending an email, querying a database, changing a file.

ANSSI reviewed the threat in its overview published on 4 February 2026. It is not aware of attacks against French organisations carried out end to end with AI, but notes that AI systems are becoming targets themselves: data poisoning, booby-trapped open-source models, flaws in connectors between agents and tools, AI accounts stolen by infostealers, data leaks by employees. AI security is therefore not a future concern for model providers only: it concerns every organisation that deploys AI.

Article 15 of Regulation (EU) 2024/1689 is one of the requirements for high-risk AI systems. It falls first on the provider, who designs the system, but it shapes what the deployer can expect. It has three parts.

Accuracy

The system achieves a level of accuracy appropriate to its intended purpose, and that level, with the metrics used to measure it, is stated in the instructions for use. The Commission is to encourage common measurement methods with metrology bodies.

Robustness

The system is as resilient as possible to errors, faults and inconsistencies, in particular those arising from its environment or its interaction with people or other systems. Redundancy, backup or fail-safe solutions can help. For a system that keeps learning after being put into service, biased outputs must not feed back into future inputs (feedback loops).

Cybersecurity

The system resists attempts by unauthorised third parties to alter its use, outputs or performance by exploiting vulnerabilities. Paragraph 5 names the AI-specific attacks that measures must prevent, detect, respond to, resolve and control: manipulation of the training data set (“data poisoning”), of pre-trained components (“model poisoning”), inputs designed to make the model err (“adversarial examples” or “model evasion”), confidentiality attacks and model flaws.

Timeline and standards

Since omnibus Regulation (EU) 2026/1744 entered into force on 27 July 2026, these requirements apply from 2 December 2027 for Annex III systems and from 2 August 2028 for those embedded in regulated products (Annex I). Compliance can be shown through harmonised standards: the one on cybersecurity, prEN 18282, prepared by the joint CEN-CENELEC technical committee JTC 21, was still a draft in October 2026. Article 42 also provides a presumption of conformity for systems certified under a European cybersecurity scheme of the Cybersecurity Act.

Providers of general-purpose AI models with systemic risk have their own cybersecurity obligation (Article 55), applicable since 2 August 2025: ensuring adequate protection of the model and its physical infrastructure.

Several frameworks classify these attacks. NIST AI 100-2 E2025, published in March 2025, distinguishes evasion, poisoning and privacy attacks for predictive AI, and adds supply chain attacks and direct or indirect prompt injection for generative AI. ANSSI, in its 2024 guide, uses three families: manipulation (misusing the system in production with malicious prompts), infection (contaminating training) and exfiltration (stealing data or model parameters). ENISA offers a three-layer framework of good practices: classic cybersecurity foundations, AI-specific measures, sector-specific measures.

The diagram below maps them onto the four links of an AI system.

A language model does not strictly separate its designer's instructions from the data it processes: it is all text. Prompt injection exploits this weakness. In its direct form, the user writes a prompt that bypasses the guardrails. In its indirect form, the malicious instruction is hidden in a web page, shared document or email that the assistant reads to do its job, without the user knowing.

The OWASP Top 10 for large language model applications (2025 edition) ranks it first, ahead of sensitive information disclosure, supply chain flaws, poisoning, improper output handling and “excessive agency”. That last point becomes central with agents: since December 2025, OWASP has published a dedicated Top 10 for them. The more a system can act, the more a successful injection matters.

No technique removes this risk. ANSSI says so in its 13 April 2026 bulletin on AI agents installed on workstations: defensive wording in the model's instructions can be bypassed, and configuration hardening remains the main lever. It recommends not deploying these tools in production until they are proven, confining them to isolated test environments without sensitive data, limiting their capabilities to what is strictly necessary and requiring human approval for any action with side effects.

Poisoning means inserting malicious data into what is used to train or fine-tune a model, to degrade its results or plant a backdoor that only activates when a specific pattern appears. ANSSI's overview cites a study finding that 250 malicious documents can be enough to poison a model. The risk also applies to the document bases an assistant queries to answer (retrieval-augmented generation): slipping a booby-trapped document into them poisons its answers.

Models themselves are third-party software components. Some file formats allow code to run on loading: ANSSI recommends safe formats and integrity checks. Models and datasets downloaded from public platforms, libraries and connectors form a supply chain to inventory, like the dependencies of conventional software; this is the logic of the software bill of materials (SBOM), which some extend to AI components. For purchased AI services, this vigilance goes through third-party risk management and a supplier questionnaire covering model provenance, use of your data and protection against these attacks.

ANSSI's guide Security recommendations for a generative AI system, published on 29 April 2024, is the French reference. It covers the whole lifecycle, from training to production, in 35 recommendations. The most structuring ones:

• carry out a risk analysis before training and build security into every stage;
• assess the trust placed in external libraries, modules and data sources;
• segregate training, deployment and production environments, and isolate the AI system in a dedicated environment;
• prefer SecNumCloud-qualified hosting for sensitive uses;
• train the model only on data its users are entitled to see, and protect the integrity of data and model files;
• filter inputs and outputs, and secure interactions with business applications;
• ban automated actions on critical operations without human approval, and limit those triggered by untrusted content;
• log all processing: prompts, plugin and data calls, filtering, responses;
• plan a fallback mode that works without AI;
• never send sensitive data to consumer online AI services.

ANSSI added a high-level analysis, Building trust in AI through a cyber risk-based approach (February 2025), written with the CNIL, Inria, LNE, PEReN and AMIAD and co-signed by foreign partners, joint recommendations with Germany's BSI on AI coding assistants (October 2024: systematic review of generated code, no generation for critical modules), then the April 2026 bulletin on agents.

Three EU texts meet on AI security. They do not target the same actors or the same object, but they refer to one another.

The most explicit bridge is Article 12 of the Cyber Resilience Act: a product with digital elements that is also a high-risk AI system and meets the CRA's essential requirements is presumed to comply with the cybersecurity requirements of Article 15, to the extent its EU declaration of conformity covers them. Conformity assessment follows the AI Act procedure, except for certain important or critical products under the CRA. A software manufacturer embedding AI should therefore run both efforts together; our guide to the Cyber Resilience Act details its obligations.

Finally, NIS 2 does not mention AI, but its risk management measures (Article 21) cover the entity's whole information system, including the AI systems it uses and the suppliers that provide them. A significant incident affecting an AI assistant, such as a data leak, falls under the notification regime.