Recent regulatory responses to the large language model (LLM) Grok regarding its use in generating deepfakes reveal something more interesting than “many tech companies behave badly.” They expose a mismatch between how platform regulation frameworks were designed and how generative AI works when built into platforms by providers themselves: ex-post content removals and user sanctions are no longer sufficient.
French prosecutors recently opened a probe following the circulation of AI-generated content, while the U.K.’s Ofcom has treated Grok as a system subject to ex-ante design duties under the Online Safety Act. Regulators in Australia, Brazil, Canada, Japan, India, and elsewhere have likewise pressured X by invoking existing sector-specific rules. These responses suggest that much effective AI regulation, at present, will come not from comprehensive, AI-specific frameworks, but from the application of existing sectoral rules to new capabilities.
The Architecture Problem That Content Policy Can’t Solve
Traditional platform governance works on the basis of a separation: the platform provides capabilities (hosting, search, curation, recommendations) and users generate content. Capability regulation places restrictions, such as training data exclusions and safety features, on what that system can do. Content regulation consists of rules about outputs, such as post removal, labeling, and downranking. This governance model assumes the platform is a content-intermediary, regardless of whether it acts in a way that is content-neutral (like a hammer that doesn’t care whether you’re building a house or breaking a window) or it doesn’t (for example, by downranking a type of content or banning it under community guidelines). The systems and the rules applied to them were largely created before generative AI became a feature of the platforms being governed.
When built into X, Grok collapsed this distinction. Multiple news outlets have reported that Grok was generating nonconsensual sexualized deepfakes of real people, including minors. The model, and by extension the platform, doesn’t just host or transmit harmful content created independently by users; its capability to generate that content itself is the proximate cause of its existence. So when Grok generates realistic, non-consensual fake images of real women after being prompted to do so by a user, the standard content moderation playbook — detect, remove, sanction the user — misses the point. The capability to readily produce unlawful outputs with trivial prompts (i.e., providing an ungated generation pathway to that content) is a content-relevant right violation, not just the misconduct of individual users. Both the content and the capability itself require addressing.
Framing these incidents of image generation as cases of individual misuse misunderstands how harm operates in socio-technical systems. Many areas of law recognize that harms can arise not from a single wrongful act, but from architectures that aggregate, coordinate, or lower the cost of conduct in ways that predictably produce harm. Some examples are products liability for design choices, rules on collusion in competition law and conspiracy in criminal law where coordination is harmful, externalities in financial regulation, aggregate and diffuse actions in environmental law, and cases of indirect discrimination, among others. Similarly, when a system is designed such that unlawful outputs can be generated on demand through trivial prompting, the relevant unit of analysis is no longer (or not only) individual user behavior, but the feature that enables repeated, scalable, and foreseeable harm. In those conditions, focusing on ex-post content removal or user sanction treats the symptom rather than the cause: the systemic problem is that, by creating the content for the user based on instructions, the system’s design collapses the distance between intent and harm.
This matters because core regulatory instruments presuppose a workable split between capability and use when imposing liability over content. The E.U.’s Digital Services Act focuses on how platforms “host” content, as evidenced in its Article 6 and its intermediary liability framework. The content duties in the U.K.’s Online Safety Act apply to platforms and search providers but, while an LLM’s integration into one of these services (like Grok on X) means that the provider will be within the Act’s scope, the Act focuses on service duties (like risk assessments, safety-by-design, and takedown) rather than prohibiting harmful model outputs.
Beyond platform governance, the E.U. AI Act‘s risk categories, while they apply to general-purpose models, focus obligations on use-specific categories. Even principles-based approaches like algorithmic impact assessments assume one can meaningfully distinguish between what a system can do and what users actually do with it. Where a general-purpose model can synthesize unlawful output on trivial prompting, this distinction has limited analytical value.
Governance tools that operate exclusively at the content layer are limited because, as they intervene only after a harmful output has been produced, they can only address individual instances rather than the conditions that enable and facilitate their production. Takedown, labeling, and user sanctions are not easy to scale and, to the extent that they can be, they scale with volume of harmful content rather than with its risk — without impeding the creation of new harmful content. As long as the underlying capability remains available, moderators are stuck chasing each new iteration of the same harm. Where a system predictably generates unlawful outputs, regulating content alone cannot meaningfully reduce risk without continuous, resource-intensive oversight that leaves the systemic cause unaffected.
Geoblocking’s Inadequacy for Capability Problems
X’s response to Grok’s outputs — geoblocking certain prompts in some jurisdictions — is interesting because it’s obviously inadequate. Where harms are capability-driven, adequate mitigation requires controls at the level of generation itself rather than territorial filters applied after the fact.
Geoblocking wrongly assumes the problem is where the generation happens. But, at least for synthetic content (i.e., content that is created, rather than collected), that assumption fails. The harm occurs when the image is created and potentially distributed, which is independent of where the user who triggered its creation is physically located. A U.K. subject, for example, can be depicted by a deepfake generated elsewhere and circulated domestically in the U.K. within minutes. Not only can geographic restrictions be trivially circumvented through VPNs but, more importantly, geoblocking mistargets the variable: an image generated in a permissive jurisdiction but depicting someone in a restrictive one can be distributed cross-border within minutes, creating cross-border enforcement problems. Geoblocking reduces domestic creation, but it does not meaningfully constrain the underlying risk.
This issue is broader than Grok. It’s about how technology (including LLMs) undermines territorial approaches to content governance. When anyone can generate harmful synthetic content depicting anyone else, anywhere, the traditional tools of jurisdictional enforcement — jurisdiction-specific blocking and filtering, adhering to territorial regulation — lose their efficacy.
The implication is that regulators need to move upstream and govern capabilities at the model level, not outputs at the content level. This is a departure from how platform regulation has mostly worked for the past three decades. To be effective, instead of limiting its prompt bans to jurisdictions that have taken issue with Grok’s harmful outputs, X should limit them for users worldwide.
What This Reveals About Regulatory Futures
Regulators’ rapid mobilization to address Grok’s capability, such as in the U.K., Canada, and France, reveals what happens when settled harms such as non-consensual intimate images and child sexual abuse material meet new production mechanisms like generative AI models without guardrails and low-friction distribution. The speed is different from traditional AI regulation and enforcement because the harm isn’t novel — only the production method is. Because of that, regulators can extend existing prohibitions to synthetic generation without redefining categories. Regulators don’t need to convince anyone that child sexual abuse material is harmful or that consent matters for intimate images. The normative consensus already exists. They just need to extend existing prohibitions to cover synthetic content generation.
This sidesteps the usual “innovation versus safety” debates. The operative question becomes whether this generation capability should be available absent proportionate gating. There’s no innovation defense for making child sexual abuse material generation more efficient, so the policy question is easier: should this capability exist at all?
It’s worth noting what regulatory responses have not included: calls for a generative AI moratoria, or efforts to regulate AI as a technology category with attempts to define it broadly, or extended consultations with the industry about feasibility. Instead, we are seeing specific prohibitions, specific capabilities, specific harms, and rapid enforcement.
The Grok case suggests that a lot of effective AI regulation may come not from comprehensive AI-specific frameworks, but from the application of existing harm-based laws to new capabilities. The U.K.’s approach, using the Online Safety Act rather than waiting for bespoke AI legislation, is one example.
For general-purpose AI developers, the implication is that, if Grok’s deepfake generation capabilities can be treated as product defects subject to regulatory prohibition, other capabilities might face similar scrutiny, including:
- Code generation models that can produce malware
- Models that can convincingly impersonate specific individuals
- Models that can generate targeted disinformation at scale
- Models that can provide detailed instructions for producing dangerous substances
A standard response to ungated generative AI capabilities has been that they are dual-use, with legitimate applications. But dual-use is a poor defense when the harmful use case is foreseeable, can be mitigated by proportionate controls, and the harms are severe. We don’t allow consumer products with dual-use explosives just because explosives have legitimate applications. If regulators adopt a product safety frame — judging models by their foreseeable harmful use rather than their theoretical beneficial use — it changes what can be deployed commercially.
Where This Goes
The Grok episode suggests AI governance is bifurcating into two separate tracks:
Track 1: Fast, harm-specific enforcement using existing regulatory frameworks focused on illegal outputs, such as non-consensual intimate images, child sexual abuse material, and fraud. This moves quickly because the normative consensus already exists. Under this approach, regulators will increasingly govern capabilities directly rather than outputs.
Track 2: Slower, framework-level regulation for long-term and systemic AI risks, such as labor displacement and epistemological harms. This approach remains in the cycle of consultation and gradual norm development.
Most attention has focused on Track 2 because it affects broader economic interests. But Track 1 is moving faster and establishing precedents that will shape how regulators think about governing AI capabilities more generally.
The lesson isn’t that AI regulation is coming faster than expected. It’s that AI regulation is coming differently than expected. The bottleneck is the normative consensus about which harms matter enough to regulate aggressively. Grok just found one. General-purpose model developers should assume there are others.
– Ignacio Cofone, Published courtesy of Just Security.
