We are no longer just worried about a classified document being pasted into a prompt; we are witnessing the erosion of the state’s cognitive perimeter.
A colleague in the government asked me a very interesting question. They wanted to understand the landscape around the usage of commercial AI chatbots in regular government work. That led me thinking and researching, and into a rabbit hole which spanned across official secrecy, informational integrity, Personal data protection, right to information, hallucinated content, model biases and even the sovereignty of the country. Consider the following scenarios. The official types a draft Cabinet note into a commercial chatbot. The provider suffers a breach—not hypothetical, as demonstrated by the OpenAI March 2023 bug that exposed conversation titles and billing details of 1.2% of Plus subscribers for nine hours.
The Cabinet note, stored on the provider’s servers, is accessible to an attacker. The OSA Section 5 crime is committed; the official faces criminal liability. Under the US CLOUD Act, a US court can compel a US company to produce data stored anywhere in the world. A foreign government investigating a matter involving India serves a legal order on the chatbot provider for all queries from Indian government accounts. The provider complies. The foreign government now has months of intelligence on Indian decision-making, negotiating positions, and departmental vulnerabilities. A government official uses a chatbot to summarise a policy file, weigh options, and draft a recommendation.
The model’s training data contains a systematic bias—against a particular region, community, or policy approach— that the official does not detect. The official acts on the model’s advice. The policy causes harm. The official is held accountable for the decision; the model’s role is invisible. An official uses a chatbot to prepare a court affidavit or a parliamentary question. The model invents a case citation or a data point. The official repeats it. The credibility of the government in court or Parliament is damaged. The official faces disciplinary action for presenting false information. A legal case involving a government department triggers a discovery process.
The opposing party serves a preservation order on the chatbot provider for all data related to the department. The official’s conversations—including candid assessments of options, criticisms of colleagues, and unformed policy ideas—are produced as evidence. The official’s career is damaged. The government’s internal deliberation process, which depends on candour, is exposed. Modern chatbots offer persistent memory: they remember previous conversations and can reference them in future sessions. An official queries the chatbot about a sensitive personnel matter. The memory persists. Weeks later, a different official—or the same official on a different device—encounters the memory in a new session. The sensitive context is exposed to an unintended recipient within the same organisation.
The access control that was supposed to be enforced by the government is bypassed by the chatbot’s own memory architecture. A chatbot’s “human review” feature, enabled for safety purposes, exposes a government official’s sensitive queries to a foreign contractor employed by the AI company. The official’s conversation is flagged for review. A contractor in another country reads the full content, including references to a sensitive defence procurement, a corruption investigation, or a diplomatic negotiation.
The contractor’s employer is not bound by Indian law. India’s Ministry of Finance, in January 2025 issued an advisory for employees to cease using AI chatbots on office devices. Following a cyber advisory from the Indian Computer Emergency Response Team (CERT-In), the Central government explicitly directed all ministries and departments to pause and restrict the use of unapproved external AI platforms (such as OpenAI and Anthropic models) for any official, confidential, or sensitive data. Government of India is definitely seized with the greater implications regarding the use of AI chatbots in official settings.
The real danger is not found in the explicit; it is found in the implicit. It is in the queries that never mention a single secret, yet, through the cumulative weight of a thousand mundane interactions, reveal the shape of a pending policy, the tremor of a negotiating position, or the specific contours of a departmental vulnerability. A model does not need to see the secret to know the secret. It only needs to see the pattern. Through model inversion attacks, the very architecture of these tools becomes a sieve, capable of reconstructing the intent behind the input.
We are facing a future where the state’s thinking is being harvested, not through a breach, but through a dialogue. Our current regulatory and legal fortifications are relics of a pre-algorithmic age. The Official Secrets Act of 1923, the Manual of Office Procedure, the cyber guidelines of 2022—they were all built for a world of physical files, discrete recipients, and identifiable boundaries. They assume a world where information is a commodity that moves from point A to point B. But a conversational AI is not a recipient; it is an endpoint that never closes.
The very notion of a “boundary” is rendered obsolete when the terms-of-service agreement becomes the de facto sovereign of the digital workspace. Even our emerging digital laws are insufficient. The Digital Personal Data Protection Act of 2023 is a shield for the person, not a shield for the state. It treats data as an individual’s property, leaving the vast, grey middle ground of institutional intelligence and statecraft entirely exposed.
We are left with a legal landscape that governs the “personal” but leaves the “governmental”—the very essence of sovereignty— floating in a regulatory noman’s land.
There is also the shadow of dependency. We see the rise of domestic efforts—the 34,000 GPUs of the IndiaAI Mission, the multilingual promise of BharatGen, the capability of Sarvam—but these are pieces of a puzzle that has yet to be assembled. Our sovereign aspirations are currently hostage to a hardware-software loop we do not control. The 2026 suspension of models like Anthropic’s Fable 5 demonstrated the fragility of this position: a single geopolitical shift or a foreign court order can, with a single stroke, decapitate the state’s access to its own digital intellect.
The government’s response has been a series of disconnected movements. We have a specialized, secure intranet through the NIC—a vital tool, but one that lacks the generalpurpose utility required to match the cognitive leap offered by frontier models. We have policies that are effectively bypassed by the sheer convenience of the tools we use. We are caught in a cycle of’reactive patching’—trying to fix a structural problem with a series of temporary, localized band-aids.
If we do not move from advisories to a coherent, structural framework, the cost will be more than just data leakage. We risk an institutional amnesia, where the state’s decision-making logic is outsourced to models whose incentives are divorced from the public interest. We risk a world where the very ‘noting’ culture that ensures administrative accountability is replaced by AI-generated echoes—untraceable, unattributable, and detached from the reality of the state.
The pieces are on the table: the compute, the legal frameworks, the safety mandates. But the pieces do not yet form a picture of sovereignty. The next step must be more than a memo; it must be a foundational reckoning, that “Eternal vigilance is the cost of liberty”.
*Brijesh Singh is a senior IPS officer and an author (@ brijeshbsingh on X). His latest book on ancient India, “The Cloud Chariot” (Penguin) is out on stands. Views are personal.
