National CERT Issues Warning Over Unsafe Use of AI Tools
The National Cyber Emergency Response Team (nCERT) has issued an advisory urging organisations to use Generative Artificial Intelligence (GenAI) tools safely, warning that uncontrolled use of public AI services could expose sensitive information and create cybersecurity risks.
The advisory covers public AI chatbots, coding assistants, browser extensions, AI-enabled applications and third-party AI services. It warns that unauthorised use of these tools, commonly known as “Shadow AI,” can expose sensitive information, intellectual property, credentials, source code and other organisational data.
AI Security Risks
National CERT highlighted several risks linked to the use of GenAI tools. These include prompt injection attacks, insecure AI-generated code, malicious integrations, inaccurate AI outputs and compromised third-party AI models.
Prompt injection can manipulate an AI system through specially crafted instructions, while insecure AI-generated code may introduce security weaknesses if it is used without proper review.
The advisory also warned that third-party integrations and models can create additional risks if they are compromised or not properly vetted.
Organisations Told to Control AI Use
National CERT has advised organisations to establish and enforce Generative AI Acceptable Use Policies.
They should prohibit employees from submitting classified, confidential, personal, proprietary or credential-related information to public or unapproved AI platforms.
Organisations have also been advised to maintain centrally vetted lists of approved AI tools, models, plugins, APIs and platforms.
The advisory calls for qualified human review of AI-generated code and other critical AI outputs before they are deployed or used in decision-making.
Monitoring and AI Governance
National CERT recommends extending Data Loss Prevention, access monitoring and endpoint security controls to AI interfaces.
These controls should be used to detect sensitive-data submissions, unauthorised AI usage, suspicious API activity and Shadow AI.
Organisations have also been advised to align their AI governance practices with recognised frameworks, including the NIST AI Risk Management Framework and the OWASP Top 10 for LLM Applications.
Regular staff training and audit trails for AI tool usage have also been recommended.
Continuous Monitoring Needed
The advisory directs organisations to continuously monitor for sensitive data exposure, unauthorised AI plugins and APIs, prompt injection, unreviewed AI-generated code, suspicious AI access and violations of internal policies.
If an AI-related security incident occurs, organisations should contain unauthorised access, preserve evidence and logs, revoke compromised credentials or API keys, investigate the exposure and take corrective measures.
Incidents involving sensitive-data exposure, prompt injection, AI supply-chain compromise or violations of GenAI policies should be reported to National CERT Pakistan through its incident reporting portal, email or UAN.
