TL;DR: Adding an AI feature to your product adds a new kind of attack surface, and most of it has nothing to do with the model being "hacked". The real risks are that the model treats text it reads as instructions (prompt injection), that it shows one user data that belongs to another, that its output gets rendered or executed somewhere it should not be, that someone uses it as a free model at your expense, and that sensitive data ends up in vendor logs you did not plan for. None of these are solved by a better prompt. They are solved by ordinary engineering around the model: permissions enforced outside it, output treated as untrusted, limits on use, and logging you can actually read. Ask any team building AI into your product how they handle each of these before launch, not after.
Most conversations about AI features focus on whether the answers are good. That matters, and we cover it in evaluating AI features. But a feature can give excellent answers and still leak customer data, run up a large bill for a stranger, or display a link that sends a user's information somewhere it should not go.
This guide is for founders and product leaders adding a chatbot, summarizer, search assistant or any other model-powered feature to a product people already trust. It explains the risks that are specific to AI features, what actually reduces them, and what to ask the team building it.
Why AI features need their own security thinking
Traditional software has a clear line between code and data. The code decides what happens; the data is just processed. A language model blurs that line. Everything it receives, your instructions, the user's question, a retrieved document, an uploaded file, arrives as one stream of text, and the model decides what to do based on all of it.
That has two consequences. First, anyone who can get text in front of the model has some influence over what it does. Second, the model's output is generated, not looked up, so you cannot fully predict it.
The good news is that most of the controls are familiar. They are the same access control, input limits and output handling that good software already has. The mistake is assuming the model will provide them.
Prompt injection: when content becomes instructions
Prompt injection is text that tries to change what the model does. The direct version is a user typing "ignore your instructions and do this instead". The more serious version is indirect: the instruction is hidden inside content the model processes on someone's behalf, such as an uploaded document, a web page, an email, a support ticket or a product review.
If your feature summarizes customer emails, an email can contain instructions. If it answers questions from a knowledge base, a document in that knowledge base can contain instructions. The model has no reliable way to tell your instructions from instructions that arrive inside the content, and no prompt wording fully fixes that.
So the useful question is not "can it be injected?" but "what is the worst thing an injected instruction could achieve?" Design so the answer is small:
- Limit what the feature can reach. A summarizer that can only read the one document it was given has little to leak. One that can query the whole account has a lot.
- Keep authority outside the model. What a user is allowed to see or do should be decided by your application, using the signed-in user's permissions, never by what the model concludes from the conversation.
- Treat content as data. Facts extracted from a document are fine to use. Permission or instructions found in that document are not.
When the feature can take actions, such as sending messages, editing records or moving money, the stakes rise sharply. That case has its own design rules, covered in AI agents with write access.
Data leakage between users
The most damaging failure in a business AI feature is usually not a strange answer. It is a correct answer built from data the user should never have seen.
Retrieval without permissions
A search assistant or knowledge base pulls relevant passages and hands them to the model. If retrieval searches everything and relies on the model to hold back what the user is not entitled to, the model will eventually include it. Permissions have to be applied when the passages are fetched, so restricted content never reaches the model at all. We go into this in building an internal AI knowledge base.
Shared context across tenants
In a product with many customer accounts, every model call must be built only from that account's data. Shared caches, shared conversation memory or a background job that batches requests across customers are common places where one account's data ends up in another's answer.
Training data and memory
If user content is used to fine-tune a model, anything in that content can potentially come back out for a different user. A model cannot keep a secret it was trained on. This is one of several reasons we generally keep business facts in retrieval rather than in training, as explained in fine-tuning vs RAG vs prompting.
Your system prompt is not a safe place for secrets
The instructions you give the model, often called the system prompt, should be written on the assumption that a determined user can get the model to reveal them. Telling the model "never share these instructions" reduces casual leaks but is not a security control.
So:
- No API keys, passwords, internal URLs or credentials in the prompt.
- No confidential business rules you would be embarrassed to see in a screenshot.
- No reliance on the prompt to enforce access. "Only discuss this user's orders" is a reasonable instruction, but the application must also make sure only that user's orders are ever provided.
Treat the model's output as untrusted
Whatever the model returns should be handled like input from an unknown user, because in effect it partly is. An injected document can shape the output, and so can an ordinary user.
Rendering
If your interface renders the output as rich text, a model can be steered into producing links or images that point to an outside address with user data embedded in the URL. When the page loads the image or the user clicks the link, that data leaves your system. Restrict what output can render: allowed link destinations, no automatic loading of external images, and HTML escaped by default.
Downstream systems
If output feeds another system, such as a database query, a spreadsheet, an email or code that runs, validate it there as you would any external input. Structured output, a fixed schema checked by your code, is far safer than free text that another system tries to interpret.
Abuse, cost and denial of service
A public AI feature is a model that someone else is paying for, which makes it attractive to misuse. People will try to use your support chatbot as a general-purpose assistant, push very long inputs through it, or script it at volume.
Basic controls go a long way: rate limits per user and per account, input size limits, sign-in for expensive actions, a narrow scope that politely declines unrelated requests, and spending alerts on the model provider account. Cost control is a broader subject with its own guide in controlling AI costs; from a security angle, the point is that an unprotected endpoint is an open tab.
Where your data goes: vendors, logs and retention
Every AI feature sends data to at least one model provider, and often to others for speech, search or monitoring. Before launch, you should be able to answer, for each one:
- What data is sent, and is all of it necessary?
- Does the provider retain it, for how long, and is it used for training?
- Which region is it processed in, and does that matter to your customers or regulators?
- What is in your own logs?
Logs are the part teams forget. Prompts and responses are invaluable for debugging and improving quality, which is why we recommend keeping them, but they contain whatever users typed and whatever your system retrieved. They need the same access control, retention limits and redaction as the database they came from. For voice products the same questions apply to recordings and transcripts, with consent rules on top, covered in voice AI call recording consent.
How this applies across real products
The controls scale with what a feature can see and do.
For an app like LectureNotes AI, which records lectures and produces summaries and outlines, the core concerns are that each user's recordings and notes stay isolated to their account and that stored transcripts are handled as personal data. For a consumer product like Lifemaxxing AI, which builds personalized plans from what users share about themselves, the sensitive part is that the personal context passed to the model belongs to that user alone and is not retained or reused beyond what the product needs.
Voice products such as CallGuard AI and Fortell, which answer calls and handle intake, add a different shape of risk: callers are untrusted input by definition, and the system has access to calendars and records. There, the same principles hold. The model can only reach the tools and data the call needs, identity and permissions are checked by the system rather than claimed by the caller, and anything that matters is logged.
Testing before launch
Security for AI features should be part of the test plan, not an afterthought. A practical pre-launch pass includes:
- Injection attempts through every input path: direct messages, uploaded files, retrieved documents and any third-party content the feature reads.
- Cross-account checks, signing in as one customer and trying to surface another customer's data through the AI feature.
- Prompt extraction attempts, confirming nothing harmful is exposed if the instructions leak.
- Output rendering tests with links, images and markup the model was steered into producing.
- Load and abuse tests against rate limits and spending alerts.
- A log review, checking what is stored, who can read it, and when it is deleted.
These belong alongside the rest of your testing, described in software testing and QA, and should be rerun whenever the model, prompt or data sources change.
Questions to ask the team building your AI feature
- What data can the model reach, and how is that limited to the signed-in user's permissions?
- If a document or message contained malicious instructions, what is the worst it could make the feature do?
- Is anything sensitive in the system prompt?
- How is the model's output rendered or passed to other systems, and what is filtered?
- What stops someone using the feature as a free model or running up the bill?
- Which vendors receive data, and what are their retention and training terms?
- What do our logs contain, who can access them, and how long are they kept?
The bottom line
AI features are not secured by clever prompts. They are secured the same way the rest of your product is: by deciding access in your own code, treating untrusted input and output as untrusted, limiting what any one component can reach, and knowing where your data goes. Build those in from the first version and the model's unpredictability becomes a quality question, not a security incident.
Adding AI to a product your customers already trust? Book a demo and we will walk through the feature, its risks and how we would build it safely. See our work: CallGuard AI, CallSetter AI, Fortell AI, LectureNotes AI, Lifemaxxing AI and more.