If you have spent any time building AI applications recently, you have probably heard some version of this:
"We need RAG."
The product needs to answer questions about company documents.
"We need RAG."
The application needs access to internal knowledge.
"We need RAG."
The model does not know about our data.
And sometimes that is exactly the right answer.
But I have started to think that RAG is often introduced too early in the conversation.
RAG is a technical solution.
It is not a product strategy.
Before deciding that a product needs retrieval-augmented generation, I think there is a more important question to answer:
What problem are we actually trying to solve?
What RAG actually does
At a high level, RAG allows an application to retrieve relevant information from an external knowledge source and provide that information as context to an LLM before generating a response.
Instead of expecting the model to know everything, we give it the information it needs at request time.
A typical flow might look something like this:
User asks a question.
The system searches a knowledge base.
Relevant documents or chunks are retrieved.
Those results are added to the model's context.
The model generates an answer based on that context.
This is useful when the information is outside the model's built-in knowledge, changes frequently, or belongs to a private data source.
That is the easy part.
The difficult part is deciding whether retrieval is actually the right solution for the product problem.
Start with the problem, not the vector database
A common AI implementation can start with technology.
We have a collection of PDFs.
Let's chunk them.
Let's generate embeddings.
Let's put them into a vector database.
Let's build a retrieval pipeline.
Then we connect an LLM.
At this point, we have built something that technically works.
But we have not necessarily built something useful.
Imagine the real user problem is:
"Our employees cannot quickly find the right company policy."
That could lead to several different solutions.
Maybe semantic search is enough.
Maybe a carefully structured search experience is better.
Maybe a traditional database query is more appropriate.
Maybe the user needs a simple question-and-answer interface.
Maybe the real problem is that the documentation itself is badly organized.
RAG is only one possible solution.
The product decision has to come first.
Search can sometimes be better than RAG
This is one of the easiest things to forget when building AI products.
Sometimes the user does not need an AI-generated answer.
They just need the right information.
If I am looking for a specific policy document, a good search system may be faster and more reliable than sending the retrieved content through an LLM.
There is no reason to introduce generation if generation does not create additional value.
This is especially important when correctness matters.
A search result can point me directly to the source.
A generated answer introduces another layer of interpretation.
That interpretation can be useful, but it can also introduce mistakes.
The question should therefore not be:
"Can we make this conversational?"
It should be:
"Does a conversational experience make this workflow better?"
Structured data is another important distinction
RAG is also not a replacement for structured data.
Suppose a user asks:
"How many active customers do we have this month?"
That is not really a retrieval problem.
It is a data query.
The right solution might be a database query, an analytics system, or a tool that allows an LLM to query structured data safely.
Trying to solve every data question by putting documents into a vector database is solving the wrong problem.
Likewise, if the application needs exact filtering, sorting, aggregation, or transactional consistency, traditional data systems remain much better suited to those jobs.
AI can sit on top of those systems.
It does not need to replace them.
Retrieval quality and answer quality are different problems
This is one of the most important things to understand when building RAG systems.
A model can only do so much with bad context.
Imagine a user asks a question and the correct document exists in your knowledge base.
But your retrieval system returns the wrong section.
The LLM may still produce a very convincing answer.
That answer can be completely wrong while looking perfectly reasonable.
This means there are at least two separate problems:
Did we retrieve the right information?
And:
Did the model use that information correctly?
People often focus heavily on prompts and models when the real problem is retrieval.
Changing the model will not magically fix a retrieval system that consistently returns the wrong context.
The quality of the knowledge base matters too
Another lesson I've found important is that RAG does not remove the need for good data.
It can actually make poor information architecture more visible.
If your documents contain:
- outdated information
- duplicated policies
- contradictory instructions
- unclear ownership
- bad formatting
- missing context
then adding embeddings will not fix the underlying problem.
You may end up with a very sophisticated system that retrieves very accurately from a collection of very bad information.
That is still a bad product.
Sometimes the highest-value engineering work is not improving the retrieval algorithm.
It is improving the source material.
Chunking is not a product decision
There is a lot of discussion around chunk sizes, overlap, embedding models, vector databases, reranking, and retrieval strategies.
These things matter.
But they are implementation details.
They should come after we understand the user workflow.
Otherwise it becomes very easy to spend days optimizing retrieval metrics for a feature that users do not actually need.
This is one of the patterns I have become more conscious of while working on AI-assisted applications.
Technical problems are interesting.
That does not automatically make them the most important problems.
When RAG actually makes sense
There are many situations where RAG is a very good fit.
For example:
A company has a large collection of internal documents and employees need to ask questions across them.
A customer-facing application needs to answer questions using frequently updated product documentation.
A support workflow needs to retrieve relevant knowledge before suggesting a response.
An application needs to combine an LLM with private information that should not be part of the model's general knowledge.
In these situations, retrieval can provide the missing context that makes an LLM useful.
But even then, I would start with the workflow.
What does the user want to accomplish?
What information is required?
Where does that information live?
How accurate does the answer need to be?
What happens when the system does not know?
Those questions determine the architecture.
You might not need fine-tuning either
There is a similar pattern with fine-tuning.
A team sees that the model does not behave exactly how they want, so the next idea is:
"Let's fine-tune it."
But sometimes the actual requirement is simply better instructions, better context, structured outputs, or a better workflow.
Fine-tuning can be useful, but it should solve a specific problem.
The same principle applies:
Don't choose the technique first.
Understand the problem first.
And you might not need an agent
The same thing is happening with agents.
A workflow needs to retrieve information, call an API, and produce an answer.
Suddenly the architecture becomes:
"We need an agent."
Maybe.
But if the workflow is predictable, a deterministic pipeline may be simpler and more reliable.
For example:
Retrieve data.
Call the relevant service.
Generate a response.
Validate the result.
Return it to the user.
There is nothing wrong with a workflow being boring.
If it solves the problem reliably, boring is often a good thing.
Autonomy should be introduced when the problem actually requires it.
Not because autonomous agents are currently interesting.
Think in terms of product value
This is where I think the distinction between AI engineering and product engineering becomes important.
As engineers, it is easy to become fascinated by the technology.
I certainly am.
LLMs are interesting.
Agents are interesting.
Vector search is interesting.
New models are interesting.
But users generally do not care which retrieval strategy we used.
They care whether they can accomplish what they came to do.
That means the success of an AI feature should eventually be measured in product terms.
Did the user find the information faster?
Did support agents resolve tickets more quickly?
Did a repetitive workflow become simpler?
Did the system reduce manual work?
Did users actually trust the results?
Did the feature improve the outcome we were trying to improve?
Those questions are much more important than whether the architecture looks impressive.
How I approach RAG now
I still use RAG.
It is a very useful pattern, and there are plenty of problems where it is the right choice.
But I try to approach it differently now.
I start with the workflow.
Then I identify the information the workflow needs.
Then I look at where that information lives and what shape it is in.
Then I decide how that information should be accessed.
Only after that do I think about whether retrieval, structured queries, tools, generation, or some combination of them is appropriate.
The technology comes after the problem.
Not before it.
The bigger lesson
The reason I call RAG a product concern rather than simply an engineering concern is that choosing RAG changes the product.
It affects:
- how users interact with information
- how quickly they get answers
- how much they can trust the system
- how errors are handled
- how sources are presented
- how the system behaves when information is missing
- how much the workflow depends on AI
Those are product decisions.
The retrieval pipeline is only part of the implementation.
Final thoughts
RAG is not bad.
Vector databases are not bad.
Agents are not bad.
Fine-tuning is not bad.
The problem is starting with any of them before understanding what we are trying to accomplish.
There is a pattern I have found useful across both traditional software and AI systems:
Understand the problem.
Understand the workflow.
Understand the constraints.
Choose the simplest solution that can solve it.
Then build.
Sometimes that solution will include RAG.
Sometimes it will be a database query.
Sometimes it will be search.
Sometimes it will be a deterministic workflow.
And sometimes it will not need AI at all.
That last possibility is worth keeping on the table.