I build agentic AI solutions with MCP, retrieval-augmented generation (RAG), Microsoft Foundry (formerly Azure AI Foundry) and the Fabric data agent. The pattern is always the same: the first demo impresses everyone, then real users ask real questions and the gaps appear. These tips are what closes those gaps.
1. Fix the content before you tune the prompt
Most bad answers start with bad source material: duplicate documents, outdated versions, scanned PDFs with no text layer, and no metadata. Clean that up first. Keep one current version of each document, and capture owner, date and document type as metadata you can filter on.
2. Chunk by structure, not by character count
Split on headings, sections and list boundaries so each chunk makes sense on its own, and keep tables intact. Add the document title and section path to every chunk. Fixed-size chunks that cut a policy clause in half are a common cause of confident but wrong answers.
3. Use hybrid retrieval and reranking
Vector search is good at meaning but weak on exact terms like product codes, error numbers and names. Combine keyword and vector search, then rerank the results. Azure AI Search supports hybrid queries and a semantic ranker out of the box, so you rarely need to build this yourself.
4. Ground every answer and allow “I don’t know”
Instruct the model to answer only from the retrieved context, to cite its sources, and to say when the context doesn’t contain the answer. Show the citations in the user interface. Users trust an assistant that admits a gap far more than one that fills it with something plausible.
5. Build an evaluation set before you go live
Collect real questions from users, with the expected answer and the source it should come from. Measure retrieval (did the right chunk come back?) separately from generation (was the answer grounded and relevant?). Microsoft Foundry includes built-in evaluators for qualities such as groundedness and relevance. Run the set on every change to chunking, prompts or models.
6. Design MCP tools like good APIs
An agent is only as good as the tools it can call. Expose a few narrow, well-named tools with clear descriptions and typed parameters, rather than one “run any query” tool. Return compact, structured results instead of huge payloads. Keep read and write tools separate, and require human confirmation for anything that changes data.
7. Enforce security in the data layer, not the prompt
A system prompt is not an access control. Filter retrieval by the user’s permissions, and pass the user’s identity through to the data source where the platform supports it. Treat retrieved documents and tool results as untrusted input that may contain prompt injection. Give tool credentials least privilege, and log every tool call for audit.
Wrapping up
Reliable RAG and agents are mostly a data engineering and security problem, not a prompt engineering one. Get the content, retrieval, evaluation and permissions right, and the model has a fair chance of giving answers your organisation can trust.
Leave a Reply