RAG Failures
Bharat VarshneyAmbassador
1w ago 160 1
6 checks that help isolate RAG failures
1️⃣ Q → C (Context Relevance): Did retrieval bring back the right information?
If not, investigate chunking, embeddings, query rewriting, filters, or the retrieval strategy before touching the prompt.
2️⃣ C → Q (Answerability): Could the retrieved context actually answer the question?
Sometimes retrieval is relevant but incomplete. The system may need another retrieval step or should simply abstain.
3️⃣ C → A (Faithfulness): Did the model stay faithful to the evidence?
If the context contains the right answer but the model invents unsupported details, you have a grounding problem.
4️⃣ A → C (Support Coverage): Can every important claim be traced back to evidence?
This catches answers that start correctly but add details the retrieved context never supported.
5️⃣ Q → A (Answer Relevance): Did the response actually solve the user's request?
An answer can be grounded and still be irrelevant because the model misunderstood the instruction.
6️⃣ A → Q (Self-Containment): Is the answer complete on its own?
A technically correct response can still fail if important steps, assumptions, or details are missing.
This breakdown is useful because it tells you which part to fix instead of treating every bad answer as an LLM problem.
1️⃣ Q → C (Context Relevance): Did retrieval bring back the right information?
If not, investigate chunking, embeddings, query rewriting, filters, or the retrieval strategy before touching the prompt.
2️⃣ C → Q (Answerability): Could the retrieved context actually answer the question?
Sometimes retrieval is relevant but incomplete. The system may need another retrieval step or should simply abstain.
3️⃣ C → A (Faithfulness): Did the model stay faithful to the evidence?
If the context contains the right answer but the model invents unsupported details, you have a grounding problem.
4️⃣ A → C (Support Coverage): Can every important claim be traced back to evidence?
This catches answers that start correctly but add details the retrieved context never supported.
5️⃣ Q → A (Answer Relevance): Did the response actually solve the user's request?
An answer can be grounded and still be irrelevant because the model misunderstood the instruction.
6️⃣ A → Q (Self-Containment): Is the answer complete on its own?
A technically correct response can still fail if important steps, assumptions, or details are missing.
This breakdown is useful because it tells you which part to fix instead of treating every bad answer as an LLM problem.
