Project Overview
Built a finance-domain RAG agent on top of FAQ and ToS data, using a dual-stage retrieval design to balance response speed and legal-text accuracy.
Problem Definition & Goals
- Problem: Semantic gaps between user wording and policy terminology, clause ambiguity, and hallucinated answers reduced trust in financial QA bots.
- Goal 1: Implement a dual-stage flow that searches FAQ first and escalates to full ToS retrieval when needed.
- Goal 2: Improve clause-level retrieval precision with hybrid search and cross-encoder reranking.
- Goal 3: Add hallucination verification and automated quality evaluation.
Key Features & Contributions
- Dual-Stage RAG: Stage 1 fast FAQ matching, Stage 2 deep policy retrieval for complex cases.
- Hybrid Search + Reranking: Weighted dense/sparse/metadata search with cross-encoder re-scoring.
- Hallucination Verifier: Added self-check logic to ensure generated answers are grounded in retrieved evidence.
- LLM Judge Pipeline: Automated scoring for accuracy, faithfulness, and completeness, including adversarial test cases.
- MCP Extensibility: Added Model Context Protocol integration for tool-calling agent workflows.
- Local Inference Ready: Deployed with vLLM/Ollama for security-friendly operation without external data transfer.
Technical Challenges & Solutions
- Term Mismatch: User terms and legal terms often differed, causing retrieval misses.
- Context Confusion: Similar clauses such as normal cancellation vs forced cancellation were misranked.
- Fix: Introduced weighted hybrid retrieval and cross-encoder reranking to improve semantic precision.
- Outcome: Improved Top-5 recall by about 20% over pure vector search and raised final answer quality.
Results & Learnings
- Result: Completed a full retrieve-verify-evaluate engineering loop beyond basic retrieve-generate design.
- Result: Achieved around +20% Top-5 recall improvement with stronger response reliability.
- Result: Secured long-term extensibility via MCP-based agent orchestration.
- Learning: Established a data-centric workflow driven by evaluation sets and automated metrics.