Local LLM with Retrieval-Augmented Generation
Development of a local large language model system using retrieval-augmented generation, combining vector search with tool-based querying and citation-aware responses.
Retrieval-Augmented Generation (RAG) with Local LLM Agents
This project was an early exploration of Retrieval-Augmented Generation (RAG), developed at a time when locally deployable large language models were still relatively limited in size, speed, and instruction-following capabilities. The primary goal was to design a system capable of producing fact-backed answers with explicit references, rather than free-form generative responses.
The core idea was to use a local LLM agent as an intermediate reasoning component. Instead of directly querying a vector database with the user input, the model was first tasked with extracting keywords and topics from the question. These keywords were then used to perform a vector-based search over a database of article paragraphs, allowing for a more targeted and interpretable retrieval step.
Retrieved paragraphs were passed back to the language model, which generated an answer constrained to the provided sources. Each statement in the output was accompanied by explicit references to the original documents, enabling traceability and manual verification. This design choice was motivated by concerns around hallucinations and the need for transparency in research-oriented question answering systems.
From a modern perspective, this architecture is intentionally conservative and somewhat rigid. If redesigned today, the system would likely rely on more capable embedding models, structured retrievers, multi-stage reranking, and stronger reasoning models. However, at the time, achieving reliable instruction-following, multilingual handling, and reference-aware generation using a fully local setup was a significant technical milestone.
Beyond the technical implementation, the project served as an important learning experience in agent decomposition, prompt engineering, and the practical limitations of early-generation LLMs. Many of the design decisions anticipated ideas that later became standard in modern RAG pipelines.
Links
Tags
Skills