The Challenge
Researchers often work with hundreds of PDFs — many of them old, scanned documents that standard tools can't even read. Finding a specific piece of information means manually searching through dozens of papers. Cloud-based AI tools solve part of this, but raise a real concern: researchers don't want their unpublished or sensitive academic data leaving their own machine.
The Solution
I built a fully local Retrieval-Augmented Generation (RAG) system. Users upload their PDFs — including scanned documents — and ask questions directly. Every step, from embedding to reasoning, runs entirely on the user's own machine using Ollama and HuggingFace models. An OCR fallback layer recovers text from scanned papers that standard libraries can't parse, while a disk-native vector database (LanceDB) keeps memory usage minimal.
Architecture
Engineering Decisions
Preventing Hallucination
A strict-context prompting strategy forces the model to answer only from retrieved document content — and explicitly say so when an answer isn't in the documents. Temperature is kept at 0 to minimize creative drift.
Resource-Conscious by Design
LanceDB's disk-native architecture and the 80MB all-MiniLM-L6-v2 embedding model keep memory usage minimal — the system runs comfortably on a researcher's personal laptop, not a server.
OCR as a Fallback, Not a Default
Native text extraction is attempted first; EasyOCR only kicks in for scanned pages. This keeps processing fast while still recovering text from older, scanned academic papers.
Source-Grounded Answers
Every response is tracked back to its source document, giving researchers a clear citation trail instead of an unverifiable AI summary.
Project Details
14 end-to-end test scenarios covering API behavior, hallucination control, security, and edge cases.
A privacy-preserving research tool with grounded, source-cited answers — built on benchmarked, research-backed architectural choices.
Why LanceDB?
Benchmarked against the most common vector database alternatives for local, resource-constrained environments.
| Feature | LanceDB | Chroma | pgvector |
|---|---|---|---|
| Architecture | Embedded / Local | Python-first | Relational extension |
| Memory Footprint | Minimal (disk-native) | High (in-memory risk) | Moderate (HNSW index) |
| Scalability | Millions of vectors | Risky past 50M | Enterprise scale |
Model Selection
| Model | Parameters | MMLU Score | RAM (4-bit) |
|---|---|---|---|
| Qwen2.5-7B-Instruct (Selected) | 7.5B | ~75.0+ | ~5.5 GB |
| Llama-3.1-8B | 8B | ~68.0 - 73.0 | ~6.0 GB |
| Mistral-7B-v0.3 | 7B | ~60.0 - 65.0 | ~5.0 GB |
| Gemma-2-9B | 9B | ~71.0 - 74.0 | ~7.5 GB |
Selected for the strongest academic reasoning capability per GB of RAM — critical for running smoothly on a researcher's personal machine.