Chat with your documents offline using on-device retrieval-augmented generation (RAG). Ask questions across your PDFs, notes, and folders—everything stays encrypted and local, with no uploads to the cloud or AI training services.
What is On-Device RAG?
Retrieval-augmented generation, or RAG, lets you ask an AI questions about your personal documents. Instead of relying only on the model's training data (which has a knowledge cutoff), RAG pulls relevant passages from your files and feeds them to the model as context. The model then answers based on what it found in your documents.
With on-device RAG, this entire process—embedding, retrieval, and generation—happens locally. You add your documents to MyBenAI, the app converts them to searchable embeddings (dense vectors representing meaning), and stores those vectors in a local vector database. When you ask a question, the app finds the most relevant passages and feeds them to the chat model. No document leaves your phone.
How On-Device RAG Works
The process has three steps:
- Embedding: When you upload a PDF, notes folder, or notebook, MyBenAI breaks it into passages (usually 100–500 words each) and converts each to a 768-dimensional vector using a local embedding model (typically E5-small or similar). This happens once; the vectors are stored locally.
- Storage: Vectors live in a local vector database (SQLite with vec0 extension). They're indexed for fast similarity search. Your documents themselves are also stored locally, encrypted at rest by your phone's file system.
- Retrieval and Answer: When you ask a question, the app converts your question to a vector using the same embedding model. It searches the local index for the top 3–5 most similar document passages. Those passages are prepended to your question and sent to the chat model. The model reads the relevant context and answers.
All three steps are on-device. No document text, vectors, or embeddings leave your phone. The AI model running locally never needs internet to answer your questions.
Supported Document Types
MyBenAI handles a wide range of local document sources:
- PDFs: Research papers, textbooks, reports, contracts. Any PDF on your device can be indexed and searched.
- Text files and notes: Markdown files, plain text notes, Evernote exports, or text copied into the app.
- Folders: Point the app at a folder on your device, and it indexes all text/PDF files within it. Useful for scanning an entire project archive or collection of notes.
- Notebooks: Obsidian, Notion exports, or custom notebooks. If it exports as text or markdown, RAG can index it.
- Web content: Optional: if you save web articles as PDFs or text, they're indexed like any other document.
Use Cases for On-Device RAG
Research and writing: Indexing a set of research papers lets you ask the AI synthesis questions: "Summarize the main criticisms of X across these papers." The model reads your documents, not internet sources, so answers are accurate and properly cited.
Study and learning: Students can index textbooks, lecture notes, and practice materials. Ask clarification questions, and the AI pulls relevant sections to explain concepts. Everything stays private—no study notes are uploaded to a third party.
Work and reference: Lawyers, doctors, and professionals can index proprietary documents, codebases, or manuals. Ask the AI how to handle a specific scenario based on your organization's guidelines or documentation. Sensitive material never leaves your device.
Personal journaling and memory: Index your past journal entries or notes. Ask the AI about your own history: "What did I say about X in previous entries?" The model draws from your local memory, not internet data.
Legal and medical privacy: Highly regulated fields require document privacy. Local RAG is the only approach that guarantees documents never reach a third-party AI service.
Speed and Accuracy
On-device RAG is fast—retrieval happens in under 1 second on modern phones. The bottleneck is the local LLM generating an answer, which is the same speed as regular chat. Accuracy depends on three factors:
- Embedding quality: E5-small is a solid embedding model trained on 1 billion document-query pairs. It captures semantic meaning accurately, so similar passages are retrieved even if the exact wording differs from your question.
- Document clarity: Well-organized, clearly written documents yield better retrieval and answers. Scanned PDFs or poor OCR reduce accuracy because the text is garbled.
- Context window: The local LLM has a fixed context window (typically 4–8K tokens). If your documents are very long or complex, the model might miss nuance. For most documents, this isn't a bottleneck.
Privacy Benefits of Local Retrieval
Cloud-based RAG (Anthropic's docs.anthropic.com, OpenAI's file upload, or similar) has a fundamental risk: your documents are in someone else's system, subject to their terms of service, backup policies, and potential law enforcement requests. Even with promises of deletion, you never fully control your data.
On-device RAG gives you total control. Your documents stay on your phone. The embeddings and vectors are stored locally. The AI model processing them is local. If you delete a document, it's gone immediately. If your phone is lost, you control the encryption. There's no third party holding your sensitive material.
This is especially critical for legal, medical, or proprietary information where exposure could have serious consequences.
Storage and Performance
Vector databases are space-efficient. A typical PDF creates 10–50 embeddings (depending on length), each about 3 KB. A 1000-page research collection might use 100–200 MB of storage for vectors. The documents themselves take up more space, but it's manageable on phones with 128+ GB of storage.
If storage is tight, you can selectively index. Index only the most important documents, or periodically archive old indexed documents to external storage.
Setting Up Document Retrieval
In MyBenAI, indexing documents is straightforward:
- Go to Documents or Knowledge settings.
- Add a document, folder, or notebook. The app shows supported formats.
- The app generates embeddings locally (this can take a minute or two for large documents).
- Once indexed, ask questions about the documents in chat. The app retrieves relevant sections automatically.
You can enable or disable RAG per chat—sometimes you want pure conversation, sometimes you want grounding in your documents.
Limitations
On-device RAG isn't magic. The model can't hallucinate facts that aren't in your documents, but it can misunderstand or over-generalize. Always review retrieved passages and verify important claims. Retrieval quality depends on document quality and clarity—garbage in, garbage out.
It's also not a replacement for full-text search. If you need exact keyword matches, use traditional search. RAG excels at semantic search and synthesis—understanding meaning, not just keywords.
Learn More
On-device RAG transforms your documents into an intelligent, private knowledge base. It's a powerful feature for researchers, students, professionals, and anyone managing sensitive information. To understand the broader ecosystem, explore how on-device RAG architecture works, see why students benefit from private AI, or dive deeper into using local AI for personal memory and reflection.
Ready to chat with your documents privately? Start with MyBenAI and index your first document today. Your knowledge base will grow over time, all encrypted and local.