A document search engine takes four moves: gather the files in one place, extract the text from each one (with OCR if they're scanned), store four fields per document (type, date, counterparty and reference), and index all of that in a database with full-text search. On top, one search box with those four filters. Nothing more to start with.
The real work isn't the search engine: it's the state of the files. An accountancy firm I worked with had twenty-three thousand PDFs spread across a server, two Drive accounts and the desktop of the person who had been there twenty years. Half were scans without a single searchable letter. Until that's sorted, no search engine will help.
Why isn't Windows or Drive search enough?
Because they search for words inside files, and what you usually need is to search by the document's data. "The maintenance contract with this company" is not a word: it's a type, a counterparty and a date. System search doesn't know what a contract is or who the supplier is, so it hands you a hundred and forty results ranked by text relevance and you get to open them.
There are two more limits. The first is scanning: a scanned PDF is an image with no text in it, so system search walks past it as if it were empty. The second is dispersion: they search where they are, and if the documents live in four places you search four times.
That said, if you have three hundred files in one well-named folder, your operating system's search is enough. Build nothing.
How do you build it, step by step?
Six steps, and the first two need no code:
- Inventory where the documents live. Server, Drive, email attachments, someone's disks. Two or three unmentioned places always turn up.
- Choose one place where they will live from now on. Plus a one-line rule about where new files go. A search over five folders breaks when someone saves into a sixth.
- Extract the text from each file, with OCR if needed. Native PDFs and Word files hand it over directly. Scans and photos need OCR. Without this step there is no search.
- Store four fields per document. Type, date, counterparty, reference. Those four filter 90% of real searches. Twenty tags get filled in by nobody.
- Index the text and build one search box. PostgreSQL with a full-text index handles tens of thousands of documents comfortably. Type part of a sentence, filter by the four fields.
- Apply permissions and measure what gets searched. Filter by role before showing results, and log the searches that return nothing: that list tells you what's missing or badly tagged.
Step four decides whether the search gets used or abandoned. The fields are chosen by watching how people actually ask, not how they ought to ask.
What do I do with scanned documents?
Run them through OCR, which turns the image into searchable text. A free engine like Tesseract works reasonably well on clean scans; with skewed documents, stamps over the text or phone photos, quality drops and a paid service pays off, which in practice costs cents per page.
Two warnings from experience. OCR never hits a hundred per cent: on normal scans it sits somewhere between ninety and ninety-eight per cent of characters, which means some exact search will fail. And processing twenty thousand pages takes hours of machine time, so you run it once in the background and then only over new files.
It's worth storing the extracted text separately from the original file. That way you can reindex, switch search engines or improve the OCR later without reprocessing everything.
Which option fits the size of my archive?
| Situation | What to use | What you'll run into |
|---|---|---|
| Hundreds of files, one folder | System or Drive search | Enough. Build nothing |
| Thousands of files, several places | Your own index on a database | Two or three weeks, mostly cleaning |
| Thousands, many of them scans | The above plus an OCR pass | Hours of processing and some recognition errors |
| Complex business searches | Index plus custom fields per document type | The types must be defined before any code |
| You want to ask in plain language | An AI layer over the existing index | Cost per query and the need to cite the source |
The temptation is to jump straight to the last row. It usually works out better to build the first four and check whether the fifth is needed. Often it isn't.
How much does it cost and what does it save?
An internal search over documents already gathered, with OCR and permissions, is usually two or three weeks of work. Hosting costs little: a small server with a database handles tens of thousands of documents for a few tens of euros a month. Paid OCR, if used, is counted in cents per page and is a one-off on the historical archive.
The saving is measured by counting searches. At the firm above, four people looked for documents between six and ten times a day, averaging around ten minutes per search when the document didn't show up first time. That's several hours a day of the team spent finding things they already had. Bringing that average down to a minute needs no exotic technology.
When is it NOT worth building?
- When the problem is order, not search. If nobody knows which version of a contract is the good one, a search engine will hand you all four versions faster. First decide where the good one lives.
- When the volume is small. With a few hundred well-named files, system search already solves it and there's nothing to maintain.
- When the document manager you already pay for does it. If your ERP or document manager already indexes and searches, what you need is to configure it, not a second system in parallel.
- When nobody will make sure new documents get in. An index that stops being updated ages within weeks and ends up giving worse results than the folder.
What mistakes do people usually make?
- Indexing without cleaning. Putting three copies of the same contract under different names into the index fixes nothing: you get three results and still don't know which one counts.
- Asking for too much metadata. A twelve-field form on upload guarantees people stop uploading. Four fields, and the ones that can be derived automatically, better.
- Leaving permissions until the end. Permissions are designed at the start. Adding them later, once the search is already returning payroll to the wrong people, means rebuilding half the system.
- Trusting OCR completely. There will always be misread characters. The search should tolerate errors, and the user should be able to filter by fields even when the text fails.
- Not looking at the empty searches. It's the best to-do list you'll ever get, and it comes free.
Frequently asked questions
How long does it take to build an internal document search?
With the documents already gathered in one place, a first version that searches text and filters by type, date and counterparty is usually running in two or three weeks. What stretches the project is almost never the search engine: it's the state of the files, especially if there are thousands of scans without OCR or spread across different disks.
Isn't Windows search or Google Drive search enough?
For a few files in a single folder, yes, and you need to build nothing. They fall short when documents are spread across several places, when there are scans with no text, or when you need to filter by business criteria such as "contracts with this supplier expiring this quarter". That is no longer searching for words, it's searching by the document's data.
Do I need artificial intelligence to search my documents?
To find a document, almost never: a good text search with four filters solves most cases and is faster and cheaper to maintain. AI helps when you want to ask in plain language and get a summary or a field pulled out of the document. It's a layer you add later, on top of a search that already works.
Can I build it without documents leaving my company?
Yes. Text extraction, OCR and the index can all run on your own server without sending anything to third parties, and that is the sensible route when there is personal data or contracts with confidentiality clauses. If you also want an AI layer, anonymise before sending or use a model hosted on your own infrastructure.
Four fields and an index solve nearly everything
Gather the documents in one place, extract their text (with OCR if scanned), store type, date, counterparty and reference, and index. With that and a search box carrying four filters you cover the vast majority of what people actually look for in a company.
The project gets stuck on the state of the files, not on the search engine. And before building anything, check whether your volume justifies more than system search: with a few hundred well-named files, it doesn't.
Is your day going into looking for documents?
If contracts and invoices are spread across the server, a Drive and the inbox, we can build a search of your own that finds them in seconds without taking them out of your infrastructure. Tell me how many documents you have and in how many places.
See the custom software service Let's talk about your case