Indexing

Indexing turns supported content into embeddings and stores the vectors and result metadata in local ChromaDB. File indexing and Photos indexing share one backend and should be run one at a time.

Watched folders

The current defaults are:

~/Documents
~/Desktop
~/Downloads
~/Pictures

The Status page lists the saved folders. Use Add Folder or Index a Folder… to choose another folder, and use the remove control to stop treating a folder as watched.

Removing a watched folder does not delete its existing entries from ChromaDB. Clear the full index and rebuild if you need those results removed immediately.

Manual indexing

The file pipeline recursively scans each folder, counts eligible files, then processes them sequentially. It upserts by a SHA-256 identifier derived from the file path, so a later successful run replaces the entry at that path.

Automatic indexing

Automatic indexing is off by default. When enabled, the backend watches the saved folders recursively.

• Creates and modifications are queued for indexing.
• Moves delete the old path entry and index the new path when supported.
• Deletes remove the old path entry from ChromaDB.
• Events are debounced for five seconds and checked every two seconds.
• Queued events wait while a manual indexing run is active.

Changes to the watched-folder list restart the watcher when automatic indexing is enabled.

Progress and status

The /status data shown by the desktop app includes:

• total items and counts by modality;
• last-indexed timestamp;
• whether an index job is active;
• indexed and error counts;
• current file or current Photos phase;
• total files found;
• per-folder totals, indexed counts, and errors; and
• the most recent Photos result or error for about 60 seconds after completion.

Photos phases can include starting, scanning the library, filtering existing assets, downloading, embedding, done, cancelled, or error.

Cancel an indexing run

The backend endpoint sets cancellation flags for both pipelines and responds with {"status":"cancelling"}. File indexing checks between scan/index operations. Photos work already blocked inside PhotoKit or an embedding request may not stop immediately.

The current desktop Stop Indexing action restarts the backend. Entries already written to ChromaDB remain, and the next run skips unchanged items. The CLI has no stop command because it runs the index operation in the foreground; use the terminal interrupt appropriate for the process.

Incremental behavior

For folder files, Memsearch skips an existing entry when the file modification time is not newer than the stored indexed_at timestamp. For Photos, it compares the Photos asset modification date when one exists; otherwise an existing asset is skipped.

This makes repeat runs incremental. A --limit applies to the number of eligible items selected for that run, not to the total size of the existing index.

File size and empty files

The default maximum file size is 50 MB. Files that are empty, exceed MEMSEARCH_MAX_FILE_SIZE_MB, cannot be stat-ed, or have no supported extension are skipped.

Images prepared for Gemini are converted to JPEG. If the JPEG exceeds 4 MB, Memsearch downsizes it before the embedding request. This embedding-payload resize is separate from the general 50 MB source-file limit.

Default exclusions

The backend excludes path components matching:

.*
__pycache__
node_modules
.git
.DS_Store
Thumbs.db

The .* entry causes hidden files and hidden directories to be excluded. Exclusions are configurable through MEMSEARCH_EXCLUDE_PATTERNS, but the desktop UI does not currently expose this setting.

Supported formats

Images

PDF

Text and source files

.txt .md .markdown .rst .csv .json .yaml .yml .toml .xml .html .css
.js .ts .py .rb .go .rs .java .c .cpp .h .hpp .swift
.sh .bash .zsh .sql .r .m .tex .log

Text processing reads UTF-8 with replacement and falls back to Latin-1 on read failure. Only the first 8,000 characters are embedded.

Errors and skipped counts

An indexing error can come from file permissions, unsupported/corrupt content, Photos downloads, Gemini, ChromaDB, or the filesystem. File runs retain up to 100 detailed errors in status. Skipped counts also include unchanged or ineligible files, so a high skipped count does not necessarily indicate failure.

Clear indexed data

Select Clear Index on Status, call DELETE /index, or run:

cd backend
uv run memsearch clear

Clearing deletes the ChromaDB collection and cached .jpg thumbnails, then recreates an empty collection. It does not delete source files, Photos assets, watched-folder settings, or the Gemini key.

Related configuration

See Configuration for MAX_FILE_SIZE_MB, MAX_CONCURRENT_EMBEDS, EXCLUDE_PATTERNS, watched-folder persistence, and the local storage paths.