Efficient procedure to maintain a sqlite backend #119

New issue

Open

opened 2025-09-01 16:38:27 +00:00 by mih · 0 comments

mih commented

2025-09-01 16:38:27 +00:00

(Migrated from github.com)

Right now, the best approach appears to be to have a record_dir collection, maintain that with regular file system operations (in particular delete records, if needed), and then convert to sqlite when needed.

However, this last step appears to be complex -- I suspect because the involved commands depend on the configuration of a service deployment and do not operate just on a backend alone. The procedure I find myself doing is

take the server down
configure the target collection to use the record_dir backend
move the new records in
rebuild the index
run dump-things-copy-store record_dir:... sqlite:...
reconfigure the collection to used the sqlite backend
restart the server

This is complex, and it require the server to be restarted, implying some downtime.

How could this be done better?

Right now, the best approach appears to be to have a `record_dir` collection, maintain that with regular file system operations (in particular delete records, if needed), and then convert to sqlite when needed. However, this last step appears to be complex -- I suspect because the involved commands depend on the configuration of a service deployment and do not operate just on a backend alone. The procedure I find myself doing is - take the server down - configure the target collection to use the `record_dir` backend - move the new records in - rebuild the index - run `dump-things-copy-store record_dir:... sqlite:...` - reconfigure the collection to used the sqlite backend - restart the server This is complex, and it require the server to be restarted, implying some downtime. How could this be done better?