-
-
Notifications
You must be signed in to change notification settings - Fork 291
fix(SyncProcess): Reduce size of continuation on disk #2140
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: develop
Are you sure you want to change the base?
Conversation
0a35501 to
27d0699
Compare
Signed-off-by: Marcel Klehr <mklehr@gmx.net>
Signed-off-by: Marcel Klehr <mklehr@gmx.net>
… time Signed-off-by: Marcel Klehr <mklehr@gmx.net>
Signed-off-by: Marcel Klehr <mklehr@gmx.net>
be09e21 to
426a078
Compare
don't throw if index doesn't exist Signed-off-by: Marcel Klehr <mklehr@gmx.net>
Signed-off-by: Marcel Klehr <mklehr@gmx.net>
…eIndex Signed-off-by: Marcel Klehr <mklehr@gmx.net>
Signed-off-by: Marcel Klehr <mklehr@gmx.net>
Signed-off-by: Marcel Klehr <mklehr@gmx.net>
Signed-off-by: Marcel Klehr <mklehr@gmx.net>
Signed-off-by: Marcel Klehr <mklehr@gmx.net>
Signed-off-by: Marcel Klehr <mklehr@gmx.net>
Signed-off-by: Marcel Klehr <mklehr@gmx.net>
Signed-off-by: Marcel Klehr <mklehr@gmx.net>
501e589 to
fa6ffaf
Compare
(important for mappings and tree index) Signed-off-by: Marcel Klehr <mklehr@gmx.net>
fa6ffaf to
e7bcdf4
Compare
Signed-off-by: Marcel Klehr <mklehr@gmx.net>
Signed-off-by: Marcel Klehr <mklehr@gmx.net>
to reduce memory pressure Signed-off-by: Marcel Klehr <mklehr@gmx.net>
Signed-off-by: Marcel Klehr <mklehr@gmx.net>
Signed-off-by: Marcel Klehr <mklehr@gmx.net>
0e4a096 to
55c8a56
Compare
Signed-off-by: Marcel Klehr <mklehr@gmx.net>
Signed-off-by: Marcel Klehr <mklehr@gmx.net>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR optimizes the continuation persistence mechanism to reduce disk storage requirements by selectively persisting only the necessary state data based on the current sync stage. The changes introduce conditional logic to determine which members need to be persisted, add async JSON deserialization for better performance, and refactor tree indexing to use incremental updates instead of full rebuilds.
Key Changes
- Modified
getMembersToPersist()methods inDefault.tsandUnidirectional.tsto conditionally include state members based on sync progress, significantly reducing continuation size - Introduced
Diff.fromJSONAsync()for asynchronous JSON parsing and addedsetProgress()methods to properly hydrate persisted state - Refactored tree index management with new
updateIndex()andremoveFromIndex()methods to incrementally maintain indexes instead of full rebuilds
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 14 comments.
Show a summary per file
| File | Description |
|---|---|
| src/lib/strategies/Default.ts | Implements conditional member persistence based on sync stage and adds async progress restoration |
| src/lib/strategies/Unidirectional.ts | Applies similar conditional persistence logic for unidirectional sync and switches to async JSON parsing |
| src/lib/Tree.ts | Adds incremental index update methods (updateIndex, removeFromIndex) and exposes index property |
| src/lib/Diff.ts | Introduces fromJSONAsync method for async deserialization and includes code formatting improvements |
| src/lib/adapters/NextcloudBookmarks.ts | Replaces full createIndex() calls with incremental updateIndex() and removeFromIndex() operations |
| src/lib/adapters/Caching.ts | Updates index management to use incremental methods and fixes orderFolder logic |
| src/lib/LocalTabs.ts | Adds helper methods for ID conversion and changes folder ID format to use string prefixes |
| src/lib/browser/BrowserAccountStorage.js | Simplifies cache/mappings initialization by using setEntry directly instead of changeEntry |
| src/lib/CachingTreeWrapper.ts | Updates to use incremental index methods and adds proper tree copying |
| src/lib/Account.ts | Simplifies cache tree initialization during sync continuation |
| src/test/test.js | Wraps plain objects with Bookmark constructor for proper type handling |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Signed-off-by: Marcel Klehr <mklehr@gmx.net>
Signed-off-by: Marcel Klehr <mklehr@gmx.net>
Signed-off-by: Marcel Klehr <mklehr@gmx.net>
instead of just the ID of the item that was passed Signed-off-by: Marcel Klehr <mklehr@gmx.net>
7ac0e1a to
b770ca8
Compare
Signed-off-by: Marcel Klehr <mklehr@gmx.net>
Signed-off-by: Marcel Klehr <mklehr@gmx.net>
Signed-off-by: Marcel Klehr <mklehr@gmx.net>
…ance Signed-off-by: Marcel Klehr <mklehr@gmx.net>
Signed-off-by: Marcel Klehr <mklehr@gmx.net>
Signed-off-by: Marcel Klehr <mklehr@gmx.net>
Signed-off-by: Marcel Klehr <mklehr@gmx.net>
Signed-off-by: Marcel Klehr <mklehr@gmx.net>
…ion tag Signed-off-by: Marcel Klehr <mklehr@gmx.net>
…t MOVEs since we're no longer using the Cache in Unidirectional it doesn't matter and is cleaner Signed-off-by: Marcel Klehr <mklehr@gmx.net>
…ance Signed-off-by: Marcel Klehr <mklehr@gmx.net>
No description provided.