1
0
Fork 0
Commit Graph

153 Commits

Author SHA1 Message Date
Markus Goetz 26b5e36351
Discovery: List local directories from thread #7456 #7439 2020-12-15 10:58:59 +01:00
Christian Kamm e37e954720
Discovery: Remove level of indent around moves
No code changes.
2020-12-15 10:58:58 +01:00
Christian Kamm 28797baa39
Discovery: If a move is forbidden, restore the source
Previously the source was deleted (or attempted to be deleted), even if
the new location was not acceptable for upload. This could make data
unavilable on the server.

For #7410
2020-12-15 10:58:58 +01:00
Hannah von Reth e97784bb9d
Don't use mutable lambdas as they hide where we modify work on a copy 2020-12-15 10:58:55 +01:00
Christian Kamm 55ee3f440b
Vfs: Remove old db record when dehydrating via rename
For #7338
2020-12-15 10:58:54 +01:00
Christian Kamm e6990bd04e
Vfs: Make move detection work with virtual files #7001
Previously a checksum computation could be done on a suffix-placeholder
file, making discovery believe that no move took place.
2020-12-15 10:58:54 +01:00
Christian Kamm 0df3b83bd2
Vfs: Add hook to allow update-metadata for unchanged files
Allows winvfs to convert files to placeholders when vfs is enabled. This
is required to mark files as in-sync #7329.
2020-12-15 10:58:53 +01:00
Christian Kamm 5acb157a7e
VfsSuffix: Wipe stale pin states #7273
Previously the pin states of deleted files stayed in the 'flags'
database and could be inadvertently reused when a new file with the same
name appeared. Now they are deleted.

To make this work right, the meaning of the 'path' column in the 'flags'
table was changed: Previously it never had the .owncloud file suffix.
Now it's the same as in metadata.path.

This takes the safe parts from #7274 for inclusion in 2.6. The more
elaborate database schema changes (why use 'path' the join the two
tables in the first place?) shall go into master.
2020-12-15 10:58:52 +01:00
Olivier Goffart c335f69a65
Discovery: Do not abort the sync in case of error 404 (or 500)
Issue: #7199
2020-12-15 10:58:51 +01:00
Christian Kamm fbe2dbf4ab
Discovery: Query data-fingerprint on root item
Previously the property wasn't queried, meaning the fingerprint logic
couldn't get triggered.
2020-12-15 10:58:48 +01:00
Christian Kamm c50f041c5b
Discovery: 403 and 503 on root cause error
Previously these result codes during remote discovery of the sync root
would not cause an error and the discovery would get stuck.

Also extends RemoteDiscovery tests to check for errors on the root item.
2020-12-15 10:58:48 +01:00
Christian Kamm 7b96321df2
PropagateIgnore: Default to NormalError for INSTRUCTION_ERROR
Previously if one set the instruction to ERROR while forgetting to set
an error status, it'd propagate as FileIgnored. Now the default is
NormalError for INSTRUCTION_ERROR and FileIgnored for
INSTRUCTION_IGNORE.
2020-12-15 10:58:47 +01:00
Christian Kamm 590db28541
Vfs: Clear up relationship between _type and pin state
The pin state is a per-item attribute that has an effect on _type:
AlwaysLocal dehydrated files will be marked for hydration and OnlineOnly
hydrated files will be marked for dehydration.

Where exactly this effect materializes depends on how the pin states are
stored. If they're stored in the db (suffix) the dbEntry._type is
changed during the discovery.

If the pin state is stored in the filesystem, the localEntry._type must
be adjusted by the plugin's stat callback.

This patch makes pin states behave more consistently between plugins.
Previously with suffix-vfs pin states only had an effect on new remote
files. Now the effect of pinning or unpinning files or directories is as
documented and similar to other plugins.
2020-12-15 10:58:46 +01:00
Christian Kamm fd9b01981b
Detect missing server data during discovery #7112
This has two positive effects:
- We can put the error on the particular file that has missing data
- We can sync all other files
2020-12-15 10:58:46 +01:00
Christian Kamm 4bab93b246
Vfs: Better handling and more tests for suffix file renames
Previously removing the vfs suffix of a file always triggered a
conflict. Now it may just cause a file download.

This was done because users expected symmetry in the rename actions and
renaming foo -> foo.owncloud already triggers the "make the file
virtual" action. Now foo.owncloud -> foo triggers the "download the
contents" action.
2020-12-15 10:58:46 +01:00
Christian Kamm 3f55f9302e
Vfs: Hydrating a virtual is SYNC not NEW #7101
Previously it'd be NEW(ItemTypeFile), but now it has changed to be
SYNC(ItemTypeVirtualFileDownload) which allows better classification.
2020-12-15 10:58:45 +01:00
Christian Kamm 0cf19123a7
VFS: Unbreak behavior for rename+hydrate #7001
Users can rename a file *and* add/remove the vfs suffix at the same time
leading to very complex sync actions. This patch doesn't add support for
them, but adds tests and makes sure these cases do not cause unintened
behavior.

The rename will be propagated, but the users's hydrate/dehydrate request
will be ignored.
2020-12-15 10:58:44 +01:00
Christian Kamm 93afc2a04b
Discovery win: Fix detection of case-only renames
Previously they were detected as DELETE+NEW because if "a" is renamed to
"A" then QFile::exists("a") will still return true on Windows.
2020-12-15 10:58:42 +01:00
Christian Kamm f24687ec93
Vfs: Make dehydration a SYNC and not a NEW action
That change will be useful for the notifications. Previously the
dehydrated files were reported as "newly downloaded", now they're
reported as "updated".
2020-12-15 10:58:40 +01:00
Christian Kamm 8a8e93827f
Vfs suffix: Fix dehydration creating the wrong db entry 2020-12-15 10:58:40 +01:00
Christian Kamm 83a818678f
PinStates cleanup
- SyncJournalDB functions now behind internalPinStates() to avoid
accidental usage, when nearly everyone should go through Vfs.
- Rename Vfs::getPinState() to Vfs::pinState()
2020-12-15 10:58:39 +01:00
Christian Kamm 7f400e3226
Pin state updates
- unspecified and inherited are different
- move enum to header in common/
- access through Vfs instead of directly in Journal
2020-12-15 10:58:38 +01:00
Christian Kamm 5820ac8b41
Discovery: Files can have dehydrate/download actions
This will be used in conjunction with vfs plugins that detect whether a
file has a pending hydration/dehydration through independent means and
communicate that to the discovery through local file type.
2020-12-15 10:58:38 +01:00
Christian Kamm af1666788e
Discovery: Add signal for silentlyExcluded files
This allows SyncFileStatusTracker to also know about these. After all
its information is used to provide icons for them too.
2020-12-15 10:58:37 +01:00
Olivier Goffart 6f4bf585f0
Move: Fix move detection in directory move on the other side 2020-12-15 10:58:35 +01:00
Olivier Goffart 1fb4c22adf
Move: add more test and fix move within moves 2020-12-15 10:58:35 +01:00
Christian Kamm 836d298141
vfs suffix: Ignore server files or synced files with the suffix #6953 2020-12-15 10:58:34 +01:00
Christian Kamm d956f518a8
vfs: Remove newFilesAreVirtual - use root PinState instead
This unifies how to deal with pin states.

Also enable reading a folders direct pin state vs its effective pin
state.
2020-12-15 10:58:30 +01:00
Olivier Goffart 62ec4c9330
Discovery: Handle the blacklistFiles from the server capabilities
Issue #434

Ideally one could add the blacklist to the exlucde reggexp, but this
is simpler
2020-12-15 10:58:30 +01:00
Christian Kamm 486c25cb47
vfs: Use PinState in sync algorithm #6815
New files are virtual if the file's pin state is OnlineOnly.
2020-12-15 10:58:29 +01:00
Olivier Goffart a29320b18d
Discovery: Set right direction when restoring deleted discovery because it has modified files
(Catched by a faillure of t1.pl)
2020-12-15 10:58:29 +01:00
Christian Kamm 305d439c41
vfs: Separate vfs availability from new-files-virtual
This helps support 2.5 settings where there are virtual files in the
tree but new files aren't created virtual.

It's also a prelude for #6815

There's currently no way of
- upgrading vfs plugins (a silent suffix->winvfs upgrade is attempted
  once only, when moving to master)
- disabling vfs capabilities outright
2020-12-15 10:58:27 +01:00
Christian Kamm b30f79edf6
vfs: Ensure SyncOptions::_vfs is never null
- Create a VfsOff derived class
- Make it a shared pointer shared with Folder::_vfs
2020-12-15 10:58:26 +01:00
Christian Kamm f502a526fa
Generalize Result<> class, add Optional<>
To make it nicer to use outside of HTTP results.
2020-12-15 10:58:24 +01:00
Christian Kamm 96f4fd46e5
vfs: Make some behaviors suffix-vfs specific 2020-12-15 10:58:23 +01:00
Christian Kamm 90827c8162
vfs: Allow remote renames to propagate as such 2020-12-15 10:58:22 +01:00
Christian Kamm bee1b0edd4
vfs: Implement dehydration tagging 2020-12-15 10:58:22 +01:00
Christian Kamm ae9a7e088f
vfs: Fix suffix detection and handling 2020-12-15 10:58:22 +01:00
Christian Kamm 0f2ef42ba2
Fixes after rebase to master
- use vfs suffix in ProcessDirectoryJob
- fix include vfs.h
- fix local vio passing vfs
- fix checksum computation
- vfs mode use
- mingw lambda compile issue
2020-12-15 10:58:21 +01:00
Christian Kamm 22e08cf6ad
Checksums: Work on QIODevice*s
Needed for cfapi where we want to feed data through a custom device
which retrieves data from the windows api.
2020-12-15 10:58:21 +01:00
Olivier Goffart 5e05b61d2f
csync_vio_locale: use QString for the path in opendir
So we don't have to convert to utf8 and back again
2020-12-15 10:58:20 +01:00
Olivier Goffart 3c3619f99a
Exclude: do everything with QString wiuthout converting to char* 2020-12-15 10:58:19 +01:00
Olivier Goffart 164051b0c9
ProcessDirectoryJob::process: optimize so there is only one map 2020-12-15 10:58:19 +01:00
Olivier Goffart 69de2d5180
Sync: optimize by removing setFileRecordMetadata
Inh most case we already have a record from before, so avoid doing a useless
lookup in the database.
In owncloudpropagator.cpp, directories do not have a checksum so no need
to call a function that preserves it
2020-12-15 10:58:19 +01:00
Olivier Goffart b79e57d1c1
Discovery: fix double emission of finished in case of error 2020-12-15 10:58:17 +01:00
Olivier Goffart 7061f31887
Discovery: Fix downloading files when database is used for local discovery
This also fix the currently failling LockedFilesTest
2020-12-15 10:58:17 +01:00
Olivier Goffart eb23776f16
Discovery: Fix renaming on windows
buf.type is ItemFileSkip because csync_vio_local_stat does not set this field
2020-12-15 10:58:16 +01:00
Olivier Goffart beee123c80
New Discovery: Fix trailing slash causing failure on windows 2020-12-15 10:58:16 +01:00
Christian Kamm 00edcf98a1
Discovery: Virtual file handling adjustments
- adjust virtual file path handing
- helpers for vfs suffix adding/removal
- helpers for isDirectory/isVirtual on SyncJournalRecords
- be clear about what PathTuple _local/_server mean
2020-12-15 10:58:14 +01:00
Olivier Goffart 58eaf9940a
Database: Add an index on the parent path
So we can quickly query the items in a parent directory

This uses a custom slite3 function, which means that when downgrading the client,
or using another tool to add entries in the database, any insertion in the metadata
table will produce an error: "unknown function: parent_hash()"
(This will crash the client 2.5)
2020-12-15 10:58:13 +01:00
Christian Kamm b431f4ef0c
Discovery: Fix log output for instructions 2020-12-15 10:58:13 +01:00
Christian Kamm ce420d77a8
Discovery: Don't rebuild invalidFilname regex each call 2020-12-15 10:58:13 +01:00
Christian Kamm 113124cde5
Discovery: Introduce smaller functions 2020-12-15 10:58:13 +01:00
Christian Kamm 4f6f706f40
Excludes: drop csyncTraversalMatchFun()
The new discovery can call the traversal match function directly.
2020-12-15 10:58:13 +01:00
Christian Kamm d1aedcfd3c
Discovery: restructure processFileAnalyzeLocalInfo 2020-12-15 10:58:10 +01:00
Christian Kamm a2839bd40a
Discovery: easy conditions first in processFileAnalyzeRemoteInfo
Removing two levels of indent that way
2020-12-15 10:58:10 +01:00
Christian Kamm 76341904e9
Discovery: Add comments 2020-12-15 10:58:10 +01:00
Olivier Goffart 5a1c93d0ac
Discovery: make sure finished is not called twice, even in case of errors 2020-12-15 10:58:10 +01:00
Olivier Goffart f666511a4b
Discovery: Remove stale DB entries
And test the Remove/Remove case.

This means we need to always query the database for all the entries.
This showed another small bug in the test in which sync item for virtual
files at the root could have a slash in front of them.
2020-12-15 10:58:10 +01:00
Olivier Goffart aa18e10ff5
Discovery: cleanups and comments 2020-12-15 10:58:09 +01:00
Olivier Goffart b10b3e5eeb
Discovery: move checkMovePermissions to its own function 2020-12-15 10:58:09 +01:00
Olivier Goffart edd866b32b
Discovery: Adjust the instruction in case of resolved conflict
When resolving a conflict because the file was just updated on the server,
we write all the metadata on the database immediatly, so INSTRUCITON_NONE
is enough and UPDATE_METADATA is not needed
2020-12-15 10:58:09 +01:00
Olivier Goffart 7cddaf82ab
ProcessDirectoryJob: always set _childModified to true, regardless the direction
This was like that to handle the case of CSYNC_INSTRUCTION_TYPE_CHANGE, but just add
a condition in this location.
2020-12-15 10:58:09 +01:00
Olivier Goffart 46510c2f39
Discovery phase: refactor some code in DiscoveryPhase::findAndCancelDeletedJob
Less code duplication
2020-12-15 10:58:09 +01:00
Olivier Goffart 1c2a3279bb
New Discovery Algorithm: more cleanups
- rename progress() to be more explicit
 - Make some more member of the discovery phase private
2020-12-15 10:58:09 +01:00
Olivier Goffart afed46afcc
New discovery algorithm: Parallel PROPFIND 2020-12-15 10:58:08 +01:00
Olivier Goffart 35c0cf4e59
New discovery algorithm: Set the originalFile for ignored files
This is used to show the name in the UI
2020-12-15 10:58:08 +01:00
Olivier Goffart 52dcfcb166
New Propagation algorithm: Fetch, and emit, the root etag
Remove the feature to concatenate etags as servers that don't
have a root etag are no longer suported
2020-12-15 10:58:07 +01:00
Olivier Goffart d25d87e92c
New Discovery Algorithm: Ge tthe size of new folders
Also add a test that this works properly
2020-12-15 10:58:07 +01:00
Olivier Goffart ef542ac83d
New Discovery algorithm: Split the process function even further
Move the finialization in its own function.
This allow to save a bit of code duplication.

Also change the order of the parameter in the constructor for consistency
2020-12-15 10:58:07 +01:00
Olivier Goffart fc69dda246
New Discovery Algo: Refactor by splitting the processFile in two 2020-12-15 10:58:07 +01:00
Olivier Goffart b7d827d6d0
New Discovery Algorithm: Handle rename of virtual files 2020-12-15 10:58:07 +01:00
Olivier Goffart 21fe54fb13
New Discovery Algo: readability improvements
As proposed by ckamm on #6738
2020-12-15 10:58:04 +01:00
Olivier Goffart 3a06a7978f
New discovery algo: data finger print 2020-12-15 10:58:04 +01:00
Olivier Goffart 5683f1c33d
New Disco Algo: Handle spurious virtual files
Port commit 0b9049e6ff
for issue #6718
2020-12-15 10:58:04 +01:00
Olivier Goffart a36ed56f01
New Discovery algorithm: Refactor a bit the way the signal are emited 2020-12-15 10:58:02 +01:00
Olivier Goffart b6487181c6
New Discovery Phase: fix a few FIXME
Some FIXME were already fixed.
2020-12-15 10:58:02 +01:00
Olivier Goffart 9863500ec1
New discovery algorithm: handle symlinks 2020-12-15 10:58:01 +01:00
Olivier Goffart a9ec521bf1
New discovery algo: Handle Database error 2020-12-15 10:58:01 +01:00
Olivier Goffart 1bcbcd407c
New discovery phase: read the direct download URL and Cookie 2020-12-15 10:58:01 +01:00
Olivier Goffart ac24cdbde6
New Discovery Algo: Permsission check 2020-12-15 10:58:01 +01:00
Olivier Goffart 57068b0fd9
New Discovery Algorithm: handle server errors 2020-12-15 10:58:00 +01:00
Olivier Goffart 4031fb6d5b
Backport 64014dd374 2020-12-15 10:58:00 +01:00
Olivier Goffart f4a5167812
New Discovery Algo: Support the DatabaseAndFilesystem mode for local discovery 2020-12-15 10:57:59 +01:00
Olivier Goffart 370923791d
New Disco algortihm: Fix TestChunkingNG::connectionDroppedBeforeEtagRecieved
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-12-15 10:57:59 +01:00
Olivier Goffart f926248937
New discovery algorithm: More work on virtual files 2020-12-15 10:57:59 +01:00
Olivier Goffart 35e40b58ca
New Disco algorithm: Type change (file to dir) 2020-12-15 10:57:59 +01:00
Olivier Goffart 1e8c37d3d6
New discovery algorithm: Virtual files
The commented tests lines were implementation details
2020-12-15 10:57:59 +01:00
Olivier Goffart d54e00488a
New Discovery algorithm
Some error handling. In particular for the case where there is a conflict
between files and directories.

SyncEngineTest and SyncMoveTest passes
2020-12-15 10:57:58 +01:00
Olivier Goffart a384a2d1cb
New Discovery algorithm: Handle of move within a moved directory 2020-12-15 10:57:58 +01:00
Olivier Goffart bdd1e72dda
New discovery algoritmh: more on Renames 2020-12-15 10:57:58 +01:00
Olivier Goffart f43d07dc05
New Discovery algorithm: Check that the original file is still on the server while renaming 2020-12-15 10:57:58 +01:00
Olivier Goffart 7e36cc3fcb
New disco algorithm: Fix some moving
Fix TestSyncMove::testSelectiveSyncMovedFolder
2020-12-15 10:57:58 +01:00
Olivier Goffart 5a57a36729
New discovery algorithm: Local rename 2020-12-15 10:57:58 +01:00
Olivier Goffart f9a9be59e6
New discovery algo: Remote move
TestSyncMove::testRemoteChangeInMovedFolder
2020-12-15 10:57:57 +01:00
Olivier Goffart 92ddc60900
Handle Encoding Problems
TestSyncEngine now passes
2020-12-15 10:57:57 +01:00
Olivier Goffart 501da58b10
Ignore Hidden Files 2020-12-15 10:57:57 +01:00
Olivier Goffart 8bf69cf0cd
Port the invalid napme regexp
TestSyncEngine::testInvalidFilenameRegex
2020-12-15 10:57:57 +01:00
Olivier Goffart 9efd21d70b
More progress
TestSyncEngine::testFakeConflict / TestSyncEngine::testSyncFileItemProperties
2020-12-15 10:57:57 +01:00
Olivier Goffart e934f6b27b
New discovery algo
Make TestSyncEngine::testSelectiveSyncBug pass
2020-12-15 10:57:57 +01:00