1
0
Fork 0
Commit Graph

298 Commits

Author SHA1 Message Date
Hannah von Reth 552427ffc3
Update windows launch on start binary location
Fixes: #7672
2020-12-30 16:17:47 +01:00
Hannah von Reth 713a429675
Add todo for Qt 5.15 2020-12-30 16:17:47 +01:00
Hannah von Reth 1144473f5d
Cleanup pathtoUNC and its test 2020-12-30 16:17:46 +01:00
Hannah Rittich 5fc778bfaa Fixed the autostart for AppImages. See #2504.
Signed-off-by: Hannah Rittich <hrittich@users.noreply.github.com>
2020-12-21 11:56:09 +01:00
Kevin Ottens c57eff6fd8
Please the clang-tidy overlord
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-12-15 11:01:53 +01:00
Hannah von Reth b2934a68bc
Fix folder permission check on NTFS
Fixes: #8187
2020-12-15 10:59:26 +01:00
Kevin Ottens 201dbd54db
Remove the plugin loader mechanism for VFS backends
We will have all the code in public anyway so it can just be compiled
in. Thus no need to go through the plugin loading dance. Replaced the
loading with factory functions. Kept mostly the same structure
otherwise.

Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-12-15 10:59:24 +01:00
Kevin Ottens 2cf76bbcbf
Repair the Windows Build
We now enforce the use of QStringLiteral and friends in some places,
but that feel through the cracks for some of the Windows specific code.

Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-12-15 10:59:21 +01:00
Hannah von Reth dd3e70b667
VirtualFiles: Ensure the target location supports vfs
Fixes: #8131
2020-12-15 10:59:20 +01:00
Hannah von Reth 8796f25994
Log error code as hex 2020-12-15 10:59:20 +01:00
Hannah von Reth 1619cd5dcc
Use winvfs by default when adding folder sync connection
Fixes: #8083
2020-12-15 10:59:19 +01:00
Hannah von Reth 1aa1ea7bea
Return empty string instead of 1 as adler32 checksum for empty files 2020-12-15 10:59:18 +01:00
Hannah von Reth 23f7c51f70
Print checksum on mismatch 2020-12-15 10:59:18 +01:00
Hannah von Reth 3caf8276a2
Remove pointless warning
We also don't have a version entry in the db if thne db is new.
As I don't expect regular updates from 1.5 this message can just be removed.

Fixes: #8004
2020-12-15 10:59:17 +01:00
Hannah von Reth 2d66025d72
csync: apply strict QString handling 2020-12-15 10:59:17 +01:00
Hannah von Reth 563b347567
csync: apply strict QString handling 2020-12-15 10:59:16 +01:00
Hannah von Reth 82dbf8b5e1
VFS: remove now unused parameter 2020-12-15 10:59:16 +01:00
Hannah von Reth c1a91e91e7
Use the servers checksum type by default 2020-12-15 10:59:16 +01:00
Hannah von Reth 2887a93c40
Win: Use full Windows paths in file watcher and improve logging 2020-12-15 10:59:16 +01:00
Hannah von Reth 7fa7bc54c4
Win: Move hresultToQString from vfs plugin to Utility::formatWinError 2020-12-15 10:59:16 +01:00
Hannah von Reth 29c6f44124
Logging: Print enum before cast in SqlQuer::bindValue 2020-12-15 10:59:15 +01:00
Hannah von Reth f932dfc648
VFS: Tell the vfs plugin whether we have multiple accounts
This allows us to decide on the presentation of the account
2020-12-15 10:59:14 +01:00
Hannah von Reth c803292829
Use Q_ENUM_NS for PinState 2020-12-15 10:59:10 +01:00
Hannah von Reth 868b05f25b
Improve logging of issues during plugin loading
If the plugin could not be loaded the client calls qFatal
Make the loading warnings critical so they get printed before we crash
2020-12-15 10:59:08 +01:00
Hannah von Reth 9176afe263
Make sure all cases are handled 2020-12-15 10:59:08 +01:00
Olivier Goffart 7fd4a280f0
Make PluginFactory virtual to silent -Wnon-virtual-dtor warning
This is not necessary, but it also shouldn't hurt.
2020-12-15 10:59:06 +01:00
Hannah von Reth 466e8abc91
[Core] Include more information about the OS in the user agent 2020-12-15 10:59:05 +01:00
Hannah von Reth 3317e354f2
Prepend PLUGINDIR else its pointless 2020-12-15 10:59:05 +01:00
Olivier Goffart 66f7b27121
VFS: Do not overwrite existing files by placeholder
For issue #7557 and #7556

Note: this change the API of the VFS plugin, so the VFS plugin needs small
adaptations
2020-12-15 10:59:03 +01:00
Christian Kamm c9476a11f4
Checksums: Explicitly close file before reporting result
To ensure it's no longer open when the finished signal fires.
2020-12-15 10:59:00 +01:00
Christian Kamm bade7aedc6
Checksums: Add back QIODevice api
Because the winvfs plugin needs it. But be more careful about the
device's lifetime this time.
2020-12-15 10:58:55 +01:00
Christian Kamm ea829f96ca
Download: Don't trigger too many concurrent hash computations
Previously the job would only become "active" when the downloads
started. That meant that arbitrarily many hash computations could be
queued at the same time.

Since the the file was opened during future creation this could lead to
a "too many open files" problem if there were lots of new-new conflicts.

To change this:
- Make PropagateDownload become active when computing a hash
  asynchronously.
- Make the computation future open the file only once it gets run. This
  will make it less likely for this problem to occur even if thousands
  of these futures are queued.

For #7372
2020-12-15 10:58:55 +01:00
Christian Kamm 8a5a185752
Fix 'unused parameter name' warnings 2020-12-15 10:58:55 +01:00
Christian Kamm c9dbe46542
Checksums: Fix crash due to threading issue
The checksum computation thread was potentially using a QFile that was
deleted in the gui thread.

For #7368
2020-12-15 10:58:54 +01:00
Christian Kamm 0df3b83bd2
Vfs: Add hook to allow update-metadata for unchanged files
Allows winvfs to convert files to placeholders when vfs is enabled. This
is required to mark files as in-sync #7329.
2020-12-15 10:58:53 +01:00
Christian Kamm 5acb157a7e
VfsSuffix: Wipe stale pin states #7273
Previously the pin states of deleted files stayed in the 'flags'
database and could be inadvertently reused when a new file with the same
name appeared. Now they are deleted.

To make this work right, the meaning of the 'path' column in the 'flags'
table was changed: Previously it never had the .owncloud file suffix.
Now it's the same as in metadata.path.

This takes the safe parts from #7274 for inclusion in 2.6. The more
elaborate database schema changes (why use 'path' the join the two
tables in the first place?) shall go into master.
2020-12-15 10:58:52 +01:00
Christian Kamm fb47419e84
Vfs: Move pin state if files move #7250
Previously renames of items didn't carry the pin state with them.
2020-12-15 10:58:52 +01:00
Christian Kamm edad7ce7d7
Vfs: Mark sqlite temporaries excluded on db-open #7141
The previous patch ensured that the sqlite temporaries weren't deleted
and recreated for every sync run, but there was still time between
client startup and the first sync run where they would have the
"needs-sync" icon.
2020-12-15 10:58:50 +01:00
Christian Kamm 7774b8049e
Vfs: Distinguish availability error kinds #7143
Previously "no-availability" meant db-error and querying the
availability of a nonexistant path returned AllHydrated.

Now, the availability has a DbError and a NoSuchItem error case.
2020-12-15 10:58:49 +01:00
Christian Kamm 7f3f13fd97
Vfs: "free space" only shows when it has an effect #7143
To do this, introduce AllDehydrated availability and rename
SomeDehydrated to Mixed - it now guarantees there are also hydrated
items.
2020-12-15 10:58:49 +01:00
Christian Kamm 40d9fc4f4b
Vfs: Adjust and centralise action text #7143
Saying "Currently available locally" sounds more like an indicator than
"Availably locally" does. Centralizing translations avoids consistency
issues between shell context menus and sync folder context menu.
2020-12-15 10:58:49 +01:00
Christian Kamm a4f357ee4b
Sqlite: Use FULL synchronous mode with non-WAL journal
According to the documentation DELETE+NORMAL isn't safe from corruption
on older file systems.
2020-12-15 10:58:48 +01:00
Christian Kamm dcf34316fd
Vfs: Add 'availability', a simplified, user-facing pin state #7111
The idea is that the user's question is "is this folder's data available
offline?" and not "does this folder have AlwaysLocal pin state?".
The the answers to the two questions can differ: an always-local
folder can have subitems that are not always-local and are dehydrated.

The new availability enum intends to describe the answer to the user's
actual question and can be derived from pin states. If pin states aren't
stored in the database the way of calculating availability will depend
on the vfs plugin.
2020-12-15 10:58:47 +01:00
Christian Kamm 590db28541
Vfs: Clear up relationship between _type and pin state
The pin state is a per-item attribute that has an effect on _type:
AlwaysLocal dehydrated files will be marked for hydration and OnlineOnly
hydrated files will be marked for dehydration.

Where exactly this effect materializes depends on how the pin states are
stored. If they're stored in the db (suffix) the dbEntry._type is
changed during the discovery.

If the pin state is stored in the filesystem, the localEntry._type must
be adjusted by the plugin's stat callback.

This patch makes pin states behave more consistently between plugins.
Previously with suffix-vfs pin states only had an effect on new remote
files. Now the effect of pinning or unpinning files or directories is as
documented and similar to other plugins.
2020-12-15 10:58:46 +01:00
Olivier Goffart 4346567a03
Async Poll: keep the size in the database
This was not required with 2.5 because a size of 0 was ignorted when comparing
size by the csync updater, to be compatible with very old version of the database.
But the we discovery will still think the file is changed if the database contains
a size of 0
2020-12-15 10:58:44 +01:00
Olivier Goffart 46bf3ed31a
Upload: asynchronious operations
Implements https://github.com/owncloud/core/pull/31851
2020-12-15 10:58:44 +01:00
Christian Kamm d496aa5933
Linux: Add autostart delay to avoid tray issues #6518
It seems that sometimes the tray implementation isn't ready on system
startup. Retrying later seems to not help. Delaying the start of the
client is the workaround that people have reported as effective.
2020-12-15 10:58:43 +01:00
Christian Kamm 4bd062f5be
OwnSql: Distinguish no-data from error #6677
This could fix a problem where the client incorrectly decides to delete
local data.

Previously any sqlite3_step() return value that wasn't SQLITE_ROW would
be interpreted as "there's no more data here". Thus an sqlite error at a
bad time could cause the remote discovery to fail to read an unchanged
subtree from the database. These files would then be deleted locally.

With this change sqlite errors from sqlite3_step are detected and
logged. For the particular case of SyncJournalDb::getFilesBelowPath()
the error will now be propagated and the sync run will fail instead of
performing spurious deletes.

Note that many other database functions still don't distinguish
not-found from error cases. Most of them won't have as severe effects on
affected sync runs though.
2020-12-15 10:58:43 +01:00
Christian Kamm 238ac53666
Ensure local discovery on selective sync changes
As far as I'm aware local discovery can be skipped on folders that are
selective-sync blacklisted, so a local discovery is required when an
entry is removed from the blacklist.

Also rename
avoidReadFromDbOnNextSync() -> schedulePathForRemoteDiscovery()
since the old name might also imply it's not read from db in the local
discovery - which is not the case. Use Folder::
schedulePathForLocalDiscovery() for that.
2020-12-15 10:58:41 +01:00
Christian Kamm 4c04351360
Fix warnings about signedness
Sizes are always qint64, not unsigned.
TransferIds are always uint.
2020-12-15 10:58:41 +01:00
Christian Kamm b06f67baf0
Vfs: Ensure VfsOff works without start() being called
In tests an un-started Vfs instance is sometimes passed to SyncEngine
via SyncOptions.
2020-12-15 10:58:40 +01:00
Christian Kamm 6a977edeee
Vfs: Remove VfsDefaults
That just complicated things. It's ok if Vfs is not a fully abstract
interface class.

The pinstate-in-db methods are instead provided directly on Vfs and
VfsSuffix and VfsOff use them to implement pin states.

The start() method is simply non-virtual and calls into startImpl() for
the plugin-specific startup code.
2020-12-15 10:58:40 +01:00
Christian Kamm e9cbe13598
Vfs: Add option to hide socketapi pin actions
Because some plugins provide alternative ui.
2020-12-15 10:58:40 +01:00
Christian Kamm 6c5fa1dadd
Vfs: dehydration is separate action
Allows for better attribute preservation.

Also add verifyFileUnchanged() call before dehydration to avoid data
loss when discovery takes a while.
2020-12-15 10:58:39 +01:00
Christian Kamm d8873c18a1
File watcher: Pin state attribute changes are valid notifications
Previously they would be discarded since the file's mtime or size hadn't
changed.
2020-12-15 10:58:39 +01:00
Christian Kamm 83a818678f
PinStates cleanup
- SyncJournalDB functions now behind internalPinStates() to avoid
accidental usage, when nearly everyone should go through Vfs.
- Rename Vfs::getPinState() to Vfs::pinState()
2020-12-15 10:58:39 +01:00
Christian Kamm 0eb4065197
Folder: Add remoteFolderTrailingSlash()
There were cases where the "/" exception wasn't handled correctly
and there'd be extra slashes in generated paths.
2020-12-15 10:58:39 +01:00
Christian Kamm 597cc60f52
Vfs: Enable propagating attributes on download 2020-12-15 10:58:38 +01:00
Christian Kamm 01f7bc7b49
SyncFileStatusTracker: Distinguish Warning and Excluded
Any folder with a (potentially deeply) contained error will have
StatusWarning. StatusExcluded marks exclusions. The difference is useful
to know for VFS.
2020-12-15 10:58:38 +01:00
Christian Kamm 7f400e3226
Pin state updates
- unspecified and inherited are different
- move enum to header in common/
- access through Vfs instead of directly in Journal
2020-12-15 10:58:38 +01:00
Christian Kamm 2722c61515
Vfs: Shell integration is enabled optionally
Needs to be disabled for tests in some cases.
2020-12-15 10:58:38 +01:00
Christian Kamm e2eea24a03
Vfs: Send SyncFileStatusTracker data to vfs plugins 2020-12-15 10:58:37 +01:00
Christian Kamm 31394f14b5
Vfs: Make files that end up in db placeholders
Since 'placeholder' just means that it's an item of the special type
that the vfs plugin can deal with - no matter whether hydrated or
dehydrated - all done items should become placeholders. Even
directories.

Now every file that passes through updateMetadata() will be converted to
a placeholder if necessary.
2020-12-15 10:58:37 +01:00
Christian Kamm fe27804afb
Move SyncFileStatus to libcommon
It'll be needed in vfs plugins so they can connect to the data coming
out of SyncFileStatusTracker.
2020-12-15 10:58:37 +01:00
Christian Kamm 22255e4976
vfs: Add vfs migration options to folder context menu
This allows enabling and disabling vfs.

To distinguish this operation from setting the root pin state, the
availability setting is adjusted as well to be similar to the
menu that shows in the shell extensions.
2020-12-15 10:58:33 +01:00
Christian Kamm 20ef0a0290
vfs: Allow retrieving of pin state paths and flags 2020-12-15 10:58:33 +01:00
Christian Kamm 7ef6e60660
Result: Add copy/move ctor/op=
It has a destructor and these operations make sense. Particularly the
move is important for code like:

Result<x, y> foo() { Result<x, y> v; return v; }

because the move-ctor will not autogenerate if x or y are not trivially
destructible.
2020-12-15 10:58:33 +01:00
Christian Kamm 4df101ed84
vfs: Allow (de-)hydrating the full sync folder 2020-12-15 10:58:33 +01:00
Christian Kamm b4e73690b7
sqlite: Set exclusive locking_mode to avoid WAL issues #6881
Can be overridden with OWNCLOUD_SQLITE_LOCKING_MODE
2020-12-15 10:58:32 +01:00
Christian Kamm aa382eda29
Db: Add wiping of pin state for subtrees 2020-12-15 10:58:30 +01:00
Christian Kamm d956f518a8
vfs: Remove newFilesAreVirtual - use root PinState instead
This unifies how to deal with pin states.

Also enable reading a folders direct pin state vs its effective pin
state.
2020-12-15 10:58:30 +01:00
Christian Kamm 68126ac208
vfs: Introduce PinState db storage #6815
The idea is to allow folders (and later maybe files?) to be
- pinned to be available locally
- pinned to be online only
- inherit their pin from the parent

Where this pinning only controls the default for new files.
Subfolders may have a different pin state, and contained files
may be hydrated or dehydrated based on user actions.

This value is stored in a new 'flags' table. The idea is to store
data there that doesn't necessarily exist for each metadata entry.
The selective sync state could be migrated to this table.
2020-12-15 10:58:29 +01:00
Christian Kamm 69144566ce
Database: Change path for new dbs to .sync_* #5904
This is to avoid issues on OSX, where the ._ prefix has special meaning.

Originally (before 2.3.2) ._ was necessary to guarantee exclusion. But
since then the .sync_ prefix is excluded as well.

This does not affect existing database files.
2020-12-15 10:58:28 +01:00
Christian Kamm bdfda460e6
Vfs plugins: Available plugins must load
A plugin that can't be loaded due to dependency issues should not be
considered as available.
2020-12-15 10:58:28 +01:00
Christian Kamm d87c2b803d
vfs: Make Vfs ctor explicit 2020-12-15 10:58:27 +01:00
Christian Kamm 1e5e884805
vfs: Don't always load plugins, check metadata 2020-12-15 10:58:26 +01:00
Christian Kamm b30f79edf6
vfs: Ensure SyncOptions::_vfs is never null
- Create a VfsOff derived class
- Make it a shared pointer shared with Folder::_vfs
2020-12-15 10:58:26 +01:00
Christian Kamm fa2450cf11
vfs: Be more careful about Vfs instance ownership 2020-12-15 10:58:25 +01:00
Christian Kamm 77e5b956d1
Plugin: Drop PluginLoader 2020-12-15 10:58:25 +01:00
Christian Kamm 9bc28e3006
Plugin: style fixes 2020-12-15 10:58:25 +01:00
Christian Kamm a5264f7740
Checksums: Make file ownership more explicit 2020-12-15 10:58:24 +01:00
Christian Kamm 9ced8dd836
vfs: Improve modeFromString() signature 2020-12-15 10:58:24 +01:00
Christian Kamm f502a526fa
Generalize Result<> class, add Optional<>
To make it nicer to use outside of HTTP results.
2020-12-15 10:58:24 +01:00
Christian Kamm e7e6b839c0
vfs: Add API docs 2020-12-15 10:58:24 +01:00
Dominik Schmidt bfe136da7a
Fix export of vfs functions 2020-12-15 10:58:23 +01:00
Christian Kamm fa6f3cd847
vfs: Fix plugin decision in wizards, sanitize loading 2020-12-15 10:58:23 +01:00
Christian Kamm 2b20985875
winvfs: initial work
Done by ckamm and dschmidt
2020-12-15 10:58:21 +01:00
Christian Kamm 22e08cf6ad
Checksums: Work on QIODevice*s
Needed for cfapi where we want to feed data through a custom device
which retrieves data from the windows api.
2020-12-15 10:58:21 +01:00
Christian Kamm 582803b6bd
cjhash: Fix build fix
See 17a4055e8d658f24b1fc47c44fd2daf66c7d5dbf
2020-12-15 10:58:20 +01:00
Olivier Goffart 175b064364
cjhash: use Q_FALLTHROUGH
This fixes the warning
   warning: unknown option after ‘#pragma GCC diagnostic’ kind [-Wpragmas]

Issue #6872
2020-12-15 10:58:20 +01:00
Olivier Goffart d6a0290058
Optimisation: Add a cache SyncJournalDb::mapChecksumType
No need to do two sql query for something that's always the same and
there are very few checksum types
2020-12-15 10:58:19 +01:00
Olivier Goffart 69de2d5180
Sync: optimize by removing setFileRecordMetadata
Inh most case we already have a record from before, so avoid doing a useless
lookup in the database.
In owncloudpropagator.cpp, directories do not have a checksum so no need
to call a function that preserves it
2020-12-15 10:58:19 +01:00
Olivier Goffart 7e1840bb2f
OwnSql: Speedup by avoiding allocating an error string on success 2020-12-15 10:58:18 +01:00
Olivier Goffart bb9ce8db21
Test that the sync behave well if there are errors while reading the database 2020-12-15 10:58:17 +01:00
Olivier Goffart b820e46805
Fix compiler warning 2020-12-15 10:58:14 +01:00
Dominik Schmidt 5d9370594d
Disable autostartCheckBox if autostart is configured system wide 2020-12-15 10:58:14 +01:00
Christian Kamm 00edcf98a1
Discovery: Virtual file handling adjustments
- adjust virtual file path handing
- helpers for vfs suffix adding/removal
- helpers for isDirectory/isVirtual on SyncJournalRecords
- be clear about what PathTuple _local/_server mean
2020-12-15 10:58:14 +01:00
Olivier Goffart 58eaf9940a
Database: Add an index on the parent path
So we can quickly query the items in a parent directory

This uses a custom slite3 function, which means that when downgrading the client,
or using another tool to add entries in the database, any insertion in the metadata
table will produce an error: "unknown function: parent_hash()"
(This will crash the client 2.5)
2020-12-15 10:58:13 +01:00
Olivier Goffart c009dae1ce
New discovery algorithm: fixups
Adapt reviews from ckamm in https://github.com/owncloud/client/pull/6738#pullrequestreview-164623532

- SyncJournalFileRecord: initialize everything inline
- Add more comments
- And some ENFORCE
2020-12-15 10:58:08 +01:00
Olivier Goffart c8eff3da2d
New Discovery algorithm: Remove the sync cleanup phase
Since we do not recurse within some directories, many files are not seen.

The stale entry will cleanup by themself as the sync engine try to remove
the files that are already removed.
Should we need to actually do this cleanup, it should be dotected in the
discovery.
2020-12-15 10:58:08 +01:00