- Put all tests in the bin directory so that DLLs can be loaded
- Add missing exports
- Skip tests that use code depending on zlib
- The "GMT" timezone is named differently, use the int constructor instead
5 tests are still failing, it's not really worth fixing at the moment
since no developper is currently using Windows as its main platform.
On macOS /var is a symlink to /private/var and we have to make sure that we
use the canonical path before and after it enters the code to make sure we
compare paths correctly.
- We need to use a QGuiApplication on macOS or else we don't get notifications
- Switch to use QSignalSpy rather than lists and sleeps
- Use system() for all modifications since we pass kFSEventStreamCreateFlagIgnoreSelf
- Keep using the local process on Windows since it catches its own events
It could be possible that _firstJob is marked as finished if
aborted before its parent PropagateDirectory was marked as finished,
allowing a posted scheduleNextJob call to schedule the child job
in-between.
We now delete subjobs as their propagation is complete. This allows us
to also release the item by making sure that nothing else is holding a
reference to it.
Remove the stored SyncFileItemVector from SyncEngine and SyncResult
and instead gather the needed info progressively as each itemCompleted
signal is emitted.
This frees some holes on the heap as propagation goes, allowing many
memory allocations without the need of requesting more virtual memory
from the OS, preventing the memory usage from increasingly growing.
This was to catch duplicate emissions for PropagateDirectory but we
don't emit this signal anymore from there.
This fixes a warning about PropagatorJob not being a registered metatype.
This reverts commit fe42c1a818.
The current logic tried to avoid a DB lookup just to fetch whether
the file is shared or not since that info is already in the
SyncFileItem. The implementation would however need to decrease the
sync count for itself (and parents) before emitting the new status,
thus emitting the OK status for parents before that last child that
ended the propagation for that folder.
Change the implementation to achieve what we want: give the
possibility to decSyncCount to use a pre-fetched sharing state while
still doing the emission for all involved files. This ensures that
the leaf file also gets its status emitted before its parents.
Issue #4797
IsMemberOf is called for every file (in the ownCloud directory or not) and
with every instance of OCOverlay (we have 5) when displaying a list of
files in Explorer.
Refactor the code to avoid copying the list of watched directories, as
well as creating a wstring from a PWWSTR for files outside watched
directories.
Also change some calls of begin_with to use isDescendantOf since it
properly handles parent paths not ending with a backslash, which could
lead to SocketAPI queries for sibling folders with a name that starts with
a watched folder name.
Since StateError == 0, if this was the status used when the path
isn't in the map already, the view would not be updated since the
new state would be the same as the default-constructed state in the
map. Fix by explicitly inserting in that case, this also avoid aving
to do two lookups in the map when a path already has an entry.
The client now tries to only push STATUS messages to connections
unless they previously requested it with a RETRIEVE_FILE_STATUS,
but this means that we now have to make sure that a new connection
will rerequest every icon that the user will see on files from that
point on.
To that end, we now send a SHChangeNotify for every file in the cache
when we lose the connection (or on UNREGISTER_PATH) to make sure that
what's on the screen matches what's in the cache. We also remove the
_oldCache logic that made this more difficult to enforce.
The client has been able to reliably push status updates on macOS
for a few versions now, and we don't need it on Windows either. The
_oldCache mechanism was to avoid sending to many update requests
when receiving an UPDATE_VIEW.
Also fixes#4766
The SocketAPI now only sends status pushes for paths requested by the
shell. We have to make sure that Finder doesn't just show the entry
from its cache after a reconnect.
Tell cmake not to escape shell strings adding backslashes before
spaces by passing the VERBATIM optiton. Also add the quotes to
whole command line arguments to prevent quotes from appearing on the
XCode side, because of the VERBATIM option.
This only affects themed clients since the default theme doesn't
have space in its APPLICATION_NAME.
We currently push the SYNC status for all files that will be propagated,
and then the OK status when those files are propagated.
On top of this, we send those statuses to all clients connected, even
if the socket is kept open by an application that only needed to show
a file open dialog. On macOS we're also using an NSConnection which
means that we have to wait for the RPC call to return from the
extension, which makes bulk status changes possibly heavy.
Reduce the time spent needlessly sending status pushes by limiting
them to files requested through that socket since it connected.
To limit the data to store, only remember the parent directory of
files requested, and store those in a bloom filter.
Note that this adds a requirement to shell extensions: they should
make sure that the status cache only contains entries that have been
requested through the socket API. In other words, the status cache
must be empty when each socket client connects to the socket API.
Otherwise the cached icon type will be shown to the user, and the
SocketAPI won't push new status for that file if it didn't receive
a RETRIEVE_FILE_STATUS.
- Use the looked-up method index also for the invocation
- Do the method name concatenation already on QByteArray since we'll
convert anyway
- Use staticMetaObject instead of metaObject()
The csync log level was only set up on startup, and for log files.
Fix the issue by making Logger::isNoop rely on being explicitly activated
for the log window instead of relying on the presence of a connected
signal, and move the csync log level logic in Logger.
The compiler seems to use signed enums and we need to reserve an extra
bit for the sign to avoid the 2 value to overflow and being interpreted
as -2 when read, and thus not being correctly compared to the full enum
value.
Qt now only build it if the system library is available since libmng
isn't active enough to make its security reliable enough. We don't
use that image format ourselves anyway.
Also get rid of the list of different version. We always only support one Qt version
for macOS and Windows, and the list of patches will be based on that supported
version.
- Replace functions that are provided by MinGW with a Win32-based
implementation
- Explicitly export needed symbols from ocsync.dll
- Rename share.h to sharemanager.h since the name clashes with one
of the Windows headers and get included from there
- Remove the timestamp from the fallback csync stderr logging, it's
not used since we always provide a log callback
This would happen if the directory would first need to be created
through an mkdir propagation job. This job's itemCompleted signal
would trigger the directory to show as SYNC even though its children
are still propagating.
Fix the issue by tracking the sync count for each file, affecting
its parents. This allows us to get rid of the O(n) vector lookup
for each status query, and properly track the hierachical sync
status of a directory.
This also removes the itemCompleted signal emission from the
PropagateDirectory job. Since we only needed for overlay icons, and
since this job doesn't do any direct propagation, we can remove it
to ensure that we won't call itemCompleted twice for the item attached
to Propagate*Mkdir jobs (since the PropagateDirectory is backed by
the same SyncFileItem, instruction and status).
The current way of tracking the need to update the metadata without
propagation using a separate flag makes it difficult to track
priorities between the local and remote tree. The logic is also
difficult to logically cover since the possibilities matrix isn't
100% covered, leaving the flag only used in a few situations
(mostly involving folders, but not only).
The reason we need to change this is to be able to track the sync
state of files for overlay icons. The instruction alone can't be
used since CSYNC_INSTRUCTION_SYNC is used for folders even though
they won't be propagated. Removing this logic is however not possible
without using something else than CSYNC_INSTRUCTION_NONE since too
many codepath interpret (rightfully) this as meaning "nothing to do".
This patch adds a new CSYNC_INSTRUCTION_UPDATE_METADATA instruction
to let the update and reconcile steps tell the SyncEngine to update
the metadata of a file without any propagation. Other flags are left
to be interpretted by the implementation as implicitly needing
metadata update or not, as this was already the case for most file
propagation jobs. For example, CSYNC_INSTRUCTION_NEW for directories
now also implicitly update the metadata.
Since it's not impossible for folders to emit CSYNC_INSTRUCTION_SYNC
or CSYNC_INSTRUCTION_CONFLICT, the corresponding code paths in the
sync engine have been removed.
Since the reconcile step can now know if the local tree needs metadata
update while the remote side might want propagation, the
localMetadataUpdate logic in SyncEngine::treewalkFile now simply use
a CSYNC_INSTRUCTION_UPDATE_METADATA for the local side, which is now
implemented as a different database query.
Add a missing call that we currently only do in slotItemCompleted.
This would normally only affect the first sync and would have
gotten properly update at the end of the sync anyway.
To be able to test the SyncEngine efficiently, a set of server
mocking classes have been implemented on top of QNetworkAccessManager.
The local disk side hasn't been mocked since this would require adding
a large abstraction layer in csync. The SyncEngine is instead pointed
to a different temporary dir in each test and we test by interacting
with files in this directory instead.
The FakeFolder object wraps the SyncEngine with those abstractions
and allow controlling the local files, and the fake remote state
through the FileModifier interface, using a FileInfo tree structure
for the remote-side implementation as well as feeding and comparing
the states on both side in tests.
Tests run fast and require no setup to be run, but each server feature
that we want to test on the client side needs to be implemented in
this fake objects library. For example, the OC-FileId header isn't
set as of this commit, and we can't test the file move logic properly
without implementing it first.
The TestSyncFileStatusTracker tests already contain a few QEXPECT_FAIL
for what I esteem being issues that need to be fixed in order to catch
up on our test coverage without making this patch too huge.
The open function expects a URL, passing only the directory name would
lead HTTP::DAV to try looking it as an hostname on the network and
only return after it timed out.
Reverts commit 622017adcf
Could be the cause of #5092 and the cost is higher than the benefit if this is the case.
A network request taking more than 30 seconds isn't something unlikely in this world
and shouldn't be a good reason to abort. We should try to untangle the threads
dependencies to properly fix this if possible instead.
Since the SyncEngine now quits and waits for the discovery thread,
the main thread can enter a deadlock where the discovery thread waits
for its directory result.
Add a 2 seconds timer to the discovery thread wait condition
to limit the deadlock time.
This allows creating a new build simply by hitting F7 in visual studio
and then committing the changed files in the binary submodules without
having to know which files to move where.
Since this isn't yet built together with the rest, we can't reliably
use the client version without having to rebuilt the extension manually
on each release. But it would still be nice to be able to tell if a user
has the right version or not loaded, so instead just tag each update of
the DLLs with a separate release version.
This means that every time we update those binaries we'll have to manually
increment the number in the header, but worst case we'll forget from time
to time but we'll be able to know if we didn't, and know if we can rely on
the number or not.
Use a QMap to avoid using a full hashtable for only a few entries, and
clear the QMap once we're done with the measuring. This saves a few
hundred bytes per job during propagation that would otherwise only be
freed at the end of the sync.
During propagation, we create a line for each file, taking memory, but
we delete all lines passed 2000 right at the beginning of the next sync.
Since the user has little chances of being able to read past those 2000
lines in the log, we might as well keep it capped at 2000 also during
propagation to prevent it from eating memory.
The SyncRunFileLog owned by the Folder must be destroyed after the
SyncEngine since the SyncEngine will abort during destruction, resulting
in all jobs being aborted.
It's possible that this crash only happens with a debug build.
The FolderWatcher inserts files to be marked as SYNC and we
currently assume that all file statuses will be updated by the
following sync. It's however possible that the FolderWatcher
notify us of a change that csync won't consider necessary to
propagate, in which case a new status wouldn't be pushed and
the file manager would continue showing this file as syncing.
Re-push the file status when emptying the dirty files list
before propagating to avoid this issue, most likely the OK
status.
Events from the crash reporter suggest that the QNAM and its
child replies might get deleted before returning from this method
and the only possible cause we can see is that the inner event
loop has something to do with it.
Try keeping a ref on the QNAM while in this method to make sure
that it won't get deleted by the inner event loop.
The assert was there to make sure that this case wasn't happening
to eventually be properly tested. Remove the assert for now but this
codepath should eventually be unit tested using this specific situation.
Since the windows implementation first does cache lookups using the
path string, directories need to be passed identically as through
RETRIEVE_FILE_STATUS.
Change the convention to never have a trailing slash for directories
in the protocol. This allows the convention to be applied without
having to access the disk (since we'd need to know if the path is
represented by a directory) and also matches the convention of the
rest of the sync engine. Individual file manager plugins are then
responsible of handling pushed paths as not ending with a trailing
slash.
This also:
- Moves the trailing slash removal logic from the SyncFileStatusTracker
to the SocketApi class
- Remove the unneeded QString::normalized call in fileStatus, since
this should already be done by the FolderWatcher and plugins
Since the statuses are cached and that we can't invalidate the cache,
sending NOP would need to be overwritten by the default OK status
once the client successfully connected. But instead of remembering
which files we NOPed, rather wait until we are ready to sync before
sending the REGISTER_PATH message to the socket API client. It will
also prevent the client from sending unnecessary RETRIEVE_FILE_STATUS
requests.
Also remove AccountState::canSync, since it does the same as
isConnected and syncing is not an account responsibility.
Go through fileStatus like other cases to make sure that all use
cases go through the same code path. This also makes sure to use
lookupProblem which will use lower_bound which is more efficient
for larger sets of sync problems.
This also fixes the issue with lookupProblem that prevented it to
properly match an empty pathToMatch, caused by the fact that the
problem map contains relative paths not starting with a slash.
Make sure that we push the new status when the status of the SyncEngine
changed. SyncEngine::started comes a bit late, only when the propagation
starts, although it's better in this case since child folders will
only switch to Sync in aboutToPropagate.
Also fix an issue with SyncEngine::findSyncItem when using an empty
fileName; this would match and return the wrong item, even though
not currently happening with the code since fileStatus won't call
it with an empty fileName anymore.
As before, we rely on metadata-update SyncFileItem entries for parent
directories to notify us that a directory contains files to propagate,
and to know when all children were propagated through its itemCompleted
signal.
Those metadata SyncFileItems however have a None direction and we need
to add a explicit directory check to show them as Sync.
This fix also handles new files as well as existing ones, so no need
to keep a separate logic for new files.
The pull request should have been done against the 2.2 branch, and
the branch would better be merged instead of squashed to keep the
history easier to lookup.
This reverts commit 0469236d80.
* Fix the sync status push for parent directories #4682
As before, we rely on metadata-update SyncFileItem entries for parent
directories to notify us that a directory contains files to propagate,
and to know when all children were propagated through its itemCompleted
signal.
Those metadata SyncFileItems however have a None direction and we need
to add a explicit directory check to show them as Sync.
This fix also handles new files as well as existing ones, so no need
to keep a separate logic for new files.
* Fix the root item sync status #4682
Make sure that we push the new status when the status of the SyncEngine
changed. SyncEngine::started comes a bit late, only when the propagation
starts, although it's better in this case since child folders will
only switch to Sync in aboutToPropagate.
Also fix an issue with SyncEngine::findSyncItem when using an empty
fileName; this would match and return the wrong item, even though
not currently happening with the code since fileStatus won't call
it with an empty fileName anymore.
* Simplify the root status logic
Go through fileStatus like other cases to make sure that all use
cases go through the same code path. This also makes sure to use
lookupProblem which will use lower_bound which is more efficient
for larger sets of sync problems.
This also fixes the issue with lookupProblem that prevented it to
properly match an empty pathToMatch, caused by the fact that the
problem map contains relative paths not starting with a slash.
* Avoid a SyncFileStatusTracker private overload with the same name
Having an overload as a private function in the same class makes the
code harder to follow. Rename the private fileStatus to syncFileItemStatus.
* [osx] Fix missing overlay icons on client startup
Since the statuses are cached and that we can't invalidate the cache,
sending NOP would need to be overwritten by the default OK status
once the client successfully connected. But instead of remembering
which files we NOPed, rather wait until we are ready to sync before
sending the REGISTER_PATH message to the socket API client. It will
also prevent the client from sending unnecessary RETRIEVE_FILE_STATUS
requests.
Also remove AccountState::canSync, since it does the same as
isConnected and syncing is not an account responsibility.
Remove all configure_files:
- Move all tests to cpp files
- Use the QTEST_MAIN macro instead of a generated main.cpp
- Include test*.moc in the cpp to let CMAKE_AUTOMOC call moc
- Pass info through add_definitions instead of generating oc_bin.h with them
This makes sure that build errors points to the original test source
file instead of the generated one in the build directory to be able to
jump and fix errors directly from the IDE's error pane.
Bring back the hardcoded status logic for excluded files.
Since the activity log doesn't even mention those files on purpose,
we can't rely on the SyncEngine to notify us about the status of those files.
Looking up a/aa while an error is present in a/aab/aaba would return
a warning status since a/aa is a substring of a/aab.
Fix the issue by checking if the following character is a slash.
This prevents having to define a Problem structure with dubious
operator overloads to accomplish the same.
Also use std::map::lower_bound to quickly iterate over the
list of problems.
This also remove all smartness from the SocketApi about the status
of a file and solely use info from the current and last sync.
This simplifies the logic a lot and prevents any discrepancy between
the status shown in the activity log and the one displayed on the
overlay icon of a file.
The main benefit of the additional simplicity is that we are able
to push all new status of a file reliably (including warnings for
parent folders) to properly update the icon on overlay implementations
that don't allow us invalidating the status cache, like on OS X.
Both errors and warning from the last sync are now kept in a set,
which is used to also affect parent folders of an error.
To make sure that errors don't become warning icons on a second
sync, SyncFileItem::_hasBlacklistEntry is also interpreted as an error.
This also renames StatusIgnore to StatusWarning to match this semantic.
SyncEngine::aboutToPropagate is used in favor of SyncEngine::syncItemDiscovered
since the latter is emitted before file permission warnings are set on the
SyncFileItem. SyncEngine::finished is not used since we have all the
needed information in SyncEngine::itemCompleted.
SyncFileStatus' purpose is to track overlay icon status.
Instead of putting comments and default: clauses in switch
on both sides about unused enums, use different enums.
This also remove STATUS_NEW which is the equivalent of
STATUS_SYNC in all shell extension implementations, and
remove STATUS_UPDATED and STATUS_STAT_ERROR which have
the same semantic as STATUS_UPTODATE and STATUS__ERROR.
This currently is no-op code since the socket API isn't notified
that the tainted folder list changed, and the result is the same
since a sync will be triggered within the next 5 seconds and the
modified folder will be shown as SYNC at that point anyway.
Removing the dependency to the file watcher allows moving the
status estimation logic to libsync.
Jocelyn Turcotte