The current implementation would return the same value whether the query failed
or if no row would be found. This is something that is currently checked by csync
and needs to be provided if we want to use SyncJournalDB there.
Adjusted all call sites to also check the return value even though they
could still just rely on rec.isValid(), but makes it more explicit as to what
happens for database errors in those cases, if we ever want to gracefully handle
them.
Create a specific type that parses the permissions so we can store
it in a short rather than in a QByteArray
Note: in RemotePermissions::toString, we make sure the string is not
empty by adding a space, this was already existing before commit
e8f7adc7ca where it was removed by mistake.
This is motivated by the fact that QMetaObject::noralizeSignature takes 7.35%
CPU of the LargeSyncBench. (Mostly from ABstractNetworkJob::setupConnections and
PropagateUploadFileV1::startNextChunk). It could be fixed by using normalized
signature in the connection statement, but i tought it was a good oportunity
to modernize the code.
This commit only contains calls that were automatically converted with clazy.
This remove the remaining "other" fields of the sync log to save a
bit of memory.
other_etag and other_fileId don't give much information to the users
and other_instruction will always be INST_NONE anyway.
other_modtime and other_size are kept since they are sometimes used.
They were renamed to have a bit more meaningful name.
SyncEngine::checkPermissions will now fetch its information from the
csync trees since they are now preserved until right after this point.
Fixes#3213
Now that csync builds as C++, this will avoid having to implement
functionalities needed by csync mandatorily in csync itself.
This library is built as part of libocsync and symbols exported
through it.
This requires a relicense of Utility as LGPL. All classes moved into
this library from src/libsync will need to be relicensed as well.
We need Qt 5.9 for HTTP2 because, even if Qt 5.8 already has support
for it, there is some critical bug in the HTTP2 implementation which
make it unusable [ https://codereview.qt-project.org/186050 and
https://codereview.qt-project.org/186066 ]
When using HTTP2, we can use many more parallel network request, this
is especially good for small file handling
Lower the priority of the GET and PUT propagation jobs, so the quota
or selective sync ui PROPFIND will not be blocked by them
For now we use them for:
* csync errors: This allows them to appear in the sync issues tab
* insufficient local disk space, as a summary of individual file errors
Insufficient remote space will use them too, as might other issues that
are bigger than a single sync item.
* For conflicts where mtime and size are identical:
a) If there's no remote checksum, skip (unchanged)
b) If there's a remote checksum that's a useful hash, create a
PropagateDownload job and compute the local hash. If the hashes
are identical, don't download the file and just update metadata.
* Avoid exposing the existence of checksumTypeId beyond the database
layer. This makes handling checksums easier in general because they
can usually be treated as a single blob.
This change was prompted by the difficulty of producing file_stat_t
entries uniformly from PROPFINDs and the database.
See owncloud/enterprise#1966
If the server and the client's database go out of sync, there could be
persistent 404 errors. This change ensures that the problem corrects
itself eventually by triggering a remote discovery of the file's
parent folders.
It does not address the root cause that might have lead to the
divergence.
Use qCInfo for anything that has general value for support and
development. Use qCWarning for any recoverable error and qCCritical
for anything that could result in data loss or would identify a serious
issue with the code.
Issue #5647
This gives more insight about the logs and allow setting fine-tuned
logging rules. The categories are set to only output Info by default
so this allows us to provide more concise logging while keeping the
ability to extract more information for a specific category when
developping or debugging customer issues.
Issue #5647
By default QNetworkReply::errorString() often produces messages like
"Error downloading <url> - server replied: <reason>"
but the "downloading" part invariably confuses people since the
error might very well have been produced by a PUT request.
This commit produces clearer error messages for HTTP errors.
Additionally:
* Remove some unnecessary null checks from slots connected to
network job signals and document that these signals never send
null replies.
* There was a bug where AbstractNetworkJob::_timedout wasn't
set when derived classes overrode slotTimeout. We now ensure
it's always set by disallowing overrides of slotTimeout.
Instead it now calls onTimedOut, which allows custom handling.
* Several subclasses declared errorString, isTimedOut. Move
these to AbstractNetworkJob.
* Unify handling of OC-ErrorString (via the new, general
Job::errorString)
* Add documentation in various places.
Backtrace looks like this:
Crash: EXCEPTION_ACCESS_VIOLATION_READ at 0x0
File "propagatedownload.cpp", line 234, in OCC::GETFileJob::slotReadyRead
File "moc_propagatedownl_CA5CFSHZDTX34X.cpp", line 86, in OCC::GETFileJob::qt_static_metacall
File "qobject.cpp", line 495, in QMetaCallEvent::placeMetaCall
File "qobject.cpp", line 1256, in QObject::event
File "qapplication.cpp", line 3804, in QApplicationPrivate::notify_helper
GETFileJob::slotReadyRead can be called with a QueuedConnection when the
bendwith manager is involved. In that case, if the QNAM was reset
in between, the reply might have been destroyed.
(This is only speculation based on the backtrace)
* For requests:
- reuse the original QNetworkRequest, so headers and attributes
are the same as in the original request
- determine the original http method from the reply and the request
attributes
- keep the original request body around such that it can be sent
again in case the request is redirected
* Simplify the interface that is used for creating new requests in
AbstractNetworkJob.
Avoid using connections to report up the job tree for signals
that we can directly communicate to the OwncloudPropagator.
This slightly reduces the memory usage and avoid passing those calls
through the whole parent chain.
This could make sure that the network job gets deleted if the parent job gets
deleted, and would avoid crashes like:
Crash: EXCEPTION_ACCESS_VIOLATION_READ at 0xffffffff8b008a04
File "qiodevice.cpp", line 1617, in QIODevice::errorString
File "propagatedownload.cpp", line 264, in OCC::GETFileJob::slotReadyRead
File "moc_propagatedownload.cpp", line 85, in OCC::GETFileJob::qt_static_metacall
File "qobject.cpp", line 3716, in QMetaObject::activate
File "moc_qiodevice.cpp", line 154, in QIODevice::readyRead
File "qnetworkreplyhttpimpl.cpp", line 1045, in QNetworkReplyHttpImplPrivate::replyDownloadData
(#5329)
- Replace functions that are provided by MinGW with a Win32-based
implementation
- Explicitly export needed symbols from ocsync.dll
- Rename share.h to sharemanager.h since the name clashes with one
of the Windows headers and get included from there
- Remove the timestamp from the fallback csync stderr logging, it's
not used since we always provide a log callback
This is a move away from the original policy where jobs
would only follow redirects in special cases.
Two restrictions are in place:
1. We do not allow protocol downgrades (https -> http)
2. We stop redirects after we find them looping (e.g. old = new url, or
indirectly when looping 10 times).
This is closer to RFC conforming behavior, although currently
we will treat 301 replies like they were 302. This is for a separate
commit.
Error handling (and display) also needs improvement.
Addresses #2791
When a conflict-rename or a temporary-rename fails, notify the
LockWatcher. It'll regularly check whether the file has become
accesible again. When it has, another sync is triggered.
owncloud/enterprise#1288
If the downloaded file is empty but the PROPFIND previously announced it
should not have been empty, this might mean the file was somehow corrupted
because of a bug on the server and that we should therefore not accept
the file.
Normaly we accept a change between the actual size of the file and what we
got during discovery because the file might have been updated to a new version
inbetween. But after this patch we won't accept the file if it was replaced
by an empty file.
Will help for issue #4583
Also requested by IL for issue 548
Added in previous commit from pull request #4663
As discussed, we do not need this option so no need to introduce
a new dependency on the config file in the sync engine
* Add checksums/supportedTypes and checksums/preferredUploadType
capabilities. The default is that no checksum types are supported.
* Remove the transmissionChecksum config option. Servers must now
use the capabilities to indicate that they are fine with the
client sending checksums.
Note: This intentionally breaks brandings that overrode
Theme::transmissionChecksum. The override must be removed and the
server's capabilities must be adjusted to include the new values.
Helps with small file sync #331
When I benchmarked this, it went up to 6 parallelism and
was about 1/3 faster than the previous fixed 3 parallelism.
Doing more than 6 is dangerous because QNAM limits to 6 TCP
connections and also the server might become a bottleneck.
Should also help for #4081
* Ensure every time a file becomes a directory or the other way around
the item is flagged as INSTRUCTION_TYPE_CHANGE.
* Delete the badly-typed entity if necessary in the propagation jobs.
* Compute the content checksum (in addition to the optional
transmission checksum) during upload (.eml files only)
* Add hook to compute and compare the checksum in csync_update
* Add content checksum to database, remove transmission checksum
Use a QSharedPointer to keep the same ownership and
continue passing the SyncFileItems as a const& when
ownership isn't taken. This allows sharing the same
allocations between the jobs and the result vectors.
This saves about 20MB of memory (off 120MB) once all
jobs are created.
Because file system like FAT only have two second accuracy and would result
in a upload if the mtime in the database is not the same as the one that was
downloaded
Issue #3103
QFileInfo has to be refreshed if the underlying file has been
modified in between. That is dangerous so ckamm and me decided
to eliminate the QFileInfo based implementations.
This was triggered by a bug that the client uploaded files that
it should not have.
There could be a race condition if the file was updated on the server
between the discovery and the propagate phase. By taking the mtime from
the server, we make sure that we do not have a race.
This is tested by t6.pl with BIG3.file because the script was modifying
the file between the two phases
* Use a shared pointer to Account everywhere to ensure
the instance stays alive long enough for a sync to terminate
* Folder is now tied to an AccountState
* SyncEngine and OwncloudPropagator tie to an Account and use that
for all jobs they run
Issue: Since the setup wizard currently always replaces the
account, it will always wipe all folder definitions, even when
the actual changes to the account were minor.
The server might support resuming, so don't always erase the temporary file
and pass the startSize, so the temporary file will be remove if the server
does not support it after all (because it is not sending the "bytes" header
Also pass the expected etag for consistency even if it's not used in this case.
Jocelyn Turcotte