Welcome! Log In Create A New Profile

Advanced

fsync()-in webdav PUT

Posted by Nagy, Attila 
Nagy, Attila
fsync()-in webdav PUT
February 27, 2018 11:30AM
Hi,

I would like to make sure when a WebDAV (with ngx_http_dav) PUT returns
the file is reliably on the storage. This needs an fsync() on the file.
It would be easy to put that into the module, but it would block the
whole nginx process.

Now, that nginx supports running threads, are there plans to convert at
least DAV PUTs into it's own thread(pool), so make it possible to do
non-blocking (from nginx's event loop PoV) fsync on the uploaded file?

Thanks,

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Maxim Dounin
Re: fsync()-in webdav PUT
February 27, 2018 02:30PM
Hello!

On Tue, Feb 27, 2018 at 11:21:23AM +0100, Nagy, Attila wrote:

> I would like to make sure when a WebDAV (with ngx_http_dav) PUT returns
> the file is reliably on the storage. This needs an fsync() on the file.
> It would be easy to put that into the module, but it would block the
> whole nginx process.
>
> Now, that nginx supports running threads, are there plans to convert at
> least DAV PUTs into it's own thread(pool), so make it possible to do
> non-blocking (from nginx's event loop PoV) fsync on the uploaded file?

No, there are no such plans.

(Also, trying to do fsync() might not be the best idea even in
threads. A reliable server might be a better option.)

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Nagy, Attila
Re: fsync()-in webdav PUT
February 28, 2018 10:40AM
On 02/27/2018 02:24 PM, Maxim Dounin wrote:
>
>> Now, that nginx supports running threads, are there plans to convert at
>> least DAV PUTs into it's own thread(pool), so make it possible to do
>> non-blocking (from nginx's event loop PoV) fsync on the uploaded file?
> No, there are no such plans.
>
> (Also, trying to do fsync() might not be the best idea even in
> threads. A reliable server might be a better option.)
>
What do you mean by a reliable server?
I want to make sure when the HTTP operation returns, the file is on the
disk, not just in a buffer waiting for an indefinite amount of time to
be flushed.
This is what fsync is for.

Why doing this in a thread is not a good idea? It would'nt block nginx
that way.
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Aziz Rozyev
Re: fsync()-in webdav PUT
February 28, 2018 11:10AM
While it’s not clear why one may need to flush the data on each http operation,
I can imagine to what performance degradation that may lead of.

if it’s not a some kind of funny clustering among nodes, I wouldn't care much
where actual data is, RAM still should be much faster, than disk I/O.


br,
Aziz.





> On 28 Feb 2018, at 12:30, Nagy, Attila <[email protected]> wrote:
>
> On 02/27/2018 02:24 PM, Maxim Dounin wrote:
>>
>>> Now, that nginx supports running threads, are there plans to convert at
>>> least DAV PUTs into it's own thread(pool), so make it possible to do
>>> non-blocking (from nginx's event loop PoV) fsync on the uploaded file?
>> No, there are no such plans.
>>
>> (Also, trying to do fsync() might not be the best idea even in
>> threads. A reliable server might be a better option.)
>>
> What do you mean by a reliable server?
> I want to make sure when the HTTP operation returns, the file is on the disk, not just in a buffer waiting for an indefinite amount of time to be flushed.
> This is what fsync is for.
>
> Why doing this in a thread is not a good idea? It would'nt block nginx that way.
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Maxim Dounin
Re: fsync()-in webdav PUT
February 28, 2018 03:10PM
Hello!

On Wed, Feb 28, 2018 at 10:30:08AM +0100, Nagy, Attila wrote:

> On 02/27/2018 02:24 PM, Maxim Dounin wrote:
> >
> >> Now, that nginx supports running threads, are there plans to convert at
> >> least DAV PUTs into it's own thread(pool), so make it possible to do
> >> non-blocking (from nginx's event loop PoV) fsync on the uploaded file?
> > No, there are no such plans.
> >
> > (Also, trying to do fsync() might not be the best idea even in
> > threads. A reliable server might be a better option.)
> >
> What do you mean by a reliable server?
> I want to make sure when the HTTP operation returns, the file is on the
> disk, not just in a buffer waiting for an indefinite amount of time to
> be flushed.
> This is what fsync is for.

The question here is - why you want the file to be on disk, and
not just in a buffer? Because you expect the server to die in a
few seconds without flushing the file to disk? How probable it
is, compared to the probability of the disk to die? A more
reliable server can make this probability negligible, hence the
suggestion.

(Also, another question is what "on the disk" meas from physical
point of view. In many cases this in fact means "somewhere in the
disk buffers", and a power outage can easily result in the file
being not accessible even after fsync().)

> Why doing this in a thread is not a good idea? It would'nt block nginx
> that way.

Because even in threads, fsync() is likely to cause performance
degradation. It might be a better idea to let the OS manage
buffers instead.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Valery Kholodkov
Re: fsync()-in webdav PUT
February 28, 2018 07:30PM
It's completely clear why someone would need to flush file's data and
metadata upon a WebDAV PUT operation. That is because many architectures
expect a PUT operation to be completely settled before a reply is returned.

Without fsyncing file's data and metadata a client will receive a
positive reply before data has reached the storage, thus leaving
non-zero probability that states of two systems involved into a web
transaction end up inconsistent.

Further, the exact moment when the data of certain specific file reaches
the storage depends on numerous factors, for example, I/O contention.
Consequently, the exact moment when the data of a file being uploaded
reaches the storage can be only determined by executing fsync.

val

On 28-02-18 11:04, Aziz Rozyev wrote:
> While it’s not clear why one may need to flush the data on each http operation,
> I can imagine to what performance degradation that may lead of.
>
> if it’s not a some kind of funny clustering among nodes, I wouldn't care much
> where actual data is, RAM still should be much faster, than disk I/O.
>
>
> br,
> Aziz.
>
>
>
>
>
>> On 28 Feb 2018, at 12:30, Nagy, Attila <[email protected]> wrote:
>>
>> On 02/27/2018 02:24 PM, Maxim Dounin wrote:
>>>
>>>> Now, that nginx supports running threads, are there plans to convert at
>>>> least DAV PUTs into it's own thread(pool), so make it possible to do
>>>> non-blocking (from nginx's event loop PoV) fsync on the uploaded file?
>>> No, there are no such plans.
>>>
>>> (Also, trying to do fsync() might not be the best idea even in
>>> threads. A reliable server might be a better option.)
>>>
>> What do you mean by a reliable server?
>> I want to make sure when the HTTP operation returns, the file is on the disk, not just in a buffer waiting for an indefinite amount of time to be flushed.
>> This is what fsync is for.
>>
>> Why doing this in a thread is not a good idea? It would'nt block nginx that way.

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Valery Kholodkov
Re: fsync()-in webdav PUT
February 28, 2018 08:00PM
On 28-02-18 15:08, Maxim Dounin wrote:
>> What do you mean by a reliable server?
>> I want to make sure when the HTTP operation returns, the file is on the
>> disk, not just in a buffer waiting for an indefinite amount of time to
>> be flushed.
>> This is what fsync is for.
>
> The question here is - why you want the file to be on disk, and
> not just in a buffer? Because you expect the server to die in a
> few seconds without flushing the file to disk? How probable it
> is, compared to the probability of the disk to die? A more
> reliable server can make this probability negligible, hence the
> suggestion.

I think the point here is that lack of fsync leaves some questions
unanswered. Adding fsync will simply put all dots above the "i"s.

> (Also, another question is what "on the disk" meas from physical
> point of view. In many cases this in fact means "somewhere in the
> disk buffers", and a power outage can easily result in the file
> being not accessible even after fsync().)
>
>> Why doing this in a thread is not a good idea? It would'nt block nginx
>> that way.
>
> Because even in threads, fsync() is likely to cause performance
> degradation. It might be a better idea to let the OS manage
> buffers instead.

fsync does not cause performance degradation. fsync simply instructs OS
to ensure consistency of a file. What causes performance degradation is
expenditure of resources necessary to ensure consistency.

val
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
itpp2012
Re: fsync()-in webdav PUT
February 28, 2018 09:30PM
Not waiting for fsync to complete makes calling fsync pointless, waiting for
fsync is blocking, thread based or otherwise.
The only midway solution is to implement fsync as a cgi, ea. a none-blocking
(background)fc call in combination with an OS resource lock.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,278788,278847#msg-278847

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Aziz Rozyev
Re: fsync()-in webdav PUT
February 28, 2018 10:50PM
Valery,

may you please suggest how you came to the conclusion that

“fsync simply instructs OS to ensure consistency of a file"?

As far as understand simply instructing OS staff come at no cost, right?

> Without fsyncing file's data and metadata a client will receive a positive reply before data has reached the storage, thus leaving non-zero probability that states of two systems involved into a web transaction end up inconsistent.


I understand why one may need consistency, but doing so with fsyncing is non-sense.

Here is what man page says in that regard:


fsync() transfers ("flushes") all modified in-core data of (i.e., modified buffer cache pages for) the file referred to by the file descriptor fd to the disk device (or other permanent
storage device) so that all changed information can be retrieved even after the system crashed or was rebooted. This includes writing through or flushing a disk cache if present. The call
blocks until the device reports that the transfer has completed. It also flushes metadata information associated with the file (see stat(2)).




br,
Aziz.





> On 28 Feb 2018, at 21:24, Valery Kholodkov <[email protected]> wrote:
>
> It's completely clear why someone would need to flush file's data and metadata upon a WebDAV PUT operation. That is because many architectures expect a PUT operation to be completely settled before a reply is returned.
>
> Without fsyncing file's data and metadata a client will receive a positive reply before data has reached the storage, thus leaving non-zero probability that states of two systems involved into a web transaction end up inconsistent.
>
> Further, the exact moment when the data of certain specific file reaches the storage depends on numerous factors, for example, I/O contention. Consequently, the exact moment when the data of a file being uploaded reaches the storage can be only determined by executing fsync.
>
> val
>
> On 28-02-18 11:04, Aziz Rozyev wrote:
>> While it’s not clear why one may need to flush the data on each http operation,
>> I can imagine to what performance degradation that may lead of.
>> if it’s not a some kind of funny clustering among nodes, I wouldn't care much
>> where actual data is, RAM still should be much faster, than disk I/O.
>> br,
>> Aziz.
>>> On 28 Feb 2018, at 12:30, Nagy, Attila <[email protected]> wrote:
>>>
>>> On 02/27/2018 02:24 PM, Maxim Dounin wrote:
>>>>
>>>>> Now, that nginx supports running threads, are there plans to convert at
>>>>> least DAV PUTs into it's own thread(pool), so make it possible to do
>>>>> non-blocking (from nginx's event loop PoV) fsync on the uploaded file?
>>>> No, there are no such plans.
>>>>
>>>> (Also, trying to do fsync() might not be the best idea even in
>>>> threads. A reliable server might be a better option.)
>>>>
>>> What do you mean by a reliable server?
>>> I want to make sure when the HTTP operation returns, the file is on the disk, not just in a buffer waiting for an indefinite amount of time to be flushed.
>>> This is what fsync is for.
>>>
>>> Why doing this in a thread is not a good idea? It would'nt block nginx that way.
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Aziz Rozyev
Re: fsync()-in webdav PUT
February 28, 2018 11:10PM
here is a synthetic test on vm, not perfect, but representative:


[[email protected] ~]# dd if=/dev/zero of=/writetest bs=8k count=30960
30960+0 records in
30960+0 records out
253624320 bytes (254 MB) copied, 0.834861 s, 304 MB/s

[[email protected] ~]# dd if=/dev/zero of=/writetest bs=8k count=30960 conv=fsync
30960+0 records in
30960+0 records out
253624320 bytes (254 MB) copied, 0.854208 s, 297 MB/s

[[email protected] ~]# dd if=/dev/zero of=/writetest bs=8k count=61960
61960+0 records in
61960+0 records out
507576320 bytes (508 MB) copied, 1.71833 s, 295 MB/s
[[email protected] ~]# dd if=/dev/zero of=/writetest bs=8k count=61960 conv=fsync
61960+0 records in
61960+0 records out
507576320 bytes (508 MB) copied, 1.74482 s, 291 MB/s


br,
Aziz.





> On 1 Mar 2018, at 00:41, Aziz Rozyev <[email protected]> wrote:
>
> Valery,
>
> may you please suggest how you came to the conclusion that
>
> “fsync simply instructs OS to ensure consistency of a file"?
>
> As far as understand simply instructing OS staff come at no cost, right?
>
>> Without fsyncing file's data and metadata a client will receive a positive reply before data has reached the storage, thus leaving non-zero probability that states of two systems involved into a web transaction end up inconsistent.
>
>
> I understand why one may need consistency, but doing so with fsyncing is non-sense.
>
> Here is what man page says in that regard:
>
>
> fsync() transfers ("flushes") all modified in-core data of (i.e., modified buffer cache pages for) the file referred to by the file descriptor fd to the disk device (or other permanent
> storage device) so that all changed information can be retrieved even after the system crashed or was rebooted. This includes writing through or flushing a disk cache if present. The call
> blocks until the device reports that the transfer has completed. It also flushes metadata information associated with the file (see stat(2)).
>
>
>
>
> br,
> Aziz.
>
>
>
>
>
>> On 28 Feb 2018, at 21:24, Valery Kholodkov <[email protected]> wrote:
>>
>> It's completely clear why someone would need to flush file's data and metadata upon a WebDAV PUT operation. That is because many architectures expect a PUT operation to be completely settled before a reply is returned.
>>
>> Without fsyncing file's data and metadata a client will receive a positive reply before data has reached the storage, thus leaving non-zero probability that states of two systems involved into a web transaction end up inconsistent.
>>
>> Further, the exact moment when the data of certain specific file reaches the storage depends on numerous factors, for example, I/O contention. Consequently, the exact moment when the data of a file being uploaded reaches the storage can be only determined by executing fsync.
>>
>> val
>>
>> On 28-02-18 11:04, Aziz Rozyev wrote:
>>> While it’s not clear why one may need to flush the data on each http operation,
>>> I can imagine to what performance degradation that may lead of.
>>> if it’s not a some kind of funny clustering among nodes, I wouldn't care much
>>> where actual data is, RAM still should be much faster, than disk I/O.
>>> br,
>>> Aziz.
>>>> On 28 Feb 2018, at 12:30, Nagy, Attila <[email protected]> wrote:
>>>>
>>>> On 02/27/2018 02:24 PM, Maxim Dounin wrote:
>>>>>
>>>>>> Now, that nginx supports running threads, are there plans to convert at
>>>>>> least DAV PUTs into it's own thread(pool), so make it possible to do
>>>>>> non-blocking (from nginx's event loop PoV) fsync on the uploaded file?
>>>>> No, there are no such plans.
>>>>>
>>>>> (Also, trying to do fsync() might not be the best idea even in
>>>>> threads. A reliable server might be a better option.)
>>>>>
>>>> What do you mean by a reliable server?
>>>> I want to make sure when the HTTP operation returns, the file is on the disk, not just in a buffer waiting for an indefinite amount of time to be flushed.
>>>> This is what fsync is for.
>>>>
>>>> Why doing this in a thread is not a good idea? It would'nt block nginx that way.
>>
>> _______________________________________________
>> nginx mailing list
>> nginx@nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx
>

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Peter Booth
Re: fsync()-in webdav PUT
February 28, 2018 11:40PM
This discussion is interesting, educational, and thought provoking. Web architects
only learn “the right way” by first doing things “the wrong way” and seeing what happens.
Attila and Valery asked questions that sound logical, and I think there's value in exploring
what would happen if their suggestions were implemented.

First caveat - nginx is deployed in all manner different scenarios on different hardware
and operating systems. Physical servers and VMs behave very differently, as do local
and remote storage. When an application writes to NFS mounted storage there's no guarantee
that even and synch will correctly enforce a write barrier. Still, if we consider real numbers:

On current model quad socket hosts, nginx can support well over 1 million requests per second (see TechEmpower benchmarks)
On the same hardware, a web app that writes to a Postgresql DB can do at least a few thousand writes per second.
A SATA drive might support 300 write IOPS, whilst an SSD will support 100x that.
What this means that doing fully synchronous writes can reduce your potential throughput
by a factor of 100 or more. So it’s not a great way to ensure consistency.

But there are cheaper ways to achieve the same consistency and reliability characteristics:

If you are using Linux then your reads and write swill occur through the page cache - so the actual disk itself really doesn’t matter (whilst your host is up).
If you want to protect against loss of physical disk then use RAID.
If you want to protect against a random power failure then use drives with battery backed caches, so writes will get persisted when a server restarts after a power failure
If you want to protect against a crazy person hitting your server with an axe then write to two servers ...
But the bottom line is separation of concerns. Nginx should not use fsync because it isn’t nginx's business.

My two cents,

Peter


> On Feb 28, 2018, at 4:41 PM, Aziz Rozyev <[email protected]> wrote:
>
> Hello!
>
> On Wed, Feb 28, 2018 at 10:30:08AM +0100, Nagy, Attila wrote:
>
>> On 02/27/2018 02:24 PM, Maxim Dounin wrote:
>>>
>>>> Now, that nginx supports running threads, are there plans to convert at
>>>> least DAV PUTs into it's own thread(pool), so make it possible to do
>>>> non-blocking (from nginx's event loop PoV) fsync on the uploaded file?
>>> No, there are no such plans.
>>>
>>> (Also, trying to do fsync() might not be the best idea even in
>>> threads. A reliable server might be a better option.)
>>>
>> What do you mean by a reliable server?
>> I want to make sure when the HTTP operation returns, the file is on the
>> disk, not just in a buffer waiting for an indefinite amount of time to
>> be flushed.
>> This is what fsync is for.
>
> The question here is - why you want the file to be on disk, and
> not just in a buffer? Because you expect the server to die in a
> few seconds without flushing the file to disk? How probable it
> is, compared to the probability of the disk to die? A more
> reliable server can make this probability negligible, hence the
> suggestion.
>
> (Also, another question is what "on the disk" meas from physical
> point of view. In many cases this in fact means "somewhere in the
> disk buffers", and a power outage can easily result in the file
> being not accessible even after fsync().)
>
>> Why doing this in a thread is not a good idea? It would'nt block nginx
>> that way.
>
> Because even in threads, fsync() is likely to cause performance
> degradation. It might be a better idea to let the OS manage
> buffers instead.
>
> --
> Maxim Dounin
> http://mdounin.ru/ http://mdounin.ru/
> _______________________________________________
> nginx mailing list
> nginx@nginx.org <mailto:[email protected]>
> http://mailman.nginx.org/mailman/listinfo/nginx http://mailman.nginx.org/mailman/listinfo/nginx
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Valery Kholodkov
Re: fsync()-in webdav PUT
March 01, 2018 01:30PM
I admire your wise approach to this discussion, as well your technical
expertise! I see the value in people who know the right way, but I see
the value in people who dare to explore and want learning the right way.
Coincidently, at Qubiq Labs we're looking for that kind of kick-ass
Systems and Performance Architect to run and scale our software and
infrastructure.

If you're challenged by the intricates of online marketing industry and
tons of traffic, we'd love to get your application at info@qubiqlabs.com

It's a funded and growing startup with a lots of interesting projects
that you always dreamed of if you're into nginx.

So, make sure to shoot us an email and don't forget to mention "I want
that job" in the subject!

On 28-02-18 23:33, Peter Booth wrote:
> This discussion is interesting, educational, and thought provoking. Web
> architects
> only learn “the right way” by first doing things “the wrong way” and
> seeing what happens.
> Attila and Valery asked questions that sound logical, and I think
> there's value in exploring
> what would happen if their suggestions were implemented.
>
> First caveat - nginx is deployed in all manner different scenarios on
> different hardware
> and operating systems. Physical servers and VMs behave very differently,
> as do local
> and remote storage. When an application writes to NFS mounted storage
> there's no guarantee
> that even and synch will correctly enforce a write barrier. Still, if we
> consider real numbers:
>
> * On current model quad socket hosts, nginx can support well over 1
> million requests per second (see TechEmpower benchmarks)
> * On the same hardware, a web app that writes to a Postgresql DB can
> do at least a few thousand writes per second.
> * A SATA drive might support 300 write IOPS, whilst an SSD will
> support 100x that.
>
> What this means that doing fully synchronous writes can reduce your
> potential throughput
> by a factor of 100 or more. So it’s not a great way to ensure consistency.
>
> But there are cheaper ways to achieve the same consistency and
> reliability characteristics:
>
> * If you are using Linux then your reads and write swill occur through
> the page cache - so the actual disk itself really doesn’t matter
> (whilst your host is up).
> * If you want to protect against loss of physical disk then use RAID.
> * If you want to protect against a random power failure then use
> drives with battery backed caches, so writes will get persisted when
> a server restarts after a power failure
> * If you want to protect against a crazy person hitting your server
> with an axe then write to two servers ...
>
> *But the bottom line is separation of concerns.* Nginx should not use
> fsync because it isn’t nginx's business.
>
> My two cents,
>
> Peter
>
>
>> On Feb 28, 2018, at 4:41 PM, Aziz Rozyev <[email protected]
>> <mailto:[email protected]>> wrote:
>>
>> Hello!
>>
>> On Wed, Feb 28, 2018 at 10:30:08AM +0100, Nagy, Attila wrote:
>>
>>> On 02/27/2018 02:24 PM, Maxim Dounin wrote:
>>>>
>>>>> Now, that nginx supports running threads, are there plans to convert at
>>>>> least DAV PUTs into it's own thread(pool), so make it possible to do
>>>>> non-blocking (from nginx's event loop PoV) fsync on the uploaded file?
>>>> No, there are no such plans.
>>>>
>>>> (Also, trying to do fsync() might not be the best idea even in
>>>> threads. A reliable server might be a better option.)
>>>>
>>> What do you mean by a reliable server?
>>> I want to make sure when the HTTP operation returns, the file is on the
>>> disk, not just in a buffer waiting for an indefinite amount of time to
>>> be flushed.
>>> This is what fsync is for.
>>
>> The question here is - why you want the file to be on disk, and
>> not just in a buffer? Because you expect the server to die in a
>> few seconds without flushing the file to disk? How probable it
>> is, compared to the probability of the disk to die? A more
>> reliable server can make this probability negligible, hence the
>> suggestion.
>>
>> (Also, another question is what "on the disk" meas from physical
>> point of view. In many cases this in fact means "somewhere in the
>> disk buffers", and a power outage can easily result in the file
>> being not accessible even after fsync().)
>>
>>> Why doing this in a thread is not a good idea? It would'nt block nginx
>>> that way.
>>
>> Because even in threads, fsync() is likely to cause performance
>> degradation. It might be a better idea to let the OS manage
>> buffers instead.
>>
>> --
>> Maxim Dounin
>> http://mdounin.ru/
>> _______________________________________________
>> nginx mailing list
>> nginx@nginx.org <mailto:[email protected]>
>> http://mailman.nginx.org/mailman/listinfo/nginx
>
>
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Valery Kholodkov
Re: fsync()-in webdav PUT
March 01, 2018 01:30PM
You can also apply online:

https://angel.co/qubiq-digital-b-v/jobs

That's more 2018-nish.

On 01-03-18 13:24, Valery Kholodkov wrote:
> I admire your wise approach to this discussion, as well your technical
> expertise! I see the value in people who know the right way, but I see
> the value in people who dare to explore and want learning the right way.
> Coincidently, at Qubiq Labs we're looking for that kind of kick-ass
> Systems and Performance Architect to run and scale our software and
> infrastructure.
>
> If you're challenged by the intricates of online marketing industry and
> tons of traffic, we'd love to get your application at info@qubiqlabs.com
>
> It's a funded and growing startup with a lots of interesting projects
> that you always dreamed of if you're into nginx.
>
> So, make sure to shoot us an email and don't forget to mention "I want
> that job" in the subject!
>
> On 28-02-18 23:33, Peter Booth wrote:
>> This discussion is interesting, educational, and thought provoking. Web
>> architects
>> only learn “the right way” by first doing things “the wrong way” and
>> seeing what happens.
>> Attila and Valery asked questions that sound logical, and I think
>> there's value in exploring
>> what would happen if their suggestions were implemented.
>>
>> First caveat - nginx is deployed in all manner different scenarios on
>> different hardware
>> and operating systems. Physical servers and VMs behave very differently,
>> as do local
>> and remote storage. When an application writes to NFS mounted storage
>> there's no guarantee
>> that even and synch will correctly enforce a write barrier. Still, if we
>> consider real numbers:
>>
>> * On current model quad socket hosts, nginx can support well over 1
>> million requests per second (see TechEmpower benchmarks)
>> * On the same hardware, a web app that writes to a Postgresql DB can
>> do at least a few thousand writes per second.
>> * A SATA drive might support 300 write IOPS, whilst an SSD will
>> support 100x that.
>>
>> What this means that doing fully synchronous writes can reduce your
>> potential throughput
>> by a factor of 100 or more. So it’s not a great way to ensure
>> consistency.
>>
>> But there are cheaper ways to achieve the same consistency and
>> reliability characteristics:
>>
>> * If you are using Linux then your reads and write swill occur through
>> the page cache - so the actual disk itself really doesn’t matter
>> (whilst your host is up).
>> * If you want to protect against loss of physical disk then use RAID.
>> * If you want to protect against a random power failure then use
>> drives with battery backed caches, so writes will get persisted when
>> a server restarts after a power failure
>> * If you want to protect against a crazy person hitting your server
>> with an axe then write to two servers ...
>>
>> *But the bottom line is separation of concerns.* Nginx should not use
>> fsync because it isn’t nginx's business.
>>
>> My two cents,
>>
>> Peter
>>
>>
>>> On Feb 28, 2018, at 4:41 PM, Aziz Rozyev <[email protected]
>>> <mailto:[email protected]>> wrote:
>>>
>>> Hello!
>>>
>>> On Wed, Feb 28, 2018 at 10:30:08AM +0100, Nagy, Attila wrote:
>>>
>>>> On 02/27/2018 02:24 PM, Maxim Dounin wrote:
>>>>>
>>>>>> Now, that nginx supports running threads, are there plans to
>>>>>> convert at
>>>>>> least DAV PUTs into it's own thread(pool), so make it possible to do
>>>>>> non-blocking (from nginx's event loop PoV) fsync on the uploaded
>>>>>> file?
>>>>> No, there are no such plans.
>>>>>
>>>>> (Also, trying to do fsync() might not be the best idea even in
>>>>> threads. A reliable server might be a better option.)
>>>>>
>>>> What do you mean by a reliable server?
>>>> I want to make sure when the HTTP operation returns, the file is on the
>>>> disk, not just in a buffer waiting for an indefinite amount of time to
>>>> be flushed.
>>>> This is what fsync is for.
>>>
>>> The question here is - why you want the file to be on disk, and
>>> not just in a buffer? Because you expect the server to die in a
>>> few seconds without flushing the file to disk? How probable it
>>> is, compared to the probability of the disk to die? A more
>>> reliable server can make this probability negligible, hence the
>>> suggestion.
>>>
>>> (Also, another question is what "on the disk" meas from physical
>>> point of view. In many cases this in fact means "somewhere in the
>>> disk buffers", and a power outage can easily result in the file
>>> being not accessible even after fsync().)
>>>
>>>> Why doing this in a thread is not a good idea? It would'nt block nginx
>>>> that way.
>>>
>>> Because even in threads, fsync() is likely to cause performance
>>> degradation. It might be a better idea to let the OS manage
>>> buffers instead.
>>>
>>> --
>>> Maxim Dounin
>>> http://mdounin.ru/
>>> _______________________________________________
>>> nginx mailing list
>>> nginx@nginx.org <mailto:[email protected]>
>>> http://mailman.nginx.org/mailman/listinfo/nginx
>>
>>
>>
>> _______________________________________________
>> nginx mailing list
>> nginx@nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx
>>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Nagy, Attila
Re: fsync()-in webdav PUT
March 02, 2018 10:00AM
On 02/28/2018 11:04 AM, Aziz Rozyev wrote:
> While it’s not clear why one may need to flush the data on each http operation,
> I can imagine to what performance degradation that may lead of.
I store data on HTTP servers in a distributed manner and have a catalog
of where each file is. If I get back a successful HTTP response for a
PUT operation, I want it to be true, so the file must be on stable storage.
If I just write it to a buffer and something happens with the machine
while the data is still in the buffer, I can't trust that response and I
have to make sure the file is there in its entirety from time to time,
which is much much much much more of a performance degradation.
With clever file systems and/or good hardware (battery backed write
cache) it won't cost you much.
Anyways, it's completely irrelevant how fast you can write to RAM. The
task here is to write reliably. And you can make it fast if you want
with software and hardware.

>
> if it’s not a some kind of funny clustering among nodes, I wouldn't care much
> where actual data is, RAM still should be much faster, than disk I/O.
>
Let's turn the question this way: if you write to RAM, you can't make
sure that the file really made it's way to the storage.
Why do you upload files to an HTTP server if you don't care whether they
are there or not?
You could use /dev/null too. It's even more faster...
Or just make your upload_file() function to a dummy "return immediately"
call.
That's even more faster. :)

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Nagy, Attila
Re: fsync()-in webdav PUT
March 02, 2018 10:10AM
On 02/28/2018 11:33 PM, Peter Booth wrote:
> This discussion is interesting, educational, and thought provoking.
>  Web architects
> only learn “the right way” by first doing things “the wrong way” and
> seeing what happens.
> Attila and Valery asked questions that sound logical, and I think
> there's value in exploring
> what would happen if their suggestions were implemented.
>
> First caveat - nginx is deployed in all manner different scenarios on
> different hardware
> and operating systems. Physical servers and VMs behave very
> differently, as do local
> and remote storage. When an application writes to NFS mounted storage
> there's no guarantee
> that even and synch will correctly enforce a write barrier. Still, if
> we consider  real numbers:
>
> * On current model quad socket hosts, nginx can support well over 1
> million requests per second (see TechEmpower benchmarks)
> * On the same hardware, a web app that writes to a Postgresql DB can
> do at least a few thousand writes per second.
> * A SATA drive might support  300 write IOPS, whilst an SSD will
> support 100x that.
>
> What this means that doing fully synchronous writes can reduce your
> potential throughput
> by a factor of 100 or more. So it’s not a great way to ensure consistency.
>
> But there are cheaper ways to achieve the same consistency and
> reliability characteristics:
>
> * If you are using Linux then your reads and write swill occur
> through the page cache - so the actual disk itself really doesn’t
> matter (whilst your host is up).
> * If you want to protect against loss of physical disk then use RAID.
> * If you want to protect against a random power failure then use
> drives with battery backed caches, so writes will get persisted
> when a server restarts after a power failure
>
Sorry, but this point shows that you don't understand the problem. A
BBWC won't save you from random power failure. Because the data is still
in RAM!
BBWC will save you when you do an fsync and the end of the write (and
that fsync will still write RAM, but it will be the controller's RAM
which is protected by battery).
But nginx doesn't do this today. And that's what this discussion is all
about...

> * If you want to protect against a crazy person hitting your server
> with an axe then write to two servers ...
>
And still you won't have it reliably on your disks.


> *But the bottom line is separation of concerns.* Nginx should not use
> fsync because it isn’t nginx's business.
>
Please suggest at least working solution, which is compatible with
nginx's asynchronous architecture and it ensures that a successful HTTP
PUT will mean the data written to a reliable store.

There are several filesystems which can be turned "fsync by default",
but that will fail miserably because nginx does the writes in the same
process in the same thread. That's what could be solved by doing at
least the fsyncs in different threads, so they wouldn't block the main
thread.

BTW, I'm not proposing this to be the default. It should be an optional
setting, so if somebody want to maintain the current situation, they
could do that.
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Nagy, Attila
Re: fsync()-in webdav PUT
March 02, 2018 10:20AM
On 02/28/2018 03:08 PM, Maxim Dounin wrote:
> The question here is - why you want the file to be on disk, and
> not just in a buffer? Because you expect the server to die in a
> few seconds without flushing the file to disk? How probable it
> is, compared to the probability of the disk to die? A more
> reliable server can make this probability negligible, hence the
> suggestion.
Because the files I upload to nginx servers are important to me. Please
step back a little and forget that we are talking about nginx or an HTTP
server.
We have data which we want to write to somewhere.
Check any of the database servers. Would you accept a DB server which
can loose confirmed data or couldn't be configured that way that a
write/insert/update/commit/whatever you use to modify or put data into
it operation is reliably written by the time you receive acknowledgement?
Now try to use this example. I would like to use nginx to store files.
That's what HTTP PUT is for.
Of course I'm not expecting that the server will die every day. But when
that happens, I want to make sure that the confirmed data is there.
Let's take a look at various object storage systems, like ceph. Would
you accept a confirmed write to be lost there? They make a great deal of
work to make that impossible.
Now try to imagine that somebody doesn't need the complexity of -for
example- ceph, but wants to store data with plain HTTP. And you got
there. If you store data, then you want to make sure the data is there.
If you don't, why do you store it anyways?

> (Also, another question is what "on the disk" meas from physical
> point of view. In many cases this in fact means "somewhere in the
> disk buffers", and a power outage can easily result in the file
> being not accessible even after fsync().)
Not with good software/hardware. (and it doesn't really have to be super
good, but average)

>
>> Why doing this in a thread is not a good idea? It would'nt block nginx
>> that way.
> Because even in threads, fsync() is likely to cause performance
> degradation. It might be a better idea to let the OS manage
> buffers instead.
>
Sure, it will cause some (not much BTW in a good configuration). But if
my primary goal is to store files reliably, why should I care?
I can solve that by using SSDs for logs, BBWCs and a lot more thing. But
in the current way, I can't make sure that a HTTP PUT was really
successful or it will be successful in some seconds or it will fail badly.

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Nagy, Attila
Re: fsync()-in webdav PUT
March 02, 2018 11:10AM
On 02/28/2018 10:41 PM, Aziz Rozyev wrote:
>> Without fsyncing file's data and metadata a client will receive a positive reply before data has reached the storage, thus leaving non-zero probability that states of two systems involved into a web transaction end up inconsistent.
>
> I understand why one may need consistency, but doing so with fsyncing is non-sense.
>
> Here is what man page says in that regard:
>
>
> fsync() transfers ("flushes") all modified in-core data of (i.e., modified buffer cache pages for) the file referred to by the file descriptor fd to the disk device (or other permanent
> storage device) so that all changed information can be retrieved even after the system crashed or was rebooted. This includes writing through or flushing a disk cache if present. The call
> blocks until the device reports that the transfer has completed. It also flushes metadata information associated with the file (see stat(2)).
>
>
Could you please elaborate what do you mean by calling this a nonsense?
Also I don't understand why you cited the man page. It clearly says this
is what ensures that when fsync return, the file will be on stable storage.

What else method do you recommend if somebody wants to get an
acknowledgement to the HTTP PUT only after the file is safely stored?
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Aziz Rozyev
Re: fsync()-in webdav PUT
March 02, 2018 11:50AM
Atilla,

man page quote is related to the Valery’s argument that fsync wont affect performance, forget it.

It’s nonsense because you’re trying to solve the reliability problem at the different level,
it has been multiple times suggested here already by maxim and Paul, that it’s better
to invest to the good server/storage infrastructure, instead of fsyncing each PUT.

Regarding the DB server analogy, you’re still not save from the power outages as long as your
transaction isn’t in a transaction log.

If you’re still consent with syncing and ready to sacrifice your time, try mounting a file system
with ‘sync’ option.


br,
Aziz.





> On 2 Mar 2018, at 12:12, Nagy, Attila <[email protected]> wrote:
>
> On 02/28/2018 03:08 PM, Maxim Dounin wrote:
>> The question here is - why you want the file to be on disk, and
>> not just in a buffer? Because you expect the server to die in a
>> few seconds without flushing the file to disk? How probable it
>> is, compared to the probability of the disk to die? A more
>> reliable server can make this probability negligible, hence the
>> suggestion.
> Because the files I upload to nginx servers are important to me. Please step back a little and forget that we are talking about nginx or an HTTP server.
> We have data which we want to write to somewhere.
> Check any of the database servers. Would you accept a DB server which can loose confirmed data or couldn't be configured that way that a write/insert/update/commit/whatever you use to modify or put data into it operation is reliably written by the time you receive acknowledgement?
> Now try to use this example. I would like to use nginx to store files. That's what HTTP PUT is for.
> Of course I'm not expecting that the server will die every day. But when that happens, I want to make sure that the confirmed data is there.
> Let's take a look at various object storage systems, like ceph. Would you accept a confirmed write to be lost there? They make a great deal of work to make that impossible.
> Now try to imagine that somebody doesn't need the complexity of -for example- ceph, but wants to store data with plain HTTP. And you got there. If you store data, then you want to make sure the data is there.
> If you don't, why do you store it anyways?
>
>> (Also, another question is what "on the disk" meas from physical
>> point of view. In many cases this in fact means "somewhere in the
>> disk buffers", and a power outage can easily result in the file
>> being not accessible even after fsync().)
> Not with good software/hardware. (and it doesn't really have to be super good, but average)
>
>>
>>> Why doing this in a thread is not a good idea? It would'nt block nginx
>>> that way.
>> Because even in threads, fsync() is likely to cause performance
>> degradation. It might be a better idea to let the OS manage
>> buffers instead.
>>
> Sure, it will cause some (not much BTW in a good configuration). But if my primary goal is to store files reliably, why should I care?
> I can solve that by using SSDs for logs, BBWCs and a lot more thing. But in the current way, I can't make sure that a HTTP PUT was really successful or it will be successful in some seconds or it will fail badly.
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Nagy, Attila
Re: fsync()-in webdav PUT
March 02, 2018 12:40PM
On 03/02/2018 11:42 AM, Aziz Rozyev wrote:
> man page quote is related to the Valery’s argument that fsync wont affect performance, forget it.
Of course it affects performance. But as for how much: it depends on
many factors. It's possible to build servers where the overall effect
will be negligible.
>
> It’s nonsense because you’re trying to solve the reliability problem at the different level,
> it has been multiple times suggested here already by maxim and Paul, that it’s better
> to invest to the good server/storage infrastructure, instead of fsyncing each PUT.
Yes, it has been suggested multiple times, the only problem is it's not
true. No matter how good server/storage you have, if you write to
unbacked memory buffers (which nginx does), you are toast.

>
> Regarding the DB server analogy, you’re still not save from the power outages as long as your
> transaction isn’t in a transaction log.
>
> If you’re still consent with syncing and ready to sacrifice your time, try mounting a file system
> with ‘sync’ option.
>
That's what really kills performance, because of the async nature of
nginx. That's why I'm proposing an option to do the fsync at the end of
the PUT (or maybe even the whole operation) in a thread(pool).

If you care about performance and reliability, that's the way it has to
be solved.
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Maxim Dounin
Re: fsync()-in webdav PUT
March 02, 2018 05:10PM
Hello!

On Fri, Mar 02, 2018 at 10:12:02AM +0100, Nagy, Attila wrote:

> On 02/28/2018 03:08 PM, Maxim Dounin wrote:
> > The question here is - why you want the file to be on disk, and
> > not just in a buffer? Because you expect the server to die in a
> > few seconds without flushing the file to disk? How probable it
> > is, compared to the probability of the disk to die? A more
> > reliable server can make this probability negligible, hence the
> > suggestion.
> Because the files I upload to nginx servers are important to me. Please
> step back a little and forget that we are talking about nginx or an HTTP
> server.

If file are indeed important to you, you have to keep a second
copy in a different location, or even in multiple different
locations. Trying to do fsync() won't save your data in a lot of
quite realistic scenarios, but certainly will imply performance
(and complexity, from nginx code point of view) costs.

> We have data which we want to write to somewhere.
> Check any of the database servers. Would you accept a DB server which
> can loose confirmed data or couldn't be configured that way that a
> write/insert/update/commit/whatever you use to modify or put data into
> it operation is reliably written by the time you receive acknowledgement?

The "can loose confirmed data" claim applies to all database
servers, all installations in the world. There is no such thing
as 100% reliability. And the question is how probable data loss
is, and if we can ignore a particular probability or not.

> Now try to use this example. I would like to use nginx to store files.
> That's what HTTP PUT is for.
> Of course I'm not expecting that the server will die every day. But when
> that happens, I want to make sure that the confirmed data is there.
> Let's take a look at various object storage systems, like ceph. Would
> you accept a confirmed write to be lost there? They make a great deal of
> work to make that impossible.
> Now try to imagine that somebody doesn't need the complexity of -for
> example- ceph, but wants to store data with plain HTTP. And you got
> there. If you store data, then you want to make sure the data is there.
> If you don't, why do you store it anyways?

So, given the fact that there is no such thing as 100%
reliability, you suggest to do not store files at all? I don't
think it's viable approach - and clearly you are already doing the
opposite. Rather, you want to consider various scenarios and
their probabilities, and minimize probabilities of loosing data
where possible and makes sense.

And that's why I asked you to compare the probability you are
trying to avoid with other probabilities which can cause data loss -
for example, the probability of the disk to die. Just for the
reference, assuming you are using a commodity HDD to store your
files, the probability that it will fail within a year is about 2%
(see, for example, Backblaze data, recent stats are available at
https://www.backblaze.com/blog/hard-drive-stats-for-2017/).

Moreover, even if you have numbers on hand and this numbers will
show that you indeed need to ensure syncing files to disk to reach
greater reliability, doing fsync() might not be the best way to
achieve this. For example, doing sync() instead after loading
multiple files might be a better solution, both due to lower
complexity and higher performance.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Valery Kholodkov
Re: fsync()-in webdav PUT
March 02, 2018 08:50PM
On 02-03-18 17:06, Maxim Dounin wrote:
>>> The question here is - why you want the file to be on disk, and
>>> not just in a buffer? Because you expect the server to die in a
>>> few seconds without flushing the file to disk? How probable it
>>> is, compared to the probability of the disk to die? A more
>>> reliable server can make this probability negligible, hence the
>>> suggestion.
>> Because the files I upload to nginx servers are important to me. Please
>> step back a little and forget that we are talking about nginx or an HTTP
>> server.
>
> If file are indeed important to you, you have to keep a second
> copy in a different location, or even in multiple different
> locations. Trying to do fsync() won't save your data in a lot of
> quite realistic scenarios, but certainly will imply performance
> (and complexity, from nginx code point of view) costs.

But do you understand that even in a replicated setup the time interval
when data reaches permanent storage might be significantly long and
according to your assumptions is random and unpredictable.

In other words, without fsync() it's not possible to make any judgments
about consistency of your data, consequently it's not possible to
implement a program, that tells if your data is consistent or not.

Don't you think that your arguments are fundamentally flawed because you
insist on probabilistic nature of the problem, while it is actually
deterministic?

By the way, even LevelDB has options for synchronous writes:

https://github.com/google/leveldb/blob/master/doc/index.md#synchronous-writes

and it implements them with fsync()

Bitcoin Core varies these options depending on operation mode (see
src/validation.cpp, src/txdb.cpp, src/dbwrapper.cpp).

Oh, I forgot, Bitcoin it's nonsense...

val
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Maxim Dounin
Re: fsync()-in webdav PUT
March 03, 2018 01:50AM
Hello!

On Fri, Mar 02, 2018 at 08:47:17PM +0100, Valery Kholodkov wrote:

> On 02-03-18 17:06, Maxim Dounin wrote:
> >>> The question here is - why you want the file to be on disk, and
> >>> not just in a buffer? Because you expect the server to die in a
> >>> few seconds without flushing the file to disk? How probable it
> >>> is, compared to the probability of the disk to die? A more
> >>> reliable server can make this probability negligible, hence the
> >>> suggestion.
> >> Because the files I upload to nginx servers are important to me. Please
> >> step back a little and forget that we are talking about nginx or an HTTP
> >> server.
> >
> > If file are indeed important to you, you have to keep a second
> > copy in a different location, or even in multiple different
> > locations. Trying to do fsync() won't save your data in a lot of
> > quite realistic scenarios, but certainly will imply performance
> > (and complexity, from nginx code point of view) costs.
>
> But do you understand that even in a replicated setup the time interval
> when data reaches permanent storage might be significantly long and
> according to your assumptions is random and unpredictable.
>
> In other words, without fsync() it's not possible to make any judgments
> about consistency of your data, consequently it's not possible to
> implement a program, that tells if your data is consistent or not.
>
> Don't you think that your arguments are fundamentally flawed because you
> insist on probabilistic nature of the problem, while it is actually
> deterministic?

In no particular order:

1. There are no "my assumptions".

2. This is not about consistency, it's about fault tolerance.
Everything is consistent unless a server crash happens.

3. Using fsync() can increase the chance that your data will
survive a server crash / power outage. It doesn't matter in many
other scenarios though, for example, if your disk dies.

4. Trying to insist that reliability is deterministic looks unwise
to me, but it's up to you to insist on anything you want.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Reinis Rozitis
RE: fsync()-in webdav PUT
March 04, 2018 01:50PM
> That's what really kills performance, because of the async nature of
> nginx. That's why I'm proposing an option to do the fsync at the end of
> the PUT (or maybe even the whole operation) in a thread(pool).

Then again this way you make it again "asynchronous" (since it is/could be waiting in some thread/pool (forever)).

In general this whole thing reminds me of Linus rant about how ".. the only reason O_DIRECT exists is because database people are too used to it" ..

rr

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Nagy, Attila
Re: fsync()-in webdav PUT
March 05, 2018 11:40AM
On 03/04/2018 12:45 PM, Reinis Rozitis wrote:
>> That's what really kills performance, because of the async nature of
>> nginx. That's why I'm proposing an option to do the fsync at the end of
>> the PUT (or maybe even the whole operation) in a thread(pool).
> Then again this way you make it again "asynchronous" (since it is/could be waiting in some thread/pool (forever)).
Jesus, why? You start the fsync in a thread and you wait for it to be
completed with the HTTP response. Until this happens, the main thread
can service other requests.
Have you ever seen an async program which uses threads to run blocking
operations?

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Reinis Rozitis
RE: fsync()-in webdav PUT
March 05, 2018 01:00PM
> Jesus, why? You start the fsync in a thread and you wait for it to be completed
> with the HTTP response. Until this happens, the main thread can service other
> requests.
> Have you ever seen an async program which uses threads to run blocking
> operations?

The point was that it's odd that you are going to "trust" the userland daemon to finish the sync operation (which obviously has the possibility to fail) in some background thread while not trusting the OS/kernel to do the buffer/vm/pagecache flush at some "later" / "better" time (which you can even finetune to do it immediately - vm.dirty_ratio / vm.dirty_expire_centisecs etc).

Besides even with sync you don't get 100% of guarantees that the write actually ends (correctly) "on the iron" - coming from the land of ZFS ("lots of cheksuming") people will confirm that there are quite a few parts (like drive cache/firmware, controller cache/firmware) which occasionally lie about state of things.



p.s. then again nginx is also an opensource project and one can implement/propose whatever changes you need for your application even they don't align with authors (for example I also use nginx's webdav module but I do remove everything related to directory delete (to be more safe) just because of the way the app operates)


Just my 2 cents ..

rr

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Valery Kholodkov
Re: fsync()-in webdav PUT
March 05, 2018 02:20PM
On 05-03-18 12:54, Reinis Rozitis wrote:
>> Have you ever seen an async program which uses threads to run blocking
>> operations?
>
> The point was that it's odd that you are going to "trust" the userland daemon to finish the sync operation (which obviously has the possibility to fail) in some background thread while not trusting the OS/kernel to do the buffer/vm/pagecache flush at some "later" / "better" time (which you can even finetune to do it immediately - vm.dirty_ratio / vm.dirty_expire_centisecs etc).

And so is odd to return a positive reply when you only speculated about
its positiveness.

> Besides even with sync you don't get 100% of guarantees that the write actually ends (correctly) "on the iron" - coming from the land of ZFS ("lots of cheksuming") people will confirm that there are quite a few parts (like drive cache/firmware, controller cache/firmware) which occasionally lie about state of things.

Not lie, but speculate. The speculative behavior is then isolated in the
"iron", don't you think?

val

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Nagy, Attila
Re: fsync()-in webdav PUT
March 05, 2018 03:00PM
On 03/05/2018 12:54 PM, Reinis Rozitis wrote:
>> Jesus, why? You start the fsync in a thread and you wait for it to be completed
>> with the HTTP response. Until this happens, the main thread can service other
>> requests.
>> Have you ever seen an async program which uses threads to run blocking
>> operations?
> The point was that it's odd that you are going to "trust" the userland daemon to finish the sync operation (which obviously has the possibility to fail) in some background thread while not trusting the OS/kernel to do the buffer/vm/pagecache flush at some "later" / "better" time (which you can even finetune to do it immediately - vm.dirty_ratio / vm.dirty_expire_centisecs etc).
The point here is you are completely on the wrong track. I'm not talking
about a "background thread".
I was talking about:
1. placing the WHOLE PUT operation into its own thread OR
2. placing just the fsync into a thread (and waiting it to complete with
the HTTP response of course)
(or doing something equivalent, which won't block nginx's main thread)
This means the HTTP success or fail response will only be given back
AFTER the fsync was successfully completed.

There is absolutely no problem with a failing fsync. It means the
operation couldn't complete and the file is not written successfully.
An information, which you'll never know with current nginx.

> Besides even with sync you don't get 100% of guarantees that the write actually ends (correctly) "on the iron" - coming from the land of ZFS ("lots of cheksuming") people will confirm that there are quite a few parts (like drive cache/firmware, controller cache/firmware) which occasionally lie about state of things.
This is not relevant here. As somebody in this thread said: this has
nothing to do with nginx. It's the matter of choosing the right hardware.

> p.s. then again nginx is also an opensource project and one can implement/propose whatever changes you need for your application even they don't align with authors (for example I also use nginx's webdav module but I do remove everything related to directory delete (to be more safe) just because of the way the app operates)
Sure. But this also doesn't really add to this discussion.
>
>
> Just my 2 cents ..
>
I'm not sure it worths 2 cents. You clearly don't understand the problem
here, so I'm not sure why you have to speculate about something which I
didn't write...
It's completely useless...
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Richard Demeny
Re: fsync()-in webdav PUT
March 05, 2018 03:10PM
This needs to stop

On Monday, March 5, 2018, Nagy, Attila <[email protected]fsn.hu> wrote:

> On 03/05/2018 12:54 PM, Reinis Rozitis wrote:
>
>> Jesus, why? You start the fsync in a thread and you wait for it to be
>>> completed
>>> with the HTTP response. Until this happens, the main thread can service
>>> other
>>> requests.
>>> Have you ever seen an async program which uses threads to run blocking
>>> operations?
>>>
>> The point was that it's odd that you are going to "trust" the userland
>> daemon to finish the sync operation (which obviously has the possibility to
>> fail) in some background thread while not trusting the OS/kernel to do the
>> buffer/vm/pagecache flush at some "later" / "better" time (which you can
>> even finetune to do it immediately - vm.dirty_ratio /
>> vm.dirty_expire_centisecs etc).
>>
> The point here is you are completely on the wrong track. I'm not talking
> about a "background thread".
> I was talking about:
> 1. placing the WHOLE PUT operation into its own thread OR
> 2. placing just the fsync into a thread (and waiting it to complete with
> the HTTP response of course)
> (or doing something equivalent, which won't block nginx's main thread)
> This means the HTTP success or fail response will only be given back AFTER
> the fsync was successfully completed.
>
> There is absolutely no problem with a failing fsync. It means the
> operation couldn't complete and the file is not written successfully.
> An information, which you'll never know with current nginx.
>
> Besides even with sync you don't get 100% of guarantees that the write
>> actually ends (correctly) "on the iron" - coming from the land of ZFS
>> ("lots of cheksuming") people will confirm that there are quite a few parts
>> (like drive cache/firmware, controller cache/firmware) which occasionally
>> lie about state of things.
>>
> This is not relevant here. As somebody in this thread said: this has
> nothing to do with nginx. It's the matter of choosing the right hardware.
>
> p.s. then again nginx is also an opensource project and one can
>> implement/propose whatever changes you need for your application even they
>> don't align with authors (for example I also use nginx's webdav module but
>> I do remove everything related to directory delete (to be more safe) just
>> because of the way the app operates)
>>
> Sure. But this also doesn't really add to this discussion.
>
>>
>>
>> Just my 2 cents ..
>>
>> I'm not sure it worths 2 cents. You clearly don't understand the problem
> here, so I'm not sure why you have to speculate about something which I
> didn't write...
> It's completely useless...
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Sorry, only registered users may post in this forum.

Click here to login