Welcome! Log In Create A New Profile

Advanced

Recovering from partial writes

Posted by Anonymous User 
Anonymous User
Recovering from partial writes
June 22, 2018 08:50PM
I have an nginx proxy through which clients pass a large POST payload to
the upstream server. Sometimes, the upstream server is slow and so
writing the POST data will fail with a writev() not ready (EAGAIN)
error. But of course, that's a very common situation when dealing with
non-blocking I/O, and I'd expect the rest of the data to be written when
the socket is again ready for writing.

In fact, it seems like the basic structure of that is in place; when
ngx_writev gets the EAGAIN, it passes that to calling functions, which
modify the chain buffers. Yet somewhere along the line (seemingly in
ngx_http_upstream_send_request_body) the partially-written buffer is
freed, and although the socket later indicates that it is ready to write
(and the ngx epoll module does detect that), there is no longer any data
to write and so everything fails.

I realize this is not the dev mailing list so an answer to how that is
programmed isn't necessarily what I'm after -- again, the partial write
of data to a socket is such a common thing that I can't think I'm the
first to encounter it and find a basic bug, so I assume that something
else is going on. I have tried this with proxy_request_buffering off and
on, and the failure is essentially the same. The http section of my conf
looks like this:

http {
max_ranges 1;
#map $http_accept $file_extension {
# default ".html";
# "~*json" ".json";
#}
map $http_upgrade $connection_upgrade {
default upgrade;
'' "";
}
server_names_hash_bucket_size 512;
server_names_hash_max_size 2048;
variables_hash_bucket_size 512;
variables_hash_max_size 2048;
client_header_buffer_size 8k;
large_client_header_buffers 4 16k;
proxy_buffering off;
proxy_request_buffering off; # Tried on, and various sizes
#proxy_buffer_size 16k;
#proxy_buffers 4 128k;
#proxy_busy_buffers_size 256k;
#proxy_headers_hash_bucket_size 256;
client_max_body_size 0;
ssl_session_cache shared:SSL:20m;
ssl_session_timeout 60m;

include /u01/data/config/nginx/mime.types;
default_type application/octet-stream;

log_format main '"$remote_addr" "-" "$remote_user" "[$time_local]" "$request" '
'"$status" "$body_bytes_sent" "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';

log_format opcroutingtier '"$remote_addr" "-" "$remote_user" [$time_local] "$request" "$status" '
'"$body_bytes_sent" "$http_referer" "$http_user_agent" "$bytes_sent" "$request_length" "-" '
'"$host" "$http_x_forwarded_for" "$server_name" "$server_port" "$request_time" "$upstream_addr" '
'"$upstream_connect_time" "$upstream_header_time" "$upstream_response_time" "$upstream_status" "$ssl_cipher" "$ssl_protocol" '
'"-" "-" "-"';

access_log /u01/data/logs/nginx_logs/access_logs/access.log opcroutingtier;
sendfile off; # also tried on
keepalive_timeout 60s;
keepalive_requests 2000000;
open_file_cache max=2000 inactive=20s;
open_file_cache_valid 60s;
open_file_cache_min_uses 5;
open_file_cache_errors off;
gzip on;
gzip_types text/plain text/css text/javascript text/xml application/x-javascript application/xml;
gzip_min_length 500;
gzip_comp_level 7;

Everything works fine if the upstream reads data fast enough; it's only
when nginx gets a partial write upstream that there is a problem. Am I
missing something here?

-Scott

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Peter Booth
Re: Recovering from partial writes
June 22, 2018 10:20PM
How large is a large POST payload?
Are the nginx and upstream systems physical hosts in same data center?
What are approx best case / typical case / worst case latency for the post to upstream?

Sent from my iPhone

> On Jun 22, 2018, at 2:40 PM, scott.oaks@oracle.com wrote:
>
> I have an nginx proxy through which clients pass a large POST payload to the upstream server. Sometimes, the upstream server is slow and so writing the POST data will fail with a writev() not ready (EAGAIN) error. But of course, that's a very common situation when dealing with non-blocking I/O, and I'd expect the rest of the data to be written when the socket is again ready for writing.
>
> In fact, it seems like the basic structure of that is in place; when ngx_writev gets the EAGAIN, it passes that to calling functions, which modify the chain buffers. Yet somewhere along the line (seemingly in ngx_http_upstream_send_request_body) the partially-written buffer is freed, and although the socket later indicates that it is ready to write (and the ngx epoll module does detect that), there is no longer any data to write and so everything fails.
>
> I realize this is not the dev mailing list so an answer to how that is programmed isn't necessarily what I'm after -- again, the partial write of data to a socket is such a common thing that I can't think I'm the first to encounter it and find a basic bug, so I assume that something else is going on. I have tried this with proxy_request_buffering off and on, and the failure is essentially the same. The http section of my conf looks like this:
>
> http {
> max_ranges 1;
> #map $http_accept $file_extension {
> # default ".html";
> # "~*json" ".json";
> #}
> map $http_upgrade $connection_upgrade {
> default upgrade;
> '' "";
> }
> server_names_hash_bucket_size 512;
> server_names_hash_max_size 2048;
> variables_hash_bucket_size 512;
> variables_hash_max_size 2048;
> client_header_buffer_size 8k;
> large_client_header_buffers 4 16k;
> proxy_buffering off;
> proxy_request_buffering off; # Tried on, and various sizes
> #proxy_buffer_size 16k;
> #proxy_buffers 4 128k;
> #proxy_busy_buffers_size 256k;
> #proxy_headers_hash_bucket_size 256;
> client_max_body_size 0;
> ssl_session_cache shared:SSL:20m;
> ssl_session_timeout 60m;
>
> include /u01/data/config/nginx/mime.types;
> default_type application/octet-stream;
>
> log_format main '"$remote_addr" "-" "$remote_user" "[$time_local]" "$request" '
> '"$status" "$body_bytes_sent" "$http_referer" '
> '"$http_user_agent" "$http_x_forwarded_for"';
>
> log_format opcroutingtier '"$remote_addr" "-" "$remote_user" [$time_local] "$request" "$status" '
> '"$body_bytes_sent" "$http_referer" "$http_user_agent" "$bytes_sent" "$request_length" "-" '
> '"$host" "$http_x_forwarded_for" "$server_name" "$server_port" "$request_time" "$upstream_addr" '
> '"$upstream_connect_time" "$upstream_header_time" "$upstream_response_time" "$upstream_status" "$ssl_cipher" "$ssl_protocol" '
> '"-" "-" "-"';
>
> access_log /u01/data/logs/nginx_logs/access_logs/access.log opcroutingtier;
> sendfile off; # also tried on
> keepalive_timeout 60s;
> keepalive_requests 2000000;
> open_file_cache max=2000 inactive=20s;
> open_file_cache_valid 60s;
> open_file_cache_min_uses 5;
> open_file_cache_errors off;
> gzip on;
> gzip_types text/plain text/css text/javascript text/xml application/x-javascript application/xml;
> gzip_min_length 500;
> gzip_comp_level 7;
>
> Everything works fine if the upstream reads data fast enough; it's only when nginx gets a partial write upstream that there is a problem. Am I missing something here?
>
> -Scott
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Anonymous User
Re: Recovering from partial writes
June 22, 2018 10:30PM
The POST payload varies but can be as much as 20M

nginx and the upstream are in the same data center now, but that isn't
necessarily a requirement, and even in the data center speeds will vary
depending on network congestion. Hence I cannot guarantee the worst-case
latency. If the upstream java server does a 5 second GC, then there
could be a long pause in its reading data.

Your questions lead me to believe that you'd like to suggest things to
make the writev() not do a partial write. That is helpful, but the real
point isn't to find a config that happens to work so that the writev()
never gets a partial write -- it is to make the partial writev scenario
actually work.

-Scott

On 6/22/18 4:18 PM, Peter Booth wrote:
> How large is a large POST payload?
> Are the nginx and upstream systems physical hosts in same data center?
> What are approx best case / typical case / worst case latency for the post to upstream?
>
> Sent from my iPhone
>
>> On Jun 22, 2018, at 2:40 PM, scott.oaks@oracle.com wrote:
>>
>> I have an nginx proxy through which clients pass a large POST payload to the upstream server. Sometimes, the upstream server is slow and so writing the POST data will fail with a writev() not ready (EAGAIN) error. But of course, that's a very common situation when dealing with non-blocking I/O, and I'd expect the rest of the data to be written when the socket is again ready for writing.
>>
>> In fact, it seems like the basic structure of that is in place; when ngx_writev gets the EAGAIN, it passes that to calling functions, which modify the chain buffers. Yet somewhere along the line (seemingly in ngx_http_upstream_send_request_body) the partially-written buffer is freed, and although the socket later indicates that it is ready to write (and the ngx epoll module does detect that), there is no longer any data to write and so everything fails.
>>
>> I realize this is not the dev mailing list so an answer to how that is programmed isn't necessarily what I'm after -- again, the partial write of data to a socket is such a common thing that I can't think I'm the first to encounter it and find a basic bug, so I assume that something else is going on. I have tried this with proxy_request_buffering off and on, and the failure is essentially the same. The http section of my conf looks like this:
>>
>> http {
>> max_ranges 1;
>> #map $http_accept $file_extension {
>> # default ".html";
>> # "~*json" ".json";
>> #}
>> map $http_upgrade $connection_upgrade {
>> default upgrade;
>> '' "";
>> }
>> server_names_hash_bucket_size 512;
>> server_names_hash_max_size 2048;
>> variables_hash_bucket_size 512;
>> variables_hash_max_size 2048;
>> client_header_buffer_size 8k;
>> large_client_header_buffers 4 16k;
>> proxy_buffering off;
>> proxy_request_buffering off; # Tried on, and various sizes
>> #proxy_buffer_size 16k;
>> #proxy_buffers 4 128k;
>> #proxy_busy_buffers_size 256k;
>> #proxy_headers_hash_bucket_size 256;
>> client_max_body_size 0;
>> ssl_session_cache shared:SSL:20m;
>> ssl_session_timeout 60m;
>>
>> include /u01/data/config/nginx/mime.types;
>> default_type application/octet-stream;
>>
>> log_format main '"$remote_addr" "-" "$remote_user" "[$time_local]" "$request" '
>> '"$status" "$body_bytes_sent" "$http_referer" '
>> '"$http_user_agent" "$http_x_forwarded_for"';
>>
>> log_format opcroutingtier '"$remote_addr" "-" "$remote_user" [$time_local] "$request" "$status" '
>> '"$body_bytes_sent" "$http_referer" "$http_user_agent" "$bytes_sent" "$request_length" "-" '
>> '"$host" "$http_x_forwarded_for" "$server_name" "$server_port" "$request_time" "$upstream_addr" '
>> '"$upstream_connect_time" "$upstream_header_time" "$upstream_response_time" "$upstream_status" "$ssl_cipher" "$ssl_protocol" '
>> '"-" "-" "-"';
>>
>> access_log /u01/data/logs/nginx_logs/access_logs/access.log opcroutingtier;
>> sendfile off; # also tried on
>> keepalive_timeout 60s;
>> keepalive_requests 2000000;
>> open_file_cache max=2000 inactive=20s;
>> open_file_cache_valid 60s;
>> open_file_cache_min_uses 5;
>> open_file_cache_errors off;
>> gzip on;
>> gzip_types text/plain text/css text/javascript text/xml application/x-javascript application/xml;
>> gzip_min_length 500;
>> gzip_comp_level 7;
>>
>> Everything works fine if the upstream reads data fast enough; it's only when nginx gets a partial write upstream that there is a problem. Am I missing something here?
>>
>> -Scott
>>
>> _______________________________________________
>> nginx mailing list
>> nginx@nginx.org
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.nginx.org_mailman_listinfo_nginx&d=DwICAg&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=rd78Sm7B8x7Jg86BzKn7cn1a_8HKt26SFIE05r0bOD0&m=tOSPAWsYbWZKUrwdK1PErzJygcIkiJzSm3gAK6UYZRQ&s=1Ko-8EEz4a8Ukl0ELKr1jnR5-sZe5qTh_cWP19eVye4&e=
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.nginx.org_mailman_listinfo_nginx&d=DwICAg&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=rd78Sm7B8x7Jg86BzKn7cn1a_8HKt26SFIE05r0bOD0&m=tOSPAWsYbWZKUrwdK1PErzJygcIkiJzSm3gAK6UYZRQ&s=1Ko-8EEz4a8Ukl0ELKr1jnR5-sZe5qTh_cWP19eVye4&e=

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Anonymous User
Re: Recovering from partial writes
June 22, 2018 10:40PM
I should have added -- I know that there are 60 second (default)
timeouts in place so if nginx cannot write upstream for 60 seconds it
will abort the request. That's fine; it's the shorter scenarios I am
worried about. In fact, what happens when nginx doesn't send the
remaining data in my case is that the upstream server times out when it
doesn't read data for 30 seconds, so there is a 30-second period where
the socket buffers are clear on both sides (yet nginx doesn't continue
to send up the data).

-Scott

On 6/22/18 4:25 PM, scott.oaks@oracle.com wrote:
> The POST payload varies but can be as much as 20M
>
> nginx and the upstream are in the same data center now, but that isn't
> necessarily a requirement, and even in the data center speeds will
> vary depending on network congestion. Hence I cannot guarantee the
> worst-case latency. If the upstream java server does a 5 second GC,
> then there could be a long pause in its reading data.
>
> Your questions lead me to believe that you'd like to suggest things to
> make the writev() not do a partial write. That is helpful, but the
> real point isn't to find a config that happens to work so that the
> writev() never gets a partial write -- it is to make the partial
> writev scenario actually work.
>
> -Scott
>
> On 6/22/18 4:18 PM, Peter Booth wrote:
>> How large is a large POST payload?
>> Are the nginx and upstream systems physical hosts in same data center?
>> What are approx best case / typical case / worst case latency for the
>> post to upstream?
>>
>> Sent from my iPhone
>>
>>> On Jun 22, 2018, at 2:40 PM, scott.oaks@oracle.com wrote:
>>>
>>> I have an nginx proxy through which clients pass a large POST
>>> payload to the upstream server. Sometimes, the upstream server is
>>> slow and so writing the POST data will fail with a writev() not
>>> ready (EAGAIN) error. But of course, that's a very common situation
>>> when dealing with non-blocking I/O, and I'd expect the rest of the
>>> data to be written when the socket is again ready for writing.
>>>
>>> In fact, it seems like the basic structure of that is in place; when
>>> ngx_writev gets the EAGAIN, it passes that to calling functions,
>>> which modify the chain buffers. Yet somewhere along the line
>>> (seemingly in ngx_http_upstream_send_request_body) the
>>> partially-written buffer is freed, and although the socket later
>>> indicates that it is ready to write (and the ngx epoll module does
>>> detect that), there is no longer any data to write and so everything
>>> fails.
>>>
>>> I realize this is not the dev mailing list so an answer to how that
>>> is programmed isn't necessarily what I'm after -- again, the partial
>>> write of data to a socket is such a common thing that I can't think
>>> I'm the first to encounter it and find a basic bug, so I assume that
>>> something else is going on. I have tried this with
>>> proxy_request_buffering off and on, and the failure is essentially
>>> the same. The http section of my conf looks like this:
>>>
>>> http {
>>>     max_ranges 1;
>>>     #map $http_accept $file_extension {
>>>     #   default   ".html";
>>>     #    "~*json"  ".json";
>>>     #}
>>>     map $http_upgrade $connection_upgrade {
>>>         default upgrade;
>>>         '' "";
>>>     }
>>>     server_names_hash_bucket_size 512;
>>>     server_names_hash_max_size 2048;
>>>     variables_hash_bucket_size 512;
>>>     variables_hash_max_size 2048;
>>>     client_header_buffer_size 8k;
>>>     large_client_header_buffers 4 16k;
>>>     proxy_buffering off;
>>>     proxy_request_buffering off; # Tried on, and various sizes
>>>     #proxy_buffer_size 16k;
>>>     #proxy_buffers 4 128k;
>>>     #proxy_busy_buffers_size 256k;
>>>     #proxy_headers_hash_bucket_size 256;
>>>     client_max_body_size 0;
>>>     ssl_session_cache shared:SSL:20m;
>>>     ssl_session_timeout 60m;
>>>
>>>     include       /u01/data/config/nginx/mime.types;
>>>     default_type  application/octet-stream;
>>>
>>>     log_format  main  '"$remote_addr" "-" "$remote_user"
>>> "[$time_local]" "$request" '
>>>                       '"$status" "$body_bytes_sent" "$http_referer" '
>>>                       '"$http_user_agent" "$http_x_forwarded_for"';
>>>
>>>     log_format  opcroutingtier  '"$remote_addr" "-" "$remote_user"
>>> [$time_local] "$request" "$status" '
>>>                                 '"$body_bytes_sent" "$http_referer"
>>> "$http_user_agent" "$bytes_sent" "$request_length" "-" '
>>>                                 '"$host" "$http_x_forwarded_for"
>>> "$server_name" "$server_port" "$request_time" "$upstream_addr" '
>>>                                 '"$upstream_connect_time"
>>> "$upstream_header_time" "$upstream_response_time" "$upstream_status"
>>> "$ssl_cipher" "$ssl_protocol" '
>>>                                 '"-" "-" "-"';
>>>
>>>     access_log /u01/data/logs/nginx_logs/access_logs/access.log
>>> opcroutingtier;
>>>     sendfile        off;  # also tried on
>>>     keepalive_timeout 60s;
>>>     keepalive_requests 2000000;
>>>     open_file_cache max=2000 inactive=20s;
>>>     open_file_cache_valid 60s;
>>>     open_file_cache_min_uses 5;
>>>     open_file_cache_errors off;
>>>     gzip on;
>>>     gzip_types text/plain text/css text/javascript text/xml
>>> application/x-javascript application/xml;
>>>     gzip_min_length 500;
>>>     gzip_comp_level 7;
>>>
>>> Everything works fine if the upstream reads data fast enough; it's
>>> only when nginx gets a partial write upstream that there is a
>>> problem. Am I missing something here?
>>>
>>> -Scott
>>>
>>> _______________________________________________
>>> nginx mailing list
>>> nginx@nginx.org
>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.nginx.org_mailman_listinfo_nginx&d=DwICAg&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=rd78Sm7B8x7Jg86BzKn7cn1a_8HKt26SFIE05r0bOD0&m=tOSPAWsYbWZKUrwdK1PErzJygcIkiJzSm3gAK6UYZRQ&s=1Ko-8EEz4a8Ukl0ELKr1jnR5-sZe5qTh_cWP19eVye4&e=
>>>
>> _______________________________________________
>> nginx mailing list
>> nginx@nginx.org
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.nginx.org_mailman_listinfo_nginx&d=DwICAg&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=rd78Sm7B8x7Jg86BzKn7cn1a_8HKt26SFIE05r0bOD0&m=tOSPAWsYbWZKUrwdK1PErzJygcIkiJzSm3gAK6UYZRQ&s=1Ko-8EEz4a8Ukl0ELKr1jnR5-sZe5qTh_cWP19eVye4&e=
>>
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.nginx.org_mailman_listinfo_nginx&d=DwICAg&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=rd78Sm7B8x7Jg86BzKn7cn1a_8HKt26SFIE05r0bOD0&m=W78uVaGLNes6XZNk3hrr9Rhn3aiguEruIhv09PHI64I&s=H2bt0GJ1kNPn3cOgJSaV1y2ER5GErOy6k0cT7PiS0cY&e=
>

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Sorry, only registered users may post in this forum.

Click here to login