Welcome! Log In Create A New Profile

Advanced

Proxy Request buffering not working as expected

Posted by Mohammad Puyandeh via nginx 
Mohammad Puyandeh via nginx
Proxy Request buffering not working as expected
November 06, 2017 02:00PM
If I try to upload a 200M file using POST method, nginx will pass chunks
to back-end in less than 3 meg chunks


sample nginx config:

worker_processes auto;
user nginx;

events {

}

http {
    upstream servers {
        server                127.0.0.1:9999;
    }



 server {
   listen 80;
   client_max_body_size 200M;
   client_body_buffer_size 200M;

   server_name           localhost;

        location / {
            proxy_request_buffering   on;
            proxy_pass http://servers;
        }

 }
}


a quote about proxy_request_buffering  from docs:

"""When buffering is enabled, the entire request body is read from the
client before sending the request to a proxied server"""


but this behavior which I am facing is the opposite of this quote, any
ideas what's going on ?

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Francis Daly
Re: Proxy Request buffering not working as expected
November 06, 2017 03:30PM
On Mon, Nov 06, 2017 at 04:23:06PM +0330, Mohammad Puyandeh via nginx wrote:

Hi there,

> If I try to upload a 200M file using POST method, nginx will pass
> chunks to back-end in less than 3 meg chunks

proxy_request_buffering says nothing about chunks being sent to the
back-end.

> a quote about proxy_request_buffering  from docs:
>
> """When buffering is enabled, the entire request body is read from
> the client before sending the request to a proxied server"""
>
>
> but this behavior which I am facing is the opposite of this quote,

You haven't explicitly reported any behaviour that contradicts this quote.

> any ideas what's going on ?

If you report that nginx starts to send the content to the back-end
before nginx has received all of the content from the client, then
proxy_request_buffering is not working as intended.

When I try a quick test here, nginx receives the full content from the
client before it tries to connect to the back-end.

f
--
Francis Daly francis@daoine.org
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Mohammad Puyandeh via nginx
Re: Proxy Request buffering not working as expected
November 06, 2017 04:50PM
>> If you report that nginx starts to send the content to the back-end
>> before nginx has received all of the content from the client, then
>> proxy_request_buffering is not working as intended.
Yes, that's what I am reporting, I put together a very simple tornado
(python) script, I used both tcp connection and a unix socket, no
differences
I used curl and my own upload python client, result were the same

>> When I try a quick test here, nginx receives the full content from the
>> client before it tries to connect to the back-end.
I prepared a minimal working sample which is easy to run and test
this is python script: https://paste.ubuntu.com/25903714/
(see lines 16 to 23 to know how to run it, it's so simple, you need a
Linux based OS and python)
and this is nginx config: https://paste.ubuntu.com/25903719/

with the above python client you can easily produce the problem, it will
print chunk sizes along with the number of requests, use a file larger
than 3M to get better results
to make it complete, here is an example to upload file using curl: curl
--request POST --data-binary "@/file/path" http://localhost/upload/

This is my first time that I'm using this service, So I thought putting
codes somewhere else is cleaner and better

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Francis Daly
Re: Proxy Request buffering not working as expected
November 06, 2017 08:00PM
On Mon, Nov 06, 2017 at 07:13:53PM +0330, Mohammad Puyandeh via nginx wrote:

Hi there,

> >>If you report that nginx starts to send the content to the back-end
> >>before nginx has received all of the content from the client, then
> >>proxy_request_buffering is not working as intended.

> Yes, that's what I am reporting,

I'm not seeing it in my testing (but I am not using your not-nginx part).

> >>When I try a quick test here, nginx receives the full content from the
> >>client before it tries to connect to the back-end.

> with the above python client you can easily produce the problem, it
> will print chunk sizes along with the number of requests, use a file
> larger than 3M to get better results

"chunk size" and "number of requests" are not related to
proxy_request_buffering.

Only "the time at which nginx starts to send traffic to the back-end"
is related to proxy_request_buffering..

> to make it complete, here is an example to upload file using curl:
> curl --request POST --data-binary "@/file/path"
> http://localhost/upload/

If you add "--limit-rate 16000" to that curl command, do you see output
from your back-end server immediately, or only after enough time has
passed that your client has sent all of the content to nginx?

f
--
Francis Daly francis@daoine.org
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Mohammad Puyandeh via nginx
Re: Proxy Request buffering not working as expected
November 07, 2017 03:40PM
>>If you add "--limit-rate 16000" to that curl command, do you see output
>>from your back-end server immediately, or only after enough time has
>>passed that your client has sent all of the content to nginx?

It's after, so Nginx buffers correctly here and I was mistaken

but now the question is, this behavior is not much different than when
proxy_request_buffering is set to off

the whole point of using  nginx is to buffer request and pass request
body to back-end server at once in a single chunk, but nginx pass it in
small chunks, there is some other configuration for that ? how can we
achieve this goal then ?

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Francis Daly
Re: Proxy Request buffering not working as expected
November 09, 2017 02:40PM
On Tue, Nov 07, 2017 at 06:04:26PM +0330, Mohammad Puyandeh via nginx wrote:

Hi there,

> >>If you add "--limit-rate 16000" to that curl command, do you see output
> >>from your back-end server immediately, or only after enough time has
> >>passed that your client has sent all of the content to nginx?
>
> It's after, so Nginx buffers correctly here and I was mistaken
>
> but now the question is, this behavior is not much different than
> when proxy_request_buffering is set to off

If you repeat the test with "proxy_request_buffering off" in nginx.conf,
you should see that the back-end server starts receiving content sooner.

That is pretty much the only thing that proxy_request_buffering is for.

> the whole point of using  nginx is to buffer request and pass
> request body to back-end server at once in a single chunk, but nginx
> pass it in small chunks, there is some other configuration for that
> ? how can we achieve this goal then ?

What do you mean by "single chunk" and "small chunks"?

Do you mean "nginx uses Transfer-Encoding: chunked in its request to
upstream", or do you mean something else?

Because if you mean "Transfer-Encoding: chunked", you can disable
that by using HTTP/1.0 in the connection from nginx to upstream
(http://nginx.org/r/proxy_http_version); but that is the default and
you did not show it having been changed.

f
--
Francis Daly francis@daoine.org
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Sorry, only registered users may post in this forum.

Click here to login