Mohammad Puyandeh via nginx
Proxy Request buffering not working as expected
November 06, 2017 02:00PM
If I try to upload a 200M file using POST method, nginx will pass chunks
to back-end in less than 3 meg chunks


sample nginx config:

worker_processes auto;
user nginx;

events {

}

http {
    upstream servers {
        server                127.0.0.1:9999;
    }



 server {
   listen 80;
   client_max_body_size 200M;
   client_body_buffer_size 200M;

   server_name           localhost;

        location / {
            proxy_request_buffering   on;
            proxy_pass http://servers;
        }

 }
}


a quote about proxy_request_buffering  from docs:

"""When buffering is enabled, the entire request body is read from the
client before sending the request to a proxied server"""


but this behavior which I am facing is the opposite of this quote, any
ideas what's going on ?

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Francis Daly
Re: Proxy Request buffering not working as expected
November 06, 2017 03:30PM
On Mon, Nov 06, 2017 at 04:23:06PM +0330, Mohammad Puyandeh via nginx wrote:

Hi there,

> If I try to upload a 200M file using POST method, nginx will pass
> chunks to back-end in less than 3 meg chunks

proxy_request_buffering says nothing about chunks being sent to the
back-end.

> a quote about proxy_request_buffering  from docs:
>
> """When buffering is enabled, the entire request body is read from
> the client before sending the request to a proxied server"""
>
>
> but this behavior which I am facing is the opposite of this quote,

You haven't explicitly reported any behaviour that contradicts this quote.

> any ideas what's going on ?

If you report that nginx starts to send the content to the back-end
before nginx has received all of the content from the client, then
proxy_request_buffering is not working as intended.

When I try a quick test here, nginx receives the full content from the
client before it tries to connect to the back-end.

f
--
Francis Daly francis@daoine.org
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Mohammad Puyandeh via nginx
Re: Proxy Request buffering not working as expected
November 06, 2017 04:50PM
>> If you report that nginx starts to send the content to the back-end
>> before nginx has received all of the content from the client, then
>> proxy_request_buffering is not working as intended.
Yes, that's what I am reporting, I put together a very simple tornado
(python) script, I used both tcp connection and a unix socket, no
differences
I used curl and my own upload python client, result were the same

>> When I try a quick test here, nginx receives the full content from the
>> client before it tries to connect to the back-end.
I prepared a minimal working sample which is easy to run and test
this is python script: https://paste.ubuntu.com/25903714/
(see lines 16 to 23 to know how to run it, it's so simple, you need a
Linux based OS and python)
and this is nginx config: https://paste.ubuntu.com/25903719/

with the above python client you can easily produce the problem, it will
print chunk sizes along with the number of requests, use a file larger
than 3M to get better results
to make it complete, here is an example to upload file using curl: curl
--request POST --data-binary "@/file/path" http://localhost/upload/

This is my first time that I'm using this service, So I thought putting
codes somewhere else is cleaner and better

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Francis Daly
Re: Proxy Request buffering not working as expected
November 06, 2017 08:00PM
On Mon, Nov 06, 2017 at 07:13:53PM +0330, Mohammad Puyandeh via nginx wrote:

Hi there,

> >>If you report that nginx starts to send the content to the back-end
> >>before nginx has received all of the content from the client, then
> >>proxy_request_buffering is not working as intended.

> Yes, that's what I am reporting,

I'm not seeing it in my testing (but I am not using your not-nginx part).

> >>When I try a quick test here, nginx receives the full content from the
> >>client before it tries to connect to the back-end.

> with the above python client you can easily produce the problem, it
> will print chunk sizes along with the number of requests, use a file
> larger than 3M to get better results

"chunk size" and "number of requests" are not related to
proxy_request_buffering.

Only "the time at which nginx starts to send traffic to the back-end"
is related to proxy_request_buffering..

> to make it complete, here is an example to upload file using curl:
> curl --request POST --data-binary "@/file/path"
> http://localhost/upload/

If you add "--limit-rate 16000" to that curl command, do you see output
from your back-end server immediately, or only after enough time has
passed that your client has sent all of the content to nginx?

f
--
Francis Daly francis@daoine.org
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Mohammad Puyandeh via nginx
Re: Proxy Request buffering not working as expected
November 07, 2017 03:40PM
>>If you add "--limit-rate 16000" to that curl command, do you see output
>>from your back-end server immediately, or only after enough time has
>>passed that your client has sent all of the content to nginx?

It's after, so Nginx buffers correctly here and I was mistaken

but now the question is, this behavior is not much different than when
proxy_request_buffering is set to off

the whole point of using  nginx is to buffer request and pass request
body to back-end server at once in a single chunk, but nginx pass it in
small chunks, there is some other configuration for that ? how can we
achieve this goal then ?

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Francis Daly
Re: Proxy Request buffering not working as expected
November 09, 2017 02:40PM
On Tue, Nov 07, 2017 at 06:04:26PM +0330, Mohammad Puyandeh via nginx wrote:

Hi there,

> >>If you add "--limit-rate 16000" to that curl command, do you see output
> >>from your back-end server immediately, or only after enough time has
> >>passed that your client has sent all of the content to nginx?
>
> It's after, so Nginx buffers correctly here and I was mistaken
>
> but now the question is, this behavior is not much different than
> when proxy_request_buffering is set to off

If you repeat the test with "proxy_request_buffering off" in nginx.conf,
you should see that the back-end server starts receiving content sooner.

That is pretty much the only thing that proxy_request_buffering is for.

> the whole point of using  nginx is to buffer request and pass
> request body to back-end server at once in a single chunk, but nginx
> pass it in small chunks, there is some other configuration for that
> ? how can we achieve this goal then ?

What do you mean by "single chunk" and "small chunks"?

Do you mean "nginx uses Transfer-Encoding: chunked in its request to
upstream", or do you mean something else?

Because if you mean "Transfer-Encoding: chunked", you can disable
that by using HTTP/1.0 in the connection from nginx to upstream
(http://nginx.org/r/proxy_http_version); but that is the default and
you did not show it having been changed.

f
--
Francis Daly francis@daoine.org
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Sorry, only registered users may post in this forum.

Click here to login