Welcome! Log In Create A New Profile

Advanced

upstream (tcp stream mode) doesn't detect connecton failure

Posted by Cecile, Adam 
Cecile, Adam
upstream (tcp stream mode) doesn't detect connecton failure
January 08, 2018 01:40PM
Hello,


I'm using this quite complicated setup involving SNI routing and proxy_protocol but I'm stuck on something.


Here is the configuration file:

http://paste.debian.net/hidden/62e13f9c/


Routing, proxy_protocol, logging stuff is working just fine, the only (quite critical issue) is that the "mag" upstream doesn't see connection failures and does not switch to the second server.


In the mag.log file I just see:

98.98.98.98 [08/Jan/2018:10:56:10 +0100] proxying to "mag":10.0.0.1:443 TCP 500 0 239 1.01


But instead of blacklisting this server and moving to 10.0.0.2 I receive a connection closed error on the client.


Thanks in advance for your help,


Best regards, Adam.
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Hello!

On Mon, Jan 08, 2018 at 12:37:41PM +0000, Cecile, Adam wrote:

> Hello,
>
>
> I'm using this quite complicated setup involving SNI routing and proxy_protocol but I'm stuck on something.
>
>
> Here is the configuration file:
>
> http://paste.debian.net/hidden/62e13f9c/
>
>
> Routing, proxy_protocol, logging stuff is working just fine, the only (quite critical issue) is that the "mag" upstream doesn't see connection failures and does not switch to the second server.
>
>
> In the mag.log file I just see:
>
> 98.98.98.98 [08/Jan/2018:10:56:10 +0100] proxying to "mag":10.0.0.1:443 TCP 500 0 239 1.01
>
>
> But instead of blacklisting this server and moving to 10.0.0.2 I receive a connection closed error on the client.

As far as I understand your configuration, you have two stream
proxy layers:

1. The first one uses ssl_preread to obtain SNI name and tries to
do some routing based on it. This layer also adds to the PROXY
protocol to backend connections.

2. The second one strips PROXY protocol header.

The problem with "upstream doesn't see connection failures" is
because connection failures are only seen at the second layer (the
log line above belongs to the second layer). The first layer will
only see a connection close, and it won't know if there was an
error or not.

Also note:

- You use $proxy_protocol_addr in the "upstream mag {...}" block,
but the upstream block is used only in the first layer, where
$proxy_protocol_addr won't be available according to your
configuration.

- You use $name in the logs of the second layer. It will always
point to "map", as there is no ssl_preread in the second layer,
hence $ssl_preread_server_name will be not available.

Depending on what you actually want to achieve, the most
straightforward solution might be to actually remove the second
proxy layer.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
On 01/09/2018 02:46 PM, Maxim Dounin wrote:
> Hello!
>
> On Mon, Jan 08, 2018 at 12:37:41PM +0000, Cecile, Adam wrote:
>
>> Hello,
>>
>>
>> I'm using this quite complicated setup involving SNI routing and proxy_protocol but I'm stuck on something.
>>
>>
>> Here is the configuration file:
>>
>> http://paste.debian.net/hidden/62e13f9c/
>>
>>
>> Routing, proxy_protocol, logging stuff is working just fine, the only (quite critical issue) is that the "mag" upstream doesn't see connection failures and does not switch to the second server.
>>
>>
>> In the mag.log file I just see:
>>
>> 98.98.98.98 [08/Jan/2018:10:56:10 +0100] proxying to "mag":10.0.0.1:443 TCP 500 0 239 1.01
>>
>>
>> But instead of blacklisting this server and moving to 10.0.0.2 I receive a connection closed error on the client.
> As far as I understand your configuration, you have two stream
> proxy layers:
>
> 1. The first one uses ssl_preread to obtain SNI name and tries to
> do some routing based on it. This layer also adds to the PROXY
> protocol to backend connections.
>
> 2. The second one strips PROXY protocol header.
>
> The problem with "upstream doesn't see connection failures" is
> because connection failures are only seen at the second layer (the
> log line above belongs to the second layer). The first layer will
> only see a connection close, and it won't know if there was an
> error or not.
>
> Also note:
>
> - You use $proxy_protocol_addr in the "upstream mag {...}" block,
> but the upstream block is used only in the first layer, where
> $proxy_protocol_addr won't be available according to your
> configuration.
>
> - You use $name in the logs of the second layer. It will always
> point to "map", as there is no ssl_preread in the second layer,
> hence $ssl_preread_server_name will be not available.
>
> Depending on what you actually want to achieve, the most
> straightforward solution might be to actually remove the second
> proxy layer.
Hello,

The proxy protocol was used for the "non-stream" routing on SNI when
forwarding to nginx itself as "local_https". At this point it's using
regular https vhost, that's why I added proxy_protocol to easily be able
to extract the original client address.

Aim of the two servers on 8080 and 8181 are only to strip proxy_protocol
before going to upstream mag. I'd be happy to remove them but if I do
that I need a way to strip out proxy_protocol inside the "upstream mag"
block. Is it possible ?

Thanks a lot,

Adam.


_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Hello!

On Tue, Jan 09, 2018 at 11:48:54PM +0100, Adam Cecile wrote:

> On 01/09/2018 02:46 PM, Maxim Dounin wrote:
> > Hello!
> >
> > On Mon, Jan 08, 2018 at 12:37:41PM +0000, Cecile, Adam wrote:
> >
> >> Hello,
> >>
> >>
> >> I'm using this quite complicated setup involving SNI routing and proxy_protocol but I'm stuck on something.
> >>
> >>
> >> Here is the configuration file:
> >>
> >> http://paste.debian.net/hidden/62e13f9c/
> >>
> >>
> >> Routing, proxy_protocol, logging stuff is working just fine, the only (quite critical issue) is that the "mag" upstream doesn't see connection failures and does not switch to the second server.
> >>
> >>
> >> In the mag.log file I just see:
> >>
> >> 98.98.98.98 [08/Jan/2018:10:56:10 +0100] proxying to "mag":10.0.0.1:443 TCP 500 0 239 1.01
> >>
> >>
> >> But instead of blacklisting this server and moving to 10.0.0.2 I receive a connection closed error on the client.
> > As far as I understand your configuration, you have two stream
> > proxy layers:
> >
> > 1. The first one uses ssl_preread to obtain SNI name and tries to
> > do some routing based on it. This layer also adds to the PROXY
> > protocol to backend connections.
> >
> > 2. The second one strips PROXY protocol header.
> >
> > The problem with "upstream doesn't see connection failures" is
> > because connection failures are only seen at the second layer (the
> > log line above belongs to the second layer). The first layer will
> > only see a connection close, and it won't know if there was an
> > error or not.
> >
> > Also note:
> >
> > - You use $proxy_protocol_addr in the "upstream mag {...}" block,
> > but the upstream block is used only in the first layer, where
> > $proxy_protocol_addr won't be available according to your
> > configuration.
> >
> > - You use $name in the logs of the second layer. It will always
> > point to "map", as there is no ssl_preread in the second layer,
> > hence $ssl_preread_server_name will be not available.
> >
> > Depending on what you actually want to achieve, the most
> > straightforward solution might be to actually remove the second
> > proxy layer.
> Hello,
>
> The proxy protocol was used for the "non-stream" routing on SNI when
> forwarding to nginx itself as "local_https". At this point it's using
> regular https vhost, that's why I added proxy_protocol to easily be able
> to extract the original client address.
>
> Aim of the two servers on 8080 and 8181 are only to strip proxy_protocol
> before going to upstream mag. I'd be happy to remove them but if I do
> that I need a way to strip out proxy_protocol inside the "upstream mag"
> block. Is it possible ?

Ok, so you use multiple proxy layers to be able to combine
backends which support/need PROXY protocol and ones which do not,
right? This looks like a valid reason, as "proxy_protocol" is
either on or off in a particular server.

If you want nginx to switch to a different backend while
maintaining two proxy layers, consider moving balancing to the
second layer instead. This way balancing will happen where
connection errors can be seen, and so nginx will be able to switch
to a different server on errors.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
On 01/10/2018 05:54 PM, Maxim Dounin wrote:
> Hello!
>
> On Tue, Jan 09, 2018 at 11:48:54PM +0100, Adam Cecile wrote:
>
>> On 01/09/2018 02:46 PM, Maxim Dounin wrote:
>>> Hello!
>>>
>>> On Mon, Jan 08, 2018 at 12:37:41PM +0000, Cecile, Adam wrote:
>>>
>>>> Hello,
>>>>
>>>>
>>>> I'm using this quite complicated setup involving SNI routing and proxy_protocol but I'm stuck on something.
>>>>
>>>>
>>>> Here is the configuration file:
>>>>
>>>> http://paste.debian.net/hidden/62e13f9c/
>>>>
>>>>
>>>> Routing, proxy_protocol, logging stuff is working just fine, the only (quite critical issue) is that the "mag" upstream doesn't see connection failures and does not switch to the second server.
>>>>
>>>>
>>>> In the mag.log file I just see:
>>>>
>>>> 98.98.98.98 [08/Jan/2018:10:56:10 +0100] proxying to "mag":10.0.0.1:443 TCP 500 0 239 1.01
>>>>
>>>>
>>>> But instead of blacklisting this server and moving to 10.0.0.2 I receive a connection closed error on the client.
>>> As far as I understand your configuration, you have two stream
>>> proxy layers:
>>>
>>> 1. The first one uses ssl_preread to obtain SNI name and tries to
>>> do some routing based on it. This layer also adds to the PROXY
>>> protocol to backend connections.
>>>
>>> 2. The second one strips PROXY protocol header.
>>>
>>> The problem with "upstream doesn't see connection failures" is
>>> because connection failures are only seen at the second layer (the
>>> log line above belongs to the second layer). The first layer will
>>> only see a connection close, and it won't know if there was an
>>> error or not.
>>>
>>> Also note:
>>>
>>> - You use $proxy_protocol_addr in the "upstream mag {...}" block,
>>> but the upstream block is used only in the first layer, where
>>> $proxy_protocol_addr won't be available according to your
>>> configuration.
>>>
>>> - You use $name in the logs of the second layer. It will always
>>> point to "map", as there is no ssl_preread in the second layer,
>>> hence $ssl_preread_server_name will be not available.
>>>
>>> Depending on what you actually want to achieve, the most
>>> straightforward solution might be to actually remove the second
>>> proxy layer.
>> Hello,
>>
>> The proxy protocol was used for the "non-stream" routing on SNI when
>> forwarding to nginx itself as "local_https". At this point it's using
>> regular https vhost, that's why I added proxy_protocol to easily be able
>> to extract the original client address.
>>
>> Aim of the two servers on 8080 and 8181 are only to strip proxy_protocol
>> before going to upstream mag. I'd be happy to remove them but if I do
>> that I need a way to strip out proxy_protocol inside the "upstream mag"
>> block. Is it possible ?
> Ok, so you use multiple proxy layers to be able to combine
> backends which support/need PROXY protocol and ones which do not,
> right? This looks like a valid reason, as "proxy_protocol" is
> either on or off in a particular server.
Yes exactly !

Aim of this setup is to do SNI routing to TCP endpoints (with failover)
or HTTPS virtual hosts.
>
> If you want nginx to switch to a different backend while
> maintaining two proxy layers, consider moving balancing to the
> second layer instead. This way balancing will happen where
> connection errors can be seen, and so nginx will be able to switch
> to a different server on errors.

Could you be more specific and show me how to do this with my current
configuration ? I'm a bit lost...

Thanks !
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Hello!

On Wed, Jan 10, 2018 at 07:18:36PM +0100, Adam Cecile wrote:

[...]

> > Ok, so you use multiple proxy layers to be able to combine
> > backends which support/need PROXY protocol and ones which do not,
> > right? This looks like a valid reason, as "proxy_protocol" is
> > either on or off in a particular server.
> Yes exactly !
>
> Aim of this setup is to do SNI routing to TCP endpoints (with failover)
> or HTTPS virtual hosts.
> >
> > If you want nginx to switch to a different backend while
> > maintaining two proxy layers, consider moving balancing to the
> > second layer instead. This way balancing will happen where
> > connection errors can be seen, and so nginx will be able to switch
> > to a different server on errors.
>
> Could you be more specific and show me how to do this with my current
> configuration ? I'm a bit lost...

At the first level, differentiate between hosts based on
$ssl_preread_server_name. Proxy to either "local_https" or to a
second-level server, say 8080. On the second level server, proxy
to an upstream group with servers you want to balance. Example
configuration (completely untested):

map $ssl_preread_server_name $name {
default local_https;
"" second;
pub.domain.com second;
}

upstream local_https {
server 127.0.0.1:8443;
}

upstream second {
server 127.0.0.1:8080;
}

upstream u {
server 10.0.0.1:443;
server 10.0.0.2:443;
}

server {
listen 443;
ssl_preread on;
proxy_pass $name;
proxy_protocol on;
}

server {
listen 127.0.0.1:8080 proxy_protocol;
proxy_pass u;
}

Logging and timeouts omitted for clarity.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
On 01/10/2018 07:58 PM, Maxim Dounin wrote:
> Hello!
>
> On Wed, Jan 10, 2018 at 07:18:36PM +0100, Adam Cecile wrote:
>
> [...]
>
>>> Ok, so you use multiple proxy layers to be able to combine
>>> backends which support/need PROXY protocol and ones which do not,
>>> right? This looks like a valid reason, as "proxy_protocol" is
>>> either on or off in a particular server.
>> Yes exactly !
>>
>> Aim of this setup is to do SNI routing to TCP endpoints (with failover)
>> or HTTPS virtual hosts.
>>> If you want nginx to switch to a different backend while
>>> maintaining two proxy layers, consider moving balancing to the
>>> second layer instead. This way balancing will happen where
>>> connection errors can be seen, and so nginx will be able to switch
>>> to a different server on errors.
>> Could you be more specific and show me how to do this with my current
>> configuration ? I'm a bit lost...
> At the first level, differentiate between hosts based on
> $ssl_preread_server_name. Proxy to either "local_https" or to a
> second-level server, say 8080. On the second level server, proxy
> to an upstream group with servers you want to balance. Example
> configuration (completely untested):
>
> map $ssl_preread_server_name $name {
> default local_https;
> "" second;
> pub.domain.com second;
> }
>
> upstream local_https {
> server 127.0.0.1:8443;
> }
>
> upstream second {
> server 127.0.0.1:8080;
> }
>
> upstream u {
> server 10.0.0.1:443;
> server 10.0.0.2:443;
> }
>
> server {
> listen 443;
> ssl_preread on;
> proxy_pass $name;
> proxy_protocol on;
> }
>
> server {
> listen 127.0.0.1:8080 proxy_protocol;
> proxy_pass u;
> }
>
> Logging and timeouts omitted for clarity.
>
Very nice !

I'll give a try tomorrow morning and let you know, thanks.

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Hello,


I tried the following configuration but it's still not working as expected. Client browser receive connection closed and the failing server does not seem to get blacklisted...


Any idea ?


Thanks



map $ssl_preread_server_name $name {
default local_https;
"" mag_strip_proxy_protocol;
maintenance.domain.com mag_strip_proxy_protocol;
}


upstream mag {
server 10.0.0.32:443 max_conns=10 weight=100 max_fails=1 fail_timeout=300;

server 10.0.0.33:443 max_conns=10 weight=50 max_fails=1 fail_timeout=300;

server 10.0.0.26:443 max_conns=10 weight=10 max_fails=1 fail_timeout=300 backup;
server 10.0.0.27:443 max_conns=10 weight=10 max_fails=1 fail_timeout=300 backup;
}

upstream mag_strip_proxy_protocol {
server 127.0.0.1:8080;
}

upstream local_https {
server 127.0.0.1:8443;
}


log_format stream_routing '$remote_addr [$time_local] '
'with SNI name "$ssl_preread_server_name" '
'proxying to "$name" '
'$protocol $status $bytes_sent $bytes_received '
'$session_time';

log_format upstream_routing '$proxy_protocol_addr [$time_local] '
'proxying to "mag":$upstream_addr '
'$protocol $status $bytes_sent $bytes_received '
'$session_time';


server {
listen 443;
ssl_preread on;
proxy_pass $name;
proxy_protocol on;
access_log /var/log/nginx/stream_443.log stream_routing;
}

server {
listen 8080 proxy_protocol;
proxy_pass mag;
access_log /var/log/nginx/mag.log upstream_routing;
}


________________________________
De : nginx <[email protected]> de la part de Adam Cecile <[email protected]>
Envoyé : mercredi 10 janvier 2018 20:02:59
À : nginx@nginx.org; Maxim Dounin
Objet : Re: upstream (tcp stream mode) doesn't detect connecton failure

[This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing]

On 01/10/2018 07:58 PM, Maxim Dounin wrote:
> Hello!
>
> On Wed, Jan 10, 2018 at 07:18:36PM +0100, Adam Cecile wrote:
>
> [...]
>
>>> Ok, so you use multiple proxy layers to be able to combine
>>> backends which support/need PROXY protocol and ones which do not,
>>> right? This looks like a valid reason, as "proxy_protocol" is
>>> either on or off in a particular server.
>> Yes exactly !
>>
>> Aim of this setup is to do SNI routing to TCP endpoints (with failover)
>> or HTTPS virtual hosts.
>>> If you want nginx to switch to a different backend while
>>> maintaining two proxy layers, consider moving balancing to the
>>> second layer instead. This way balancing will happen where
>>> connection errors can be seen, and so nginx will be able to switch
>>> to a different server on errors.
>> Could you be more specific and show me how to do this with my current
>> configuration ? I'm a bit lost...
> At the first level, differentiate between hosts based on
> $ssl_preread_server_name. Proxy to either "local_https" or to a
> second-level server, say 8080. On the second level server, proxy
> to an upstream group with servers you want to balance. Example
> configuration (completely untested):
>
> map $ssl_preread_server_name $name {
> default local_https;
> "" second;
> pub.domain.com second;
> }
>
> upstream local_https {
> server 127.0.0.1:8443;
> }
>
> upstream second {
> server 127.0.0.1:8080;
> }
>
> upstream u {
> server 10.0.0.1:443;
> server 10.0.0.2:443;
> }
>
> server {
> listen 443;
> ssl_preread on;
> proxy_pass $name;
> proxy_protocol on;
> }
>
> server {
> listen 127.0.0.1:8080 proxy_protocol;
> proxy_pass u;
> }
>
> Logging and timeouts omitted for clarity.
>
Very nice !

I'll give a try tomorrow morning and let you know, thanks.

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Sorry, only registered users may post in this forum.

Click here to login