Welcome! Log In Create A New Profile

Advanced

Resolver not re-resolving new ip address of an AW ELB

Posted by RKGood 
Hello,

We are trying to find a solution from past couple of days but nothing seems
to work so far. Our server in prod going down everyday when the AWS ELB
changes ip address. Our's little complex proxy (don't ask me why we have to
do this :), there is a strong reason for it), our clients will send requests
apache (legacy), apache proxies to nginx (new) and nginx decides whether
proxy back to apache or serve the request with new micro services. Resolver
re-resolves the new micro services (internal alb) ip address but fail to
re-resolve the legacy apache (has a ELB with route 53 entry in front). We
are using https endpoint to proxy apache request. The request flows thru ELB
(legacy) -> Apache -> ELB (new) -> nginx -> ELB (legacy) -> apache

Can you please provide feedback on what are we doing wrong, this is only
happening in production. Our load is normal few fundred requests per second.
We aren't able to simulate it in test environment.

Here is the configuration:


user nginx;
worker_processes auto;
worker_rlimit_nofile 5120;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;

events {
worker_connections 2048;
}

http {
include /etc/nginx/mime.types;
default_type text/plain;

resolver 10....2 172.16.0.23 valid=30s ipv6=off;
resolver_timeout 5s;

log_format main '$proxy_protocol_addr - [$status] - [$request_time] -
[$upstream_response_time] - $server_name $upstream_addr $request';

access_log /var/log/nginx/error.log main;

rewrite_log on;

client_body_timeout 60s;
client_header_timeout 30s;
send_timeout 60s;
sendfile off;

tcp_nodelay on;
tcp_nopush on;
reset_timedout_connection on;

server_names_hash_bucket_size 128;
client_body_buffer_size 64k;
client_max_body_size 10m;

server {

listen 443 ssl proxy_protocol default_server;
server_name mydomain.com;
ssl_certificate mydomain.crt;
ssl_certificate_key mydomain.key;

set $alb_upstream aws-internal-alb;
set $apache_upstream legacy.domain.com;

proxy_buffers 8 24k;
proxy_buffer_size 2k;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header X-Real-IP $proxy_protocol_addr;
proxy_next_upstream off;


location /services/(migrated1|migrated2)/ {

proxy_set_header Host $host;
proxy_connect_timeout 302;
proxy_read_timeout 302;

rewrite /services/(.*) /$1?$args break;
proxy_pass http://alb_upstream;
}

location /services/ {
proxy_set_header x-nginx-rejected true;
proxy_set_header Host legacy.domain.com;
proxy_connect_timeout 302;
proxy_read_timeout 302;

rewrite /services/(.*) /$1?$args break;
proxy_pass https://$apache_upstream;
}

}
}

Thanks in advance.
RK

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,277101,277101#msg-277101

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Forgot to mention imporant information: we are using dockerized nginx,
deployed in ECS cluster. Docker nginx image version is 1.11

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,277101,277102#msg-277102

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Can someone help please. We been having issues everyday and left with no
other options to try

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,277101,277132#msg-277132

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Nishikubo Minoru
Re: Resolver not re-resolving new ip address of an AW ELB
October 31, 2017 02:20AM
Hello,

I checked my old note and configured as follows, but I doubt myself that
will help you...

- Configured the resolver of internal reverse proxy to internal ALB/ELB to
AWS dedicated DNS server.
- Configured the resolver cache 30 seconds.
- Set proxy_pass arguments to ALB/ELB endpoint names.

Here is my old note:

The resolver 172.31.0.2 is the address of AWS dedicated DNS server. I
checked the response time of resolve accross the availability zone with
dig, the AWS dedicated DNS server will respond without difference in spite
of our cache resolver.

TTL of DNS record in AWS is 60seconds, ALB(ELB) will scale in/out and
update their address, so the internal resolver in nginx will keep 30
seconds, that is valid=30s with resolver directive.

resolver 172.31.0.2 valid=30s;
resolver_timeout 10s;
set $backendelb "
backendelb-87654321.elasticloaodbalancing.region.amazonaws.com";
location / { proxy_pass http://$backendelb; }

On Sat, Oct 28, 2017 at 3:58 AM, RKGood <[email protected]> wrote:

> Hello,
>
> We are trying to find a solution from past couple of days but nothing seems
> to work so far. Our server in prod going down everyday when the AWS ELB
> changes ip address. Our's little complex proxy (don't ask me why we have to
> do this :), there is a strong reason for it), our clients will send
> requests
> apache (legacy), apache proxies to nginx (new) and nginx decides whether
> proxy back to apache or serve the request with new micro services. Resolver
> re-resolves the new micro services (internal alb) ip address but fail to
> re-resolve the legacy apache (has a ELB with route 53 entry in front). We
> are using https endpoint to proxy apache request. The request flows thru
> ELB
> (legacy) -> Apache -> ELB (new) -> nginx -> ELB (legacy) -> apache
>
> Can you please provide feedback on what are we doing wrong, this is only
> happening in production. Our load is normal few fundred requests per
> second.
> We aren't able to simulate it in test environment.
>
> Here is the configuration:
>
>
> user nginx;
> worker_processes auto;
> worker_rlimit_nofile 5120;
> error_log /var/log/nginx/error.log warn;
> pid /var/run/nginx.pid;
>
> events {
> worker_connections 2048;
> }
>
> http {
> include /etc/nginx/mime.types;
> default_type text/plain;
>
> resolver 10....2 172.16.0.23 valid=30s ipv6=off;
> resolver_timeout 5s;
>
> log_format main '$proxy_protocol_addr - [$status] - [$request_time] -
> [$upstream_response_time] - $server_name $upstream_addr $request';
>
> access_log /var/log/nginx/error.log main;
>
> rewrite_log on;
>
> client_body_timeout 60s;
> client_header_timeout 30s;
> send_timeout 60s;
> sendfile off;
>
> tcp_nodelay on;
> tcp_nopush on;
> reset_timedout_connection on;
>
> server_names_hash_bucket_size 128;
> client_body_buffer_size 64k;
> client_max_body_size 10m;
>
> server {
>
> listen 443 ssl proxy_protocol default_server;
> server_name mydomain.com;
> ssl_certificate mydomain.crt;
> ssl_certificate_key mydomain.key;
>
> set $alb_upstream aws-internal-alb;
> set $apache_upstream legacy.domain.com;
>
> proxy_buffers 8 24k;
> proxy_buffer_size 2k;
> proxy_http_version 1.1;
> proxy_set_header Connection "";
> proxy_set_header X-Real-IP $proxy_protocol_addr;
> proxy_next_upstream off;
>
>
> location /services/(migrated1|migrated2)/ {
>
> proxy_set_header Host $host;
> proxy_connect_timeout 302;
> proxy_read_timeout 302;
>
> rewrite /services/(.*) /$1?$args break;
> proxy_pass http://alb_upstream;
> }
>
> location /services/ {
> proxy_set_header x-nginx-rejected true;
> proxy_set_header Host legacy.domain.com;
> proxy_connect_timeout 302;
> proxy_read_timeout 302;
>
> rewrite /services/(.*) /$1?$args break;
> proxy_pass https://$apache_upstream;
> }
>
> }
> }
>
> Thanks in advance.
> RK
>
> Posted at Nginx Forum: https://forum.nginx.org/read.
> php?2,277101,277101#msg-277101
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
you have two choices if the variable use is not working and you specifically
need to use upstream config option.

1. Subscribe to Nginx Plus.
2. Compile this module along with Nginx ..
https://github.com/aziontech/nginx-upstream-dynamic-servers

I have personally used this module and it works.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,277101,277147#msg-277147

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Thank you for your replies. I think we have found the root cause. We have
found below:

When you are using variables in a proxy_pass directive, nginx will use
runtime resolving except if :
the target server is declared as an IP address
the target server name is part of an upstream server group
the target server name has already been resolved (e.g. it matches a
server name in another server block)

In our case we have one server block, with default set. Apache is sending
server name as legacy.domain.com and nginx also proxying to
legacy.domain.com, so I belive because both server name and proxy host is
same, nginx not trying to re-resolve the ip address.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,277101,277164#msg-277164

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Ruslan Ermilov
Re: Resolver not re-resolving new ip address of an AW ELB
November 02, 2017 11:10AM
On Tue, Oct 31, 2017 at 01:26:45PM -0400, RKGood wrote:
> Thank you for your replies. I think we have found the root cause. We have
> found below:
>
> When you are using variables in a proxy_pass directive, nginx will use
> runtime resolving except if :
> the target server is declared as an IP address

There's nothing to resolve if it's an address.

> the target server name is part of an upstream server group

Not part, but the upstream group name. This is documented in the
http://nginx.org/r/proxy_pass directive as follows:

: Parameter value can contain variables. In this case, if an address
: is specified as a domain name, the name is searched among the
: described server groups, and, if not found, is determined using
: a resolver.

> the target server name has already been resolved (e.g. it matches a
> server name in another server block)

nginx has its own resolver cache. By default, nginx uses DNS TTL.
If that's not desirable for some reasons, you can control the validity
of cached entries by using the "valid" parameter of the "resolver"
directive, please see http://nginx.org/r/resolver for details.
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Sorry, only registered users may post in this forum.

Click here to login