Welcome! Log In Create A New Profile

Advanced

Using the mirror module

Posted by Kenny Meyer 
Kenny Meyer
Using the mirror module
March 13, 2018 09:40PM
Hi,

I’m having trouble using the new mirror module. I want to mirror incoming requests from Nginx to other upstream servers. 1) a production server 2) a staging server

This is my config:

server {
listen 80 default_server;
listen [::]:80 default_server;

location / {
mirror /mirror;
proxy_pass http://www.example.com;
}

location /mirror {
internal;
proxy_pass http://staging.example.com$request_uri;
}
}

So, I request http://myserver.com (where Nginx is hosted) and it successfully redirects me to www.example.com, however I don’t see any requests hitting staging.example.com.

What could be the error?
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Roman Arutyunyan
Re: Using the mirror module
March 13, 2018 10:40PM
Hi Kenny,

On Tue, Mar 13, 2018 at 05:37:52PM -0300, Kenny Meyer wrote:
> Hi,
>
> I’m having trouble using the new mirror module. I want to mirror incoming requests from Nginx to other upstream servers. 1) a production server 2) a staging server
>
> This is my config:
>
> server {
> listen 80 default_server;
> listen [::]:80 default_server;
>
> location / {
> mirror /mirror;
> proxy_pass http://www.example.com;
> }
>
> location /mirror {
> internal;
> proxy_pass http://staging.example.com$request_uri;
> }
> }
>
> So, I request http://myserver.com (where Nginx is hosted) and it successfully redirects me to www.example.com, however I don’t see any requests hitting staging.example.com.
>
> What could be the error?

The configuration looks fine.
Are there any errors in error.log?
And what happens if you switch www.example.com and staging.example.com?

--
Roman Arutyunyan
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Kenny Meyer
Re: Using the mirror module
March 13, 2018 11:10PM
Hi Roman,

> Are there any errors in error.log?
No errors…

> And what happens if you switch www.example.com and staging.example.com?
Then I get redirected to staging.example.com and I don’t see any requests being logged on example.com



> On 13 Mar, 2018, at 18:34, Roman Arutyunyan <[email protected]> wrote:
>
> Hi Kenny,
>
> On Tue, Mar 13, 2018 at 05:37:52PM -0300, Kenny Meyer wrote:
>> Hi,
>>
>> I’m having trouble using the new mirror module. I want to mirror incoming requests from Nginx to other upstream servers. 1) a production server 2) a staging server
>>
>> This is my config:
>>
>> server {
>> listen 80 default_server;
>> listen [::]:80 default_server;
>>
>> location / {
>> mirror /mirror;
>> proxy_pass http://www.example.com;
>> }
>>
>> location /mirror {
>> internal;
>> proxy_pass http://staging.example.com$request_uri;
>> }
>> }
>>
>> So, I request http://myserver.com (where Nginx is hosted) and it successfully redirects me to www.example.com, however I don’t see any requests hitting staging.example.com.
>>
>> What could be the error?
>
> The configuration looks fine.
> Are there any errors in error.log?
> And what happens if you switch www.example.com and staging.example.com?
>
> --
> Roman Arutyunyan
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

Kenny Meyer
www.kennymeyer.net

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Peter Booth
Re: Using the mirror module
March 13, 2018 11:40PM
This is the point where I would jump to using the debug log. You need to build you nginx binary
with —with-debug switch and change the log level to debug innginx.conf. Debug generates a *huge*
amount of logs but it really is invaluable.

I would also want to double check what is actually happening and use ss or tcpdump
to confirm that no request is sent to your staging destination.

I’m assuming that both ww.example.com and staging.example.com are hosted
on different hosts, different IPs and are both functional.

Peter



> On Mar 13, 2018, at 5:58 PM, Kenny Meyer <[email protected]> wrote:
>
> Hi Roman,
>
>> Are there any errors in error.log?
> No errors…
>
>> And what happens if you switch www.example.com and staging.example.com?
> Then I get redirected to staging.example.com and I don’t see any requests being logged on example.com
>
>
>
>> On 13 Mar, 2018, at 18:34, Roman Arutyunyan <[email protected]> wrote:
>>
>> Hi Kenny,
>>
>> On Tue, Mar 13, 2018 at 05:37:52PM -0300, Kenny Meyer wrote:
>>> Hi,
>>>
>>> I’m having trouble using the new mirror module. I want to mirror incoming requests from Nginx to other upstream servers. 1) a production server 2) a staging server
>>>
>>> This is my config:
>>>
>>> server {
>>> listen 80 default_server;
>>> listen [::]:80 default_server;
>>>
>>> location / {
>>> mirror /mirror;
>>> proxy_pass http://www.example.com;
>>> }
>>>
>>> location /mirror {
>>> internal;
>>> proxy_pass http://staging.example.com$request_uri;
>>> }
>>> }
>>>
>>> So, I request http://myserver.com (where Nginx is hosted) and it successfully redirects me to www.example.com, however I don’t see any requests hitting staging.example.com.
>>>
>>> What could be the error?
>>
>> The configuration looks fine.
>> Are there any errors in error.log?
>> And what happens if you switch www.example.com and staging.example.com?
>>
>> --
>> Roman Arutyunyan
>> _______________________________________________
>> nginx mailing list
>> nginx@nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx
>
> Kenny Meyer
> www.kennymeyer.net
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Roman Arutyunyan
Re: Using the mirror module
March 13, 2018 11:40PM
On Tue, Mar 13, 2018 at 06:58:25PM -0300, Kenny Meyer wrote:
> Hi Roman,
>
> > Are there any errors in error.log?
> No errors…
>
> > And what happens if you switch www.example.com and staging.example.com?
> Then I get redirected to staging.example.com and I don’t see any requests being logged on example.com

Do you have a resolver defined in the http{} block?
Proxying to "http://staging.example.com$request_uri"; requires a resolver.
Normally, you would have "no resolver defined to resolve ..."
error logged if resolver is missing, but you may not see it if, for example,
your log level is too high.

> > On 13 Mar, 2018, at 18:34, Roman Arutyunyan <[email protected]> wrote:
> >
> > Hi Kenny,
> >
> > On Tue, Mar 13, 2018 at 05:37:52PM -0300, Kenny Meyer wrote:
> >> Hi,
> >>
> >> I’m having trouble using the new mirror module. I want to mirror incoming requests from Nginx to other upstream servers. 1) a production server 2) a staging server
> >>
> >> This is my config:
> >>
> >> server {
> >> listen 80 default_server;
> >> listen [::]:80 default_server;
> >>
> >> location / {
> >> mirror /mirror;
> >> proxy_pass http://www.example.com;
> >> }
> >>
> >> location /mirror {
> >> internal;
> >> proxy_pass http://staging.example.com$request_uri;
> >> }
> >> }
> >>
> >> So, I request http://myserver.com (where Nginx is hosted) and it successfully redirects me to www.example.com, however I don’t see any requests hitting staging.example.com.
> >>
> >> What could be the error?
> >
> > The configuration looks fine.
> > Are there any errors in error.log?
> > And what happens if you switch www.example.com and staging.example.com?
> >
> > --
> > Roman Arutyunyan
> > _______________________________________________
> > nginx mailing list
> > nginx@nginx.org
> > http://mailman.nginx.org/mailman/listinfo/nginx
>
> Kenny Meyer
> www.kennymeyer.net
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

--
Roman Arutyunyan
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Kenny Meyer
Re: Using the mirror module
March 14, 2018 01:00PM
How do you define a resolver?

> On 13 Mar, 2018, at 19:36, Roman Arutyunyan <[email protected]> wrote:
>
> On Tue, Mar 13, 2018 at 06:58:25PM -0300, Kenny Meyer wrote:
>> Hi Roman,
>>
>>> Are there any errors in error.log?
>> No errors…
>>
>>> And what happens if you switch www.example.com and staging.example.com?
>> Then I get redirected to staging.example.com and I don’t see any requests being logged on example.com
>
> Do you have a resolver defined in the http{} block?
> Proxying to "http://staging.example.com$request_uri"; requires a resolver.
> Normally, you would have "no resolver defined to resolve ..."
> error logged if resolver is missing, but you may not see it if, for example,
> your log level is too high.
>
>>> On 13 Mar, 2018, at 18:34, Roman Arutyunyan <[email protected]> wrote:
>>>
>>> Hi Kenny,
>>>
>>> On Tue, Mar 13, 2018 at 05:37:52PM -0300, Kenny Meyer wrote:
>>>> Hi,
>>>>
>>>> I’m having trouble using the new mirror module. I want to mirror incoming requests from Nginx to other upstream servers. 1) a production server 2) a staging server
>>>>
>>>> This is my config:
>>>>
>>>> server {
>>>> listen 80 default_server;
>>>> listen [::]:80 default_server;
>>>>
>>>> location / {
>>>> mirror /mirror;
>>>> proxy_pass http://www.example.com;
>>>> }
>>>>
>>>> location /mirror {
>>>> internal;
>>>> proxy_pass http://staging.example.com$request_uri;
>>>> }
>>>> }
>>>>
>>>> So, I request http://myserver.com (where Nginx is hosted) and it successfully redirects me to www.example.com, however I don’t see any requests hitting staging.example.com.
>>>>
>>>> What could be the error?
>>>
>>> The configuration looks fine.
>>> Are there any errors in error.log?
>>> And what happens if you switch www.example.com and staging.example.com?
>>>
>>> --
>>> Roman Arutyunyan
>>> _______________________________________________
>>> nginx mailing list
>>> nginx@nginx.org
>>> http://mailman.nginx.org/mailman/listinfo/nginx
>>
>> Kenny Meyer
>> www.kennymeyer.net
>>
>> _______________________________________________
>> nginx mailing list
>> nginx@nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx
>
> --
> Roman Arutyunyan
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

Kenny Meyer
www.kennymeyer.net

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Peter Booth
Re: Using the mirror module
March 14, 2018 01:40PM
Suggestion:

Define two more locations - one that proxies www.example.com and another that proxies staging.example.com. If both locations work then your problem is probably mirroring. If one doesn’t work then the issue is your configuration and not mirroring. Either way you have reduced the size of your problem space.

Peter

Sent from my iPhone

> On Mar 13, 2018, at 5:58 PM, Kenny Meyer <[email protected]> wrote:
>
> Hi Roman,
>
>> Are there any errors in error.log?
> No errors…
>
>> And what happens if you switch www.example.com and staging.example.com?
> Then I get redirected to staging.example.com and I don’t see any requests being logged on example.com
>
>
>
>> On 13 Mar, 2018, at 18:34, Roman Arutyunyan <[email protected]> wrote:
>>
>> Hi Kenny,
>>
>>> On Tue, Mar 13, 2018 at 05:37:52PM -0300, Kenny Meyer wrote:
>>> Hi,
>>>
>>> I’m having trouble using the new mirror module. I want to mirror incoming requests from Nginx to other upstream servers. 1) a production server 2) a staging server
>>>
>>> This is my config:
>>>
>>> server {
>>> listen 80 default_server;
>>> listen [::]:80 default_server;
>>>
>>> location / {
>>> mirror /mirror;
>>> proxy_pass http://www.example.com;
>>> }
>>>
>>> location /mirror {
>>> internal;
>>> proxy_pass http://staging.example.com$request_uri;
>>> }
>>> }
>>>
>>> So, I request http://myserver.com (where Nginx is hosted) and it successfully redirects me to www.example.com, however I don’t see any requests hitting staging.example.com.
>>>
>>> What could be the error?
>>
>> The configuration looks fine.
>> Are there any errors in error.log?
>> And what happens if you switch www.example.com and staging.example.com?
>>
>> --
>> Roman Arutyunyan
>> _______________________________________________
>> nginx mailing list
>> nginx@nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx
>
> Kenny Meyer
> www.kennymeyer.net
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Roman Arutyunyan
Re: Using the mirror module
March 14, 2018 01:40PM
On Wed, Mar 14, 2018 at 08:55:39AM -0300, Kenny Meyer wrote:
> How do you define a resolver?

Use the "resolver" directive:

http://nginx.org/en/docs/http/ngx_http_core_module.html#resolver

This only makes sense if your nginx resolves domain names in runtime.
In your configuration "staging.example.com" is resolved in runtime because
the entire URL contains a variable ($request_uri). However, if you put an ip
address instead, you will not need a resolver.

> > On 13 Mar, 2018, at 19:36, Roman Arutyunyan <[email protected]> wrote:
> >
> > On Tue, Mar 13, 2018 at 06:58:25PM -0300, Kenny Meyer wrote:
> >> Hi Roman,
> >>
> >>> Are there any errors in error.log?
> >> No errors…
> >>
> >>> And what happens if you switch www.example.com and staging.example.com?
> >> Then I get redirected to staging.example.com and I don’t see any requests being logged on example.com
> >
> > Do you have a resolver defined in the http{} block?
> > Proxying to "http://staging.example.com$request_uri"; requires a resolver.
> > Normally, you would have "no resolver defined to resolve ..."
> > error logged if resolver is missing, but you may not see it if, for example,
> > your log level is too high.
> >
> >>> On 13 Mar, 2018, at 18:34, Roman Arutyunyan <[email protected]> wrote:
> >>>
> >>> Hi Kenny,
> >>>
> >>> On Tue, Mar 13, 2018 at 05:37:52PM -0300, Kenny Meyer wrote:
> >>>> Hi,
> >>>>
> >>>> I’m having trouble using the new mirror module. I want to mirror incoming requests from Nginx to other upstream servers. 1) a production server 2) a staging server
> >>>>
> >>>> This is my config:
> >>>>
> >>>> server {
> >>>> listen 80 default_server;
> >>>> listen [::]:80 default_server;
> >>>>
> >>>> location / {
> >>>> mirror /mirror;
> >>>> proxy_pass http://www.example.com;
> >>>> }
> >>>>
> >>>> location /mirror {
> >>>> internal;
> >>>> proxy_pass http://staging.example.com$request_uri;
> >>>> }
> >>>> }
> >>>>
> >>>> So, I request http://myserver.com (where Nginx is hosted) and it successfully redirects me to www.example.com, however I don’t see any requests hitting staging.example.com.
> >>>>
> >>>> What could be the error?
> >>>
> >>> The configuration looks fine.
> >>> Are there any errors in error.log?
> >>> And what happens if you switch www.example.com and staging.example.com?
> >>>
> >>> --
> >>> Roman Arutyunyan
> >>> _______________________________________________
> >>> nginx mailing list
> >>> nginx@nginx.org
> >>> http://mailman.nginx.org/mailman/listinfo/nginx
> >>
> >> Kenny Meyer
> >> www.kennymeyer.net
> >>
> >> _______________________________________________
> >> nginx mailing list
> >> nginx@nginx.org
> >> http://mailman.nginx.org/mailman/listinfo/nginx
> >
> > --
> > Roman Arutyunyan
> > _______________________________________________
> > nginx mailing list
> > nginx@nginx.org
> > http://mailman.nginx.org/mailman/listinfo/nginx
>
> Kenny Meyer
> www.kennymeyer.net
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

--
Roman Arutyunyan
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
jlangr
Re: Using the mirror module
August 09, 2018 09:20PM
Thanks for the posts Kenny & pbooth.

I have this problem as well. I am running nginx 1.15 under MacOS and also
experience this under Windows.

Requests successfully go to the server specified under the location but are
not received at the proxy_pass defined in the mirror location's proxy_pass.
Switching between the two servers (swapping the mirror for the proxy_pass in
the primary location) exhibits the same problem.

My config is:
```
http {
server {
listen 3333;
server_name localhost;

location / {
mirror /other;
proxy_pass http://localhost:3001;
}

location = /other {
internal;
proxy_pass http://localhost:3002;
}
}
}

events {
}
```
Based on your other post, it does not look like I need a resolver.

I've also tried with two servers--one localhost and one established server
on another machine.

Any help greatly appreciated!

Jeff

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,279036,280828#msg-280828

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
jlangr
Re: Using the mirror module
August 09, 2018 11:10PM
hmm looks like the mirror kills the url. I need to understand how to do URL
rewrites, probably, but the original request's path isn't coming across to
the mirror location.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,279036,280829#msg-280829

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Francis Daly
Re: Using the mirror module
August 10, 2018 12:30AM
On Thu, Aug 09, 2018 at 05:08:25PM -0400, jlangr wrote:

Hi there,

> hmm looks like the mirror kills the url. I need to understand how to do URL
> rewrites, probably, but the original request's path isn't coming across to
> the mirror location.

Does it work for you if you use the Example Configuration in the
documentation?

http://nginx.org/en/docs/http/ngx_http_mirror_module.html

f
--
Francis Daly francis@daoine.org
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
jlangr
Re: Using the mirror module
August 10, 2018 01:10AM
Hi Francis--

We got this working to wrap up the day. My ignorance of nginx meant that I
was interpreting the example literally--I viewed the "/mirror" as the name
for the internal route, but that actually becomes part of the URL the mirror
location proxy forwards to.

http {
server {
listen 3333;
server_name localhost;

location / {
mirror /;
resolver 8.8.8.8; # I shouldn't need this, will test tomorrow
proxy_pass http://1.2.3.4:3001;
}

location = / {
internal;
resolver 8.8.8.8;
set $upstream_endpoint http://1.2.3.4:3002;
proxy_pass $upstream_endpoint$request_uri;
}
}
}

That works like a champ. Initially I was trying to use localhost as the
proxy_pass destination, and had turned on port forwarding at my router, but
for whatever reason that was having troubles.

Many thanks in any case!

Jeff

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,279036,280832#msg-280832

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
jlangr
Re: Using the mirror module
August 10, 2018 01:10AM
ps how do you format code in these posts? (It's not markdown)

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,279036,280833#msg-280833

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Francis Daly
Re: Using the mirror module
August 14, 2018 09:50AM
On Thu, Aug 09, 2018 at 07:04:50PM -0400, jlangr wrote:

Hi there,

> We got this working to wrap up the day.

Good that you have something working for you.

> My ignorance of nginx meant that I
> was interpreting the example literally--I viewed the "/mirror" as the name
> for the internal route, but that actually becomes part of the URL the mirror
> location proxy forwards to.

It's actually a bit more subtle than that.

The argument to the "mirror" directive is the local url of the subrequest;
it should start with "/" or "@".

Then, in the matching location{}, if you choose to proxy_pass, then the
proxy_pass rules come in to play. And it is proxy_pass that decides what
request to make to the upstream server.

http://nginx.org/r/proxy_pass

> mirror /;

That means that the subrequest will be exactly "/"...

> location = / {

....which will be handled in this location...

> set $upstream_endpoint http://1.2.3.4:3002;
> proxy_pass $upstream_endpoint$request_uri;

....and the proxy_pass argument includes something after the host:port,
so that is what the request to upstream will be.

An alternative could have been to use a named location -- "mirror
@mirror;", with "location @mirror { proxy_pass http://1.2.3.4:3002; }"
-- which would avoid the need for some of the extra variables.


But, once you have something that works, there is no need to change it :-)

Cheers,

f
--
Francis Daly francis@daoine.org
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
jlangr
Re: Using the mirror module
August 14, 2018 04:10PM
thanks Francis for the additional info, very helpful!

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,279036,280872#msg-280872

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Sorry, only registered users may post in this forum.

Click here to login