Welcome! Log In Create A New Profile

Advanced

Propagating agent-check weight change to tracking servers

Posted by Michał 
Hello,

We use track's in haproxy to minimize check traffic in some situations and
after my last patch we are probably going to switch to agent-checks for
live management of weights and statuses. One problem I see now - track
don't propagate weight setting to trackers, so if we set agent-check on
track we can manage status only.

My first PoC solution works good, so I thought about introducing something
like agent-track or track-agent directive set on backends (or maybe it
should be default, non-configurable behaviour) to propagate agent-check
responses from main check to all tracking backends. Both default behaviour
and directive approach are small changes in code, but a little bigger in
functionality.

In my opinion if we set agent-check on backend which is tracked by others -
it should propagate agent-check weight response to those tracking backend
and set weight on them too. Configurable or not - it will be good feature.

Michał
Willy Tarreau
Re: Propagating agent-check weight change to tracking servers
January 20, 2017 11:00AM
Hi Michal,

On Thu, Jan 19, 2017 at 11:45:57PM +0100, Micha?? wrote:
> Hello,
>
> We use track's in haproxy to minimize check traffic in some situations and
> after my last patch we are probably going to switch to agent-checks for
> live management of weights and statuses. One problem I see now - track
> don't propagate weight setting to trackers, so if we set agent-check on
> track we can manage status only.
>
> My first PoC solution works good, so I thought about introducing something
> like agent-track or track-agent directive set on backends (or maybe it
> should be default, non-configurable behaviour) to propagate agent-check
> responses from main check to all tracking backends. Both default behaviour
> and directive approach are small changes in code, but a little bigger in
> functionality.
>
> In my opinion if we set agent-check on backend which is tracked by others -
> it should propagate agent-check weight response to those tracking backend
> and set weight on them too. Configurable or not - it will be good feature.

I think we at least propagate the DRAIN state which is equivalent to
weight == 0. If so I too think we should propagate *relative* weights.
Agent-checks can return a relative weight (eg: 50%, 100%, 150%) or an
absolute weight (eg: 10, 20). If you have two farms configured like this :

backend farm1
server new1 1.1.1.1:8000 weight 10 agent-check
server new2 1.1.1.2:8000 weight 10 agent-check

backend farm2
server new1 1.1.1.1:8000 weight 20 track farm1/new1
server new2 1.1.1.2:8000 weight 20 track farm1/new2
server old1 1.1.1.3:8000 weight 10 check
server old2 1.1.1.4:8000 weight 10 check

Then you want the weight changes on farm1 to be applied proportionally
to farm2 (ie: a ratio of the configured absolute weight, which is iweight
IIRC).

Otherwise that sounds quite reasonable to me given that the agent-check's
purpose is to provide a more accurate vision of the server's health, and
that tracking is made to share the same vision across multiple farms.

Regards,
Willy
Hello,

So here's patch, which includes all functionalities I think about.
It propagates the response for every tracking server without changing it
and without intercepting it. In my opinion we should propagate relative
and absolute weights, because if you use weight=0 server's to offload
checks then you can send relative weight change and 0 will stay where it is..

Regards,
Michał


2017-01-20 10:54 GMT+01:00 Willy Tarreau <[email protected]>:

> Hi Michal,
>
> On Thu, Jan 19, 2017 at 11:45:57PM +0100, Micha?? wrote:
> > Hello,
> >
> > We use track's in haproxy to minimize check traffic in some situations
> and
> > after my last patch we are probably going to switch to agent-checks for
> > live management of weights and statuses. One problem I see now - track
> > don't propagate weight setting to trackers, so if we set agent-check on
> > track we can manage status only.
> >
> > My first PoC solution works good, so I thought about introducing
> something
> > like agent-track or track-agent directive set on backends (or maybe it
> > should be default, non-configurable behaviour) to propagate agent-check
> > responses from main check to all tracking backends. Both default
> behaviour
> > and directive approach are small changes in code, but a little bigger in
> > functionality.
> >
> > In my opinion if we set agent-check on backend which is tracked by
> others -
> > it should propagate agent-check weight response to those tracking backend
> > and set weight on them too. Configurable or not - it will be good
> feature.
>
> I think we at least propagate the DRAIN state which is equivalent to
> weight == 0. If so I too think we should propagate *relative* weights..
> Agent-checks can return a relative weight (eg: 50%, 100%, 150%) or an
> absolute weight (eg: 10, 20). If you have two farms configured like this :
>
> backend farm1
> server new1 1.1.1.1:8000 weight 10 agent-check
> server new2 1.1.1.2:8000 weight 10 agent-check
>
> backend farm2
> server new1 1.1.1.1:8000 weight 20 track farm1/new1
> server new2 1.1.1.2:8000 weight 20 track farm1/new2
> server old1 1.1.1.3:8000 weight 10 check
> server old2 1.1.1.4:8000 weight 10 check
>
> Then you want the weight changes on farm1 to be applied proportionally
> to farm2 (ie: a ratio of the configured absolute weight, which is iweight
> IIRC).
>
> Otherwise that sounds quite reasonable to me given that the agent-check's
> purpose is to provide a more accurate vision of the server's health, and
> that tracking is made to share the same vision across multiple farms.
>
> Regards,
> Willy
>
Attachments:
open | download - 0001-MINOR-checks-propagate-agent-check-weight-to-tracker.patch (1.3 KB)
Hi,
I checked it and during synthetic tests it worked. I use same
mechanism as origin agent-check, so it's ready to merge.

It doesn't need to be backported.

2017-01-27 15:38 GMT+01:00 Michał <[email protected]>:

> Hello,
>
> So here's patch, which includes all functionalities I think about.
> It propagates the response for every tracking server without changing it
> and without intercepting it. In my opinion we should propagate relative
> and absolute weights, because if you use weight=0 server's to offload
> checks then you can send relative weight change and 0 will stay where it
> is.
>
> Regards,
> Michał
>
>
> 2017-01-20 10:54 GMT+01:00 Willy Tarreau <[email protected]>:
>
>> Hi Michal,
>>
>> On Thu, Jan 19, 2017 at 11:45:57PM +0100, Micha?? wrote:
>> > Hello,
>> >
>> > We use track's in haproxy to minimize check traffic in some situations
>> and
>> > after my last patch we are probably going to switch to agent-checks for
>> > live management of weights and statuses. One problem I see now - track
>> > don't propagate weight setting to trackers, so if we set agent-check on
>> > track we can manage status only.
>> >
>> > My first PoC solution works good, so I thought about introducing
>> something
>> > like agent-track or track-agent directive set on backends (or maybe it
>> > should be default, non-configurable behaviour) to propagate agent-check
>> > responses from main check to all tracking backends. Both default
>> behaviour
>> > and directive approach are small changes in code, but a little bigger in
>> > functionality.
>> >
>> > In my opinion if we set agent-check on backend which is tracked by
>> others -
>> > it should propagate agent-check weight response to those tracking
>> backend
>> > and set weight on them too. Configurable or not - it will be good
>> feature.
>>
>> I think we at least propagate the DRAIN state which is equivalent to
>> weight == 0. If so I too think we should propagate *relative* weights.
>> Agent-checks can return a relative weight (eg: 50%, 100%, 150%) or an
>> absolute weight (eg: 10, 20). If you have two farms configured like this :
>>
>> backend farm1
>> server new1 1.1.1.1:8000 weight 10 agent-check
>> server new2 1.1.1.2:8000 weight 10 agent-check
>>
>> backend farm2
>> server new1 1.1.1.1:8000 weight 20 track farm1/new1
>> server new2 1.1.1.2:8000 weight 20 track farm1/new2
>> server old1 1.1.1.3:8000 weight 10 check
>> server old2 1.1.1.4:8000 weight 10 check
>>
>> Then you want the weight changes on farm1 to be applied proportionally
>> to farm2 (ie: a ratio of the configured absolute weight, which is iweight
>> IIRC).
>>
>> Otherwise that sounds quite reasonable to me given that the agent-check's
>> purpose is to provide a more accurate vision of the server's health, and
>> that tracking is made to share the same vision across multiple farms.
>>
>> Regards,
>> Willy
>>
>
>
>
Hello!
Any news in this topic? Is there anything wrong with my patch?

Michał

2017-02-04 9:38 GMT+01:00 Michał <[email protected]>:

> Hi,
> I checked it and during synthetic tests it worked. I use same
> mechanism as origin agent-check, so it's ready to merge.
>
> It doesn't need to be backported.
>
> 2017-01-27 15:38 GMT+01:00 Michał <[email protected]>:
>
>> Hello,
>>
>> So here's patch, which includes all functionalities I think about.
>> It propagates the response for every tracking server without changing it
>> and without intercepting it. In my opinion we should propagate relative
>> and absolute weights, because if you use weight=0 server's to offload
>> checks then you can send relative weight change and 0 will stay where it
>> is.
>>
>> Regards,
>> Michał
>>
>>
>> 2017-01-20 10:54 GMT+01:00 Willy Tarreau <[email protected]>:
>>
>>> Hi Michal,
>>>
>>> On Thu, Jan 19, 2017 at 11:45:57PM +0100, Micha?? wrote:
>>> > Hello,
>>> >
>>> > We use track's in haproxy to minimize check traffic in some situations
>>> and
>>> > after my last patch we are probably going to switch to agent-checks for
>>> > live management of weights and statuses. One problem I see now - track
>>> > don't propagate weight setting to trackers, so if we set agent-check on
>>> > track we can manage status only.
>>> >
>>> > My first PoC solution works good, so I thought about introducing
>>> something
>>> > like agent-track or track-agent directive set on backends (or maybe it
>>> > should be default, non-configurable behaviour) to propagate agent-check
>>> > responses from main check to all tracking backends. Both default
>>> behaviour
>>> > and directive approach are small changes in code, but a little bigger
>>> in
>>> > functionality.
>>> >
>>> > In my opinion if we set agent-check on backend which is tracked by
>>> others -
>>> > it should propagate agent-check weight response to those tracking
>>> backend
>>> > and set weight on them too. Configurable or not - it will be good
>>> feature.
>>>
>>> I think we at least propagate the DRAIN state which is equivalent to
>>> weight == 0. If so I too think we should propagate *relative* weights.
>>> Agent-checks can return a relative weight (eg: 50%, 100%, 150%) or an
>>> absolute weight (eg: 10, 20). If you have two farms configured like this
>>> :
>>>
>>> backend farm1
>>> server new1 1.1.1.1:8000 weight 10 agent-check
>>> server new2 1.1.1.2:8000 weight 10 agent-check
>>>
>>> backend farm2
>>> server new1 1.1.1.1:8000 weight 20 track farm1/new1
>>> server new2 1.1.1.2:8000 weight 20 track farm1/new2
>>> server old1 1.1.1.3:8000 weight 10 check
>>> server old2 1.1.1.4:8000 weight 10 check
>>>
>>> Then you want the weight changes on farm1 to be applied proportionally
>>> to farm2 (ie: a ratio of the configured absolute weight, which is iweight
>>> IIRC).
>>>
>>> Otherwise that sounds quite reasonable to me given that the agent-check's
>>> purpose is to provide a more accurate vision of the server's health, and
>>> that tracking is made to share the same vision across multiple farms.
>>>
>>> Regards,
>>> Willy
>>>
>>
>>
>>
>
Hi Michal,

On Wed, Mar 15, 2017 at 10:13:01PM +0100, Michal wrote:
> Hello!
> Any news in this topic? Is there anything wrong with my patch?

Not yet, we're just totally drowning under complex bugs resulting in
minor features to be delayed :-/

Thanks,
Willy
Hi Michal,

On Wed, Mar 15, 2017 at 10:13:01PM +0100, Michal wrote:
> Hello!
> Any news in this topic? Is there anything wrong with my patch?

So I checked it but it still has the problem of propagating absolute
weights, which, as I explained earlier, will break lots of setups. I
tend to think that doing it only for relative weight changes could be
OK (provided this is properly documented in the "track" and "agent-check"
keyword sections). The principle of the relative weight change is what
most users are seeking : the server wants to say "I'm running my backups
now, please cut my load in half" or "I'm swapping, I estimate that by
shrinking my load by 33% it will be OK". Regardless of the configured
weigths in different farms, we could propagate this relative weight
change.

Also I'm seeing that your patch only propagates to the first layer of
tracking servers, and stops there without updating the next layer, you
need a recursive propagation here.

Last, if you implement this, it's absolutely mandatory that the same is
done for the CLI since the CLI is the only way to fix the bad effects
of wrong agent changes. Thus having a function dedicated to propagating
relative weight changes would help, it would solve the case for the CLI
and for the agent.

I continue to think that such a change will definitely reintroduce problems
that took several years to get rid of, but hopefully with proper documentation
that can be worked around. Ie when a use complains that the weight change
applied to a server seems to regularly be ignored, it probably is because
of the fact that they're tracking another server whose weight changes.

Thanks,
Willy
Sorry, only registered users may post in this forum.

Click here to login