Welcome! Log In Create A New Profile

Advanced

Routing based on ALPN

Posted by Wiktor Kwapisiewicz via nginx 
Wiktor Kwapisiewicz via nginx
Routing based on ALPN
February 19, 2018 12:10PM
Hello,

I'm looking for a way to route traffic on port 443 based on ALPN value
without SSL termination.

ssl_preread_module [1] does something similar but the only exposed
variable ($ssl_preread_server_name) is for SNI, not ALPN.

A bit of context. I'd like to use nginx to host regular HTTPS server on port
443 but if the ALPN value is 'xmpp-client' transparently proxy the traffic
to my local Jabber server. This feature [2] is already supported by several
XMPP clients.

Is there a way to access and save ALPN value to a variable?

Thank you for your time.

Kind regards,
Wiktor

[1]: https://nginx.org/en/docs/stream/ngx_stream_ssl_preread_module.html

[2]: https://xmpp.org/extensions/xep-0368.html

--
*/metacode/*
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Vladimir Homutov
Re: Routing based on ALPN
February 19, 2018 12:50PM
On Mon, Feb 19, 2018 at 12:02:06PM +0100, Wiktor Kwapisiewicz via nginx wrote:
> Hello,
>
> I'm looking for a way to route traffic on port 443 based on ALPN value
> without SSL termination.
>
> ssl_preread_module [1] does something similar but the only exposed
> variable ($ssl_preread_server_name) is for SNI, not ALPN.
>
> A bit of context. I'd like to use nginx to host regular HTTPS server on port
> 443 but if the ALPN value is 'xmpp-client' transparently proxy the traffic
> to my local Jabber server. This feature [2] is already supported by several
> XMPP clients.
>
> Is there a way to access and save ALPN value to a variable?

Hello,

currently this is not possible; as you correctly noted, ssl_preread
module only processes SNI extension.
To achieve what you want, ssl_preread module needs to be extended to process
ALPN extension as well and export results as a variable, that could be
used to make routing decision.


_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Konstantin Pavlov
Re: Routing based on ALPN
February 19, 2018 02:20PM
On 19/02/2018 14:02, Wiktor Kwapisiewicz via nginx wrote:
> Hello,
>
> I'm looking for a way to route traffic on port 443 based on ALPN value
> without SSL termination.
>
> ssl_preread_module [1] does something similar but the only exposed
> variable ($ssl_preread_server_name) is for SNI, not ALPN.
>
> A bit of context. I'd like to use nginx to host regular HTTPS server on port
> 443 but if the ALPN value is 'xmpp-client' transparently proxy the traffic
> to my local Jabber server. This feature [2] is already supported by several
> XMPP clients.
>
> Is there a way to access and save ALPN value to a variable?

It should possible to parse the incoming buffer with https://nginx.org/r/js_filter and create a variable to make a routing decision on.

--
Konstantin Pavlov
www.nginx.com
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Wiktor Kwapisiewicz via nginx
Re: Routing based on ALPN
February 25, 2018 08:20PM
>> Is there a way to access and save ALPN value to a variable?
>
> It should possible to parse the incoming buffer with https://nginx.org/r/js_filter and create a variable to make a routing decision on.
>

Excellent idea for quickly solving this problem, thanks!

Would a long term solution involve creating a new, additional variable
in the ssl_preread module (e.g. ssl_preread_alpn)?

I've seen something similar being done by HAProxy (ssl_fc_alpn [1]).

Kind regards,
Wiktor


[1]: https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#7.3.4-ssl_fc_alpn

--
*/metacode/*
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Vladimir Homutov
Re: Routing based on ALPN
March 06, 2018 03:50PM
On Sun, Feb 25, 2018 at 08:16:18PM +0100, Wiktor Kwapisiewicz via nginx wrote:
> >> Is there a way to access and save ALPN value to a variable?
> >
> > It should possible to parse the incoming buffer with https://nginx.org/r/js_filter and create a variable to make a routing decision on.
> >
>
> Excellent idea for quickly solving this problem, thanks!
>
> Would a long term solution involve creating a new, additional variable
> in the ssl_preread module (e.g. ssl_preread_alpn)?
>

below is the initial version of patch that creates the
"$ssl_preread_alpn_protocols" variable; the content is a comma-separated
list of protocols, sent by client in ALPN extension, if present.

Any feedback is appretiated.

# HG changeset patch
# User Roman Arutyunyan <[email protected]>
# Date 1520346970 -10800
# Tue Mar 06 17:36:10 2018 +0300
# Node ID edea1fea2b3970889946d38a077c7f3ed98613f5
# Parent 265c29b0b8b8c54b1c623268481ed85324ce3c79
Stream ssl_preread: $ssl_preread_alpn_protocols variable.

The variable keeps a comma-separated list of ALPN protocol names sent by client.

diff --git a/src/stream/ngx_stream_ssl_preread_module.c b/src/stream/ngx_stream_ssl_preread_module.c
--- a/src/stream/ngx_stream_ssl_preread_module.c
+++ b/src/stream/ngx_stream_ssl_preread_module.c
@@ -17,10 +17,12 @@ typedef struct {
typedef struct {
size_t left;
size_t size;
+ size_t ext;
u_char *pos;
u_char *dst;
u_char buf[4];
ngx_str_t host;
+ ngx_str_t alpn;
ngx_log_t *log;
ngx_pool_t *pool;
ngx_uint_t state;
@@ -32,6 +34,8 @@ static ngx_int_t ngx_stream_ssl_preread_
ngx_stream_ssl_preread_ctx_t *ctx, u_char *pos, u_char *last);
static ngx_int_t ngx_stream_ssl_preread_server_name_variable(
ngx_stream_session_t *s, ngx_stream_variable_value_t *v, uintptr_t data);
+static ngx_int_t ngx_stream_ssl_preread_alpn_protocols_variable(
+ ngx_stream_session_t *s, ngx_stream_variable_value_t *v, uintptr_t data);
static ngx_int_t ngx_stream_ssl_preread_add_variables(ngx_conf_t *cf);
static void *ngx_stream_ssl_preread_create_srv_conf(ngx_conf_t *cf);
static char *ngx_stream_ssl_preread_merge_srv_conf(ngx_conf_t *cf, void *parent,
@@ -85,6 +89,9 @@ static ngx_stream_variable_t ngx_stream
{ ngx_string("ssl_preread_server_name"), NULL,
ngx_stream_ssl_preread_server_name_variable, 0, 0, 0 },

+ { ngx_string("ssl_preread_alpn_protocols"), NULL,
+ ngx_stream_ssl_preread_alpn_protocols_variable, 0, 0, 0 },
+
ngx_stream_null_variable
};

@@ -175,7 +182,7 @@ static ngx_int_t
ngx_stream_ssl_preread_parse_record(ngx_stream_ssl_preread_ctx_t *ctx,
u_char *pos, u_char *last)
{
- size_t left, n, size;
+ size_t left, n, size, ext;
u_char *dst, *p;

enum {
@@ -192,7 +199,10 @@ ngx_stream_ssl_preread_parse_record(ngx_
sw_ext_header, /* extension_type, extension_data length */
sw_sni_len, /* SNI length */
sw_sni_host_head, /* SNI name_type, host_name length */
- sw_sni_host /* SNI host_name */
+ sw_sni_host, /* SNI host_name */
+ sw_alpn_len, /* ALPN length */
+ sw_alpn_proto_len, /* ALPN protocol_name length */
+ sw_alpn_proto_data /* ALPN protocol_name */
} state;

ngx_log_debug2(NGX_LOG_DEBUG_STREAM, ctx->log, 0,
@@ -201,6 +211,7 @@ ngx_stream_ssl_preread_parse_record(ngx_
state = ctx->state;
size = ctx->size;
left = ctx->left;
+ ext = ctx->ext;
dst = ctx->dst;
p = ctx->buf;

@@ -299,10 +310,18 @@ ngx_stream_ssl_preread_parse_record(ngx_
break;

case sw_ext_header:
- if (p[0] == 0 && p[1] == 0) {
+ if (p[0] == 0 && p[1] == 0 && ctx->host.data == NULL) {
/* SNI extension */
state = sw_sni_len;
- dst = NULL;
+ dst = p;
+ size = 2;
+ break;
+ }
+
+ if (p[0] == 0 && p[1] == 16 && ctx->alpn.data == NULL) {
+ /* ALPN extension */
+ state = sw_alpn_len;
+ dst = p;
size = 2;
break;
}
@@ -313,6 +332,7 @@ ngx_stream_ssl_preread_parse_record(ngx_
break;

case sw_sni_len:
+ ext = (p[0] << 8) + p[1];
state = sw_sni_host_head;
dst = p;
size = 3;
@@ -328,6 +348,13 @@ ngx_stream_ssl_preread_parse_record(ngx_
state = sw_sni_host;
size = (p[1] << 8) + p[2];

+ if (ext < 3 + size) {
+ ngx_log_debug0(NGX_LOG_DEBUG_STREAM, ctx->log, 0,
+ "ssl preread: SNI format error");
+ return NGX_DECLINED;
+ }
+ ext -= 3 + size;
+
ctx->host.data = ngx_pnalloc(ctx->pool, size);
if (ctx->host.data == NULL) {
return NGX_ERROR;
@@ -341,7 +368,64 @@ ngx_stream_ssl_preread_parse_record(ngx_

ngx_log_debug1(NGX_LOG_DEBUG_STREAM, ctx->log, 0,
"ssl preread: SNI hostname \"%V\"", &ctx->host);
- return NGX_OK;
+
+ state = sw_ext;
+ dst = NULL;
+ size = ext;
+ break;
+
+ case sw_alpn_len:
+ ext = (p[0] << 8) + p[1];
+ dst = p;
+ size = 1;
+
+ ctx->alpn.data = ngx_pnalloc(ctx->pool, ext);
+ if (ctx->alpn.data == NULL) {
+ return NGX_ERROR;
+ }
+
+ state = sw_alpn_proto_len;
+ break;
+
+ case sw_alpn_proto_len:
+ dst = ctx->alpn.data + ctx->alpn.len;
+ size = p[0];
+
+ if (ext < 1 + size) {
+ ngx_log_debug0(NGX_LOG_DEBUG_STREAM, ctx->log, 0,
+ "ssl preread: ALPN format error");
+ return NGX_DECLINED;
+ }
+ ext -= 1 + size;
+ state = sw_alpn_proto_data;
+ break;
+
+ case sw_alpn_proto_data:
+
+ if (p[0] == 0) {
+ ngx_log_debug0(NGX_LOG_DEBUG_STREAM, ctx->log, 0,
+ "ssl preread: ALPN protocol zero length");
+ return NGX_DECLINED;
+ }
+
+ ctx->alpn.len += p[0];
+
+ ngx_log_debug1(NGX_LOG_DEBUG_STREAM, ctx->log, 0,
+ "ssl preread: ALPN protocols \"%V\"", &ctx->alpn);
+
+ if (ext) {
+ ctx->alpn.data[ctx->alpn.len++] = ',';
+
+ state = sw_alpn_proto_len;
+ dst = p;
+ size = 1;
+ break;
+ }
+
+ state = sw_ext;
+ dst = NULL;
+ size = 0;
+ break;
}

if (left < size) {
@@ -354,6 +438,7 @@ ngx_stream_ssl_preread_parse_record(ngx_
ctx->state = state;
ctx->size = size;
ctx->left = left;
+ ctx->ext = ext;
ctx->dst = dst;

return NGX_AGAIN;
@@ -384,6 +469,29 @@ ngx_stream_ssl_preread_server_name_varia


static ngx_int_t
+ngx_stream_ssl_preread_alpn_protocols_variable(ngx_stream_session_t *s,
+ ngx_variable_value_t *v, uintptr_t data)
+{
+ ngx_stream_ssl_preread_ctx_t *ctx;
+
+ ctx = ngx_stream_get_module_ctx(s, ngx_stream_ssl_preread_module);
+
+ if (ctx == NULL) {
+ v->not_found = 1;
+ return NGX_OK;
+ }
+
+ v->valid = 1;
+ v->no_cacheable = 0;
+ v->not_found = 0;
+ v->len = ctx->alpn.len;
+ v->data = ctx->alpn.data;
+
+ return NGX_OK;
+}
+
+
+static ngx_int_t
ngx_stream_ssl_preread_add_variables(ngx_conf_t *cf)
{
ngx_stream_variable_t *var, *v;
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Wiktor Kwapisiewicz via nginx
Re: Routing based on ALPN
March 07, 2018 12:40PM
> below is the initial version of patch that creates the
> "$ssl_preread_alpn_protocols" variable; the content is a comma-separated
> list of protocols, sent by client in ALPN extension, if present.
>
> Any feedback is appretiated.
>

I have just tested this patch and can confirm it's working perfectly fine.

The patch was applied against this commit: https://github.com/nginx/nginx/commit/83dceda8688fcba6da9fd12f6480606563d7b7a3
And I was using LibreSSL.

I've set up three upstream servers for tests, two using node.js (HTTPS) and one Prosody (XMPP server):

map $ssl_preread_alpn_protocols $upstream {
default node1;
"h2,http/1.1" node2;
"xmpp-client" prosody;
}

Curling with no ALPN correctly returns answer from node1:

> curl -k -i --no-alpn https://docker.local
HTTP/1.1 200 OK
Date: Wed, 07 Mar 2018 11:24:26 GMT
Connection: keep-alive
Content-Length: 23

Everything works: node1

Curling with default configuration (ALPN: h2,http/1.1) also works:

> curl -k -i https://docker.local
HTTP/1.1 200 OK
Date: Wed, 07 Mar 2018 11:24:43 GMT
Connection: keep-alive
Content-Length: 23

Everything works: node2

Then I tested XMPP by adding an SRV record:

> dig _xmpps-client._tcp.testing.metacode.biz SRV
;; ANSWER SECTION:
_xmpps-client._tcp.testing.metacode.biz. 119 IN SRV 1 1 443 docker.local.

And using Gajim to connect to testing.metacode.biz. It worked.

Nginx (web_1) logs correctly show all connection attempts with ALPN values:

prosody_1 | c2s2564890 info Client connected
web_1 | 192.168.99.1 xmpp-client [07/Mar/2018:11:21:58 +0000] TCP 200 2335 871 1.566
web_1 | 192.168.99.1 [07/Mar/2018:11:24:26 +0000] TCP 200 1546 327 0.298
web_1 | 192.168.99.1 h2,http/1.1 [07/Mar/2018:11:24:35 +0000] TCP 200 1539 262 0.324
web_1 | 192.168.99.1 h2,http/1.1 [07/Mar/2018:11:24:43 +0000] TCP 200 1539 262 0.293
prosody_1 | c2s2564890 info Authenticated as wiktor@testing.metacode.biz

I've used log_format basic '$remote_addr $ssl_preread_alpn_protocols [$time_local] '
'$protocol $status $bytes_sent $bytes_received '
'$session_time';

This looks *very good*, thanks for your time!

Kind regards,
Wiktor

--
*/metacode/*
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Maxim Konovalov
Re: Routing based on ALPN
March 07, 2018 12:50PM
On 07/03/2018 14:38, Wiktor Kwapisiewicz via nginx wrote:
[...]
> This looks *very good*, thanks for your time!

Thanks for your testing, Wiktor.

--
Maxim Konovalov
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Roman Arutyunyan
Re: Routing based on ALPN
March 13, 2018 01:10PM
Wiktor,

On Wed, Mar 07, 2018 at 12:38:51PM +0100, Wiktor Kwapisiewicz via nginx wrote:
> > below is the initial version of patch that creates the
> > "$ssl_preread_alpn_protocols" variable; the content is a comma-separated
> > list of protocols, sent by client in ALPN extension, if present.
> >
> > Any feedback is appretiated.
> >
>
> I have just tested this patch and can confirm it's working perfectly fine.

We have committed the patch.

http://hg.nginx.org/nginx/rev/79eb4f7b6725

Thanks for cooperation.

--
Roman Arutyunyan
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Sorry, only registered users may post in this forum.

Click here to login