Welcome! Log In Create A New Profile

Advanced

[PATCH 0/2] MEDIUM: stats: Add JSON output option to show (info|stat)

Posted by Simon Horman 
Hi,

this short series is an RFC implementation of adding JSON format
output to show (info|stat). It also adds a new show schema json
stats command to allow retreival of the schema which describes
the JSON output of show (info|stat).

Some areas for possible discussion:
* Use of STAT_STARTED in first patch
* Possible automatic generation of (part) of schema in 2nd patch
* Improved documentation

Some discussion of the size of JSON output is included as an appendix
to the changelog of the first patch.

Changes since RFC noted in per-patch changelogs.

Simon Horman (2):
MEDIUM: stats: Add JSON output option to show (info|stat)
MEDIUM: stats: Add show json schema

doc/management.txt | 74 ++++++--
include/types/stats.h | 6 +
src/stats.c | 506 +++++++++++++++++++++++++++++++++++++++++++++++++-
3 files changed, 570 insertions(+), 16 deletions(-)

--
2.7.0.rc3.207.g0ac5344
Simon Horman
[PATCH 2/2] MEDIUM: stats: Add show json schema
January 04, 2017 09:40AM
This may be used to output the JSON schema which describes the output of
show info json and show stats json.

The JSON output is without any extra whitespace in order to reduce the
volume of output. For human consumption passing the output through a
pretty printer may be helpful.

e.g.:
$ echo "show schema json" | socat /var/run/haproxy.stat stdio | \
python -m json.tool

The implementation does not generate the schema. Some consideration could
be given to integrating the output of the schema with the output of
typed and json info and stats. In particular the types (u32, s64, etc...)
and tags.

A sample verification of show info json and show stats json using
the schema is as follows. It uses the jsonschema python module:

cat > jschema.py << __EOF__
import json

from jsonschema import validate
from jsonschema.validators import Draft3Validator

with open('schema.txt', 'r') as f:
schema = json.load(f)
Draft3Validator.check_schema(schema)

with open('instance.txt', 'r') as f:
instance = json.load(f)
validate(instance, schema, Draft3Validator)
__EOF__

$ echo "show schema json" | socat /var/run/haproxy.stat stdio > schema.txt
$ echo "show info json" | socat /var/run/haproxy.stat stdio > instance.txt
python ./jschema.py
$ echo "show stats json" | socat /var/run/haproxy.stat stdio > instance.txt
python ./jschema.py

Signed-off-by: Simon Horman <[email protected]>
---

In this case the pretty printer increases the size of the output by
about 200% illustrating the value of output without whitespace.

$ echo "show schema json" | socat /var/run/haproxy.stat stdio | wc -c
2690
$ echo "show schema json" | socat /var/run/haproxy.stat stdio | \
python -m json.tool | wc -c
8587

Changes since RFC:
* Add errors to schema and use in the case where output exceeds
available buffer space
* Document that consideration should be given to updating
schema function if struct field is updated
* Correct typos
* Register "show", "schema", json" rather than "show", "schema".
This allows the parse callback to be omitted and some simplification
of the dump callback.
* Limit integer values to the range [-(2**53)+1, (2**53)-1] as
per the recommendation for interoperable integers in
section 6 of RFC 7159.
---
doc/management.txt | 33 ++++++-
include/types/stats.h | 5 +-
src/stats.c | 234 ++++++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 268 insertions(+), 4 deletions(-)

diff --git a/doc/management.txt b/doc/management.txt
index 623ac6375552..70af03b07271 100644
--- a/doc/management.txt
+++ b/doc/management.txt
@@ -1849,7 +1849,14 @@ show info [typed|json]
(...)

The format of JSON output is described in a schema which may be output
- using "show schema json" (to be implemented).
+ using "show schema json".
+
+ The JSON output contains no extra whitespace in order to reduce the
+ volume of output. For human consumption passing the output through a
+ pretty printer may be helpful. Example :
+
+ $ echo "show info json" | socat /var/run/haproxy.sock stdio | \
+ python -m json.tool

The JSON output contains no extra whitespace in order to reduce the
volume of output. For human consumption passing the output through a
@@ -2128,7 +2135,14 @@ show stat [{<iid>|<proxy>} <type> <sid>] [typed|json]
(...)

The format of JSON output is described in a schema which may be output
- using "show schema json" (to be implemented).
+ using "show schema json".
+
+ The JSON output contains no extra whitespace in order to reduce the
+ volume of output. For human consumption passing the output through a
+ pretty printer may be helpful. Example :
+
+ $ echo "show stat json" | socat /var/run/haproxy.sock stdio | \
+ python -m json.tool

The JSON output contains no extra whitespace in order to reduce the
volume of output. For human consumption passing the output through a
@@ -2237,6 +2251,21 @@ show tls-keys [id|*]
specified as parameter, it will dump the tickets, using * it will dump every
keys from every references.

+show schema json
+ Dump the schema used for the output of "show info json" and "show stat json".
+
+ The contains no extra whitespace in order to reduce the volume of output.
+ For human consumption passing the output through a pretty printer may be
+ helpful. Example :
+
+ $ echo "show schema json" | socat /var/run/haproxy.sock stdio | \
+ python -m json.tool
+
+ The schema follows "JSON Schema" (json-schema.org) and accordingly
+ verifiers may be used to verify the output of "show info json" and "show
+ stat json" against the schema.
+
+
shutdown frontend <frontend>
Completely delete the specified frontend. All the ports it was bound to will
be released. It will not be possible to enable the frontend anymore after
diff --git a/include/types/stats.h b/include/types/stats.h
index aad694c203c3..70224687123b 100644
--- a/include/types/stats.h
+++ b/include/types/stats.h
@@ -215,8 +215,9 @@ enum field_scope {
FS_MASK = 0xFF000000,
};

-/* Please consider updating stats_dump_fields_*() and
- * stats_dump_.*_info_fields() when modifying struct field or related enums.
+/* Please consider updating stats_dump_fields_*(),
+ * stats_dump_.*_info_fields() and stats_*_schema()
+ * when modifying struct field or related enums.
*/
struct field {
uint32_t type;
diff --git a/src/stats.c b/src/stats.c
index 0f226fca2c2e..1038f76f073e 100644
--- a/src/stats.c
+++ b/src/stats.c
@@ -3299,6 +3299,234 @@ static int stats_dump_info_to_buffer(struct stream_interface *si)
return 1;
}

+/* This function dumps the schema onto the stream interface's read buffer.
+ * It returns 0 as long as it does not complete, non-zero upon completion.
+ * No state is used.
+ *
+ * Integer values bouned to the range [-(2**53)+1, (2**53)-1] as
+ * per the recommendation for interoperable integers in section 6 of RFC 7159.
+ */
+static void stats_dump_json_schema(struct chunk *out)
+{
+
+ int old_len = out->len;
+
+ chunk_strcat(out,
+ "{"
+ "\"$schema\":\"http://json-schema.org/draft-04/schema#\","
+ "\"oneOf\":["
+ "{"
+ "\"title\":\"Info\","
+ "\"type\":\"array\","
+ "\"items\":{"
+ "\"properties\":{"
+ "\"title\":\"InfoItem\","
+ "\"type\":\"object\","
+ "\"field\":{\"$ref\":\"#/definitions/field\"},"
+ "\"processNum\":{\"$ref\":\"#/definitions/processNum\"},"
+ "\"tags\":{\"$ref\":\"#/definitions/tags\"},"
+ "\"value\":{\"$ref\":\"#/definitions/typedValue\"}"
+ "},"
+ "\"required\":[\"field\",\"processNum\",\"tags\","
+ "\"value\"]"
+ "}"
+ "},"
+ "{"
+ "\"title\":\"Stat\","
+ "\"type\":\"array\","
+ "\"items\":{"
+ "\"title\":\"InfoItem\","
+ "\"type\":\"object\","
+ "\"properties\":{"
+ "\"objType\":{"
+ "\"enum\":[\"Frontend\",\"Backend\",\"Listener\","
+ "\"Server\",\"Unknown\"]"
+ "},"
+ "\"proxyId\":{"
+ "\"type\":\"integer\","
+ "\"minimum\":0"
+ "},"
+ "\"id\":{"
+ "\"type\":\"integer\","
+ "\"minimum\":0"
+ "},"
+ "\"field\":{\"$ref\":\"#/definitions/field\"},"
+ "\"processNum\":{\"$ref\":\"#/definitions/processNum\"},"
+ "\"tags\":{\"$ref\":\"#/definitions/tags\"},"
+ "\"typedValue\":{\"$ref\":\"#/definitions/typedValue\"}"
+ "},"
+ "\"required\":[\"objType\",\"proxyId\",\"id\","
+ "\"field\",\"processNum\",\"tags\","
+ "\"value\"]"
+ "}"
+ "},"
+ "{"
+ "\"title\":\"Error\","
+ "\"type\":\"object\","
+ "\"properties\":{"
+ "\"errorStr\":{"
+ "\"type\":\"string\""
+ "},"
+ "\"required\":[\"errorStr\"]"
+ "}"
+ "}"
+ "],"
+ "\"definitions\":{"
+ "\"field\":{"
+ "\"type\":\"object\","
+ "\"pos\":{"
+ "\"type\":\"integer\","
+ "\"minimum\":0"
+ "},"
+ "\"name\":{"
+ "\"type\":\"string\""
+ "},"
+ "\"required\":[\"pos\",\"name\"]"
+ "},"
+ "\"processNum\":{"
+ "\"type\":\"integer\","
+ "\"minimum\":1"
+ "},"
+ "\"tags\":{"
+ "\"type\":\"object\","
+ "\"origin\":{"
+ "\"type\":\"string\","
+ "\"enum\":[\"Metric\",\"Status\",\"Key\","
+ "\"Config\",\"Product\",\"Unknown\"]"
+ "},"
+ "\"nature\":{"
+ "\"type\":\"string\","
+ "\"enum\":[\"Gauge\",\"Limit\",\"Min\",\"Max\","
+ "\"Rate\",\"Counter\",\"Duration\","
+ "\"Age\",\"Time\",\"Name\",\"Output\","
+ "\"Avg\", \"Unknown\"]"
+ "},"
+ "\"scope\":{"
+ "\"type\":\"string\","
+ "\"enum\":[\"Cluster\",\"Process\",\"Service\","
+ "\"System\",\"Unknown\"]"
+ "},"
+ "\"required\":[\"origin\",\"nature\",\"scope\"]"
+ "},"
+ "\"typedValue\":{"
+ "\"type\":\"object\","
+ "\"oneOf\":["
+ "{\"$ref\":\"#/definitions/typedValue/definitions/s32Value\"},"
+ "{\"$ref\":\"#/definitions/typedValue/definitions/s64Value\"},"
+ "{\"$ref\":\"#/definitions/typedValue/definitions/u32Value\"},"
+ "{\"$ref\":\"#/definitions/typedValue/definitions/u64Value\"},"
+ "{\"$ref\":\"#/definitions/typedValue/definitions/strValue\"}"
+ "],"
+ "\"definitions\":{"
+ "\"s32Value\":{"
+ "\"properties\":{"
+ "\"type\":{"
+ "\"type\":\"string\","
+ "\"enum\":[\"s32\"]"
+ "},"
+ "\"value\":{"
+ "\"type\":\"integer\","
+ "\"minimum\":-2147483648,"
+ "\"maximum\":2147483647"
+ "}"
+ "},"
+ "\"required\":[\"type\",\"value\"]"
+ "},"
+ "\"s64Value\":{"
+ "\"properties\":{"
+ "\"type\":{"
+ "\"type\":\"string\","
+ "\"enum\":[\"s64\"]"
+ "},"
+ "\"value\":{"
+ "\"type\":\"integer\","
+ "\"minimum\":-9007199254740991,"
+ "\"maximum\":9007199254740991"
+ "}"
+ "},"
+ "\"required\":[\"type\",\"value\"]"
+ "},"
+ "\"u32Value\":{"
+ "\"properties\":{"
+ "\"type\":{"
+ "\"type\":\"string\","
+ "\"enum\":[\"u32\"]"
+ "},"
+ "\"value\":{"
+ "\"type\":\"integer\","
+ "\"minimum\":0,"
+ "\"maximum\":4294967295"
+ "}"
+ "},"
+ "\"required\":[\"type\",\"value\"]"
+ "},"
+ "\"u64Value\":{"
+ "\"properties\":{"
+ "\"type\":{"
+ "\"type\":\"string\","
+ "\"enum\":[\"u64\"]"
+ "},"
+ "\"value\":{"
+ "\"type\":\"integer\","
+ "\"minimum\":0,"
+ "\"maximum\":9007199254740991"
+ "}"
+ "},"
+ "\"required\":[\"type\",\"value\"]"
+ "},"
+ "\"strValue\":{"
+ "\"properties\":{"
+ "\"type\":{"
+ "\"type\":\"string\","
+ "\"enum\":[\"str\"]"
+ "},"
+ "\"value\":{\"type\":\"string\"}"
+ "},"
+ "\"required\":[\"type\",\"value\"]"
+ "},"
+ "\"unknownValue\":{"
+ "\"properties\":{"
+ "\"type\":{"
+ "\"type\":\"integer\","
+ "\"minimum\":0"
+ "},"
+ "\"value\":{"
+ "\"type\":\"string\","
+ "\"enum\":[\"unknown\"]"
+ "}"
+ "},"
+ "\"required\":[\"type\",\"value\"]"
+ "}"
+ "}"
+ "}"
+ "}"
+ "}");
+
+ if (old_len == out->len) {
+ chunk_reset(out);
+ chunk_appendf(out,
+ "{\"errorStr\":\"output buffer too short\"}");
+ }
+}
+
+/* This function dumps the schema onto the stream interface's read buffer.
+ * It returns 0 as long as it does not complete, non-zero upon completion.
+ * No state is used.
+ */
+static int stats_dump_json_schema_to_buffer(struct stream_interface *si)
+{
+ chunk_reset(&trash);
+
+ stats_dump_json_schema(&trash);
+
+ if (bi_putchk(si_ic(si), &trash) == -1) {
+ si_applet_cant_put(si);
+ return 0;
+ }
+
+ return 1;
+}
+
static int cli_parse_clear_counters(char **args, struct appctx *appctx, void *private)
{
struct proxy *px;
@@ -3420,11 +3648,17 @@ static int cli_io_handler_dump_stat(struct appctx *appctx)
return stats_dump_stat_to_buffer(appctx->owner, NULL);
}

+static int cli_io_handler_dump_json_schema(struct appctx *appctx)
+{
+ return stats_dump_json_schema_to_buffer(appctx->owner);
+}
+
/* register cli keywords */
static struct cli_kw_list cli_kws = {{ },{
{ { "clear", "counters", NULL }, "clear counters : clear max statistics counters (add 'all' for all counters)", cli_parse_clear_counters, NULL, NULL },
{ { "show", "info", NULL }, "show info : report information about the running process", cli_parse_show_info, cli_io_handler_dump_info, NULL },
{ { "show", "stat", NULL }, "show stat : report counters for each proxy and server", cli_parse_show_stat, cli_io_handler_dump_stat, NULL },
+ { { "show", "schema", "json", NULL }, "show schema json : report schema used for stats", NULL, cli_io_handler_dump_json_schema, NULL },
{{},}
}};

--
2.7.0.rc3.207.g0ac5344
Add a json parameter to show (info|stat) which will output information
in JSON format. A follow-up patch will add a JSON schema which describes
the format of the JSON output of these commands.

The JSON output is without any extra whitespace in order to reduce the
volume of output. For human consumption passing the output through a
pretty printer may be helpful.

e.g.:
$ echo "show info json" | socat /var/run/haproxy.stat stdio | \
python -m json.tool

STAT_STARTED has bee added in order to track if show output has begun or
not. This is used in order to allow the JSON output routines to only insert
a "," between elements when needed. I would value any feedback on how this
might be done better.

Signed-off-by: Simon Horman <[email protected]>
---

For the simple configuration below a comparison of the size of info
and stats output is as follows:

$ show stat => 1654 bytes
$ show stat typed => 7081 bytes
$ show stat json => 45331 bytes
$ show stat json (pretty printed[*]) => 113390 bytes

$ show info => 527 bytes
$ show info typed => 937 bytes
$ show info json => 5330 bytes
$ show info json (pretty printed[*]) => 11456 bytes

[*] pretty printed using python -m json.tool

--- begin config ---
global
daemon
stats socket /tmp/haproxy.stat mode 600 level admin
pidfile /tmp/haproxy.pid
log /dev/log local4
tune.bufsize 16384
tune.maxrewrite 1024

defaults
mode http
balance roundrobin
timeout connect 4000
timeout client 42000
timeout server 43000
log global

listen VIP_Name
bind 127.0.0.1:10080 transparent
mode http
balance leastconn
cookie SERVERID insert nocache indirect
server backup 127.0.0.1:9081 backup non-stick
option http-keep-alive
option forwardfor
option redispatch
option abortonclose
maxconn 40000
log global
option httplog
option log-health-checks
server RIP_Name 127.0.0.1 weight 100 cookie RIP_Name agent-check agent-port 12345 agent-inter 2000 check port 80 inter 2000 rise 2 fall 3 minconn 0 maxconn 0s on-marked-down shutdown-sessions disabled
server RIP_Name 127.0.0.1 weight 100 cookie RIP_Name agent-check agent-port 12345 agent-inter 2000 check port 80 inter 2000 rise 2 fall 3 minconn 0 maxconn 0s on-marked-down shutdown-sessions
--- end config ---

Changes since RFC:
* Handle cases where output exceeds available buffer space
* Document that consideration should be given to updating
dump functions if struct field is updated
* Limit JSON integer values to the range [-(2**53)+1, (2**53)-1] as per
the recommendation for interoperable integers in section 6 of RFC 7159.
---
doc/management.txt | 45 +++++++--
include/types/stats.h | 5 +
src/stats.c | 272 +++++++++++++++++++++++++++++++++++++++++++++++++-
3 files changed, 306 insertions(+), 16 deletions(-)

diff --git a/doc/management.txt b/doc/management.txt
index 683b99790160..623ac6375552 100644
--- a/doc/management.txt
+++ b/doc/management.txt
@@ -1760,16 +1760,18 @@ show errors [<iid>|<proxy>] [request|response]
show backend
Dump the list of backends available in the running process

-show info [typed]
+show info [typed|json]
Dump info about haproxy status on current process. If "typed" is passed as an
optional argument, field numbers, names and types are emitted as well so that
external monitoring products can easily retrieve, possibly aggregate, then
report information found in fields they don't know. Each field is dumped on
- its own line. By default, the format contains only two columns delimited by a
- colon (':'). The left one is the field name and the right one is the value.
- It is very important to note that in typed output format, the dump for a
- single object is contiguous so that there is no need for a consumer to store
- everything at once.
+ its own line. If "json" is passed as an optional argument then
+ information provided by "typed" output is provided in JSON format as a
+ list of JSON objects. By default, the format contains only two columns
+ delimited by a colon (':'). The left one is the field name and the right
+ one is the value. It is very important to note that in typed output
+ format, the dump for a single object is contiguous so that there is no
+ need for a consumer to store everything at once.

When using the typed output format, each line is made of 4 columns delimited
by colons (':'). The first column is a dot-delimited series of 3 elements. The
@@ -1846,6 +1848,16 @@ show info [typed]
6.Uptime.2:MDP:str:0d 0h01m28s
(...)

+ The format of JSON output is described in a schema which may be output
+ using "show schema json" (to be implemented).
+
+ The JSON output contains no extra whitespace in order to reduce the
+ volume of output. For human consumption passing the output through a
+ pretty printer may be helpful. Example :
+
+ $ echo "show info json" | socat /var/run/haproxy.sock stdio | \
+ python -m json.tool
+
show map [<map>]
Dump info about map converters. Without argument, the list of all available
maps is returned. If a <map> is specified, its contents are dumped. <map> is
@@ -1977,11 +1989,12 @@ show sess <id>
The special id "all" dumps the states of all sessions, which must be avoided
as much as possible as it is highly CPU intensive and can take a lot of time.

-show stat [{<iid>|<proxy>} <type> <sid>] [typed]
- Dump statistics using the CSV format, or using the extended typed output
- format described in the section above if "typed" is passed after the other
- arguments. By passing <id>, <type> and <sid>, it is possible to dump only
- selected items :
+show stat [{<iid>|<proxy>} <type> <sid>] [typed|json]
+ Dump statistics using the CSV format; using the extended typed output
+ format described in the section above if "typed" is passed after the
+ other arguments; or in JSON if "json" is passed after the other arguments
+ . By passing <id>, <type> and <sid>, it is possible to dump only selected
+ items :
- <iid> is a proxy ID, -1 to dump everything. Alternatively, a proxy name
<proxy> may be specified. In this case, this proxy's ID will be used as
the ID selector.
@@ -2114,6 +2127,16 @@ show stat [{<iid>|<proxy>} <type> <sid>] [typed]
B.3.0.6.slim.2:MGP:u32:1000
(...)

+ The format of JSON output is described in a schema which may be output
+ using "show schema json" (to be implemented).
+
+ The JSON output contains no extra whitespace in order to reduce the
+ volume of output. For human consumption passing the output through a
+ pretty printer may be helpful. Example :
+
+ $ echo "show stat json" | socat /var/run/haproxy.sock stdio | \
+ python -m json.tool
+
show stat resolvers [<resolvers section id>]
Dump statistics for the given resolvers section, or all resolvers sections
if no section is supplied.
diff --git a/include/types/stats.h b/include/types/stats.h
index 48cf645fc278..aad694c203c3 100644
--- a/include/types/stats.h
+++ b/include/types/stats.h
@@ -23,11 +23,13 @@
/* Flags for applet.ctx.stats.flags */
#define STAT_FMT_HTML 0x00000001 /* dump the stats in HTML format */
#define STAT_FMT_TYPED 0x00000002 /* use the typed output format */
+#define STAT_FMT_JSON 0x00000004 /* dump the stats in JSON format */
#define STAT_HIDE_DOWN 0x00000008 /* hide 'down' servers in the stats page */
#define STAT_NO_REFRESH 0x00000010 /* do not automatically refresh the stats page */
#define STAT_ADMIN 0x00000020 /* indicate a stats admin level */
#define STAT_CHUNKED 0x00000040 /* use chunked encoding (HTTP/1.1) */
#define STAT_BOUND 0x00800000 /* bound statistics to selected proxies/types/services */
+#define STAT_STARTED 0x01000000 /* some output has occurred */

#define STATS_TYPE_FE 0
#define STATS_TYPE_BE 1
@@ -213,6 +215,9 @@ enum field_scope {
FS_MASK = 0xFF000000,
};

+/* Please consider updating stats_dump_fields_*() and
+ * stats_dump_.*_info_fields() when modifying struct field or related enums.
+ */
struct field {
uint32_t type;
union {
diff --git a/src/stats.c b/src/stats.c
index b7d030351f16..0f226fca2c2e 100644
--- a/src/stats.c
+++ b/src/stats.c
@@ -229,6 +229,7 @@ static struct field stats[ST_F_TOTAL_FIELDS];
* http_stats_io_handler()
* -> stats_dump_stat_to_buffer() // same as above, but used for CSV or HTML
* -> stats_dump_csv_header() // emits the CSV headers (same as above)
+ * -> stats_dump_json_header() // emits the JSON headers (same as above)
* -> stats_dump_html_head() // emits the HTML headers
* -> stats_dump_html_info() // emits the equivalent of "show info" at the top
* -> stats_dump_proxy_to_buffer() // same as above, valid for CSV and HTML
@@ -239,6 +240,7 @@ static struct field stats[ST_F_TOTAL_FIELDS];
* -> stats_dump_be_stats()
* -> stats_dump_html_px_end()
* -> stats_dump_html_end() // emits HTML trailer
+ * -> stats_dump_json_end() // emits JSON trailer
*/


@@ -294,6 +296,58 @@ int stats_emit_typed_data_field(struct chunk *out, const struct field *f)
}
}

+/* Limit JSON integer values to the range [-(2**53)+1, (2**53)-1] as per
+ * the recommendation for interoperable integers in section 6 of RFC 7159.
+ */
+#define JSON_INT_MAX ((1ULL << 53) - 1)
+#define JSON_INT_MIN (0 - JSON_INT_MAX)
+
+/* Emits a stats field value and its type in JSON.
+ * Returns non-zero on success, 0 on error.
+ */
+int stats_emit_json_data_field(struct chunk *out, const struct field *f)
+{
+ int old_len;
+ char buf[20];
+ const char *type, *value = buf, *quote = "";
+
+ switch (field_format(f, 0)) {
+ case FF_EMPTY: return 1;
+ case FF_S32: type = "\"s32\"";
+ snprintf(buf, sizeof(buf), "%d", f->u.s32);
+ break;
+ case FF_U32: type = "\"u32\"";
+ snprintf(buf, sizeof(buf), "%u", f->u.u32);
+ break;
+ case FF_S64: type = "\"s64\"";
+ if (f->u.s64 < JSON_INT_MIN || f->u.s64 > JSON_INT_MAX)
+ return 0;
+ type = "\"s64\"";
+ snprintf(buf, sizeof(buf), "%lld", (long long)f->u.s64);
+ break;
+ case FF_U64: if (f->u.u64 > JSON_INT_MAX)
+ return 0;
+ type = "\"u64\"";
+ snprintf(buf, sizeof(buf), "%llu",
+ (unsigned long long) f->u.u64);
+ break;
+ case FF_STR: type = "\"str\"";
+ value = field_str(f, 0);
+ quote = "\"";
+ break;
+ default: snprintf(buf, sizeof(buf), "%u", f->type);
+ type = buf;
+ value = "unknown";
+ quote = "\"";
+ break;
+ }
+
+ old_len = out->len;
+ chunk_appendf(out, ",\"value\":{\"type\":%s,\"value\":%s%s%s}",
+ type, quote, value, quote);
+ return !(old_len == out->len);
+}
+
/* Emits an encoding of the field type on 3 characters followed by a delimiter.
* Returns non-zero on success, 0 if the buffer is full.
*/
@@ -337,6 +391,55 @@ int stats_emit_field_tags(struct chunk *out, const struct field *f, char delim)
return chunk_appendf(out, "%c%c%c%c", origin, nature, scope, delim);
}

+/* Emits an encoding of the field type as JSON.
+ * Returns non-zero on success, 0 if the buffer is full.
+ */
+int stats_emit_json_field_tags(struct chunk *out, const struct field *f)
+{
+ const char *origin, *nature, *scope;
+ int old_len;
+
+ switch (field_origin(f, 0)) {
+ case FO_METRIC: origin = "Metric"; break;
+ case FO_STATUS: origin = "Status"; break;
+ case FO_KEY: origin = "Key"; break;
+ case FO_CONFIG: origin = "Config"; break;
+ case FO_PRODUCT: origin = "Product"; break;
+ default: origin = "Unknown"; break;
+ }
+
+ switch (field_nature(f, 0)) {
+ case FN_GAUGE: nature = "Gauge"; break;
+ case FN_LIMIT: nature = "Limit"; break;
+ case FN_MIN: nature = "Min"; break;
+ case FN_MAX: nature = "Max"; break;
+ case FN_RATE: nature = "Rate"; break;
+ case FN_COUNTER: nature = "Counter"; break;
+ case FN_DURATION: nature = "Duration"; break;
+ case FN_AGE: nature = "Age"; break;
+ case FN_TIME: nature = "Time"; break;
+ case FN_NAME: nature = "Name"; break;
+ case FN_OUTPUT: nature = "Output"; break;
+ case FN_AVG: nature = "Avg"; break;
+ default: nature = "Unknown"; break;
+ }
+
+ switch (field_scope(f, 0)) {
+ case FS_PROCESS: scope = "Process"; break;
+ case FS_SERVICE: scope = "Service"; break;
+ case FS_SYSTEM: scope = "System"; break;
+ case FS_CLUSTER: scope = "Cluster"; break;
+ default: scope = "Unknown"; break;
+ }
+
+ old_len = out->len;
+ chunk_appendf(out, "\"tags\":{"
+ "\"origin\":\"%s\","
+ "\"nature\":\"%s\","
+ "\"scope\":\"%s\""
+ "}", origin, nature, scope);
+ return !(old_len == out->len);
+}

/* Dump all fields from <stats> into <out> using CSV format */
static int stats_dump_fields_csv(struct chunk *out, const struct field *stats)
@@ -381,6 +484,123 @@ static int stats_dump_fields_typed(struct chunk *out, const struct field *stats)
return 1;
}

+/* Dump all fields from <stats> into <out> using the "show info json" format */
+static int stats_dump_json_info_fields(struct chunk *out,
+ const struct field *info)
+{
+ int field;
+ int started = 0;
+
+ if (!chunk_strcat(out, "["))
+ return 0;
+
+ for (field = 0; field < INF_TOTAL_FIELDS; field++) {
+ int old_len;
+
+ if (!field_format(info, field))
+ continue;
+
+ if (started && !chunk_strcat(out, ","))
+ goto err;
+ started = 1;
+
+ old_len = out->len;
+ chunk_appendf(out,
+ "{\"field\":{\"pos\":%d,\"name\":\"%s\"},"
+ "\"processNum\":%u,",
+ field, info_field_names[field],
+ info[INF_PROCESS_NUM].u.u32);
+ if (old_len == out->len)
+ goto err;
+
+ if (!stats_emit_json_field_tags(out, &info[field]))
+ goto err;
+
+ if (!stats_emit_json_data_field(out, &info[field]))
+ goto err;
+
+ if (!chunk_strcat(out, "}"))
+ goto err;
+ }
+
+ if (!chunk_strcat(out, "]"))
+ goto err;
+ return 1;
+
+err:
+ chunk_reset(out);
+ chunk_appendf(out, "{\"errorStr\":\"output buffer too short\"}");
+ return 0;
+}
+
+/* Dump all fields from <stats> into <out> using a typed "field:desc:type:value" format */
+static int stats_dump_fields_json(struct chunk *out, const struct field *stats,
+ int first_stat)
+{
+ int field;
+ int started = 0;
+
+ if (!first_stat && !chunk_strcat(out, ","))
+ return 0;
+ if (!chunk_strcat(out, "["))
+ return 0;
+
+ for (field = 0; field < ST_F_TOTAL_FIELDS; field++) {
+ const char *obj_type;
+ int old_len;
+
+ if (!stats[field].type)
+ continue;
+
+ if (started && !chunk_strcat(out, ","))
+ goto err;
+ started = 1;
+
+ switch (stats[ST_F_TYPE].u.u32) {
+ case STATS_TYPE_FE: obj_type = "Frontend"; break;
+ case STATS_TYPE_BE: obj_type = "Backend"; break;
+ case STATS_TYPE_SO: obj_type = "Listener"; break;
+ case STATS_TYPE_SV: obj_type = "Server"; break;
+ default: obj_type = "Unknown"; break;
+ }
+
+ old_len = out->len;
+ chunk_appendf(out,
+ "{"
+ "\"objType\":\"%s\","
+ "\"proxyId\":%d,"
+ "\"id\":%d,"
+ "\"field\":{\"pos\":%d,\"name\":\"%s\"},"
+ "\"processNum\":%u,",
+ obj_type, stats[ST_F_IID].u.u32,
+ stats[ST_F_SID].u.u32, field,
+ stat_field_names[field], stats[ST_F_PID].u.u32);
+ if (old_len == out->len)
+ goto err;
+
+ if (!stats_emit_json_field_tags(out, &stats[field]))
+ goto err;
+
+ if (!stats_emit_json_data_field(out, &stats[field]))
+ goto err;
+
+ if (!chunk_strcat(out, "}"))
+ goto err;
+ }
+
+ if (!chunk_strcat(out, "]"))
+ goto err;
+
+ return 1;
+
+err:
+ chunk_reset(out);
+ if (!first_stat)
+ chunk_strcat(out, ",");
+ chunk_appendf(out, "{\"errorStr\":\"output buffer too short\"}");
+ return 0;
+}
+
/* Dump all fields from <stats> into <out> using the HTML format. A column is
* reserved for the checkbox is ST_SHOWADMIN is set in <flags>. Some extra info
* are provided if ST_SHLGNDS is present in <flags>.
@@ -1022,15 +1242,26 @@ static int stats_dump_fields_html(struct chunk *out, const struct field *stats,

int stats_dump_one_line(const struct field *stats, unsigned int flags, struct proxy *px, struct appctx *appctx)
{
+ int ret;
+
if ((px->cap & PR_CAP_BE) && px->srv && (appctx->ctx.stats.flags & STAT_ADMIN))
flags |= ST_SHOWADMIN;

if (appctx->ctx.stats.flags & STAT_FMT_HTML)
- return stats_dump_fields_html(&trash, stats, flags);
+ ret = stats_dump_fields_html(&trash, stats, flags);
else if (appctx->ctx.stats.flags & STAT_FMT_TYPED)
- return stats_dump_fields_typed(&trash, stats);
+ ret = stats_dump_fields_typed(&trash, stats);
+ else if (appctx->ctx.stats.flags & STAT_FMT_JSON)
+ ret = stats_dump_fields_json(&trash, stats,
+ !(appctx->ctx.stats.flags &
+ STAT_STARTED));
else
- return stats_dump_fields_csv(&trash, stats);
+ ret = stats_dump_fields_csv(&trash, stats);
+
+ if (ret)
+ appctx->ctx.stats.flags |= STAT_STARTED;
+
+ return ret;
}

/* Fill <stats> with the frontend statistics. <stats> is
@@ -2258,6 +2489,23 @@ static void stats_dump_html_end()
chunk_appendf(&trash, "</body></html>\n");
}

+/* Dumps the stats JSON header to the trash buffer which. The caller is responsible
+ * for clearing it if needed.
+ */
+static void stats_dump_json_header()
+{
+ chunk_strcat(&trash, "[");
+}
+
+
+/* Dumps the JSON stats trailer block to the trash. The caller is responsible
+ * for clearing the trash if needed.
+ */
+static void stats_dump_json_end()
+{
+ chunk_strcat(&trash, "]");
+}
+
/* This function dumps statistics onto the stream interface's read buffer in
* either CSV or HTML format. <uri> contains some HTML-specific parameters that
* are ignored for CSV format (hence <uri> may be NULL there). It returns 0 if
@@ -2281,6 +2529,8 @@ static int stats_dump_stat_to_buffer(struct stream_interface *si, struct uri_aut
case STAT_ST_HEAD:
if (appctx->ctx.stats.flags & STAT_FMT_HTML)
stats_dump_html_head(uri);
+ else if (appctx->ctx.stats.flags & STAT_FMT_JSON)
+ stats_dump_json_header(uri);
else if (!(appctx->ctx.stats.flags & STAT_FMT_TYPED))
stats_dump_csv_header();

@@ -2329,8 +2579,11 @@ static int stats_dump_stat_to_buffer(struct stream_interface *si, struct uri_aut
/* fall through */

case STAT_ST_END:
- if (appctx->ctx.stats.flags & STAT_FMT_HTML) {
- stats_dump_html_end();
+ if (appctx->ctx.stats.flags & (STAT_FMT_HTML|STAT_FMT_JSON)) {
+ if (appctx->ctx.stats.flags & STAT_FMT_HTML)
+ stats_dump_html_end();
+ else
+ stats_dump_json_end();
if (bi_putchk(rep, &trash) == -1) {
si_applet_cant_put(si);
return 0;
@@ -2757,6 +3010,7 @@ static int stats_send_http_redirect(struct stream_interface *si)
return 1;
}

+
/* This I/O handler runs as an applet embedded in a stream interface. It is
* used to send HTTP stats over a TCP socket. The mechanism is very simple.
* appctx->st0 contains the operation in progress (dump, done). The handler
@@ -3032,6 +3286,8 @@ static int stats_dump_info_to_buffer(struct stream_interface *si)

if (appctx->ctx.stats.flags & STAT_FMT_TYPED)
stats_dump_typed_info_fields(&trash, info);
+ else if (appctx->ctx.stats.flags & STAT_FMT_JSON)
+ stats_dump_json_info_fields(&trash, info);
else
stats_dump_info_fields(&trash, info);

@@ -3108,6 +3364,8 @@ static int cli_parse_show_info(char **args, struct appctx *appctx, void *private

if (strcmp(args[2], "typed") == 0)
appctx->ctx.stats.flags |= STAT_FMT_TYPED;
+ else if (strcmp(args[2], "json") == 0)
+ appctx->ctx.stats.flags |= STAT_FMT_JSON;
return 0;
}

@@ -3138,9 +3396,13 @@ static int cli_parse_show_stat(char **args, struct appctx *appctx, void *private
appctx->ctx.stats.sid = atoi(args[4]);
if (strcmp(args[5], "typed") == 0)
appctx->ctx.stats.flags |= STAT_FMT_TYPED;
+ else if (strcmp(args[5], "json") == 0)
+ appctx->ctx.stats.flags |= STAT_FMT_JSON;
}
else if (strcmp(args[2], "typed") == 0)
appctx->ctx.stats.flags |= STAT_FMT_TYPED;
+ else if (strcmp(args[2], "json") == 0)
+ appctx->ctx.stats.flags |= STAT_FMT_JSON;

return 0;
}
--
2.7.0.rc3.207.g0ac5344
Hi Simon,

On Wed, Jan 04, 2017 at 09:37:24AM +0100, Simon Horman wrote:
> Hi,
>
> this short series is an RFC implementation of adding JSON format
> output to show (info|stat). It also adds a new show schema json
> stats command to allow retreival of the schema which describes
> the JSON output of show (info|stat).
(...)

Thanks for this. I'm seeing in stats_emit_json_field_tags() that
you have to emit the names of the various types, scopes, etc...
I think this is the reason why you mention in patch 1 that it needs
to be updated if the structure evolves. Probably that we should put
these fields in an array declared just next to the enums. This way
the declaration will be a bit more centralized.

> Some areas for possible discussion:
> * Use of STAT_STARTED in first patch
> * Possible automatic generation of (part) of schema in 2nd patch
> * Improved documentation

For now I don't see anything there which needs further discussion, and
nobody commented on your patches either, possibly indicating you're on
the right track. If you want I can merge this series, it will be easier
for you to update it later using incremental patches.

> Some discussion of the size of JSON output is included as an appendix
> to the changelog of the first patch.

I'd prefer to integrate this with your commit message because it's quite
useful as-is.

Just let me know if you want the series to get merged or if you prefer
to respin it.

Thanks,
Willy
Hi Willy,

On Sun, Jan 08, 2017 at 07:37:24AM +0100, Willy Tarreau wrote:
> Hi Simon,
>
> On Wed, Jan 04, 2017 at 09:37:24AM +0100, Simon Horman wrote:
> > Hi,
> >
> > this short series is an RFC implementation of adding JSON format
> > output to show (info|stat). It also adds a new show schema json
> > stats command to allow retreival of the schema which describes
> > the JSON output of show (info|stat).
> (...)
>
> Thanks for this. I'm seeing in stats_emit_json_field_tags() that
> you have to emit the names of the various types, scopes, etc...
> I think this is the reason why you mention in patch 1 that it needs
> to be updated if the structure evolves. Probably that we should put
> these fields in an array declared just next to the enums. This way
> the declaration will be a bit more centralized.
>
> > Some areas for possible discussion:
> > * Use of STAT_STARTED in first patch
> > * Possible automatic generation of (part) of schema in 2nd patch
> > * Improved documentation
>
> For now I don't see anything there which needs further discussion, and
> nobody commented on your patches either, possibly indicating you're on
> the right track. If you want I can merge this series, it will be easier
> for you to update it later using incremental patches.
>
> > Some discussion of the size of JSON output is included as an appendix
> > to the changelog of the first patch.
>
> I'd prefer to integrate this with your commit message because it's quite
> useful as-is.
>
> Just let me know if you want the series to get merged or if you prefer
> to respin it.

I'd prefer if you merged the series as-is
and I then provided incremental updates.

--
Simon Horman simon@horms.nl
Horms Solutions BV www.horms.nl
Parnassusweg 819, 1082 LZ Amsterdam, Netherlands
Tel: +31 (0)20 800 6155 Skype: horms7
Hi Everyone,

Simon I've been playing with this patch today and I'm sorry to say that
adding it to a 1.8 dev build as of today does not look to work.

I did have to manually update a few of the patch chunks so I'm willing to
admit that it could be me that broke it and I will be having another go
tomorrow unless Willy would be kind enough to update the 1.8 download for
us with your patch already applied.

However, the problem that I have now is that either the haproxy.stat file
does not exist at all or I just keep getting:

# echo "show info json" | socat /tmp/haproxy.stat stdio | python -m
json.tool
No JSON object could be decoded


As I said I will try again tomorrow with a fresh brain and eyes.




On 8 January 2017 at 10:52, Simon Horman <[email protected]> wrote:

> Hi Willy,
>
> On Sun, Jan 08, 2017 at 07:37:24AM +0100, Willy Tarreau wrote:
> > Hi Simon,
> >
> > On Wed, Jan 04, 2017 at 09:37:24AM +0100, Simon Horman wrote:
> > > Hi,
> > >
> > > this short series is an RFC implementation of adding JSON format
> > > output to show (info|stat). It also adds a new show schema json
> > > stats command to allow retreival of the schema which describes
> > > the JSON output of show (info|stat).
> > (...)
> >
> > Thanks for this. I'm seeing in stats_emit_json_field_tags() that
> > you have to emit the names of the various types, scopes, etc...
> > I think this is the reason why you mention in patch 1 that it needs
> > to be updated if the structure evolves. Probably that we should put
> > these fields in an array declared just next to the enums. This way
> > the declaration will be a bit more centralized.
> >
> > > Some areas for possible discussion:
> > > * Use of STAT_STARTED in first patch
> > > * Possible automatic generation of (part) of schema in 2nd patch
> > > * Improved documentation
> >
> > For now I don't see anything there which needs further discussion, and
> > nobody commented on your patches either, possibly indicating you're on
> > the right track. If you want I can merge this series, it will be easier
> > for you to update it later using incremental patches.
> >
> > > Some discussion of the size of JSON output is included as an appendix
> > > to the changelog of the first patch.
> >
> > I'd prefer to integrate this with your commit message because it's quite
> > useful as-is.
> >
> > Just let me know if you want the series to get merged or if you prefer
> > to respin it.
>
> I'd prefer if you merged the series as-is
> and I then provided incremental updates.
>
> --
> Simon Horman simon@horms.nl
> Horms Solutions BV www.horms.nl
> Parnassusweg 819, 1082 LZ Amsterdam, Netherlands
> Tel: +31 (0)20 800 6155 Skype: horms7
>
>


--
With Kind Regards.

Scott McKeown
Loadbalancer.org
http://www.loadbalancer.org
Tel (UK) - +44 (0) 3303801064 (24x7)
Tel (US) - +1 888.867.9504 (Toll Free)(24x7)
Hi Scott,

On Mon, Jan 09, 2017 at 03:10:10PM +0000, Scott McKeown wrote:
> Hi Everyone,
>
> Simon I've been playing with this patch today and I'm sorry to say that
> adding it to a 1.8 dev build as of today does not look to work.
>
> I did have to manually update a few of the patch chunks so I'm willing to
> admit that it could be me that broke it and I will be having another go
> tomorrow unless Willy would be kind enough to update the 1.8 download for
> us with your patch already applied.
>
> However, the problem that I have now is that either the haproxy.stat file
> does not exist at all or I just keep getting:
>
> # echo "show info json" | socat /tmp/haproxy.stat stdio | python -m
> json.tool
> No JSON object could be decoded
>
>
> As I said I will try again tomorrow with a fresh brain and eyes.

OK many thanks for the heads up Scott, we'll wait for your go (and
possibly for a respin).

Thanks,
Willy
Hi Scott,

thanks for testing.

For reference the code I am using is here:

https://github.com/horms/haproxy.git show-json

And my minimal config file is as follows.
Would it be possible for you to share you config with me (privately) ?

global
daemon
stats socket /tmp/haproxy.stat mode 600 level admin
pidfile /tmp/haproxy.pid
log /dev/log local4
#tune.bufsize 2048
#tune.bufsize 4096
tune.bufsize 8192
#tune.bufsize 16384
tune.maxrewrite 1024

defaults
mode http
balance roundrobin
timeout connect 4000
timeout client 42000
timeout server 43000
log global


listen VIP_Name
bind 127.0.0.1:10080 transparent
mode http
balance leastconn
cookie SERVERID insert nocache indirect
server backup 127.0.0.1:9081 backup non-stick
option http-keep-alive
option forwardfor
option redispatch
option abortonclose
maxconn 40000
log global
option httplog
option log-health-checks
server RIP1_Name 127.0.0.1 weight 100 cookie RIP_Name agent-check agent-port 12345 agent-inter 2000 check port 80 inter 2000 rise 2 fall 3 minconn 0 maxconn 0s on-marked-down shutdown-sessions disabled
server RIP2_Name 127.0.0.1 weight 100 cookie RIP_Name agent-check agent-port 12345 agent-inter 2000 check port 80 inter 2000 rise 2 fall 3 minconn 0 maxconn 0s on-marked-down shutdown-sessions
No problem I'll have another look tomorrow morning and I'll let you all
know how I get on.


On 9 January 2017 at 15:18, Simon Horman <[email protected]> wrote:

> Hi Scott,
>
> thanks for testing.
>
> For reference the code I am using is here:
>
> https://github.com/horms/haproxy.git show-json
>
> And my minimal config file is as follows.
> Would it be possible for you to share you config with me (privately) ?
>
> global
> daemon
> stats socket /tmp/haproxy.stat mode 600 level admin
> pidfile /tmp/haproxy.pid
> log /dev/log local4
> #tune.bufsize 2048
> #tune.bufsize 4096
> tune.bufsize 8192
> #tune.bufsize 16384
> tune.maxrewrite 1024
>
> defaults
> mode http
> balance roundrobin
> timeout connect 4000
> timeout client 42000
> timeout server 43000
> log global
>
>
> listen VIP_Name
> bind 127.0.0.1:10080 transparent
> mode http
> balance leastconn
> cookie SERVERID insert nocache indirect
> server backup 127.0.0.1:9081 backup non-stick
> option http-keep-alive
> option forwardfor
> option redispatch
> option abortonclose
> maxconn 40000
> log global
> option httplog
> option log-health-checks
> server RIP1_Name 127.0.0.1 weight 100 cookie RIP_Name
> agent-check agent-port 12345 agent-inter 2000 check port 80 inter 2000 rise
> 2 fall 3 minconn 0 maxconn 0s on-marked-down shutdown-sessions disabled
> server RIP2_Name 127.0.0.1 weight 100 cookie RIP_Name
> agent-check agent-port 12345 agent-inter 2000 check port 80 inter 2000 rise
> 2 fall 3 minconn 0 maxconn 0s on-marked-down shutdown-sessions
>



--
With Kind Regards.

Scott McKeown
Loadbalancer.org
http://www.loadbalancer.org
Tel (UK) - +44 (0) 3303801064 (24x7)
Tel (US) - +1 888.867.9504 (Toll Free)(24x7)
Hi Guys,
Sorry for the delay I got tied up with some other issues yesterday but I've
just finished with Simons git repo pull.
Simon are you sure this is correct as I thought this was for a 1.8-dev
build which could be why I had problems on Tuesday.

# haproxy -v
HA-Proxy version 1.6-dev1 2015/03/11
Copyright 2000-2015 Willy Tarreau <[email protected]>

I'm still getting the following though:
# echo "show info json" | socat /tmp/haproxy.stat stdio | python -m
json.tool
No JSON object could be decoded

Its a basic build with the following options:
# make TARGET=linux26 USE_STATIC_PCRE=1 USE_LINUX_TPROXY=1

Oh and I'm running with your config file now.

Anything else that I can try or detail you would like to make sure its not
me?


~Scott



On 9 January 2017 at 16:13, Scott McKeown <[email protected]> wrote:

> No problem I'll have another look tomorrow morning and I'll let you all
> know how I get on.
>
>
> On 9 January 2017 at 15:18, Simon Horman <[email protected]> wrote:
>
>> Hi Scott,
>>
>> thanks for testing.
>>
>> For reference the code I am using is here:
>>
>> https://github.com/horms/haproxy.git show-json
>>
>> And my minimal config file is as follows.
>> Would it be possible for you to share you config with me (privately) ?
>>
>> global
>> daemon
>> stats socket /tmp/haproxy.stat mode 600 level admin
>> pidfile /tmp/haproxy.pid
>> log /dev/log local4
>> #tune.bufsize 2048
>> #tune.bufsize 4096
>> tune.bufsize 8192
>> #tune.bufsize 16384
>> tune.maxrewrite 1024
>>
>> defaults
>> mode http
>> balance roundrobin
>> timeout connect 4000
>> timeout client 42000
>> timeout server 43000
>> log global
>>
>>
>> listen VIP_Name
>> bind 127.0.0.1:10080 transparent
>> mode http
>> balance leastconn
>> cookie SERVERID insert nocache indirect
>> server backup 127.0.0.1:9081 backup non-stick
>> option http-keep-alive
>> option forwardfor
>> option redispatch
>> option abortonclose
>> maxconn 40000
>> log global
>> option httplog
>> option log-health-checks
>> server RIP1_Name 127.0.0.1 weight 100 cookie RIP_Name
>> agent-check agent-port 12345 agent-inter 2000 check port 80 inter 2000 rise
>> 2 fall 3 minconn 0 maxconn 0s on-marked-down shutdown-sessions disabled
>> server RIP2_Name 127.0.0.1 weight 100 cookie RIP_Name
>> agent-check agent-port 12345 agent-inter 2000 check port 80 inter 2000 rise
>> 2 fall 3 minconn 0 maxconn 0s on-marked-down shutdown-sessions
>>
>
>
>
> --
> With Kind Regards.
>
> Scott McKeown
> Loadbalancer.org
> http://www.loadbalancer.org
> Tel (UK) - +44 (0) 3303801064 <0330%20380%201064> (24x7)
> Tel (US) - +1 888.867.9504 <+1%20888-867-9504> (Toll Free)(24x7)
>



--
With Kind Regards.

Scott McKeown
Loadbalancer.org
http://www.loadbalancer.org
Tel (UK) - +44 (0) 3303801064 (24x7)
Tel (US) - +1 888.867.9504 (Toll Free)(24x7)
Sorry forgot to show the /tmp/ folder

# ls -latrh /tmp/
total 16K
-rw-------. 1 root root 0 Jan 9 11:00 yum.log
drwxrwxrwt. 2 root root 4.0K Jan 9 11:21 .ICE-unix
dr-xr-xr-x. 22 root root 4.0K Jan 9 11:21 ..
srw-------. 1 root root 0 Jan 9 14:39 haproxy.stat
-rw-r--r--. 1 root root 5 Jan 9 14:39 haproxy.pid
drwxrwxrwt. 3 root root 4.0K Jan 9 14:44 .




On 12 January 2017 at 13:18, Scott McKeown <[email protected]> wrote:

> Hi Guys,
> Sorry for the delay I got tied up with some other issues yesterday but
> I've just finished with Simons git repo pull.
> Simon are you sure this is correct as I thought this was for a 1.8-dev
> build which could be why I had problems on Tuesday.
>
> # haproxy -v
> HA-Proxy version 1.6-dev1 2015/03/11
> Copyright 2000-2015 Willy Tarreau <[email protected]>
>
> I'm still getting the following though:
> # echo "show info json" | socat /tmp/haproxy.stat stdio | python -m
> json.tool
> No JSON object could be decoded
>
> Its a basic build with the following options:
> # make TARGET=linux26 USE_STATIC_PCRE=1 USE_LINUX_TPROXY=1
>
> Oh and I'm running with your config file now.
>
> Anything else that I can try or detail you would like to make sure its not
> me?
>
>
> ~Scott
>
>
>
> On 9 January 2017 at 16:13, Scott McKeown <[email protected]> wrote:
>
>> No problem I'll have another look tomorrow morning and I'll let you all
>> know how I get on.
>>
>>
>> On 9 January 2017 at 15:18, Simon Horman <[email protected]> wrote:
>>
>>> Hi Scott,
>>>
>>> thanks for testing.
>>>
>>> For reference the code I am using is here:
>>>
>>> https://github.com/horms/haproxy.git show-json
>>>
>>> And my minimal config file is as follows.
>>> Would it be possible for you to share you config with me (privately) ?
>>>
>>> global
>>> daemon
>>> stats socket /tmp/haproxy.stat mode 600 level admin
>>> pidfile /tmp/haproxy.pid
>>> log /dev/log local4
>>> #tune.bufsize 2048
>>> #tune.bufsize 4096
>>> tune.bufsize 8192
>>> #tune.bufsize 16384
>>> tune.maxrewrite 1024
>>>
>>> defaults
>>> mode http
>>> balance roundrobin
>>> timeout connect 4000
>>> timeout client 42000
>>> timeout server 43000
>>> log global
>>>
>>>
>>> listen VIP_Name
>>> bind 127.0.0.1:10080 transparent
>>> mode http
>>> balance leastconn
>>> cookie SERVERID insert nocache indirect
>>> server backup 127.0.0.1:9081 backup non-stick
>>> option http-keep-alive
>>> option forwardfor
>>> option redispatch
>>> option abortonclose
>>> maxconn 40000
>>> log global
>>> option httplog
>>> option log-health-checks
>>> server RIP1_Name 127.0.0.1 weight 100 cookie RIP_Name
>>> agent-check agent-port 12345 agent-inter 2000 check port 80 inter 2000 rise
>>> 2 fall 3 minconn 0 maxconn 0s on-marked-down shutdown-sessions disabled
>>> server RIP2_Name 127.0.0.1 weight 100 cookie RIP_Name
>>> agent-check agent-port 12345 agent-inter 2000 check port 80 inter 2000 rise
>>> 2 fall 3 minconn 0 maxconn 0s on-marked-down shutdown-sessions
>>>
>>
>>
>>
>> --
>> With Kind Regards.
>>
>> Scott McKeown
>> Loadbalancer.org
>> http://www.loadbalancer.org
>> Tel (UK) - +44 (0) 3303801064 <0330%20380%201064> (24x7)
>> Tel (US) - +1 888.867.9504 <+1%20888-867-9504> (Toll Free)(24x7)
>>
>
>
>
> --
> With Kind Regards.
>
> Scott McKeown
> Loadbalancer.org
> http://www.loadbalancer.org
> Tel (UK) - +44 (0) 3303801064 <0330%20380%201064> (24x7)
> Tel (US) - +1 888.867.9504 <+1%20888-867-9504> (Toll Free)(24x7)
>



--
With Kind Regards.

Scott McKeown
Loadbalancer.org
http://www.loadbalancer.org
Tel (UK) - +44 (0) 3303801064 (24x7)
Tel (US) - +1 888.867.9504 (Toll Free)(24x7)
Hi Scott,

could you send the output of the following?

echo "show info json" | socat /tmp/haproxy.stat stdio

On Thu, Jan 12, 2017 at 01:18:54PM +0000, Scott McKeown wrote:
> Hi Guys,
> Sorry for the delay I got tied up with some other issues yesterday but I've
> just finished with Simons git repo pull.
> Simon are you sure this is correct as I thought this was for a 1.8-dev
> build which could be why I had problems on Tuesday.
>
> # haproxy -v
> HA-Proxy version 1.6-dev1 2015/03/11
> Copyright 2000-2015 Willy Tarreau <[email protected]>
>
> I'm still getting the following though:
> # echo "show info json" | socat /tmp/haproxy.stat stdio | python -m
> json.tool
> No JSON object could be decoded
>
> Its a basic build with the following options:
> # make TARGET=linux26 USE_STATIC_PCRE=1 USE_LINUX_TPROXY=1
>
> Oh and I'm running with your config file now.
>
> Anything else that I can try or detail you would like to make sure its not
> me?
>
>
> ~Scott
>
>
>
> On 9 January 2017 at 16:13, Scott McKeown <[email protected]> wrote:
>
> > No problem I'll have another look tomorrow morning and I'll let you all
> > know how I get on.
> >
> >
> > On 9 January 2017 at 15:18, Simon Horman <[email protected]> wrote:
> >
> >> Hi Scott,
> >>
> >> thanks for testing.
> >>
> >> For reference the code I am using is here:
> >>
> >> https://github.com/horms/haproxy.git show-json
> >>
> >> And my minimal config file is as follows.
> >> Would it be possible for you to share you config with me (privately) ?
> >>
> >> global
> >> daemon
> >> stats socket /tmp/haproxy.stat mode 600 level admin
> >> pidfile /tmp/haproxy.pid
> >> log /dev/log local4
> >> #tune.bufsize 2048
> >> #tune.bufsize 4096
> >> tune.bufsize 8192
> >> #tune.bufsize 16384
> >> tune.maxrewrite 1024
> >>
> >> defaults
> >> mode http
> >> balance roundrobin
> >> timeout connect 4000
> >> timeout client 42000
> >> timeout server 43000
> >> log global
> >>
> >>
> >> listen VIP_Name
> >> bind 127.0.0.1:10080 transparent
> >> mode http
> >> balance leastconn
> >> cookie SERVERID insert nocache indirect
> >> server backup 127.0.0.1:9081 backup non-stick
> >> option http-keep-alive
> >> option forwardfor
> >> option redispatch
> >> option abortonclose
> >> maxconn 40000
> >> log global
> >> option httplog
> >> option log-health-checks
> >> server RIP1_Name 127.0.0.1 weight 100 cookie RIP_Name
> >> agent-check agent-port 12345 agent-inter 2000 check port 80 inter 2000 rise
> >> 2 fall 3 minconn 0 maxconn 0s on-marked-down shutdown-sessions disabled
> >> server RIP2_Name 127.0.0.1 weight 100 cookie RIP_Name
> >> agent-check agent-port 12345 agent-inter 2000 check port 80 inter 2000 rise
> >> 2 fall 3 minconn 0 maxconn 0s on-marked-down shutdown-sessions
> >>
> >
> >
> >
> > --
> > With Kind Regards.
> >
> > Scott McKeown
> > Loadbalancer.org
> > http://www.loadbalancer.org
> > Tel (UK) - +44 (0) 3303801064 <0330%20380%201064> (24x7)
> > Tel (US) - +1 888.867.9504 <+1%20888-867-9504> (Toll Free)(24x7)
> >
>
>
>
> --
> With Kind Regards.
>
> Scott McKeown
> Loadbalancer.org
> http://www.loadbalancer.org
> Tel (UK) - +44 (0) 3303801064 (24x7)
> Tel (US) - +1 888.867.9504 (Toll Free)(24x7)

--
Simon Horman simon@horms.nl
Horms Solutions BV www.horms.nl
Parnassusweg 819, 1082 LZ Amsterdam, Netherlands
Tel: +31 (0)20 800 6155 Skype: horms7
Hi Simon,

Output below:

# echo "show info json" | socat /tmp/haproxy.stat stdio
Name: HAProxy
Version: 1.6-dev1
Release_date: 2015/03/11
Nbproc: 1
Process_num: 1
Pid: 1611
Uptime: 0d 0h14m04s
Uptime_sec: 844
Memmax_MB: 0
Ulimit-n: 4037
Maxsock: 4037
Maxconn: 2000
Hard_maxconn: 2000
CurrConns: 1
CumConns: 177
CumReq: 1005
Maxpipes: 0
PipesUsed: 0
PipesFree: 0
ConnRate: 0
ConnRateLimit: 0
MaxConnRate: 7
SessRate: 0
SessRateLimit: 0
MaxSessRate: 7
CompressBpsIn: 0
CompressBpsOut: 0
CompressBpsRateLim: 0
Tasks: 9
Run_queue: 1
Idle_pct: 100
node: localhost.localdomain
description:



On 12 January 2017 at 13:23, Simon Horman <[email protected]> wrote:

> Hi Scott,
>
> could you send the output of the following?
>
> echo "show info json" | socat /tmp/haproxy.stat stdio
>
> On Thu, Jan 12, 2017 at 01:18:54PM +0000, Scott McKeown wrote:
> > Hi Guys,
> > Sorry for the delay I got tied up with some other issues yesterday but
> I've
> > just finished with Simons git repo pull.
> > Simon are you sure this is correct as I thought this was for a 1.8-dev
> > build which could be why I had problems on Tuesday.
> >
> > # haproxy -v
> > HA-Proxy version 1.6-dev1 2015/03/11
> > Copyright 2000-2015 Willy Tarreau <[email protected]>
> >
> > I'm still getting the following though:
> > # echo "show info json" | socat /tmp/haproxy.stat stdio | python -m
> > json.tool
> > No JSON object could be decoded
> >
> > Its a basic build with the following options:
> > # make TARGET=linux26 USE_STATIC_PCRE=1 USE_LINUX_TPROXY=1
> >
> > Oh and I'm running with your config file now.
> >
> > Anything else that I can try or detail you would like to make sure its
> not
> > me?
> >
> >
> > ~Scott
> >
> >
> >
> > On 9 January 2017 at 16:13, Scott McKeown <[email protected]>
> wrote:
> >
> > > No problem I'll have another look tomorrow morning and I'll let you all
> > > know how I get on.
> > >
> > >
> > > On 9 January 2017 at 15:18, Simon Horman <[email protected]> wrote:
> > >
> > >> Hi Scott,
> > >>
> > >> thanks for testing.
> > >>
> > >> For reference the code I am using is here:
> > >>
> > >> https://github.com/horms/haproxy.git show-json
> > >>
> > >> And my minimal config file is as follows.
> > >> Would it be possible for you to share you config with me (privately) ?
> > >>
> > >> global
> > >> daemon
> > >> stats socket /tmp/haproxy.stat mode 600 level admin
> > >> pidfile /tmp/haproxy.pid
> > >> log /dev/log local4
> > >> #tune.bufsize 2048
> > >> #tune.bufsize 4096
> > >> tune.bufsize 8192
> > >> #tune.bufsize 16384
> > >> tune.maxrewrite 1024
> > >>
> > >> defaults
> > >> mode http
> > >> balance roundrobin
> > >> timeout connect 4000
> > >> timeout client 42000
> > >> timeout server 43000
> > >> log global
> > >>
> > >>
> > >> listen VIP_Name
> > >> bind 127.0.0.1:10080 transparent
> > >> mode http
> > >> balance leastconn
> > >> cookie SERVERID insert nocache indirect
> > >> server backup 127.0.0.1:9081 backup non-stick
> > >> option http-keep-alive
> > >> option forwardfor
> > >> option redispatch
> > >> option abortonclose
> > >> maxconn 40000
> > >> log global
> > >> option httplog
> > >> option log-health-checks
> > >> server RIP1_Name 127.0.0.1 weight 100 cookie RIP_Name
> > >> agent-check agent-port 12345 agent-inter 2000 check port 80 inter
> 2000 rise
> > >> 2 fall 3 minconn 0 maxconn 0s on-marked-down shutdown-sessions
> disabled
> > >> server RIP2_Name 127.0.0.1 weight 100 cookie RIP_Name
> > >> agent-check agent-port 12345 agent-inter 2000 check port 80 inter
> 2000 rise
> > >> 2 fall 3 minconn 0 maxconn 0s on-marked-down shutdown-sessions
> > >>
> > >
> > >
> > >
> > > --
> > > With Kind Regards.
> > >
> > > Scott McKeown
> > > Loadbalancer.org
> > > http://www.loadbalancer.org
> > > Tel (UK) - +44 (0) 3303801064 <0330%20380%201064> (24x7)
> > > Tel (US) - +1 888.867.9504 <+1%20888-867-9504> (Toll Free)(24x7)
> > >
> >
> >
> >
> > --
> > With Kind Regards.
> >
> > Scott McKeown
> > Loadbalancer.org
> > http://www.loadbalancer.org
> > Tel (UK) - +44 (0) 3303801064 (24x7)
> > Tel (US) - +1 888.867.9504 (Toll Free)(24x7)
>
> --
> Simon Horman simon@horms.nl
> Horms Solutions BV www.horms.nl
> Parnassusweg 819, 1082 LZ Amsterdam, Netherlands
> Tel: +31 (0)20 800 6155 Skype: horms7
>



--
With Kind Regards.

Scott McKeown
Loadbalancer.org
http://www.loadbalancer.org
Tel (UK) - +44 (0) 3303801064 (24x7)
Tel (US) - +1 888.867.9504 (Toll Free)(24x7)
On Thu, Jan 12, 2017 at 01:27:37PM +0000, Scott McKeown wrote:
> Hi Simon,
>
> Output below:

....

Thanks

> On 12 January 2017 at 13:23, Simon Horman <[email protected]> wrote:
>
> > Hi Scott,
> >
> > could you send the output of the following?
> >
> > echo "show info json" | socat /tmp/haproxy.stat stdio
> >
> > On Thu, Jan 12, 2017 at 01:18:54PM +0000, Scott McKeown wrote:
> > > Hi Guys,
> > > Sorry for the delay I got tied up with some other issues yesterday but
> > I've
> > > just finished with Simons git repo pull.
> > > Simon are you sure this is correct as I thought this was for a 1.8-dev
> > > build which could be why I had problems on Tuesday.
> > >
> > > # haproxy -v
> > > HA-Proxy version 1.6-dev1 2015/03/11
> > > Copyright 2000-2015 Willy Tarreau <[email protected]>

....

The above seems wrong. I see:

../haproxy -v
HA-Proxy version 1.8-dev0-b738a8-135 2017/01/04
Copyright 2000-2016 Willy Tarreau <[email protected]>

Could you verify that the head commit you have checked out is as follows?

git log --oneline -1
b738a8b596d8 MEDIUM: stats: Add show json schema
Doh, that would do it.
Sorry wrong git branch.

# haproxy -v
HA-Proxy version 1.8-dev0 2016/11/25
Copyright 2000-2016 Willy Tarreau <[email protected]>

# echo "show info json" | socat /tmp/haproxy.stat stdio | python -m
json.tool
[
{
"field": {
"name": "Name",
"pos": 0
},
"processNum": 1,
"tags": {
"nature": "Output",
"origin": "Product",
"scope": "Service"
},
"value": {
"type": "str",
"value": "HAProxy"
}
},
{
"field": {
"name": "Version",
"pos": 1
},
"processNum": 1,
"tags": {
"nature": "Output",
"origin": "Product",
"scope": "Service"
},
"value": {

etc. etc. etc.



On 12 January 2017 at 13:34, Simon Horman <[email protected]> wrote:

> On Thu, Jan 12, 2017 at 01:27:37PM +0000, Scott McKeown wrote:
> > Hi Simon,
> >
> > Output below:
>
> ...
>
> Thanks
>
> > On 12 January 2017 at 13:23, Simon Horman <[email protected]> wrote:
> >
> > > Hi Scott,
> > >
> > > could you send the output of the following?
> > >
> > > echo "show info json" | socat /tmp/haproxy.stat stdio
> > >
> > > On Thu, Jan 12, 2017 at 01:18:54PM +0000, Scott McKeown wrote:
> > > > Hi Guys,
> > > > Sorry for the delay I got tied up with some other issues yesterday
> but
> > > I've
> > > > just finished with Simons git repo pull.
> > > > Simon are you sure this is correct as I thought this was for a
> 1.8-dev
> > > > build which could be why I had problems on Tuesday.
> > > >
> > > > # haproxy -v
> > > > HA-Proxy version 1.6-dev1 2015/03/11
> > > > Copyright 2000-2015 Willy Tarreau <[email protected]>
>
> ...
>
> The above seems wrong. I see:
>
> ./haproxy -v
> HA-Proxy version 1.8-dev0-b738a8-135 2017/01/04
> Copyright 2000-2016 Willy Tarreau <[email protected]>
>
> Could you verify that the head commit you have checked out is as follows?
>
> git log --oneline -1
> b738a8b596d8 MEDIUM: stats: Add show json schema
>



--
With Kind Regards.

Scott McKeown
Loadbalancer.org
http://www.loadbalancer.org
Tel (UK) - +44 (0) 3303801064 (24x7)
Tel (US) - +1 888.867.9504 (Toll Free)(24x7)
On Thu, Jan 12, 2017 at 03:14:27PM +0000, Scott McKeown wrote:
> Doh, that would do it.
> Sorry wrong git branch.
>
> # haproxy -v
> HA-Proxy version 1.8-dev0 2016/11/25
> Copyright 2000-2016 Willy Tarreau <[email protected]>
>
> # echo "show info json" | socat /tmp/haproxy.stat stdio | python -m
> json.tool
> [
> {
> "field": {
> "name": "Name",
> "pos": 0
> },
> "processNum": 1,
> "tags": {
> "nature": "Output",
> "origin": "Product",
> "scope": "Service"
> },
> "value": {
> "type": "str",
> "value": "HAProxy"
> }
> },
> {
> "field": {
> "name": "Version",
> "pos": 1
> },
> "processNum": 1,
> "tags": {
> "nature": "Output",
> "origin": "Product",
> "scope": "Service"
> },
> "value": {
>
> etc. etc. etc.

Thanks, that looks like what I would expect.

--
Simon Horman simon@horms.nl
Horms Solutions BV www.horms.nl
Parnassusweg 819, 1082 LZ Amsterdam, Netherlands
Tel: +31 (0)20 800 6155 Skype: horms7
Sorry, only registered users may post in this forum.

Click here to login