Welcome! Log In Create A New Profile

Advanced

[PHP-DEV] A validator module for PHP7

Posted by Yasuo Ohgaki 
Yasuo Ohgaki
[PHP-DEV] A validator module for PHP7
September 04, 2017 08:40AM
Hi all,

I spent a little time for a new input validation module. It's not totally
new module, but is based on Filter module's validation filter improvement
RFC in many ways. [1]

As all of us knew already, input validation is the most important practice
in secure coding. [2][3] Yet, we don't provide usable feature out of box.
Sadly, almost all apps do not have proper input validation at trust
boundary. Unless we improve filter's validation, we need usable basic
validator by default. IMO.

Since I didn't get much feedbacks during the RFC discussion, I cannot tell
what part is disliked. I guess too much features in filter is one reason.
Another is messed up codes/features by providing both "filter" and
"validation".

Validator for PHP7 (validate module) gets rid of unneeded features. It only
has features for basic PHP data type validations. Validation rule(spec)
array is flexible enough. Almost any types of inputs could be handled by
multiple and nested validation rules.

Except some minor features like overflow checks, most planned features are
implemented.

https://github.com/yohgaki/validate-php

Although the code is based on filter module's code, it's almost full
rewrite except validation logic came from filter. Please consider this as
under development module.
Feedbacks are appreciated.

Regards,

[1] https://wiki.php.net/rfc/add_validate_functions_to_filter
[2]
https://www.securecoding.cert.org/confluence/display/seccode/Top+10+Secure+Coding+Practices
[3]
https://www.owasp.org/index.php/OWASP_Secure_Coding_Practices_-_Quick_Reference_Guide

--
Yasuo Ohgaki
yohgaki@ohgaki.net
Rowan Collins
Re: [PHP-DEV] A validator module for PHP7
September 04, 2017 03:00PM
On 4 September 2017 07:33:41 BST, Yasuo Ohgaki <[email protected]> wrote:
>Hi all,
>
>I spent a little time for a new input validation module. It's not
>totally
>new module, but is based on Filter module's validation filter
>improvement
>RFC in many ways. [1]

Hi Yasuo,

Thanks for tackling this. I do think the current filter module is user unfriendly and it would be great to have something better. A couple of quick thoughts based on your README:

- The use of nested arrays keeps things simple in one sense, but the deep nesting can get confusing. I wonder if a ValidationRule class would make the distinction between parameters and references to existing rules clearer. This would also give you a natural home for validating the validation rules themselves (probably by throwing an exception in the constructor).

- A minor point, but most style guides would suggest function names should be verbs, not adjectives, so "validate()" rather than "valid()".

- Is there a use case for valid_id() or is it a temporary debugging thing that won't be in the final version?

In general, I like the idea, but would have to play around a bit to see if it felt easy to use in real world situations.

Regards,

--
Rowan Collins
[IMSoP]

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Crocodile
Re: [PHP-DEV] A validator module for PHP7
September 04, 2017 09:00PM
In most cases users would like more than just valid/invalid, i. e. which
particular rule(s) failed and also human-readable error messages. As of
simple validation that is almost always at hand, filter_* functions do a
good job, although I agree that they could be better. I, for one, would
like to see a full-featured validation as part of PHP. However, this RFC
only looks like a slightly better version of filter_* functions, that is,
the new module will provide almost the same functionality but with a
different interface. I would vote against it.

On Mon, Sep 4, 2017 at 2:54 PM Rowan Collins <[email protected]>
wrote:

> On 4 September 2017 07:33:41 BST, Yasuo Ohgaki <[email protected]> wrote:
> >Hi all,
> >
> >I spent a little time for a new input validation module. It's not
> >totally
> >new module, but is based on Filter module's validation filter
> >improvement
> >RFC in many ways. [1]
>
> Hi Yasuo,
>
> Thanks for tackling this. I do think the current filter module is user
> unfriendly and it would be great to have something better. A couple of
> quick thoughts based on your README:
>
> - The use of nested arrays keeps things simple in one sense, but the deep
> nesting can get confusing. I wonder if a ValidationRule class would make
> the distinction between parameters and references to existing rules
> clearer. This would also give you a natural home for validating the
> validation rules themselves (probably by throwing an exception in the
> constructor).
>
> - A minor point, but most style guides would suggest function names should
> be verbs, not adjectives, so "validate()" rather than "valid()".
>
> - Is there a use case for valid_id() or is it a temporary debugging thing
> that won't be in the final version?
>
> In general, I like the idea, but would have to play around a bit to see if
> it felt easy to use in real world situations.
>
> Regards,
>
> --
> Rowan Collins
> [IMSoP]
>
> --
> PHP Internals - PHP Runtime Development Mailing List
> To unsubscribe, visit: http://www.php.net/unsub.php
>
> --
Best regards,
Victor Bolshov
Marco Pivetta
Re: [PHP-DEV] A validator module for PHP7
September 05, 2017 01:10AM
On Mon, Sep 4, 2017 at 8:56 PM, Crocodile <[email protected]> wrote:

> In most cases users would like more than just valid/invalid, i. e. which
> particular rule(s) failed and also human-readable error messages. As of
> simple validation that is almost always at hand, filter_* functions do a
> good job, although I agree that they could be better. I, for one, would
> like to see a full-featured validation as part of PHP. However, this RFC
> only looks like a slightly better version of filter_* functions, that is,
> the new module will provide almost the same functionality but with a
> different interface. I would vote against it.
>

And also, it would need to be better than all of these to be worth writing
it in C:

https://packagist.org/search/?q=validator

Marco Pivetta

http://twitter.com/Ocramius

http://ocramius.github.com/
Paul Jones
Re: [PHP-DEV] A validator module for PHP7
September 05, 2017 05:30AM
> On Sep 4, 2017, at 18:06, Marco Pivetta <[email protected]> wrote:
>
> On Mon, Sep 4, 2017 at 8:56 PM, Crocodile <[email protected]> wrote:
>
>> In most cases users would like more than just valid/invalid, i. e. which
>> particular rule(s) failed and also human-readable error messages. As of
>> simple validation that is almost always at hand, filter_* functions do a
>> good job, although I agree that they could be better. I, for one, would
>> like to see a full-featured validation as part of PHP. However, this RFC
>> only looks like a slightly better version of filter_* functions, that is,
>> the new module will provide almost the same functionality but with a
>> different interface. I would vote against it.
>>
>
> And also, it would need to be better than all of these to be worth writing
> it in C:
>
> https://packagist.org/search/?q=validator

And these as well: https://packagist.org/search/?q=filter

(Or at least have some level of parity.)


--
Paul M. Jones
pmjones88@gmail.com
http://paul-m-jones.com

Modernizing Legacy Applications in PHP
https://leanpub.com/mlaphp

Solving the N+1 Problem in PHP
https://leanpub.com/sn1php




--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Yasuo Ohgaki
Re: [PHP-DEV] A validator module for PHP7
September 05, 2017 01:30PM
Hi all,

On Tue, Sep 5, 2017 at 12:19 PM, Paul Jones <[email protected]> wrote:

>
> > On Sep 4, 2017, at 18:06, Marco Pivetta <[email protected]> wrote:
> >
> > On Mon, Sep 4, 2017 at 8:56 PM, Crocodile <[email protected]> wrote:
> >
> >> In most cases users would like more than just valid/invalid, i. e. which
> >> particular rule(s) failed and also human-readable error messages. As of
> >> simple validation that is almost always at hand, filter_* functions do a
> >> good job, although I agree that they could be better. I, for one, would
> >> like to see a full-featured validation as part of PHP. However, this RFC
> >> only looks like a slightly better version of filter_* functions, that
> is,
> >> the new module will provide almost the same functionality but with a
> >> different interface. I would vote against it.
> >>
> >
> > And also, it would need to be better than all of these to be worth
> writing
> > it in C:
> >
> > https://packagist.org/search/?q=validator
>
> And these as well: https://packagist.org/search/?q=filter


I cannot guess people's thought. I appreciated feedback!

Why do you think basic validation module should be better than full OO
implementation(s)?

Simple and basic type support NULL/BOOL/INT/FLOAT/STRING/ARRAY/OBJECT is
enough
for C written PHP module. IMHO, PHP modules are better of to provide basic
features
that may be extended by user scripts.

I picked 1st one on the list as an example.
This kind of rule construction is inefficient compare to simple array rules,
so I doubt this is the way for basic validator module written by C.

$validator = Validation::createValidator();
$violations = $validator->validate('Bernhard', array(
new Length(array('min' => 10)),
new NotBlank(),
));

However, this particular validation class seems it could be good one that
wraps
validate module for nicer OO API and faster execution.

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net
Lester Caine
Re: [PHP-DEV] A validator module for PHP7
September 05, 2017 01:40PM
On 05/09/17 12:18, Yasuo Ohgaki wrote:
> I cannot guess people's thought. I appreciated feedback!

With a decent database layer a lot of the validation you are proposing
is already covered but PDO does not help in this area. Adding another
layer that does not integrate with a storage layer is just adding to the
current mess ...

--
Lester Caine - G8HFL
-----------------------------
Contact - http://lsces.co.uk/wiki/?page=contact
L.S.Caine Electronic Services - http://lsces.co.uk
EnquirySolve - http://enquirysolve.com/
Model Engineers Digital Workshop - http://medw.co.uk
Rainbow Digital Media - http://rainbowdigitalmedia.co.uk

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Rowan Collins
Re: [PHP-DEV] A validator module for PHP7
September 05, 2017 02:30PM
On 5 September 2017 12:36:42 BST, Lester Caine <[email protected]> wrote:
>On 05/09/17 12:18, Yasuo Ohgaki wrote:
>> I cannot guess people's thought. I appreciated feedback!
>
>With a decent database layer a lot of the validation you are proposing
>is already covered but PDO does not help in this area. Adding another
>layer that does not integrate with a storage layer is just adding to
>the
>current mess ...

Validation should have nothing to do with the storage layer. Or at least, there should be a level of validation separate from the storage layer.

Inputs may come from all sorts of sources: the HTTP request, an API, an import file, etc; and they may be going to all sorts of destinations: the HTTP response, a database, an API, an export file, an e-mail, etc.

Regardless of where it came from, and where it's going to end up, the application knows what format that input data should be in to use as or populate appropriate domain models. For instance, "age_in_years is a non-negative integer" is an invariant fact about the domain being modelled, even if it's a value that goes nowhere near any form of database.

I actually agree that this module doesn't need to replace the existing userland libraries, only to act as a useful base for them, as well as a useful fallback when writing "raw PHP". The key problem is balancing flexibility and usability such that people will reach for this tool rather than brewing their own.

Regards,

--
Rowan Collins
[IMSoP]

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Am 05.09.2017 um 13:36 schrieb Lester Caine:
> On 05/09/17 12:18, Yasuo Ohgaki wrote:
>> I cannot guess people's thought. I appreciated feedback!
>
> With a decent database layer a lot of the validation you are proposing
> is already covered but PDO does not help in this area. Adding another
> layer that does not integrate with a storage layer is just adding to the
> current mess ...

sorry, but you confuse "input validation" which this topic is about with
something different - input validation and reject bad requests belongs
some layers on top of any storage and should be done as soon as possible

that should even happen long before you open a database connection at
all because when you know the request is bad soon enough you won't talk
to any database, filesystem or whatever storage layer at all


the only question as applicaton developer is how you proceed in which cases

* reject the whole request with a error-message
* reset form-fields where you don't expect an array as input
* reset from-fields with out-of-range input values

here you go:
https://en.wikipedia.org/wiki/Data_validation

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Lester Caine
Re: [PHP-DEV] A validator module for PHP7
September 05, 2017 03:50PM
On 05/09/17 14:08, lists@rhsoft.net wrote:
>
>
> Am 05.09.2017 um 13:36 schrieb Lester Caine:
>> On 05/09/17 12:18, Yasuo Ohgaki wrote:
>>> I cannot guess people's thought. I appreciated feedback!
>>
>> With a decent database layer a lot of the validation you are proposing
>> is already covered but PDO does not help in this area. Adding another
>> layer that does not integrate with a storage layer is just adding to the
>> current mess ...
>
> sorry, but you confuse "input validation" which this topic is about with
> something different - input validation and reject bad requests belongs
> some layers on top of any storage and should be done as soon as possible
>
> that should even happen long before you open a database connection at
> all because when you know the request is bad soon enough you won't talk
> to any database, filesystem or whatever storage layer at all
>
> the only question as applicaton developer is how you proceed in which cases
>
> * reject the whole request with a error-message
> * reset form-fields where you don't expect an array as input
> * reset from-fields with out-of-range input values
>
> here you go:
> https://en.wikipedia.org/wiki/Data_validation

When the database layer provides a complete list of fields and
validation rules as part of it's meta data, it is integral to any GOOD
process. Copying all that data and manually creating filter rules is
just unnecessary work. In addition much of the VALIDATION is best done
at the browser end, and building that code is a lot easier when there is
a standard validation base across all of the layers!

Rejecting crap from hackers that have no format matching the fields on
the browser page is something else and if the data set is corrupt then
yes you can simply skip out before doing anything with it! But the
problem these days is when hackers try injecting things like SQL into
fields they think may be able to get through to the database. Provided
that the validation layer can properly filter that injection requires
knowledge that a string has reason to be rejected. Just as simply type
casting a number to integer or float is only doing a small part of the job.

Typing and validating a field by the metadata constraints has to be the
right way forward?

--
Lester Caine - G8HFL
-----------------------------
Contact - http://lsces.co.uk/wiki/?page=contact
L.S.Caine Electronic Services - http://lsces.co.uk
EnquirySolve - http://enquirysolve.com/
Model Engineers Digital Workshop - http://medw.co.uk
Rainbow Digital Media - http://rainbowdigitalmedia.co.uk

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Am 05.09.2017 um 15:44 schrieb Lester Caine:
> On 05/09/17 14:08, lists@rhsoft.net wrote:
>> the only question as applicaton developer is how you proceed in which cases
>>
>> * reject the whole request with a error-message
>> * reset form-fields where you don't expect an array as input
>> * reset from-fields with out-of-range input values
>>
>> here you go:
>> https://en.wikipedia.org/wiki/Data_validation
>
> When the database layer provides a complete list of fields and
> validation rules as part of it's meta data, it is integral to any GOOD
> process

your first error is thinking every input is related to databases at all

> Copying all that data and manually creating filter rules is
> just unnecessary work. In addition much of the VALIDATION is best done
> at the browser end, and building that code is a lot easier when there is
> a standard validation base across all of the layers!

NO VALIDATION is best done in the browser end because no attacker ever
will execute your clientside validation code or operate a browser at all

> Rejecting crap from hackers that have no format matching the fields on
> the browser page is something else and if the data set is corrupt then
> yes you can simply skip out before doing anything with it!

and that's what the whole topic is about

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Lester Caine
Re: [PHP-DEV] A validator module for PHP7
September 05, 2017 07:00PM
On 05/09/17 15:13, lists@rhsoft.net wrote:
> your first error is thinking every input is related to databases at all
So we end up with different code for different types of input?
An array that will work directly into a database save or some other
follow on process without having to think about where the input comes
from has to be the right way ...

>> Copying all that data and manually creating filter rules is
>> just unnecessary work. In addition much of the VALIDATION is best done
>> at the browser end, and building that code is a lot easier when there is
>> a standard validation base across all of the layers!
>
> NO VALIDATION is best done in the browser end because no attacker ever
> will execute your client side validation code or operate a browser at all
Again ... write different code for each area of checking?
My clients are complaining that the browser is not doing as good a job
as it could checking things that it CAN check before passing it back to
the server. YES the server needs to cross check no bugger has bypassed
the browser checks but if the set of data is EXPECTED to have a clean
format, then any corruption can be tagged as a failure, because we have
standard rules on what we are passing.

>> Rejecting crap from hackers that have no format matching the fields on
>> the browser page is something else and if the data set is corrupt then
>> yes you can simply skip out before doing anything with it!
>
> and that's what the whole topic is about
But not at the cost of writing different sets of code to play to each
area where checking SHOULD be done. Stick to a single standard method of
defining the metadata and that already exists in the database layer.

That was what the topic was all about 15 years ago and nothing much has
changed since ... and annotating that data in the code ...

--
Lester Caine - G8HFL
-----------------------------
Contact - http://lsces.co.uk/wiki/?page=contact
L.S.Caine Electronic Services - http://lsces.co.uk
EnquirySolve - http://enquirysolve.com/
Model Engineers Digital Workshop - http://medw.co.uk
Rainbow Digital Media - http://rainbowdigitalmedia.co.uk

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Am 05.09.2017 um 18:57 schrieb Lester Caine:
> But not at the cost of writing different sets of code to play to each
> area where checking SHOULD be done. Stick to a single standard method of
> defining the metadata and that already exists in the database layer

ok, to make that point clear:

not every input which needs to be validated or sanitized is *related to
a database at all* and hence input validation can't be done in the
database-layer and only there by definition for everything

frankly since 3 weeks our core-application is at a level where the
database-layer get not loaded at all until inputs are not verfified
because database stuff is on-demand and 80% of all requests when there
is some traffic within 80% seconds don't load the database layer because
of smart caching

form-input is validated, checked against CSRF-tokens, captcha and
*after* all the validations are fine the database layer becomes part of
the game - that alone brought again 43% higher requests/second on a
already highly optimizied codebase


--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Yasuo Ohgaki
Re: [PHP-DEV] A validator module for PHP7
September 05, 2017 09:00PM
Hi Lester,

On Tue, Sep 5, 2017 at 8:36 PM, Lester Caine <[email protected]> wrote:

> On 05/09/17 12:18, Yasuo Ohgaki wrote:
> > I cannot guess people's thought. I appreciated feedback!
>
> With a decent database layer a lot of the validation you are proposing
> is already covered but PDO does not help in this area. Adding another
> layer that does not integrate with a storage layer is just adding to the
> current mess ...
>

I'm fun of multiple tier and multiple layer of protections.
For instance, Microsoft's SQL injection security page states as follows.

- Never build Transact-SQL statements directly from user input; use stored
procedures to validate user input.

- Validate user input by testing type, length, format, and range. Use the
Transact-SQL QUOTENAME() function to escape system names or the REPLACE()
function to escape any character in a string.

- Implement multiple layers of validation in each tier of your application.

https://docs.microsoft.com/en-us/dotnet/framework/data/adonet/sql/writing-secure-dynamic-sql-in-sql-server

This is what secure coding practice recommends, too.
It may seem mess, but it's not. Outermost trust boundary that can be
controlled
is the most important trust boundary. For server side web app developers,
outermost
trust boundary is controller in MVC model. Input validations at model is a
bit too late
to mitigate risks involved with invalid(attacker) inputs.

Both model and controller layer Input validations (as well as in the
database, too) are
good/important to have.

There are one principle that developers are better to follow.
https://en.wikipedia.org/wiki/Fail-fast
If we follow this principle, validation at controller makes sense.

Regards,

P.S. For database administrators or web app developers who maintain
application
Models, outermost trust boundary is "database system" and "the Model layer"
respectively.
Outermost trust boundary is changed by what they can control.

This kind of discussion could result in mess. I hope I explained well
enough.

--
Yasuo Ohgaki
yohgaki@ohgaki.net
Yasuo Ohgaki
Re: [PHP-DEV] A validator module for PHP7
September 05, 2017 09:10PM
Hi Lester,

I always make some mistakes. s/are/is

There is one principle that developers is better to follow.
https://en.wikipedia.org/wiki/Fail-fast
If we follow this principle, validation at controller makes sense.

Anyway, thank you for pointer for PDO validation. I didn't notice the
project. We may cooperate so that there aren't unnecessary validaiton
rule incompatibilities.

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net
Lester Caine
Re: [PHP-DEV] A validator module for PHP7
September 05, 2017 09:50PM
On 05/09/17 20:05, Yasuo Ohgaki wrote:
> There is one principle that developers is better to follow.
> https://en.wikipedia.org/wiki/Fail-fast
> If we follow this principle, validation at controller makes sense.
Since a large proportion of the data coming in is a result of input into
a previously generated form, the data can often be validated in terms of
basic structure before even needing to decide if the data set needs to
be iterated? If things like 'maximum data size' can be established when
the form is created, any data set larger than that can simply be killed
off.

> Anyway, thank you for pointer for PDO validation. I didn't notice the
> project. We may cooperate so that there aren't unnecessary validaiton
> rule incompatibilities
I've been pushing the idea of a single method of managing metadata for a
long time. Most of the 'checking' loading down PHP now still misses the
point and the database style of rules has been around since long before
PDO and other abstractions messed it up. A single standard set of rules
that can be used across the board from variable creation to checking
data going out to forms and returns coming back and data between
operations such as database activity. This is NOT 'typing' since that
lacks the vast majority of checks that a decent validation will handle,
but the much finer details such as limits and value sets. There is a
vast discrepancy in how this is handled across databases, but the SQL
standard does provide a base which databases are slowly evolving towards.

--
Lester Caine - G8HFL
-----------------------------
Contact - http://lsces.co.uk/wiki/?page=contact
L.S.Caine Electronic Services - http://lsces.co.uk
EnquirySolve - http://enquirysolve.com/
Model Engineers Digital Workshop - http://medw.co.uk
Rainbow Digital Media - http://rainbowdigitalmedia.co.uk

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Rowan Collins
Re: [PHP-DEV] A validator module for PHP7
September 05, 2017 10:50PM
On 5 September 2017 20:39:36 BST, Lester Caine <[email protected]> wrote:
>I've been pushing the idea of a single method of managing metadata for
>a long time.
> A single standard set of rules
>that can be used across the board from variable creation to checking
>data going out to forms and returns coming back and data between
>operations such as database activity.

Sounds great, and sounds very much like you agree with Yasuo's aims, but have some ideas on how the rules could be structured or expressed differently. Can you give any examples of how you might express these rules, so we can compare with Yasuo's ext/filter based syntax, and with some of the other validation libraries which are out there?

Regards,

--
Rowan Collins
[IMSoP]

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Yasuo Ohgaki
Re: [PHP-DEV] A validator module for PHP7
September 05, 2017 11:00PM
Hi Rowan, Crocodile and Lester,

Rowan,
Thank you for helpful suggestions. I think it may better be named
'validate()'/etc.
Sole reason I named functions as 'valid*()' is name collision.
'validate*()' seems
too common, but users should use namespace to avoid collisions. I'll take
your
suggestions into the module.

Crocodile,
I agree your comment. However, my last filter module improvement RFC
with PoC patch was failed. Upside having new module is we can enforce more
strict validations and can have much simpler/extensible API with cleaner
codes.
(Having both 'filter' and 'validator' in one code is mess)
Since the last attempt was failed, new module would be the next reasonable
attempt.

On Wed, Sep 6, 2017 at 4:39 AM, Lester Caine <[email protected]> wrote:

> On 05/09/17 20:05, Yasuo Ohgaki wrote:
> > There is one principle that developers is better to follow.
> > https://en.wikipedia.org/wiki/Fail-fast
> > If we follow this principle, validation at controller makes sense.
>
Oops. A 'is' should be 'are'.

> Since a large proportion of the data coming in is a result of input into
> a previously generated form, the data can often be validated in terms of
> basic structure before even needing to decide if the data set needs to
> be iterated? If things like 'maximum data size' can be established when
> the form is created, any data set larger than that can simply be killed
> off.
>

> Anyway, thank you for pointer for PDO validation. I didn't notice the
> > project. We may cooperate so that there aren't unnecessary validaiton
> > rule incompatibilities
> I've been pushing the idea of a single method of managing metadata for a
> long time. Most of the 'checking' loading down PHP now still misses the
> point and the database style of rules has been around since long before
> PDO and other abstractions messed it up. A single standard set of rules
> that can be used across the board from variable creation to checking
> data going out to forms and returns coming back and data between
> operations such as database activity. This is NOT 'typing' since that
> lacks the vast majority of checks that a decent validation will handle,
> but the much finer details such as limits and value sets. There is a
> vast discrepancy in how this is handled across databases, but the SQL
> standard does provide a base which databases are slowly evolving towards.
>
>
It's nice to have central input type (Not only data type, but length,
chars, format, range and char encoding, as you mentioned) repository for an
app. That's the reason why I would like to cooperate with other validator
implementation(s) to avoid unnecessary incompatibilities.

(For security perspective, it can be said different types of input
validations is
better because if one failed, another may validate data correctly. However,
managing multiple validation rules is burden. Some people try to validate
inputs with WAF. However, I suggest input validation is better to be
implemented
at web apps, not WAF.
It's a lot easier and maintainable if input validation is done by app. App
developers know what the input should be exactly. In addition, app must
validate inputs regardless of WAF. I'm not saying WAF is useless. WAF
is still useful as network layer protection. I'm saying multiple layer
validations
are recommended, but it can be unmanageable easily unless some trade
off is taken)

I agree that PDO level validation is almost mandatory, especially for
SQLite3.
SQLite3 data type is pseudo type and allows any "characters". e.g. Int type
can store '<script>alert("XSS")</script>', etc.

My validate module allows users to have central input type repository by
simple native PHP array. Array could be stored anywhere users want.
I think it will work as you wish if 'validate' module is compatible with
PDO
validator.

For the time being, I'm not sure what it should be. Data specifications for
database may be stored as additional info in 'validate' spec array, or PDO
validator may simply assume data types specified in 'validate' spec array
should be enforced, and use 'string' data spec as required format,
or 'validate' module may expose API so that PDO validator can use it for
basic PHP data type validation.

Regards,

--
Yasuo Ohgaki
Lester Caine
Re: [PHP-DEV] A validator module for PHP7
September 06, 2017 10:40AM
On 05/09/17 21:45, Rowan Collins wrote:
>> I've been pushing the idea of a single method of managing metadata for
>> a long time.
>> A single standard set of rules
>> that can be used across the board from variable creation to checking
>> data going out to forms and returns coming back and data between
>> operations such as database activity.
> Sounds great, and sounds very much like you agree with Yasuo's aims, but have some ideas on how the rules could be structured or expressed differently. Can you give any examples of how you might express these rules, so we can compare with Yasuo's ext/filter based syntax, and with some of the other validation libraries which are out there?

My only problem with Yasuo's latest offering is once again it adds a
whole new set of defines that have to be mapped to existing metadata
definitions ... That and it is a lot of longhand code using a different
style to existing arrays. We need yet another wrapper to build these
arrays from existing code ...

--
Lester Caine - G8HFL
-----------------------------
Contact - http://lsces.co.uk/wiki/?page=contact
L.S.Caine Electronic Services - http://lsces.co.uk
EnquirySolve - http://enquirysolve.com/
Model Engineers Digital Workshop - http://medw.co.uk
Rainbow Digital Media - http://rainbowdigitalmedia.co.uk

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Rowan Collins
Re: [PHP-DEV] A validator module for PHP7
September 06, 2017 01:20PM
On 6 September 2017 09:29:37 BST, Lester Caine <[email protected]> wrote:
>My only problem with Yasuo's latest offering is once again it adds a
>whole new set of defines that have to be mapped to existing metadata
>definitions ... That and it is a lot of longhand code using a different
>style to existing arrays. We need yet another wrapper to build these
>arrays from existing code ...

The rules have to be defined somehow, and I'm not aware of a standard format that current code is likely to follow. Unless there is already such a standard, I can't see any way to avoid existing code having to be wrapped or amended.

Which is why Yasuo and I have both suggested we work together to come up with such a standard format that can be used or adapted for these different parts of the application. If you have suggestions for how the format should look, we are eager to hear them and see some examples.

Regards,

--
Rowan Collins
[IMSoP]

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Dan Ackroyd
Re: [PHP-DEV] A validator module for PHP7
September 06, 2017 01:40PM
On 6 September 2017 at 12:15, Rowan Collins <[email protected]> wrote:

> If you have suggestions for how the format should look

Don't use a format. Just write code - see below.

> Which is why Yasuo and I have both suggested we work together

If you're going to work together and continue the conversation, please
can you move this conversation elsewhere?

It doesn't appear to be actually anything to do with PHP internals.

On 4 September 2017 at 07:33, Yasuo Ohgaki <[email protected]> wrote:
>
> Since I didn't get much feedbacks during the RFC discussion, I cannot tell
> what part is disliked.

Yes you did. You got feedback during the discussion and also during
the vote. For example: http://news.php.net/php.internals/95164

However you continually choose to ignore that feedback.

I will attempt once more, to get the main point through to you.
Perhaps a small amount of repetition, will get it through:

This type of library should be done in PHP, not in C.
This type of library should be done in PHP, not in C.
This type of library should be done in PHP, not in C.
This type of library should be done in PHP, not in C.
This type of library should be done in PHP, not in C.
This type of library should be done in PHP, not in C.
This type of library should be done in PHP, not in C.
This type of library should be done in PHP, not in C.
This type of library should be done in PHP, not in C.
This type of library should be done in PHP, not in C.
This type of library should be done in PHP, not in C.

cheers
Dan
Ack


function validateOrderAmount($value) : int {
$count = preg_match("/[^0-9]*/", $value);

if ($count) {
throw new InvalidOrderAmount("The order value must contain
only digits.");
}

$value = intval($value);

if ($value < 1) {
throw new InvalidOrderAmount("The order value must be one or more.");
}

if ($value >= MAX_ORDER_AMOUNT) {
throw new InvalidOrderAmount(
"Order value to big. Maximum allowed value is ".MAX_ORDER_AMOUNT
);
}

return $value;
}

(i'd probably recommend not using exceptions, but instead return
[$valid, $value] to allow validating multiple items without having to
use exceptions for flow control.)

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Lester Caine
Re: [PHP-DEV] A validator module for PHP7
September 06, 2017 02:00PM
On 06/09/17 12:15, Rowan Collins wrote:
> On 6 September 2017 09:29:37 BST, Lester Caine <[email protected]> wrote:
>> My only problem with Yasuo's latest offering is once again it adds a
>> whole new set of defines that have to be mapped to existing metadata
>> definitions ... That and it is a lot of longhand code using a different
>> style to existing arrays. We need yet another wrapper to build these
>> arrays from existing code ...
> The rules have to be defined somehow, and I'm not aware of a standard format that current code is likely to follow. Unless there is already such a standard, I can't see any way to avoid existing code having to be wrapped or amended.
>
> Which is why Yasuo and I have both suggested we work together to come up with such a standard format that can be used or adapted for these different parts of the application. If you have suggestions for how the format should look, we are eager to hear them and see some examples.

The likes of ADOdb datadict are still used as a base for metadata in
projects, but PDO destroyed the standardisation that used to exist by
spawning a number of competing wrappers. https://github.com/ADOdb/ADOdb
has evolved from a private project to being supported by it's own
community and is worth reconsidering as a proper cross database standard
to build on. Validation rules simply build on that base.

--
Lester Caine - G8HFL
-----------------------------
Contact - http://lsces.co.uk/wiki/?page=contact
L.S.Caine Electronic Services - http://lsces.co.uk
EnquirySolve - http://enquirysolve.com/
Model Engineers Digital Workshop - http://medw.co.uk
Rainbow Digital Media - http://rainbowdigitalmedia.co.uk

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Am 06.09.2017 um 13:52 schrieb Lester Caine:
> The likes of ADOdb datadict are still used as a base for metadata in
> projects, but PDO destroyed the standardisation that used to exist by
> spawning a number of competing wrappers. https://github.com/ADOdb/ADOdb
> has evolved from a private project to being supported by it's own
> community and is worth reconsidering as a proper cross database standard
> to build on. Validation rules simply build on that base

frankly - why don't you realize that input validation DOES NOT turn
around databases at all - databases and SQL injection are *just one*
subset of it and not every application works with databases at all

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Stephen Reay
Re: [PHP-DEV] A validator module for PHP7
September 06, 2017 02:10PM
> On 6 Sep 2017, at 18:15, Rowan Collins <[email protected]> wrote:
>
>> On 6 September 2017 09:29:37 BST, Lester Caine <[email protected]> wrote:
>> My only problem with Yasuo's latest offering is once again it adds a
>> whole new set of defines that have to be mapped to existing metadata
>> definitions ... That and it is a lot of longhand code using a different
>> style to existing arrays. We need yet another wrapper to build these
>> arrays from existing code ...
>
> The rules have to be defined somehow, and I'm not aware of a standard format that current code is likely to follow. Unless there is already such a standard, I can't see any way to avoid existing code having to be wrapped or amended.
>
> Which is why Yasuo and I have both suggested we work together to come up with such a standard format that can be used or adapted for these different parts of the application. If you have suggestions for how the format should look, we are eager to hear them and see some examples.
>
> Regards,
>
> --
> Rowan Collins
> [IMSoP]
>
> --
> PHP Internals - PHP Runtime Development Mailing List
> To unsubscribe, visit: http://www.php.net/unsub.php
>

Does this proposal actually offer any improvement over what's available with filter_var/etc and a userland wrapper class?

If there are deficiencies in how the existing filters work, maybe work on detailing the issues and then fixing/improving them?

That way, all the userland validation currently built around them, needs *possibly* minor changes to make use of new flags/options, as opposed to a completely new API to use.


Cheers
Stephen




--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Lester Caine
Re: [PHP-DEV] A validator module for PHP7
September 06, 2017 02:20PM
On 06/09/17 13:00, lists@rhsoft.net wrote:
> Am 06.09.2017 um 13:52 schrieb Lester Caine:
>> The likes of ADOdb datadict are still used as a base for metadata in
>> projects, but PDO destroyed the standardisation that used to exist by
>> spawning a number of competing wrappers. https://github.com/ADOdb/ADOdb
>> has evolved from a private project to being supported by it's own
>> community and is worth reconsidering as a proper cross database standard
>> to build on. Validation rules simply build on that base
>
> frankly - why don't you realize that input validation DOES NOT turn
> around databases at all - databases and SQL injection are *just one*
> subset of it and not every application works with databases at all

Metadata describing a data set should be standard across all interfaces.
PHP has had standards on the database interfaces for a long time and
writing different standards yet again for other interfaces is all I am
objecting to. If you build metadata for a form that can then simply be
dropped into the interface for a database or to pass into another
storage method then it will be a lot more usable.

--
Lester Caine - G8HFL
-----------------------------
Contact - http://lsces.co.uk/wiki/?page=contact
L.S.Caine Electronic Services - http://lsces.co.uk
EnquirySolve - http://enquirysolve.com/
Model Engineers Digital Workshop - http://medw.co.uk
Rainbow Digital Media - http://rainbowdigitalmedia.co.uk

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Rowan Collins
Re: [PHP-DEV] A validator module for PHP7
September 06, 2017 02:40PM
On 6 September 2017 12:38:03 BST, Dan Ackroyd <[email protected]> wrote:
>On 6 September 2017 at 12:15, Rowan Collins <[email protected]>
>wrote:
>
>> If you have suggestions for how the format should look
>
>Don't use a format. Just write code - see below.

I'm going to assume that the code you posted was something of a straw man, and you're not actually advocating people copy 20 lines of code for every variable they want to validate.

As soon as you have some shared functions, you have some design decisions about how those functions are named, what arguments they take, etc. When used for a lot of variables, that will resemble some kind of DSL, and that's all I meant by "format".


>This type of library should be done in PHP, not in C.

I can certainly see the argument for that. In that case, how would you like to see the language evolve:

Should we deprecate ext/filter, to encourage people to use better implementations in userland?

Should the language include some set of basic primitives that these userland implementations can use, rather than having to endlessly reimplement things like a regex for detecting a valid integer?

Or do you view ext/filter's current scope as about right, and it should be neither extended nor reduced?

Regards,

--
Rowan Collins
[IMSoP]

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Rowan Collins
Re: [PHP-DEV] A validator module for PHP7
September 06, 2017 02:50PM
On 6 September 2017 12:52:24 BST, Lester Caine <[email protected]> wrote:
>On 06/09/17 12:15, Rowan Collins wrote:
>> On 6 September 2017 09:29:37 BST, Lester Caine <[email protected]>
>wrote:
>> Which is why Yasuo and I have both suggested we work together to come
>up with such a standard format that can be used or adapted for these
>different parts of the application. If you have suggestions for how the
>format should look, we are eager to hear them and see some examples.
>
>The likes of ADOdb datadict are still used as a base for metadata in
>projects

I'm going to cut you off there, and say this one more time: please can you show us an example of how you would like it to look. I have no idea what an "ADOdb datadict" is, so if that's what you're advocating, you'll need to show us what it looks like.

Spare the commentary on what decisions we could have made differently 15 years ago, and concentrate on what we can do right now, specifically, for this particular piece of functionality.

Thank you,

--
Rowan Collins
[IMSoP]

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Dan Ackroyd
Re: [PHP-DEV] A validator module for PHP7
September 06, 2017 10:40PM
On 6 September 2017 at 13:31, Rowan Collins <[email protected]> wrote:
> I'm going to assume that the code you posted was something of a straw
> man, and you're not actually advocating people copy 20 lines of code for
> every variable they want to validate.

You assume wrong. No it's not, and yes I am.

I can point a junior developer at the function and they can understand it.

If I ask that junior developer to add an extra rule that doesn't
currently exist, they can without having to dive into a full library
of validation code.

If I need to modify the validation based on extra input (e.g whether
the user has already made several purchases, or whether they're a
brand new signup), it's trivial to add that to the function.

This is one of the times where code re-use through copying and pasting
is far superior to trying to make stuff "simple" by going through an
array based 'specification'. It turns out that that doesn't save much
time to begin with, and then becomes hard to manage when your
requirements get more complication.

> I can certainly see the argument for that. In that case, how would
> you like to see the language evolve:

I do not believe that trying to design features through an email
conversation, is a productive use of individual peoples time. But
also, this list is sent to thousands of people. The number of back and
forth messages required to (possibly) come up with a decent design
mulitplied thousands of people is a huge time sink.

I believe it is far more productive to come up with a good idea
off-list, and then present an almost finished version for discussion,
rather than design features from scratch via this list.

cheers
Dan
Ack

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Rowan Collins
Re: [PHP-DEV] A validator module for PHP7
September 07, 2017 12:50AM
On 6 September 2017 21:33:53 BST, Dan Ackroyd <[email protected]> wrote:
>On 6 September 2017 at 13:31, Rowan Collins <[email protected]>
>wrote:
>> I'm going to assume that the code you posted was something of a straw
>> man, and you're not actually advocating people copy 20 lines of code
>for
>> every variable they want to validate.
>
>You assume wrong. No it's not, and yes I am.
>
>I can point a junior developer at the function and they can understand
>it.
>
>If I ask that junior developer to add an extra rule that doesn't
>currently exist, they can without having to dive into a full library
>of validation code.

I can certainly agree that a complex DSL might be more pain than it's worth, but copying around a regex and all its scaffolding is surely worse than a clearly named function like validate_non_negative_int?

That's why I asked in my last email if you thought ext/filter should be removed, and perhaps replaced by some more straightforward primitives, or just left as it is. If a broad strategic question like that feels too much like "wasting time on the list designing features" to you, I'm not sure what you would find acceptable discussion.

Regards,

--
Rowan Collins
[IMSoP]

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Yasuo Ohgaki
Re: [PHP-DEV] A validator module for PHP7
September 07, 2017 06:30AM
Hi Dan


On Wed, Sep 6, 2017 at 8:38 PM, Dan Ackroyd <[email protected]> wrote:

> On 6 September 2017 at 12:15, Rowan Collins <[email protected]>
> wrote:
>
> > If you have suggestions for how the format should look
>
> Don't use a format. Just write code - see below.
>
> > Which is why Yasuo and I have both suggested we work together
>
> If you're going to work together and continue the conversation, please
> can you move this conversation elsewhere?
>
> It doesn't appear to be actually anything to do with PHP internals.
>
> On 4 September 2017 at 07:33, Yasuo Ohgaki <[email protected]> wrote:
> >
> > Since I didn't get much feedbacks during the RFC discussion, I cannot
> tell
> > what part is disliked.
>
> Yes you did. You got feedback during the discussion and also during
> the vote. For example: http://news.php.net/php.internals/95164
>
> However you continually choose to ignore that feedback.
>
> I will attempt once more, to get the main point through to you.
> Perhaps a small amount of repetition, will get it through:
>
> This type of library should be done in PHP, not in C.
> This type of library should be done in PHP, not in C.
> This type of library should be done in PHP, not in C.
> This type of library should be done in PHP, not in C.
> This type of library should be done in PHP, not in C.
> This type of library should be done in PHP, not in C.
> This type of library should be done in PHP, not in C.
> This type of library should be done in PHP, not in C.
> This type of library should be done in PHP, not in C.
> This type of library should be done in PHP, not in C.
> This type of library should be done in PHP, not in C.
>
> cheers
> Dan
> Ack
>
>
> function validateOrderAmount($value) : int {
> $count = preg_match("/[^0-9]*/", $value);
>
> if ($count) {
> throw new InvalidOrderAmount("The order value must contain
> only digits.");
> }
>
> $value = intval($value);
>
> if ($value < 1) {
> throw new InvalidOrderAmount("The order value must be one or
> more.");
> }
>
> if ($value >= MAX_ORDER_AMOUNT) {
> throw new InvalidOrderAmount(
> "Order value to big. Maximum allowed value is ".MAX_ORDER_AMOUNT
> );
> }
>
> return $value;
> }
>


You seems mixing up responsibility between
- Input validation that should be input handling code's responsibility.
- Logical validation that should be model code's responsibility .

Please stick to single responsibility principle that you should be well
aware of.
Input handling code should only accepts valid inputs and let other codes
use it.
Other responsibilities are not input handling code's responsibilities.

Your example code should be in logic, not input handling, that is written
by PHP script.

As I wrote in README.md, there are only 3 types of inputs.

1. Valid data should accepted.
2. Valid data should accepted, but user's mistake. e.g. Logical error like
your example above.
3. Invalid. Anything other than 1 and 2 (i.e. Client cannot send these
value)

"validate" module is supposed to take care 3 which is nothing to do with
models, etc.
It should validate against input data spec, not logical meaning of the
input. If
programmer did this, single responsibility principle is broken.

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net
Sorry, only registered users may post in this forum.

Click here to login