[PHP-DEV] PHP test suite

Hi internals,

after working for some time with the PHP test suite I've realized - it
needs an improvement of a special kind. What I'm talking about is the
server based testing. Very urgent cases have shown them up working with
APC. Most of the current bugs reported only reproduceable with tests where
multiple requests are involved. Where tests work fine on the first
request, all the subsequent requests produce memory leaks or other errors.
That's where PHP 5.4 with its built-in cli server comes into the game. As
how it works now - a CLI executable is run only once and due to its nature
only one pure PHP lifecircle is tested - MINIT-RINIT-RSHUTDOWN-MSHUTDOWN.

For now what I did - based my APC tests on sapi/cli/tests/php_cli_server*.
This has some limitations however. As such, this test don't care about any
valgrind params usually could be passed with make test TESTS=-m. Well,
this can be solved with a custom env var. Another thing - this tests
doesn't produce any coverage information, i suspect because the only way
it can be run now is using exec (talking about linux) shell command only.
You can convince yourself here
http://gcov.php.net/PHP_5_4/lcov_html/var/php_gcov/PHP_5_4/sapi/cli/index.php
despite built-in cli server has 17 tests, no code coverage is shown.
Several tries to fix this (including fork) didn't give me a solution to
produce coverage for APC either. However, a test runnt manually adds to
the coverage data.

Please look at this examples illustrating how the built-in CLI server is
used now with APC
http://svn.php.net/viewvc/pecl/apc/trunk/tests/server_test.inc?view=markup
http://svn.php.net/viewvc/pecl/apc/trunk/tests/apc54_002.phpt?view=markup

What I see now many particular tests do - relying on some external hosts
hoping them being online forever and/or a mashine running test havig
internet available. Nevertheless many tests make more sense having an
opposing party available (say a database extension for instance), another
plenty would require just a mock. Therefore the idea I have to bring here
- we need an improvement on the standard test suites since PHP 5.4
allowing to use the built-in CLI server native inside a phpt (and may be a
TCP server?). This should care about valgrind, about coverage, windows
compatibility as well as about beyond things available in the test suite.
Most likely it should be run from the make file, so on an upper lever to
run-tests.php. Having this would give more qualitative tests and at the
end of the day - more quality for PHP. It could be looking like --SERVER--
block in phpt or what ever.

That's it, an idea I wantend just throw into the air here :)

Regards

Anatoliy


--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php

On Fri, 2012-07-27 at 22:07 +0200, Anatoliy Belsky wrote:
>
> after working for some time with the PHP test suite I've realized - it
> needs an improvement of a special kind. What I'm talking about is the
> server based testing. Very urgent cases have shown them up working
> with APC. Most of the current bugs reported only reproduceable with
> tests where multiple requests are involved.

For that I once created a special SAPI, which can also do
multi-threading:
https://github.com/johannes/pconn-sapi
http://schlueters.de/blog/archives/166-Testing-persistent-connection-and-thread-safety-features-in-PHP.html

johannes


--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php

On 07/28/2012 07:09 AM, Johannes Schlüter wrote:
> On Fri, 2012-07-27 at 22:07 +0200, Anatoliy Belsky wrote:
>>
>> after working for some time with the PHP test suite I've realized - it
>> needs an improvement of a special kind. What I'm talking about is the
>> server based testing. Very urgent cases have shown them up working
>> with APC. Most of the current bugs reported only reproduceable with
>> tests where multiple requests are involved.
>
> For that I once created a special SAPI, which can also do
> multi-threading:
> https://github.com/johannes/pconn-sapi
> http://schlueters.de/blog/archives/166-Testing-persistent-connection-and-thread-safety-features-in-PHP.html

While that is cool, one of the reasons we added the built-in web server
was to use it in exactly this sort of scenario. So Anatoliy, go for it.
Make whatever changes you see fit to make it easy to do a --SERVER--
type mechanism in a phpt file. Or perhaps there are some volunteers here
who have ideas and you can team up and either flash it out in a wiki doc
or on irc.

-Rasmus

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php

On Sat, Jul 28, 2012 at 7:08 PM, Rasmus Lerdorf <[email protected]> wrote:

> On 07/28/2012 07:09 AM, Johannes Schlüter wrote:
> > On Fri, 2012-07-27 at 22:07 +0200, Anatoliy Belsky wrote:
> >>
> >> after working for some time with the PHP test suite I've realized - it
> >> needs an improvement of a special kind. What I'm talking about is the
> >> server based testing. Very urgent cases have shown them up working
> >> with APC. Most of the current bugs reported only reproduceable with
> >> tests where multiple requests are involved.
> >
> > For that I once created a special SAPI, which can also do
> > multi-threading:
> > https://github.com/johannes/pconn-sapi
> >
> http://schlueters.de/blog/archives/166-Testing-persistent-connection-and-thread-safety-features-in-PHP.html
>
> While that is cool, one of the reasons we added the built-in web server
> was to use it in exactly this sort of scenario. So Anatoliy, go for it.
> Make whatever changes you see fit to make it easy to do a --SERVER--
> type mechanism in a phpt file. Or perhaps there are some volunteers here
> who have ideas and you can team up and either flash it out in a wiki doc
> or on irc.
>
> -Rasmus
>
> --
> PHP Internals - PHP Runtime Development Mailing List
> To unsubscribe, visit: http://www.php.net/unsub.php
>
>
ccing the qa list, albeit I guess most people who follow that also follow
internals also.

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

On Sat, 2012-07-28 at 10:08 -0700, Rasmus Lerdorf wrote:
> > For that I once created a special SAPI, which can also do
> > multi-threading:
> > https://github.com/johannes/pconn-sapi
> > http://schlueters.de/blog/archives/166-Testing-persistent-connection-and-thread-safety-features-in-PHP.html
>
> While that is cool, one of the reasons we added the built-in web server
> was to use it in exactly this sort of scenario. So Anatoliy, go for it.
> Make whatever changes you see fit to make it easy to do a --SERVER--
> type mechanism in a phpt file. Or perhaps there are some volunteers here
> who have ideas and you can team up and either flash it out in a wiki doc
> or on irc.

I agree the server is a good way for that, I mostly shared my thing so
people maybe can take ideas from there, not to stop people doing
undoubtedly useful things.

johannes



--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php

On 07/28/2012 11:24 AM, Johannes Schlüter wrote:
> On Sat, 2012-07-28 at 10:08 -0700, Rasmus Lerdorf wrote:
>>> For that I once created a special SAPI, which can also do
>>> multi-threading:
>>> https://github.com/johannes/pconn-sapi
>>> http://schlueters.de/blog/archives/166-Testing-persistent-connection-and-thread-safety-features-in-PHP.html
>>
>> While that is cool, one of the reasons we added the built-in web server
>> was to use it in exactly this sort of scenario. So Anatoliy, go for it.
>> Make whatever changes you see fit to make it easy to do a --SERVER--
>> type mechanism in a phpt file. Or perhaps there are some volunteers here
>> who have ideas and you can team up and either flash it out in a wiki doc
>> or on irc.
>
> I agree the server is a good way for that, I mostly shared my thing so
> people maybe can take ideas from there, not to stop people doing
> undoubtedly useful things.

Yup, sorry I did realize that. But I wanted to make sure Anatoliy didn't
take your reply as a "do it like this instead" response.

-Rasmus


--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php

Thanks for encouraging me on this idea. I'm putting it on my list then.
Will check for what Johannes did as it possibly could be combined with the
existing stuff.

And if someone is also interested on adding this feature - so much the
better, just drop me a line :)

Am Sa, 28.07.2012, 20:33 schrieb Rasmus Lerdorf:
> On 07/28/2012 11:24 AM, Johannes Schlüter wrote:
>> On Sat, 2012-07-28 at 10:08 -0700, Rasmus Lerdorf wrote:
>>>> For that I once created a special SAPI, which can also do
>>>> multi-threading:
>>>> https://github.com/johannes/pconn-sapi
>>>> http://schlueters.de/blog/archives/166-Testing-persistent-connection-and-thread-safety-features-in-PHP.html
>>>
>>> While that is cool, one of the reasons we added the built-in web server
>>> was to use it in exactly this sort of scenario. So Anatoliy, go for it.
>>> Make whatever changes you see fit to make it easy to do a --SERVER--
>>> type mechanism in a phpt file. Or perhaps there are some volunteers
>>> here
>>> who have ideas and you can team up and either flash it out in a wiki
>>> doc
>>> or on irc.
>>
>> I agree the server is a good way for that, I mostly shared my thing so
>> people maybe can take ideas from there, not to stop people doing
>> undoubtedly useful things.
>
> Yup, sorry I did realize that. But I wanted to make sure Anatoliy didn't
> take your reply as a "do it like this instead" response.
>
> -Rasmus
>
>
> --
> PHP Internals - PHP Runtime Development Mailing List
> To unsubscribe, visit: http://www.php.net/unsub.php
>
>



--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php