Re: [PHP-DEV] default charset confusion

Daniel Convissor — Sun, 01 Apr 2012 18:10:03 +0200

Hi Folks:

This topic appears to have been quietly tabled. I didn't notice a
decision here or a commit.

On Mon, Mar 12, 2012 at 01:12:03PM -0700, Rasmus Lerdorf wrote:
>
> So maybe a way to tackle this is to use the
> mbstring internal encoding when it is set as the htmlspecialchars
> default when it is called without an encoding arg.

This seems like the clearest indicator of the programmer's intent.

Thanks,

--Dan

--
T H E A N A L Y S I S A N D S O L U T I O N S C O M P A N Y
data intensive web and database programming
http://www.AnalysisAndSolutions.com/
4015 7th Ave #4, Brooklyn NY 11232 v: 718-854-0335 f: 718-854-0409

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP-DEV] default charset confusion

Ángel González — Thu, 15 Mar 2012 00:40:02 +0100

On 13/03/12 00:25, Stas Malyshev wrote:
> Hi!
>
>> Still, that API is likely wrong: a library function written by someone
>> completely unrelated to the main application shouldn't be echoing
>> anything through the output. And if it's not generating the html, the
>> htmlspecialchars is better done from the return at the calling
>> application (probably after converting the internal charset).
>
> Again, you making a huge amount of assumptions about how ALL the
> applications must work, which means you are wrong in 99.(9)% of cases,
> because there's infinitely many applications which don't work exactly
> like yours does, and we have no idea how they work.
No. I'm saying how I consider they should work, saying that an API doing
otherwise is likely* wrong (aka. has a bad design), very much as I'd
consider insane a company policy stating "PHP function arguments shall
be named $a, $b, $c...".
That's obviously my opinion, but I think most applications will conform
to that, just as most apps will use more descriptive argument names than
"$c"**.

* There might be some very very special application where it turns out
to be an appropiate design, but that would be the exception.
** Even though there are 26!/(26-n)! ways to name so badly the arguments
of a n-ary function.

> The main point is that having global state (and yet worse, changeable
> global state) significantly influence how basic functions are working
> is dangerous. It's like keeping everything in globals and instead of
> passing parameters between functions just change some globals and
> expect functions to pick it up.
I agree with you, in the general case. Yet, I consider the html charset
to be a global state. And passing the global variables as parameters on
each function call would be nearly as bad as passing parameters as globals.
I just positioned the opposite way for parse_str(), while being fully
aware of that.

>> Such interfaces may be well served by switching the setting many times.
> That's exactly what I am trying to avoid, and you are just
> illustrating why this proposal is dangerous - because that's exactly
> what is going to happen in the code, instead of passing proper
> arguments to htmlspecialchars people will start changing INI settings
> left and right, and then nobody would know what htmlspecialchars()
> call actually does without tracking all the INI changes along the way.
That's assuming people would need to use different output charsets,
which I don't consider to be the case. How many people is using now the
third htmlspecialchars() parameter?
What makes you think that they would need to change the default global,
*several times per request*?

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP-DEV] default charset confusion

jpauli — Wed, 14 Mar 2012 16:00:02 +0100

On Wed, Mar 14, 2012 at 3:37 PM, Gustavo Lopes wrote:

> On Wed, 14 Mar 2012 14:55:17 +0100, jpauli wrote:
>
> I would then propose to make mbstring compile time mandatory.
>>
>>
> I'm completely against these kind of lazy solutions. Yes, let's add strong
> coupling (already starting to smell) to one of the largest extensions and
> make it compile time mandatory because it simplifies the implementation of
> a dubiously useful feature like Zend multibyte. Remember PHP is sometimes
> used in environments with limited memory/disk space.
>
> Also mbstring takes a long time to build (relatively speaking). Just that
> would be a strong argument against making it mandatory, at least for people
> like me that compile PHP with --disable-all very frequently.
>
>
> I'm against yet another global ini setting, I find the actual ini
>> settings confusing enough to add one more that would moreover reflect
>> mbstring one's (and add more and more confusion).
>> Why not turn ext/mbstring mandatory at compile time, for all future PHP
>> versions, like preg or spl are ?
>>
>> We do need multibyte handling either. ZendEngine takes advantage of
>> mbstring for internal encoding as well, so I probably missed something as
>> why it is still possible to --disable-mbstring (or not add
>> --enable-mbstring) when compiling ? Has it a huge performance impact ?
>>
>>
> mbstring hooks to basically all phases of PHP process/request
> startup/shutdown. Some efforts were made to mitigate the impact of this in
> 5.4 (see e.g. r301068), but at least some impact is inevitable. Of course,
> if you start enabling certain features of mbstring (zend multibyte hooks,
> translation of input variables, function overload) then it starts to be
> significant. However, there are other more compelling reasons not to make
> it required (see above).
>
> --
> Gustavo Lopes
>

That makes sense to me :-)

But we should think about complexity in the final choice.
Having something like "internal_encoding" adding in PHP.ini will confuse
people, at least, if we dont clearly explain them what the setting is for.
The name is nearly the same as mbstring's.

I recently opened a doc bug about multibyte handling in 5.4 (#61373) , as
the documentation is really light on that point

Julien.P

Re: [PHP-DEV] default charset confusion

Gustavo Lopes — Wed, 14 Mar 2012 15:40:01 +0100

On Wed, 14 Mar 2012 14:55:17 +0100, jpauli wrote:

> I would then propose to make mbstring compile time mandatory.
>

I'm completely against these kind of lazy solutions. Yes, let's add strong
coupling (already starting to smell) to one of the largest extensions and
make it compile time mandatory because it simplifies the implementation of
a dubiously useful feature like Zend multibyte. Remember PHP is sometimes
used in environments with limited memory/disk space.

Also mbstring takes a long time to build (relatively speaking). Just that
would be a strong argument against making it mandatory, at least for
people like me that compile PHP with --disable-all very frequently.

> I'm against yet another global ini setting, I find the actual ini
> settings confusing enough to add one more that would moreover reflect
> mbstring one's (and add more and more confusion).
> Why not turn ext/mbstring mandatory at compile time, for all future PHP
> versions, like preg or spl are ?
>
> We do need multibyte handling either. ZendEngine takes advantage of
> mbstring for internal encoding as well, so I probably missed something as
> why it is still possible to --disable-mbstring (or not add
> --enable-mbstring) when compiling ? Has it a huge performance impact ?
>

mbstring hooks to basically all phases of PHP process/request
startup/shutdown. Some efforts were made to mitigate the impact of this in
5.4 (see e.g. r301068), but at least some impact is inevitable. Of course,
if you start enabling certain features of mbstring (zend multibyte hooks,
translation of input variables, function overload) then it starts to be
significant. However, there are other more compelling reasons not to make
it required (see above).

--
Gustavo Lopes

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP-DEV] default charset confusion

Ferenc Kovacs — Wed, 14 Mar 2012 15:40:01 +0100

On Wed, Mar 14, 2012 at 3:29 PM, Michael Stowe wrote:

> Correct me if I'm wrong, but I believe Zend Multibyte is now enabled by
> default in PHP 5.4.
>
> - Mike
>
>
http://lxr.php.net/opengrok/xref/PHP_5_4/UPGRADING#91
http://lxr.php.net/opengrok/xref/PHP_5_4/Zend/zend.c#108
http://lxr.php.net/opengrok/xref/PHP_5_4/php.ini-development#358
http://lxr.php.net/opengrok/xref/PHP_5_4/php.ini-production#358

we just moved the switch from compilation time to runtime, so the code is
there, if you want to enable it, you don't have to recompile php but only
have to change an ini setting, but it isn't turned on by default.
AFAIK
--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

Re: [PHP-DEV] default charset confusion

Michael Stowe — Wed, 14 Mar 2012 15:40:01 +0100

Correct me if I'm wrong, but I believe Zend Multibyte is now enabled by
default in PHP 5.4.

- Mike

On Wed, Mar 14, 2012 at 9:24 AM, Ferenc Kovacs wrote:

> >
> >
> > I would then propose to make mbstring compile time mandatory.
> >
> > I'm against yet another global ini setting, I find the actual ini
> settings
> > confusing enough to add one more that would moreover reflect mbstring
> one's
> > (and add more and more confusion).
> > Why not turn ext/mbstring mandatory at compile time, for all future PHP
> > versions, like preg or spl are ?
> >
> > We do need multibyte handling either. ZendEngine takes advantage of
> > mbstring for internal encoding as well, so I probably missed something as
> > why it is still possible to --disable-mbstring (or not add
> > --enable-mbstring) when compiling ? Has it a huge performance impact ?
> >
> > Thank you :)
> >
> > Julien.P
> >
>
> see
> http://www.mail-archive.com/internals@lists.php.net/msg48452.html
> http://lxr.php.net/opengrok/xref/PHP_5_4/UPGRADING#91
> and
> http://www.mail-archive.com/internals@lists.php.net/msg53863.html
>
> basically the mbstring code in the ZE is only used if you
> enable zend.multibyte, which is disabled by default, so it isn't mandatory
> to have ext/mbstring for the default build/setup.
> as you can see from the last link, I would support having ext/mbstring
> builtin and always enabled, but I would like to hear from more people about
> the pros and cons.
>
> --
> Ferenc Kovács
> @Tyr43l - http://tyrael.hu
>

--
-----------------------

"My command is this: Love each other as I
have loved you." John 15:12

-----------------------

Re: [PHP-DEV] default charset confusion

Ferenc Kovacs — Wed, 14 Mar 2012 15:30:02 +0100

>
>
> I would then propose to make mbstring compile time mandatory.
>
> I'm against yet another global ini setting, I find the actual ini settings
> confusing enough to add one more that would moreover reflect mbstring one's
> (and add more and more confusion).
> Why not turn ext/mbstring mandatory at compile time, for all future PHP
> versions, like preg or spl are ?
>
> We do need multibyte handling either. ZendEngine takes advantage of
> mbstring for internal encoding as well, so I probably missed something as
> why it is still possible to --disable-mbstring (or not add
> --enable-mbstring) when compiling ? Has it a huge performance impact ?
>
> Thank you :)
>
> Julien.P
>

see
http://www.mail-archive.com/internals@lists.php.net/msg48452.html
http://lxr.php.net/opengrok/xref/PHP_5_4/UPGRADING#91
and
http://www.mail-archive.com/internals@lists.php.net/msg53863.html

basically the mbstring code in the ZE is only used if you
enable zend.multibyte, which is disabled by default, so it isn't mandatory
to have ext/mbstring for the default build/setup.
as you can see from the last link, I would support having ext/mbstring
builtin and always enabled, but I would like to hear from more people about
the pros and cons.

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

Re: [PHP-DEV] default charset confusion

jpauli — Wed, 14 Mar 2012 15:00:03 +0100

On Tue, Mar 13, 2012 at 1:52 AM, Yasuo Ohgaki wrote:

> 2012/3/13 Rasmus Lerdorf :
> > On 03/12/2012 03:05 AM, Yasuo Ohgaki wrote:
> >> I thought default_charset became UTF-8, so I was expecting
> >> following HTTP header.
> >>
> >> content-type text/html; charset=UTF-8
> >>
> >> However, I got empty charset (missing 'charset=UTF-8').
> >> So I looked up to source and found the line in SAPI.h
> >>
> >> 293 #define SAPI_DEFAULT_CHARSET ""
> >>
> >> Empty string should be "UTF-8", isn't it?
> >
> > No, we can't force an output charset on people since it would end up
> > breaking a lot of sites.
>
> Right, so may be for the next major release? 5.5.0?
>
> As the first XSS advisory in 2000 states, explicitly setting char coding
> will
> prevent certain XSS. Recent browsers have much better encoding handing,
> but setting encoding explicitly is better for security still.
>
> > PHP 5.3's determine_charset behaves exactly like 5.4's. In 5.3 we have:
> >
> > if (charset_hint == NULL)
> > return cs_8859_1;
> >
> > and in 5.4 we have:
> >
> > if (charset_hint == NULL)
> > return cs_utf_8;
> >
> > So there is no difference in their guessing when there is no hint, the
> > only difference is that in 5.4 we choose utf8 and in 5.3 we choose
> > 8859-1 in that case.
>
> I got this with 5.3
> > echo htmlentities('<日本語UTF-8>',ENT_QUOTES);
> echo htmlentities('<日本語UTF-8>',ENT_QUOTES, 'UTF-8');
>
> <æ�¥æ�¬èª�UTF8
> ><日本語UTF-8>
>
> So people migrating from 5.3 to 5.4 should not have problems.
> Migration older than 5.3 to 5.4 will be problematic.
>
> I always set all parameters for htmlentities/htmlspecialchars, therefore
> I haven't noticed this was changed from 5.3. They may be migrating from
> 5.2 or older. (RHEL5 uses 5.1)
>
> Since PHP does not have default multibyte module, it may be good for having
>
> input_encoding
> internal_encoding
> output_encoding
>
>
I would then propose to make mbstring compile time mandatory.

I'm against yet another global ini setting, I find the actual ini settings
confusing enough to add one more that would moreover reflect mbstring one's
(and add more and more confusion).
Why not turn ext/mbstring mandatory at compile time, for all future PHP
versions, like preg or spl are ?

We do need multibyte handling either. ZendEngine takes advantage of
mbstring for internal encoding as well, so I probably missed something as
why it is still possible to --disable-mbstring (or not add
--enable-mbstring) when compiling ? Has it a huge performance impact ?

Thank you :)

Julien.P

Re: [PHP-DEV] default charset confusion

Tomas Kuliavas — Tue, 13 Mar 2012 18:50:02 +0100

2012.03.13 16:38 Richard Lynch rašė:
> I'd have to agree with Stas that everybody should start passing in a
> variable there, that can be set somewhere in a config, or, perhaps,
> would DEFAULT to, errrr...

You do realize that suggestions on this thread and original bug reporter
failed to make correct decisions about values that should be used to
migrate original function to PHP 5.4 compatible syntax?

htmlspecialchars without arguments does not default to ENT_QUOTES or NULL.

Failure to choose proper second argument value will lead to different
exploit or data corruption.

> You can't default to a function call.

Changing default in function was bad idea.

Ignoring bug reports about f....ed up documentation and closing them with
bogus explanations might not be bad idea, but it really helps in
alienating your developer base.

--
Tomas

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP-DEV] default charset confusion

Richard Lynch — Tue, 13 Mar 2012 15:40:04 +0100

On Mon, March 12, 2012 2:44 pm, Rasmus Lerdorf wrote:
> But you can't necessarily hardcode the encoding if you are writing
> portable code. That's a bit like hardcoding a timezone. In order to
> write portable code you need to give people the ability to localize
> it.

If you wanted it portable, wouldn't you need to have a variable there,
so it can survive the ISO-8859-1 to UTF-8 change, and to allow people
to change it despite whatever non-standard setting might happen to be
in somebody else's php.ini?

I mean, sure, it's nice if it "just works" for the folks who want to
install and have it localized for their own charset hard-coded in
php.ini, but if it's being multi-national website, you have to pass in
a variable there, which seems the more portable option to this naive
reader.

Having it default to whatever happens to be in php.ini only solves the
use case of people who only want to serve up their content in their
own charset.

I'd have to agree with Stas that everybody should start passing in a
variable there, that can be set somewhere in a config, or, perhaps,
would DEFAULT to, errrr...

You can't default to a function call.

ANOTHER magic constant like __INI_CHARSET__ ???

That's probably a bad idea...

--
brain cancer update:
http://richardlynch.blogspot.com/search/label/brain%20tumor
Donate:
https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=FS9NLTNEEKWBE

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP-DEV] default charset confusion

Christian Schneider — Tue, 13 Mar 2012 15:30:02 +0100

Am 13.03.2012, 02:34 Uhr, schrieb Rasmus Lerdorf :
> On 03/12/2012 05:52 PM, Yasuo Ohgaki wrote:
>> I always set all parameters for htmlentities/htmlspecialchars, therefore
>> I haven't noticed this was changed from 5.3. They may be migrating from
>> 5.2 or older. (RHEL5 uses 5.1)
>
> No, like I showed, moving from 5.3 to 5.4 breaks because the new default
> UTF-8 encoding validates the input and 8859-1 in 5.3 does not. So for
> charsets that are actually safe for the low-ascii chars that are
> significant to html htmlspecialchars() now returns false in 5.4 because
> their chars fail the UTF8 validity check. For people who explicitly set
> all the parameters nothing has changed, of course.

I second that. It causes us big PITA because we're still using 8859-1
(shame
on us) and it is made even worse because the encoding parameter is after
the
(optional) flags parameter which now has to be given too.

The sane version from my naive point of view would be to honor
default_charset
if nothing is given. That's what I expected when I read the migration
guide.

- Chris

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP-DEV] default charset confusion

Rasmus Lerdorf — Tue, 13 Mar 2012 02:40:02 +0100

On 03/12/2012 05:52 PM, Yasuo Ohgaki wrote:
> I always set all parameters for htmlentities/htmlspecialchars, therefore
> I haven't noticed this was changed from 5.3. They may be migrating from
> 5.2 or older. (RHEL5 uses 5.1)

No, like I showed, moving from 5.3 to 5.4 breaks because the new default
UTF-8 encoding validates the input and 8859-1 in 5.3 does not. So for
charsets that are actually safe for the low-ascii chars that are
significant to html htmlspecialchars() now returns false in 5.4 because
their chars fail the UTF8 validity check. For people who explicitly set
all the parameters nothing has changed, of course.

-Rasmus

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP-DEV] default charset confusion

Yasuo Ohgaki — Tue, 13 Mar 2012 02:00:01 +0100

2012/3/13 Rasmus Lerdorf :
> On 03/12/2012 03:05 AM, Yasuo Ohgaki wrote:
>> I thought default_charset became UTF-8, so I was expecting
>> following HTTP header.
>>
>> content-type text/html; charset=UTF-8
>>
>> However, I got empty charset (missing 'charset=UTF-8').
>> So I looked up to source and found the line in SAPI.h
>>
>> 293 #define SAPI_DEFAULT_CHARSET ""
>>
>> Empty string should be "UTF-8", isn't it?
>
> No, we can't force an output charset on people since it would end up
> breaking a lot of sites.

Right, so may be for the next major release? 5.5.0?

As the first XSS advisory in 2000 states, explicitly setting char coding will
prevent certain XSS. Recent browsers have much better encoding handing,
but setting encoding explicitly is better for security still.

> PHP 5.3's determine_charset behaves exactly like 5.4's. In 5.3 we have:
>
> if (charset_hint == NULL)
> return cs_8859_1;
>
> and in 5.4 we have:
>
> if (charset_hint == NULL)
> return cs_utf_8;
>
> So there is no difference in their guessing when there is no hint, the
> only difference is that in 5.4 we choose utf8 and in 5.3 we choose
> 8859-1 in that case.

I got this with 5.3
echo htmlentities('<日本語UTF-8>',ENT_QUOTES);
echo htmlentities('<日本語UTF-8>',ENT_QUOTES, 'UTF-8');

<æ�¥æ�¬èª�UTF8
><日本語UTF-8>

So people migrating from 5.3 to 5.4 should not have problems.
Migration older than 5.3 to 5.4 will be problematic.

I always set all parameters for htmlentities/htmlspecialchars, therefore
I haven't noticed this was changed from 5.3. They may be migrating from
5.2 or older. (RHEL5 uses 5.1)

Since PHP does not have default multibyte module, it may be good for having

input_encoding
internal_encoding
output_encoding

php.ini settings and make multibyte modules use them when they are set.
Or just make mbstring default, alternatively.

Rather big change for released version, but this is simple easy change.

Regards,

--
Yasuo Ohgaki

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP-DEV] default charset confusion

Stas Malyshev — Tue, 13 Mar 2012 00:30:01 +0100

Hi!

> Still, that API is likely wrong: a library function written by someone
> completely unrelated to the main application shouldn't be echoing
> anything through the output. And if it's not generating the html, the
> htmlspecialchars is better done from the return at the calling
> application (probably after converting the internal charset).

Again, you making a huge amount of assumptions about how ALL the
applications must work, which means you are wrong in 99.(9)% of cases,
because there's infinitely many applications which don't work exactly
like yours does, and we have no idea how they work.

The main point is that having global state (and yet worse, changeable
global state) significantly influence how basic functions are working is
dangerous. It's like keeping everything in globals and instead of
passing parameters between functions just change some globals and expect
functions to pick it up.

> Such interfaces may be well served by switching the setting many times.

That's exactly what I am trying to avoid, and you are just illustrating
why this proposal is dangerous - because that's exactly what is going to
happen in the code, instead of passing proper arguments to
htmlspecialchars people will start changing INI settings left and right,
and then nobody would know what htmlspecialchars() call actually does
without tracking all the INI changes along the way.
--
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP-DEV] default charset confusion

Stas Malyshev — Mon, 12 Mar 2012 22:40:03 +0100

Hi!

> If you are a framework developer, and really want to shield against a
> bad php.ini setting, you could ini_set() to your prefered charset at the
> beginning of the request.

That assuming "the request" is completely processed by your framework
and you never call any outside code and any outside code never calls you
- otherwise your messing with INI setting may very well break that code
or that code's messing with INI settings may very well break yours.
--
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP-DEV] default charset confusion

Ángel González — Mon, 12 Mar 2012 22:30:02 +0100

On 12/03/12 20:51, Stas Malyshev wrote:
> Hi!
>
>> But you can't necessarily hardcode the encoding if you are writing
>> portable code. That's a bit like hardcoding a timezone. In order to
>> write portable code you need to give people the ability to localize it.
>
> No, it's not like timezone at all. I have to support all timezones in
> a global app, but I don't have to internally support every encoding on
> Earth - having everything internally in UTF-8 works quite well, and a
> lot of applications do exactly that - they have everything internally
> in UTF-8 and only may convert when importing or exporting the data. I
> don't see anything in using UTF-8 throughout the app/library that
> makes it non-portable. However, if we allow to change defaults in
> htmlspecialchars() etc. that essentially makes having defaults useless
> as I'd have so explicitly specify UTF-8 each time - otherwise it's a
> gamble what encoding I'd actually get.
If you are a framework developer, and really want to shield against a
bad php.ini setting, you could ini_set() to your prefered charset at the
beginning of the request.

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP-DEV] default charset confusion

Pierre Joye — Mon, 12 Mar 2012 21:30:01 +0100

hi Rasmus,

On Mon, Mar 12, 2012 at 9:12 PM, Rasmus Lerdorf wrote:

> If everything was UTF-8 we wouldn't have any of these issues.
> Unfortunately that isn't the case. The question is what to do with apps
> that need to deal with non UTF-8 data. Are we going to provide any help
> to them beyond just telling them to convert everything to UTF-8?

That's not really an acceptable solution, obviously.

> We took steps in 5.4 to improve htmlspecialchars to understand more
> encodings and we have the concept of script_encoding and
> internal_encoding that is used both in the engine and in mbstring.
>
> Currently internal_encoding isn't checked by htmlspecialchars. If you
> pass it '' it checks script_encoding and default_charset which is a bit
> odd since neither directly relate to the encoding of the internal data
> you are feeding to it. So maybe a way to tackle this is to use the
> mbstring internal encoding when it is set as the htmlspecialchars
> default when it is called without an encoding arg.

That's why I would prefer to use an existing setting and clearly
document it instead of creating a new ini settings with a totally
different impact than the existing ones. Not sure which one would fit
best tho'.

Reading these last two paragraphs gave me a headache and I did not
know anymore which encoding we were talking about ;-)

Cheers,
--
Pierre

@pierrejoye | http://blog.thepimp.net | http://www.libgd.org

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP-DEV] default charset confusion

Rasmus Lerdorf — Mon, 12 Mar 2012 21:20:03 +0100

On 03/12/2012 12:51 PM, Stas Malyshev wrote:
> Hi!
>
>> But you can't necessarily hardcode the encoding if you are writing
>> portable code. That's a bit like hardcoding a timezone. In order to
>> write portable code you need to give people the ability to localize it.
>
> No, it's not like timezone at all. I have to support all timezones in a
> global app, but I don't have to internally support every encoding on
> Earth - having everything internally in UTF-8 works quite well, and a
> lot of applications do exactly that - they have everything internally in
> UTF-8 and only may convert when importing or exporting the data. I don't
> see anything in using UTF-8 throughout the app/library that makes it
> non-portable. However, if we allow to change defaults in
> htmlspecialchars() etc. that essentially makes having defaults useless
> as I'd have so explicitly specify UTF-8 each time - otherwise it's a
> gamble what encoding I'd actually get.

If everything was UTF-8 we wouldn't have any of these issues.
Unfortunately that isn't the case. The question is what to do with apps
that need to deal with non UTF-8 data. Are we going to provide any help
to them beyond just telling them to convert everything to UTF-8?

We took steps in 5.4 to improve htmlspecialchars to understand more
encodings and we have the concept of script_encoding and
internal_encoding that is used both in the engine and in mbstring.
Currently internal_encoding isn't checked by htmlspecialchars. If you
pass it '' it checks script_encoding and default_charset which is a bit
odd since neither directly relate to the encoding of the internal data
you are feeding to it. So maybe a way to tackle this is to use the
mbstring internal encoding when it is set as the htmlspecialchars
default when it is called without an encoding arg.

-Rasmus

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP-DEV] default charset confusion

Stas Malyshev — Mon, 12 Mar 2012 21:00:02 +0100

Hi!

> But you can't necessarily hardcode the encoding if you are writing
> portable code. That's a bit like hardcoding a timezone. In order to
> write portable code you need to give people the ability to localize it.

No, it's not like timezone at all. I have to support all timezones in a
global app, but I don't have to internally support every encoding on
Earth - having everything internally in UTF-8 works quite well, and a
lot of applications do exactly that - they have everything internally in
UTF-8 and only may convert when importing or exporting the data. I don't
see anything in using UTF-8 throughout the app/library that makes it
non-portable. However, if we allow to change defaults in
htmlspecialchars() etc. that essentially makes having defaults useless
as I'd have so explicitly specify UTF-8 each time - otherwise it's a
gamble what encoding I'd actually get.
--
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP-DEV] default charset confusion

Rasmus Lerdorf — Mon, 12 Mar 2012 20:50:02 +0100

On 03/12/2012 12:40 PM, Stas Malyshev wrote:
> Hi!
>
>> And yes, it may very well be dangerous to use the wrong charset and now
>> that we have better support for GB2312 and other asian charsets in the
>> entities functions in 5.4 it is even more prudent to choose the right
>> one so we should provide some way to help people get it right short of
>> changing every call.
>
> I'm not sure "changing every call" is such a big problem - it's one grep
> and one replace, can be done in one line of sed/awk/perl/php probably.
> But a bigger issue is here that people insist on using wrong charsets
> and expect language to have some magical external defaults that work for
> exactly their use case, instead of doing what they should be doing all
> along - putting charset right there in the argument.
> We need to get people off this mindset fast, since it is not a good one.
> Having tons of hidden defaults that modify behavior of functions called
> with the same arguments in hundreds of different ways is a coding and
> maintenance nightmare. Now if I write htmlspecialchars() I can never be
> sure if works right and uses UTF-8 - what if somebody messed with the
> INI setting because of some other broken library that required that to
> work?

But you can't necessarily hardcode the encoding if you are writing
portable code. That's a bit like hardcoding a timezone. In order to
write portable code you need to give people the ability to localize it.

-Rasmus

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP-DEV] default charset confusion

Stas Malyshev — Mon, 12 Mar 2012 20:50:02 +0100

Hi!

> And yes, it may very well be dangerous to use the wrong charset and now
> that we have better support for GB2312 and other asian charsets in the
> entities functions in 5.4 it is even more prudent to choose the right
> one so we should provide some way to help people get it right short of
> changing every call.

I'm not sure "changing every call" is such a big problem - it's one grep
and one replace, can be done in one line of sed/awk/perl/php probably.
But a bigger issue is here that people insist on using wrong charsets
and expect language to have some magical external defaults that work for
exactly their use case, instead of doing what they should be doing all
along - putting charset right there in the argument.
We need to get people off this mindset fast, since it is not a good one.
Having tons of hidden defaults that modify behavior of functions called
with the same arguments in hundreds of different ways is a coding and
maintenance nightmare. Now if I write htmlspecialchars() I can never be
sure if works right and uses UTF-8 - what if somebody messed with the
INI setting because of some other broken library that required that to work?
--
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP-DEV] default charset confusion

Michael Stowe — Mon, 12 Mar 2012 16:50:25 +0100

I think the ini directive, while adding another to the list, may be the
most unobtrusive method to address this issue, at least for developers.

I definitely agree with Rasmus that this could be one of the bigger
headaches in transitioning to 5.4 (for non-UTF8 sites) and unless we can
come up with a better solution, I say let's move forward with it for 5.4.1.

- Mike

On Mon, Mar 12, 2012 at 10:27 AM, Rasmus Lerdorf wrote:

> On 03/12/2012 03:05 AM, Yasuo Ohgaki wrote:
> > Hi
> >
> > I think following PHP 5.4.0 NEWS entry is misleading.
> >
> > . Changed default value of "default_charset" php.ini option from
> ISO-8859-1 to
> > UTF-8. (Rasmus)
>
> Yes, I have fixed that now.
>
> > I thought default_charset became UTF-8, so I was expecting
> > following HTTP header.
> >
> > content-type text/html; charset=UTF-8
> >
> > However, I got empty charset (missing 'charset=UTF-8').
> > So I looked up to source and found the line in SAPI.h
> >
> > 293 #define SAPI_DEFAULT_CHARSET ""
> >
> > Empty string should be "UTF-8", isn't it?
>
> No, we can't force an output charset on people since it would end up
> breaking a lot of sites.
>
> > - php.ini's default_charset should be UTF-8.
> > - determine_charset() should not blindly default to UTF-8 when there
> > are no hint.
> >
> > Old htmlentities/htmlspecialchars actually determines charset from
> > default_charset/mbstring.internal_encoding/etc. I think old behavior
> > is better than now.
> >
> > How about make determine_charset() behaves like 5.3 and set the
> > SAPI_DEFAULT_CHARSET to "UTF-8"?
>
> PHP 5.3's determine_charset behaves exactly like 5.4's. In 5.3 we have:
>
> if (charset_hint == NULL)
> return cs_8859_1;
>
> and in 5.4 we have:
>
> if (charset_hint == NULL)
> return cs_utf_8;
>
> So there is no difference in their guessing when there is no hint, the
> only difference is that in 5.4 we choose utf8 and in 5.3 we choose
> 8859-1 in that case.
>
> -Rasmus
>
> --
> PHP Internals - PHP Runtime Development Mailing List
> To unsubscribe, visit: http://www.php.net/unsub.php
>
>

--
-----------------------

"My command is this: Love each other as I
have loved you." John 15:12

-----------------------

Re: [PHP-DEV] default charset confusion

Rasmus Lerdorf — Mon, 12 Mar 2012 16:30:02 +0100

On 03/12/2012 03:05 AM, Yasuo Ohgaki wrote:
> Hi
>
> I think following PHP 5.4.0 NEWS entry is misleading.
>
> . Changed default value of "default_charset" php.ini option from ISO-8859-1 to
> UTF-8. (Rasmus)

Yes, I have fixed that now.

> I thought default_charset became UTF-8, so I was expecting
> following HTTP header.
>
> content-type text/html; charset=UTF-8
>
> However, I got empty charset (missing 'charset=UTF-8').
> So I looked up to source and found the line in SAPI.h
>
> 293 #define SAPI_DEFAULT_CHARSET ""
>
> Empty string should be "UTF-8", isn't it?

No, we can't force an output charset on people since it would end up
breaking a lot of sites.

> - php.ini's default_charset should be UTF-8.
> - determine_charset() should not blindly default to UTF-8 when there
> are no hint.
>
> Old htmlentities/htmlspecialchars actually determines charset from
> default_charset/mbstring.internal_encoding/etc. I think old behavior
> is better than now.
>
> How about make determine_charset() behaves like 5.3 and set the
> SAPI_DEFAULT_CHARSET to "UTF-8"?

PHP 5.3's determine_charset behaves exactly like 5.4's. In 5.3 we have:

if (charset_hint == NULL)
return cs_8859_1;

and in 5.4 we have:

if (charset_hint == NULL)
return cs_utf_8;

So there is no difference in their guessing when there is no hint, the
only difference is that in 5.4 we choose utf8 and in 5.3 we choose
8859-1 in that case.

-Rasmus

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP-DEV] default charset confusion

Laruence — Mon, 12 Mar 2012 13:50:02 +0100

On Mon, Mar 12, 2012 at 6:21 PM, Yasuo Ohgaki wrote:
> Hi,
>
> I think motivation of
>
> /* Default is now UTF-8 */
> if (charset_hint == NULL)
> return cs_utf_8;
>
> is for better performance and I think it's good for better performance.
> Alternative of my suggestion is introduce new php.ini entry as Rusmus
> mentioned.
>
> The name may be "default_html_escape_encoding"?
Hi:
in consideration of succinctness, I think run_time_encoding is better.

and we should also separate the determine_output_charset and
determine_run_time_charset(there is only one determin_charset now)

thanks
>
> We should document this behavior very well, since it affects all of
> non UTF-8 web sites.
>
> Regards,
>
> --
> Yasuo Ohgaki
> yohgaki@ohgaki.net
>
> --
> PHP Internals - PHP Runtime Development Mailing List
> To unsubscribe, visit: http://www.php.net/unsub.php
>

--
Laruence Xinchen Hui
http://www.laruence.com/

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP-DEV] default charset confusion

Yasuo Ohgaki — Mon, 12 Mar 2012 11:30:02 +0100

Hi,

I think motivation of

/* Default is now UTF-8 */
if (charset_hint == NULL)
return cs_utf_8;

is for better performance and I think it's good for better performance.
Alternative of my suggestion is introduce new php.ini entry as Rusmus
mentioned.

The name may be "default_html_escape_encoding"?

We should document this behavior very well, since it affects all of
non UTF-8 web sites.

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP-DEV] default charset confusion

Yasuo Ohgaki — Mon, 12 Mar 2012 11:10:02 +0100

Hi

I think following PHP 5.4.0 NEWS entry is misleading.

. Changed default value of "default_charset" php.ini option from ISO-8859-1 to
UTF-8. (Rasmus)

I thought default_charset became UTF-8, so I was expecting
following HTTP header.

content-type text/html; charset=UTF-8

However, I got empty charset (missing 'charset=UTF-8').
So I looked up to source and found the line in SAPI.h

293 #define SAPI_DEFAULT_CHARSET ""

Empty string should be "UTF-8", isn't it?

BTW, empty charset in HTTP header does not mean the default will
be ISO-8859-1, but it let browser guess the encoding is used.
Guessing encoding may cause XSS under certain conditions.

Anyway, I was curious so I've checked ext/standard/html.c and found

/* {{{ entity_charset determine_charset
* returns the charset identifier based on current locale or a hint.
* defaults to UTF-8 */
static enum entity_charset determine_charset(char *charset_hint TSRMLS_DC)
{
int i;
enum entity_charset charset = cs_utf_8;
int len = 0;
const zend_encoding *zenc;

/* Default is now UTF-8 */
if (charset_hint == NULL)
return cs_utf_8;

There are 2 problems.

- php.ini's default_charset should be UTF-8.
- determine_charset() should not blindly default to UTF-8 when there
are no hint.

Old htmlentities/htmlspecialchars actually determines charset from
default_charset/mbstring.internal_encoding/etc. I think old behavior
is better than now.

How about make determine_charset() behaves like 5.3 and set the
SAPI_DEFAULT_CHARSET to "UTF-8"?

Then PHP will behave like as NEWS mentions, htmlentities/htmlspecialchars
default encoding became 'UTF-8' and users will have control for default
htmlenties/htmlspecialchars encoding.

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP-DEV] default charset confusion

Rasmus Lerdorf — Mon, 12 Mar 2012 09:20:01 +0100

On 03/12/2012 12:52 AM, Stas Malyshev wrote:
> Hi!
>
>> Ignoring 5.4 for a second, if you in 5.3 do this:
>>
>> echo htmlspecialchars($string);
>> echo htmlspecialchars($string, NULL, "ISO-8859-1");
>> echo htmlspecialchars($string, NULL, "UTF-8");
>>
>> You will see that the first two output the escaped string with the
>> GB2312 bytes intact within it and the UTF-8 calls returns false because
>> it correctly recognizes that GB2312 is not UTF-8. We don't have any such
>> check for 8859-1, so yes, saying UTF-8 and 8859-1 are the same for
>> htmlspecialchars() is wrong for PHP 5.3 as well as for 5.4.
>
> So the difference is that ISO8859-1 does not validate but UTF-8 validates?
> I'm not sure what GB2312 encoding does but isn't it dangerous to do
> htmlspecialchars() with wrong encoding? Wouldn't htmlentities() also
> produce wrong result when used with wrong encoding?

Not sure you can validate 8859-1 since it isn't multibyte, can you? Is
there any byte that is explicitly forbidden in 8859-1?

And yes, it may very well be dangerous to use the wrong charset and now
that we have better support for GB2312 and other asian charsets in the
entities functions in 5.4 it is even more prudent to choose the right
one so we should provide some way to help people get it right short of
changing every call.

Gustavo suggested we could use the multibyte encoding setting.
Unfortunately only zend.script_encoding is available and I think
internal_encoding is closer to what we need here, but that is only
available as mbstring.internal_encoding.

-Rasmus

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP-DEV] default charset confusion

Adam Jon Richardson — Mon, 12 Mar 2012 09:11:03 +0100

On Mon, Mar 12, 2012 at 3:52 AM, Stas Malyshev wrote:

> Hi!
>
>
> Ignoring 5.4 for a second, if you in 5.3 do this:
>>
>> echo htmlspecialchars($string);
>> echo htmlspecialchars($string, NULL, "ISO-8859-1");
>> echo htmlspecialchars($string, NULL, "UTF-8");
>>
>> You will see that the first two output the escaped string with the
>> GB2312 bytes intact within it and the UTF-8 calls returns false because
>> it correctly recognizes that GB2312 is not UTF-8. We don't have any such
>> check for 8859-1, so yes, saying UTF-8 and 8859-1 are the same for
>> htmlspecialchars() is wrong for PHP 5.3 as well as for 5.4.
>>
>
> So the difference is that ISO8859-1 does not validate but UTF-8 validates?
> I'm not sure what GB2312 encoding does but isn't it dangerous to do
> htmlspecialchars() with wrong encoding? Wouldn't htmlentities() also
> produce wrong result when used with wrong encoding?

The EUC-CN encoding appears to ensure compatibility with ascii by avoiding
the ascii range for each of its two bytes, so it seems that
htmlspecialchars should work OK:

http://en.wikipedia.org/wiki/GB_2312#EUC-CN
http://php.net/manual/en/mbstring.supported-encodings.php

Adam

Adam

Re: [PHP-DEV] default charset confusion

Stas Malyshev — Mon, 12 Mar 2012 09:00:02 +0100

Hi!

> Ignoring 5.4 for a second, if you in 5.3 do this:
>
> echo htmlspecialchars($string);
> echo htmlspecialchars($string, NULL, "ISO-8859-1");
> echo htmlspecialchars($string, NULL, "UTF-8");
>
> You will see that the first two output the escaped string with the
> GB2312 bytes intact within it and the UTF-8 calls returns false because
> it correctly recognizes that GB2312 is not UTF-8. We don't have any such
> check for 8859-1, so yes, saying UTF-8 and 8859-1 are the same for
> htmlspecialchars() is wrong for PHP 5.3 as well as for 5.4.

So the difference is that ISO8859-1 does not validate but UTF-8 validates?
I'm not sure what GB2312 encoding does but isn't it dangerous to do
htmlspecialchars() with wrong encoding? Wouldn't htmlentities() also
produce wrong result when used with wrong encoding?

--
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP-DEV] default charset confusion

Rasmus Lerdorf — Mon, 12 Mar 2012 08:50:03 +0100

On 03/12/2012 12:41 AM, Rasmus Lerdorf wrote:

> $string = $string = "

$gb2312

";

Sorry typo there obviously. Just one $string

-Rasmus

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php