patch for in tree iconv

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

patch for in tree iconv

John-Mark Gurney-2
I've worked at fixing a few of iconv's locking issues, and I've attached
this patch.  This should fix most/all of the locking related issues
relating to the global lock...

Though I ran the tests in tools/test/iconv, and it looks like the
checked in reference conversion in ref are wrong...  The
UTF-32 -> ASCII map (ref/ASCII-rev) has:
0x00A3 = 0x626C
0x00A5 = 0x6E6579

I'm pretty sure 0x626C and 0x6E6579 are not valid ASCII characters.. :)

w/ my updates, they are properly(?) mapped to 0x3F...  I'm not sure
what they were w/o my changes...

I already committed a fixed for an issue w/ svn keywords in
tablegen/cmp.sh...  Hence, why I'm able to run the tests..

Comments?

Please CC me, I'm not on the list..

--
  John-Mark Gurney Voice: +1 415 225 5579

     "All that I will do, has been done, All that I have, has not."
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-i18n
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: patch for in tree iconv

Xin LI-5
I think the list ate your patch, could you please send it inline or
(ideally) post it on a web server?

Cheers,


On Wed, Feb 27, 2013 at 1:32 PM, John-Mark Gurney <[hidden email]> wrote:

> I've worked at fixing a few of iconv's locking issues, and I've attached
> this patch.  This should fix most/all of the locking related issues
> relating to the global lock...
>
> Though I ran the tests in tools/test/iconv, and it looks like the
> checked in reference conversion in ref are wrong...  The
> UTF-32 -> ASCII map (ref/ASCII-rev) has:
> 0x00A3 = 0x626C
> 0x00A5 = 0x6E6579
>
> I'm pretty sure 0x626C and 0x6E6579 are not valid ASCII characters.. :)
>
> w/ my updates, they are properly(?) mapped to 0x3F...  I'm not sure
> what they were w/o my changes...
>
> I already committed a fixed for an issue w/ svn keywords in
> tablegen/cmp.sh...  Hence, why I'm able to run the tests..
>
> Comments?
>
> Please CC me, I'm not on the list..
>
> --
>   John-Mark Gurney                              Voice: +1 415 225 5579
>
>      "All that I will do, has been done, All that I have, has not."
> _______________________________________________
> [hidden email] mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-i18n
> To unsubscribe, send any mail to "[hidden email]"
>



--
Xin LI <[hidden email]> https://www.delphij.net/
FreeBSD - The Power to Serve! Live free or die
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-i18n
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: patch for in tree iconv

John-Mark Gurney-2
In reply to this post by John-Mark Gurney-2
John-Mark Gurney wrote this message on Wed, Feb 27, 2013 at 13:32 -0800:
> I've worked at fixing a few of iconv's locking issues, and I've attached
> this patch.  This should fix most/all of the locking related issues
> relating to the global lock...

I should of included the patch... Oops...

In case the patch gets stripped:
http://people.FreeBSD.org/~jmg/iconv.patch

--
  John-Mark Gurney Voice: +1 415 225 5579

     "All that I will do, has been done, All that I have, has not."

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-i18n
To unsubscribe, send any mail to "[hidden email]"

iconv.patch (8K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: patch for in tree iconv

Gabor Kovesdan-3
In reply to this post by Xin LI-5
Em 27-02-2013 22:38, Xin LI escreveu:

>
>
> On Wed, Feb 27, 2013 at 1:32 PM, John-Mark Gurney <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     I've worked at fixing a few of iconv's locking issues, and I've
>     attached
>     this patch.  This should fix most/all of the locking related issues
>     relating to the global lock...
>
Thank you, I have had this on my TODO list for a long time just haven't
got time for it!

>
>
>     Though I ran the tests in tools/test/iconv, and it looks like the
>     checked in reference conversion in ref are wrong...  The
>     UTF-32 -> ASCII map (ref/ASCII-rev) has:
>     0x00A3 = 0x626C
>     0x00A5 = 0x6E6579
>
>     I'm pretty sure 0x626C and 0x6E6579 are not valid ASCII
>     characters.. :)
>
>     w/ my updates, they are properly(?) mapped to 0x3F...  I'm not sure
>     what they were w/o my changes...
>
Seems like a good solution. However, the ref tables were generated with
GNU libiconv (since the goal is a good compatibility) so we should also
track down why does it emit this.
>
>
>     I already committed a fixed for an issue w/ svn keywords in
>     tablegen/cmp.sh...  Hence, why I'm able to run the tests..
>
Thanks for doing this!

Please also keep me CC'd, I'm not on the list but I'm interested in
iconv-related things.

Gabor
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-i18n
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: patch for in tree iconv

John-Mark Gurney-2
Gabor Kovesdan wrote this message on Thu, Feb 28, 2013 at 08:38 +0100:

> Em 27-02-2013 22:38, Xin LI escreveu:
> >
> >
> >On Wed, Feb 27, 2013 at 1:32 PM, John-Mark Gurney <[hidden email]
> ><mailto:[hidden email]>> wrote:
> >
> >    I've worked at fixing a few of iconv's locking issues, and I've
> >    attached
> >    this patch.  This should fix most/all of the locking related issues
> >    relating to the global lock...
> >
> Thank you, I have had this on my TODO list for a long time just haven't
> got time for it!
> >
> >
> >    Though I ran the tests in tools/test/iconv, and it looks like the
> >    checked in reference conversion in ref are wrong...  The
> >    UTF-32 -> ASCII map (ref/ASCII-rev) has:
> >    0x00A3 = 0x626C
> >    0x00A5 = 0x6E6579
> >
> >    I'm pretty sure 0x626C and 0x6E6579 are not valid ASCII
> >    characters.. :)
> >
> >    w/ my updates, they are properly(?) mapped to 0x3F...  I'm not sure
> >    what they were w/o my changes...
> >
> Seems like a good solution. However, the ref tables were generated with
> GNU libiconv (since the goal is a good compatibility) so we should also
> track down why does it emit this.

It's because we generated the ref w/ //TRANSLIT, and our iconv doesn't
have an advanced set of characters for transliteration...  Heck, it doesn't
handle the difference between //TRANSLIT and //IGNORE...

-0x00A0 = 0x0020 Non-braking space -> space
-0x00A1 = 0x0021 inverted exclamation mark -> exclamation mark
-0x00A2 = 0x0063 cent sign -> c
-0x00A3 = 0x626C pound sign -> lb

I'm "convinced" that the ref is right for proper transliteration...

So, do we need transliteration support?

--
  John-Mark Gurney Voice: +1 415 225 5579

     "All that I will do, has been done, All that I have, has not."
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-i18n
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: patch for in tree iconv

Gabor Kovesdan-3
On 2013.02.28. 19:48, John-Mark Gurney wrote:

> It's because we generated the ref w/ //TRANSLIT, and our iconv doesn't
> have an advanced set of characters for transliteration...  Heck, it doesn't
> handle the difference between //TRANSLIT and //IGNORE...
>
> -0x00A0 = 0x0020 Non-braking space -> space
> -0x00A1 = 0x0021 inverted exclamation mark -> exclamation mark
> -0x00A2 = 0x0063 cent sign -> c
> -0x00A3 = 0x626C pound sign -> lb
>
> I'm "convinced" that the ref is right for proper transliteration...
>
> So, do we need transliteration support?
Ok, now I remember about //TRANSLIT. The default choice of BSD iconv was
to use transliteration by default so I followed this convention. And
once we have it, it should be as complete as possible, imo. Personally,
I would prefer making transliteration optional but that requires heavier
changes to iconv.

Gabor
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-i18n
To unsubscribe, send any mail to "[hidden email]"