UTF-8 problem in 8.0

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

UTF-8 problem in 8.0

Derek Kulinski-2
Hello,

After installing FreeBSD 8.0 I noticed that some characters aren't
displayed correctly (it worked fine for 7.2).

Mainly the dash character is most noticeable (though it's possible
it's more of them).

Example:
env LANG=C man sh
env LANG=pl_PL.UTF-8 man sh
env LANG=en_US.UTF-8 man sh

The first command shows everything fine, the later two seem to show
some unprintable character in place of every dash (i.e. in the second
paragraph word "built-in", or the dashes that break up the words at
the end of the line)

--
Best regards,
 Derek                          mailto:[hidden email]

Daddy, why doesn't this magnet pick up this floppy disk?

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-i18n
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: UTF-8 problem in 8.0

J.R. Oldroyd-2
On Fri, 1 Jan 2010 15:47:36 -0800, Derek Kulinski <[hidden email]> wrote:

>
> Hello,
>
> After installing FreeBSD 8.0 I noticed that some characters aren't
> displayed correctly (it worked fine for 7.2).
>
> Mainly the dash character is most noticeable (though it's possible
> it's more of them).
>
> Example:
> env LANG=C man sh
> env LANG=pl_PL.UTF-8 man sh
> env LANG=en_US.UTF-8 man sh
>
> The first command shows everything fine, the later two seem to show
> some unprintable character in place of every dash (i.e. in the second
> paragraph word "built-in", or the dashes that break up the words at
> the end of the line)
>

My guess is you need to rebuild xterm with WITH_WIDE_CHARS
defined.  This used to be default but in Jan 2009, the option
was removed in the default build.

        -jr
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-i18n
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: UTF-8 problem in 8.0

Ulrich Spörlein-3
In reply to this post by Derek Kulinski-2
On Fri, 01.01.2010 at 15:47:36 -0800, Derek Kulinski wrote:

> Hello,
>
> After installing FreeBSD 8.0 I noticed that some characters aren't
> displayed correctly (it worked fine for 7.2).
>
> Mainly the dash character is most noticeable (though it's possible
> it's more of them).
>
> Example:
> env LANG=C man sh
> env LANG=pl_PL.UTF-8 man sh
> env LANG=en_US.UTF-8 man sh
>
> The first command shows everything fine, the later two seem to show
> some unprintable character in place of every dash (i.e. in the second
> paragraph word "built-in", or the dashes that break up the words at
> the end of the line)

me too, though I'm only setting LC_CTYPE to de_DE.UTF-8 and I don't
always see it. It must be some combination of xterm/ssh and/or putty
that breaks this.

On the other hand, I thought there was a commit done, that would render
- (the dash) always as - (0x2d) so you can copy&paste the examples from
the manpage into your shell.

But perhaps I was just dreaming this up?

Bye,
Uli
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-i18n
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: UTF-8 problem in 8.0

Derek Kulinski-2
In reply to this post by J.R. Oldroyd-2
Hello J.R.,

Saturday, January 2, 2010, 1:21:53 PM, you wrote:

> My guess is you need to rebuild xterm with WITH_WIDE_CHARS
> defined.  This used to be default but in Jan 2009, the option
> was removed in the default build.

Well, the thing is that I'm not using xterm at all. I don't even have
X11 installed there. My only communication with that box is through
putty (it is set to UTF-8). I also tried it through a console. In
console the character is also displayed incorrectly, though unlike
putty is shown as 2 characters.

--
Best regards,
 Derek                            mailto:[hidden email]

As I said before, I never repeat myself.

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-i18n
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: UTF-8 problem in 8.0

Jonathan Chen
In reply to this post by Ulrich Spörlein-3
On Sun, Jan 03, 2010 at 02:38:32AM +0100, Ulrich Sp??rlein wrote:

> On Fri, 01.01.2010 at 15:47:36 -0800, Derek Kulinski wrote:
> > Hello,
> >
> > After installing FreeBSD 8.0 I noticed that some characters aren't
> > displayed correctly (it worked fine for 7.2).
> >
> > Mainly the dash character is most noticeable (though it's possible
> > it's more of them).
> >
> > Example:
> > env LANG=C man sh
> > env LANG=pl_PL.UTF-8 man sh
> > env LANG=en_US.UTF-8 man sh
> >
> > The first command shows everything fine, the later two seem to show
> > some unprintable character in place of every dash (i.e. in the second
> > paragraph word "built-in", or the dashes that break up the words at
> > the end of the line)
>
> me too, though I'm only setting LC_CTYPE to de_DE.UTF-8 and I don't
> always see it. It must be some combination of xterm/ssh and/or putty
> that breaks this.

Actually, I've just got en_NZ.UTF8, and I see it on the console as
well.
--
Jonathan Chen <[hidden email]>
----------------------------------------------------------------------
"With sufficient thrust, pigs fly just fine. However, this is not necessarily
a good idea. It is hard to be sure where they are going to land, and it
could be dangerous sitting under them as they fly overhead." -- RFC 1925
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-i18n
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: UTF-8 problem in 8.0

Derek Kulinski-2
Hello Jonathan,

Saturday, January 2, 2010, 7:28:07 PM, you wrote:

>> me too, though I'm only setting LC_CTYPE to de_DE.UTF-8 and I don't
>> always see it. It must be some combination of xterm/ssh and/or putty
>> that breaks this.

> Actually, I've just got en_NZ.UTF8, and I see it on the console as
> well.

Put en_NZ.UTF-8 (with the dash), if you set an invalid value it
switches to default.

--
Best regards,
 Derek                            mailto:[hidden email]

If you have a procedure with ten parameters, you probably missed some.
-- Alan J. Perlis

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-i18n
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: UTF-8 problem in 8.0

Jonathan Chen
On Sat, Jan 02, 2010 at 07:37:15PM -0800, Derek Kulinski wrote:

> Hello Jonathan,
>
> Saturday, January 2, 2010, 7:28:07 PM, you wrote:
>
> >> me too, though I'm only setting LC_CTYPE to de_DE.UTF-8 and I don't
> >> always see it. It must be some combination of xterm/ssh and/or putty
> >> that breaks this.
>
> > Actually, I've just got en_NZ.UTF8, and I see it on the console as
> > well.
>
> Put en_NZ.UTF-8 (with the dash), if you set an invalid value it
> switches to default.

Sorry. I actually meant that if I had en_NZ.UTF-8 on the console, some
man pages' characters are mangled as well. If it uses the default of C,
everything is fine.

Cheers.
--
Jonathan Chen <[hidden email]>
----------------------------------------------------------------------
                   "Lots of folks confuse bad management with destiny"
                                                         - Kin Hubbard
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-i18n
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: UTF-8 problem in 8.0

Sean C. Farley-2
In reply to this post by Ulrich Spörlein-3
On Sun, 3 Jan 2010, Ulrich Spörlein wrote:

> On Fri, 01.01.2010 at 15:47:36 -0800, Derek Kulinski wrote:
>> Hello,
>>
>> After installing FreeBSD 8.0 I noticed that some characters aren't
>> displayed correctly (it worked fine for 7.2).
>>
>> Mainly the dash character is most noticeable (though it's possible
>> it's more of them).
>>
>> Example:
>> env LANG=C man sh
>> env LANG=pl_PL.UTF-8 man sh
>> env LANG=en_US.UTF-8 man sh
>>
>> The first command shows everything fine, the later two seem to show
>> some unprintable character in place of every dash (i.e. in the second
>> paragraph word "built-in", or the dashes that break up the words at
>> the end of the line)
>
> me too, though I'm only setting LC_CTYPE to de_DE.UTF-8 and I don't
> always see it. It must be some combination of xterm/ssh and/or putty
> that breaks this.
>
> On the other hand, I thought there was a commit done, that would
> render - (the dash) always as - (0x2d) so you can copy&paste the
> examples from the manpage into your shell.
>
> But perhaps I was just dreaming this up?
The last commit that I recall (r192561) fixed dashes in front of options
(.Fl macro) but not dashes present within a man page.  Personally, I
just create an alias for man to call 'LANG=C man' to protect against
examples with non-ASCII (U+002D) dashes.

Read the commit message[1] for a better explanation of unresolved
issues.

Sean
   1. http://svn.freebsd.org/viewvc/base?view=revision&revision=192561
--
[hidden email]
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-i18n
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: UTF-8 problem in 8.0

Derek Kulinski-2
Hello Sean,

Sunday, January 3, 2010, 9:03:01 AM, you wrote:

> The last commit that I recall (r192561) fixed dashes in front of options
> (.Fl macro) but not dashes present within a man page.  Personally, I
> just create an alias for man to call 'LANG=C man' to protect against
> examples with non-ASCII (U+002D) dashes.

> Read the commit message[1] for a better explanation of unresolved
> issues.

> Sean
>    1. http://svn.freebsd.org/viewvc/base?view=revision&revision=192561

What caused this to be broken? Everything was fine up until 7.2 (or at
least I didn't notice this earlier)

--
Best regards,
 Derek                            mailto:[hidden email]

A seminar on Time Travel will be held two weeks ago.

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-i18n
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: UTF-8 problem in 8.0

Sean C. Farley-2
On Sun, 3 Jan 2010, Derek Kulinski wrote:

> Hello Sean,
>
> Sunday, January 3, 2010, 9:03:01 AM, you wrote:
>
>> The last commit that I recall (r192561) fixed dashes in front of
>> options (.Fl macro) but not dashes present within a man page.
>> Personally, I just create an alias for man to call 'LANG=C man' to
>> protect against examples with non-ASCII (U+002D) dashes.
>
>> Read the commit message[1] for a better explanation of unresolved
>> issues.
>
>> Sean
>>    1. http://svn.freebsd.org/viewvc/base?view=revision&revision=192561
>
> What caused this to be broken? Everything was fine up until 7.2 (or at
> least I didn't notice this earlier)

Previously (to some commit revision), man (nroff) did not have NLS
enabled, so the output was in ASCII.

Sean
--
[hidden email]
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-i18n
To unsubscribe, send any mail to "[hidden email]"