Field delimiter in sort program in freebsd

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Field delimiter in sort program in freebsd

Shayan Pooya
Hello,

How can I use the ascii character 0xff as the field delimiter for the
FreeBSD sort program?
On bash on Linux I use the following:
$ sort -t $'\xff' -z
to sort a set of null terminated lines where each field is separated by a
\xff.

The FreeBSD 10 manual page says it has the -t option, but I am not sure how
I should give it the option to use a specific character.

On tcsh:
% sort -t $'\xff'
Illegal variable name.


On bash:
sort -t $'\xfe'
works fine, but:

$ sort -t $'\xff'

complains:

sort: Invalid argument


(Just in case, this is the piece of code that runs on MacOS and Linux but
not on FreeBSD:
https://github.com/discoproject/disco/blob/develop/lib/disco/compat.py#L79because
of this issue).

Thanks.
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Field delimiter in sort program in freebsd

Bernt Hansson-2
On 2014-01-25 14:59, Shayan Pooya wrote:

> Hello,
>
> How can I use the ascii character 0xff as the field delimiter for the
> FreeBSD sort program?
> On bash on Linux I use the following:
> $ sort -t $'\xff' -z
> to sort a set of null terminated lines where each field is separated by a
> \xff.
>
> The FreeBSD 10 manual page says it has the -t option, but I am not sure how
> I should give it the option to use a specific character.
>
> On tcsh:
> % sort -t $'\xff'
> Illegal variable name.

$ is a variable, and you have not defined it.
$ by it self means the first word on the line ie sort.

So your line expands to: $ sort -t sort'\xff'

NB I AM NOT A PROGRAMMER!
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Field delimiter in sort program in freebsd

Shayan Pooya
$ is a variable, and you have not defined it.

> $ by it self means the first word on the line ie sort.
>
> So your line expands to: $ sort -t sort'\xff'
>
> NB I AM NOT A PROGRAMMER!
>

Thanks. That explains the error message.  What is the correct way of doing
it in tcsh though?
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Field delimiter in sort program in freebsd

Polytropon
On Sat, 25 Jan 2014 09:51:39 -0500, Shayan Pooya wrote:
> Thanks. That explains the error message.  What is the correct way of doing
> it in tcsh though?

I know it's probably one of the most overcomplicated and stupid
approaches, but it seems to work:

        % sort -t `awk 'BEGIN { printf("%c", 0); }'` <in-data>

Please note that I've not tested this. The null byte has a
special meaning, it's interpreted as "end of string" in C-style
strings ("null terminated strings"), so having to use it as a
field delimiter looks a bit strange.

Maybe you could use a search and replace step _prior_ to sorting
to get this resolved? Just a suggestion...



--
Polytropon
Magdeburg, Germany
Happy FreeBSD user since 4.0
Andra moi ennepe, Mousa, ...
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Field delimiter in sort program in freebsd

Shayan Pooya
In reply to this post by Shayan Pooya
I took a look at the source code of sort and saw this condition:

                                if (sort_opts_vals.field_sep == WEOF) {
                                        errno = EINVAL;
                                        err(2, NULL);
                                }


https://github.com/freebsd/freebsd/blob/master/usr.bin/sort/sort.c#L1063

Removing this condition and remaking sort fixes the issue (at least I can
use bash to give 0xff to the sort program, no luck with other shells
though).

Using \xff as the record separator makes sense for binary files. Any idea
why there is such a condition in the sort program?

Thanks.
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Field delimiter in sort program in freebsd

John Levine
>Using \xff as the record separator makes sense for binary files. Any idea
>why there is such a condition in the sort program?

Probably because its authors implemented the POSIX spec for the sort utility,
which sorts text files.

If you want to sort binary stuff using the standard sort program, turn it into text,
sort it, and turn it back.  Or write your own specialized binary sort.

R's,
John


_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[hidden email]"