Speculative: Rust for base system components

classic Classic list List threaded Threaded
132 messages Options
12345 ... 7
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Enji Cooper

> On Dec 31, 2018, at 7:02 AM, Warner Losh <[hidden email]> wrote:

….

> TBH, there's a stronger case for python than rust: there's actual python
> scripts in the tree today that we have to install a port to use. And there
> the benefit, while not zero, is small and the effort is large compared to
> just dragging it in as a port, so it hasn't been done. It's another fast
> evolving language that requires multiple versions as well...
>
> So write something that everybody wants, that must be in base, and that
> requires rust, and then we can have the conversation…

As someone who has been using python extensively over the past decade, I think that using python versions less than 3.7 without type hints to ensure that the data types are correct [1], mocked tests to ensure that the code functions and is tested, is the wrong path to make.

Rust is still a young language, but it has a number of benefits in terms of:
i. Being able to scale past JIT python. This fact doesn’t matter on workstations/servers, but it definitely matters on the low end with embedded systems and the upper end with distributed systems at scale (there’s a reason why a number of critical services at my previous longterm employment were written in C++, not python. Some argue Rust can outperform C/C++ [2]. C++ I can see (managed pointers were about an order of magnitude less performant in a microbenchmark I wrote for grabbing the time in the Linux kernel vs malloc in C using llvm36). However, outperforming C is up for debate.
ii. It is easier to grok than C++ (even the most recent versions of the C++ spec, the language is cryptic in areas).
iii. It is more reusable than C out of the box. How often do we need to rewrite common logic/routines in C and mimic an OOP language like C++ (see libarchive, pkgng)?

<offtopic>
At the end of the day, I think the key is that the FreeBSD project needs to start expressing more complicated subsystems in terms of OOP languages, like C++, Rust, etc, instead of expressing most of the code in C. I do think (for instance) a service management system would be a good candidate for modern C++ or Rust.
</offtopic>

Cheers,
-Enji

1. https://www.python.org/dev/peps/pep-0484/ <https://www.python.org/dev/peps/pep-0484/>
2. https://www.reddit.com/r/rust/comments/5urar1/is_rust_likely_the_next_fastest_language_after_c/ <https://www.reddit.com/r/rust/comments/5urar1/is_rust_likely_the_next_fastest_language_after_c/>

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Eric McCorkle-2
In reply to this post by Conrad Meyer-2
On 12/31/18 1:56 AM, Conrad Meyer wrote:
> Rust has many of the exact same problems C++ has.  Sure, it avoids a
> few classes of issue.  But it doesn't fix the classic C++ problems of
> extremely slow build times (if you think C++ is bad, Rust is worse)
> and having a kitchen sink of language features that make Rust and C++
> code difficult to grok.

I would debate the kitchen sink claim.  For one, Rust benefits from a
solid understanding of type systems that didn't exist when C++ was
created.  Proper parameterized types are a significant improvement over
C++ templates (the same holds for Java's generics, but that's
tangential).  This alone reduces the complexity of the language (and its
error messages) considerably.  While I give C++ slack on the issue of
templates because Somebody Had To Go First, that doesn't mean I relish
the idea of writing C++ code.

Beyond that, the C++ standardization process these days is seemingly
aiming at bringing everything under the sun into the language itself,
whereas Rust went for a syntax extension system and an overall language
design that avoids the need to graft everything into the language
itself.  (Side note: as much as I loathe macros in programming
languages, Rust actually seems to have produced a reasonable macro-like
system)


signature.asc (235 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Eric McCorkle-2
In reply to this post by Igor Mozolevsky-2
On 12/31/18 9:21 AM, Igor Mozolevsky wrote:

> Does the generated code size increase or decrease and does the
> data-path get longer or shorter with switch from C to Rust with all
> the "great features" of Rust? Are there any metrics for these? If a
> CPU cache gets trashed repeatedly or data takes massive d-tours that
> would be a significant downside, wouldn't it?

Since most of the deltas of Rust over C are accomplished through
type-checking, I don't imagine this would be an issue.


signature.asc (235 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Diane Bruce
In reply to this post by Enji Cooper
On Mon, Dec 31, 2018 at 09:36:18AM -0800, Enji Cooper wrote:
>
> > On Dec 31, 2018, at 7:02 AM, Warner Losh <[hidden email]> wrote:
>
...
> <offtopic>
> At the end of the day, I think the key is that the FreeBSD project needs to start expressing more complicated subsystems in terms of OOP languages, like C++, Rust, etc, instead of expressing most of the code in C. I do think (for instance) a service management system would be a good candidate for modern C++ or Rust.
> </offtopic>

Well, as someone who remembers when Fortran was King and much systems(!)
programming such as editors etc. was done in Fortran, I cheer on
any effort moving on from the failed Fortran compiler 'C'.

FWIW I happen to like Rust and Python. So um I'll just step back
and let others do the painting.

P.S. I prefer plaid green and orange with purple stripes.

<back into my hole with my popcorn>


>
> Cheers,
> -Enji
>
> 1. https://www.python.org/dev/peps/pep-0484/ <https://www.python.org/dev/peps/pep-0484/>
> 2. https://www.reddit.com/r/rust/comments/5urar1/is_rust_likely_the_next_fastest_language_after_c/ <https://www.reddit.com/r/rust/comments/5urar1/is_rust_likely_the_next_fastest_language_after_c/>



--
- [hidden email] [hidden email] http://artemis.db.net/~db
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Eric McCorkle-2
In reply to this post by Enji Cooper
On 12/31/18 12:36 PM, Enji Cooper wrote:

> Rust is still a young language, but it has a number of benefits in terms of:
> i. Being able to scale past JIT python. This fact doesn’t matter on workstations/servers, but it definitely matters on the low end with embedded systems and the upper end with distributed systems at scale (there’s a reason why a number of critical services at my previous longterm employment were written in C++, not python. Some argue Rust can outperform C/C++ [2]. C++ I can see (managed pointers were about an order of magnitude less performant in a microbenchmark I wrote for grabbing the time in the Linux kernel vs malloc in C using llvm36). However, outperforming C is up for debate.

In terms of outperforming C++, you can outrun it on anything that
involves virtual calls or runtime type information.  C++ compilers have
gotten pretty good at optimizing this stuff, but that's all an
approximation of just not doing it in the first place.

In terms of C, I think the more accurate notion is that you converge to
C-like performance over time.  There's not a lot of opportunity to pull
out ahead of C in terms of performance, afterall, at least in purely
sequential code.


signature.asc (235 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Steve Kargl
In reply to this post by Diane Bruce
On Mon, Dec 31, 2018 at 01:05:24PM -0500, Diane Bruce wrote:

> On Mon, Dec 31, 2018 at 09:36:18AM -0800, Enji Cooper wrote:
> >
> > > On Dec 31, 2018, at 7:02 AM, Warner Losh <[hidden email]> wrote:
> >
> ...
> > <offtopic>
> > At the end of the day, I think the key is that the FreeBSD project needs to start expressing more complicated subsystems in terms of OOP languages, like C++, Rust, etc, instead of expressing most of the code in C. I do think (for instance) a service management system would be a good candidate for modern C++ or Rust.
> > </offtopic>
>
> Well, as someone who remembers when Fortran was King and much systems(!)
> programming such as editors etc. was done in Fortran, I cheer on
> any effort moving on from the failed Fortran compiler 'C'.
>

All those Fortran codes from years ago, if written in
standard conforming Fortran, will still compile today.
J3 (Fortran standardization committee) goes to great
length to retain backwards compatiability with older
standards.  The latest revision to the language was
ratified as an international standard only a few weeks
ago.  https://wg5-fortran.org/f2018.html 

--
Steve
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Alexandre C. Guimarães-2
In reply to this post by Eric McCorkle-2
Hi.

Just to add my 2c. ;-)

If the objective is *security* and *reliability*, the said options: C++
(assuming HIC++), Rust, and Haskwell are all surpassed in these regards by
Ada/SPARK (yes, there is the compiler problem) and OCaml, both in fact used
by high integrity safe-critical industries.

There is a new Ada standard coming (Ada2020), and the Ada standard
`updates` are not the hell of C++ ones.

The SPARK language is being formally verified by INRIA using the Coq proof
assistant (which is written in OCaml).

http://sworthodoxy.blogspot.com/2017/03/comparing-ada-and-high-integrity-c.html

PS. oh, there is also a CompCert frontend being written for SPARK by INRIA
(at least they were prototyping it in 2014).

Cheers!

--
Best Regards,
Alexandre C. Guimarães.
https://bitbucket.org/rigoletto-freebsd/

signature.asc (981 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Rozhuk Ivan-2
In reply to this post by Volker Lendecke
On Mon, 31 Dec 2018 17:07:38 +0100
Volker Lendecke <[hidden email]> wrote:

> Not being involved in FreeBSD but in Samba, I could see a migration of
> parts of Samba to Rust if it had a Rust->C translator. I think for
> FreeBSD base components similar arguments would apply. You could ship
> the C output for the platforms that don't have native Rust.

Rust is modern geeks toy by mozilla.
Just waste ot time, without any real profits.

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Rozhuk Ivan-2
In reply to this post by Enji Cooper
On Mon, 31 Dec 2018 09:36:18 -0800
Enji Cooper <[hidden email]> wrote:

> iii. It is more reusable
> than C out of the box. How often do we need to rewrite common
> logic/routines in C and mimic an OOP language like C++ (see
> libarchive, pkgng)?

This is not a problem.


> <offtopic>
> At the end of the day, I think the key is that the FreeBSD project
> needs to start expressing more complicated subsystems in terms of OOP
> languages, like C++, Rust, etc, instead of expressing most of the
> code in C. I do think (for instance) a service management system
> would be a good candidate for modern C++ or Rust. </offtopic>
>

C - 100 pages
C++ - 1000 pages.
Rust - unknown, not stable.

Why some one should learn things so big to do same thing that can be done with simple C?

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Rozhuk Ivan-2
In reply to this post by Eric McCorkle-2
On Mon, 31 Dec 2018 12:58:51 -0500
Eric McCorkle <[hidden email]> wrote:

> I would debate the kitchen sink claim.  For one, Rust benefits from a
> solid understanding of type systems that didn't exist when C++ was
> created.  Proper parameterized types are a significant improvement
> over C++ templates (the same holds for Java's generics, but that's
> tangential).  This alone reduces the complexity of the language (and
> its error messages) considerably.  While I give C++ slack on the
> issue of templates because Somebody Had To Go First, that doesn't
> mean I relish the idea of writing C++ code.
>

Who cares!?
Who will support all this staff?

I want that programs works, I do not care types, classes and other things.
Now most things works, so why we should waste time to rewrite it to more difficult to understand and support languages?
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Steffen Nurpmeso-2
In reply to this post by Rozhuk Ivan-2
Rozhuk Ivan wrote in <20181231221030.6471937e@rimwks>:
 |On Mon, 31 Dec 2018 17:07:38 +0100
 |Volker Lendecke <[hidden email]> wrote:
 |
 |> Not being involved in FreeBSD but in Samba, I could see a migration of
 |> parts of Samba to Rust if it had a Rust->C translator. I think for
 |> FreeBSD base components similar arguments would apply. You could ship
 |> the C output for the platforms that don't have native Rust.
 |
 |Rust is modern geeks toy by mozilla.
 |Just waste ot time, without any real profits.

I vote for Nim (former Nimrod is no more).

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Conrad Meyer-2
In reply to this post by Eric McCorkle-2
Hi Eric,

I don't disagree that C++ is worse than Rust in that regard.  I just
don't believe Rust is enough better to be a suitable *C* replacement
(it's a perfectly fine C++ replacement).

Things that are closer to suitable might be:

1. Enabling stronger C semantics and runtime enforcement over existing
code.  It might be possible to enable globally and selectively disable
only on performance sensitive code, while still covering the majority
of the system.  Some work has been done by the Cambridge academics as
part of CHERI / Cerberus / REMS ("Rigorous Engineering for Mainstream
Systems").  A demo of Cerberus is online at
http://cerberus.cl.cam.ac.uk/cerberus ; press 'forward' a few times.
Or it could be something else (UBSan and ASan on security-critical
userspace programs?).  But the migration path from existing code is
more clear and requires less "throw away the world" as step 1.

2. Ziglang seems somewhat promising.  It's still immature and I am
certainly not claiming the pros of rewriting components or including
new compilers in base outweigh the cons.  But its aims are closer to C
than C++ or Rust (simplicity, no operator overloading) and that seems
a better fit.  It is also based on LLVM, which comes with some of the
same drawbacks as Rust (or Clang, for that matter) re: toolchain build
time and supported architectures.  https://github.com/ziglang/zig

Best,
Conrad

On Mon, Dec 31, 2018 at 9:59 AM Eric McCorkle <[hidden email]> wrote:

>
> On 12/31/18 1:56 AM, Conrad Meyer wrote:
> > Rust has many of the exact same problems C++ has.  Sure, it avoids a
> > few classes of issue.  But it doesn't fix the classic C++ problems of
> > extremely slow build times (if you think C++ is bad, Rust is worse)
> > and having a kitchen sink of language features that make Rust and C++
> > code difficult to grok.
>
> I would debate the kitchen sink claim.  For one, Rust benefits from a
> solid understanding of type systems that didn't exist when C++ was
> created.  Proper parameterized types are a significant improvement over
> C++ templates (the same holds for Java's generics, but that's
> tangential).  This alone reduces the complexity of the language (and its
> error messages) considerably.  While I give C++ slack on the issue of
> templates because Somebody Had To Go First, that doesn't mean I relish
> the idea of writing C++ code.
>
> Beyond that, the C++ standardization process these days is seemingly
> aiming at bringing everything under the sun into the language itself,
> whereas Rust went for a syntax extension system and an overall language
> design that avoids the need to graft everything into the language
> itself.  (Side note: as much as I loathe macros in programming
> languages, Rust actually seems to have produced a reasonable macro-like
> system)
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Wojciech Puchar-8
In reply to this post by Eric McCorkle-2
it's nothing wrong with C. no need for another modern language that
"fixes" problems that doesn't exist when properly programming

On Sun, 30 Dec 2018, Eric McCorkle wrote:

> Before I begin, I want to be clear that everything here is in the realm
> of speculative, long-term discussion.  My goal is to start a
> conversation, not to propose anything concrete right now.
>
>
>
> I've talked at several conferences about the possibility of bringing the
> Rust programming language into the base system, with the intent of
> making it a viable implementation language for parts of the base system.
> I believe the potential security and stability benefits of doing this
> are *substantial*.  This would obviously be a serious undertaking, but I
> think the benefits are worth the cost.  For the rest of this mail, I'll
> outline my position and present the important facts that support it.
>
> I of course welcome any discussion, criticism, or input.
>
> = Why Rust? =
>
> Rust is a true systems programming language that incorporates results
> from programming language research that took place in the late 90's and
> 2000's, and can be viewed as a culmination of this work.  (Specifically,
> it draws on work on region types as well as substructural types, but
> that's its own survey article).
>
> This manifests most clearly in Rust's lifetime type system, which is
> able to perform quite sophisticated reasoning about memory allocation
> and ownership.  Rust programs do not require you to explicitly manage
> memory, but neither do they use garbage collection.  Memory management
> is automatically tracked by the lifetime types and allocations/frees are
> added during compilation.  Moreover, threaded programs are able to use
> the same type system features to track ownership.  This effectively
> prevents a large class of memory- and concurrency-related errors *at the
> type system level*, resulting in dramatically safer programs.  Rust
> provides an "unsafe" language facility for accomplishing low-level
> tasks, so you're not prevented from getting things done; however, almost
> all programs can be implemented without relying on unsafe, and doing so
> eliminates what has probably been the two worst classes of bugs in the
> history of computing.
>
> Because this is done at the type system level, it has no cost at
> runtime.  This theme of "zero-cost abstractions" runs throughout the
> language, which makes it suitable as a replacement for C in the long
> term.  It also incorporates features befitting a modern programming
> language such as parameterized types (aka generics), pattern-matching,
> and trait-based polymorphism.  Finally, it can interface with C/C++
> binaries and libraries relatively easily, making it a suitable systems
> implementation language.
>
> Because of these benefits, several projects (Mozilla and Tor) have begun
> to reimplement their systems using Rust.  Additionally, there is a
> master's thesis about writing FreeBSD device drivers using Rust out
> there somewhere (forgive me for not chasing it down).
>
> A secondary benefit is that Rust's packaging system, cargo, is built
> around a more modern concept of modularity.  This is in line with
> FreeBSD's goals of modularizing the base system and reducing redundancy.
>
> = Downsides =
>
> There are a couple of downsides, which I'll outline briefly, and provide
> some counterargumentation for them.
>
> * It's a big component
>
> Adding the entire Rust platform would add more to the buildworld time,
> which is already quite long due to LLVM.  OTOH, LLVM itself is a big
> chunk of the rust build time, and we already build that as part of the
> base system now.  Moreover, it stands to reason that the compiler would
> end up being the majority of an open-source operating system's build
> time, with second place being the kernel itself.  Compilers are an
> active area of research with no signs of slowing down, and more hardware
> platforms means more backends, which means the compiler ends up growing
> over time, while most of the rest of the system doesn't.
>
> * The language is still evolving
>
> While this is true, the language has gotten *way* more mature in the
> past year and a half, and I think it's ready to hold up major codebases
> and production use, or at the very least it will be by the time we'd be
> seriously looking at bringing it in.
>
> * Rust binaries are large
>
> This issue can be avoided with the proper use of build flags.  The
> default settings use static linking for everything, and bring in the
> entire runtime library (which isn't necessary in most cases).  See this
> article:
> https://lifthrasiir.github.io/rustlog/why-is-a-rust-executable-large.html.
> With LTO, the size can be cut down to that of a C++ binary, and with
> dynamic linking, the executable size is comparable to a C program.  If
> we were seriously using rust, we would probably change the default flags
> to use dynamic linking/LTO by default, in fact.
>
>
> = Alternatives =
>
> There have been several alternatives that come up when I mention this.
> I'll discuss the two big ones: go and C++
>
> == C++ ==
>
> C++ is already in the base system, of course.  It could be brought into
> the kernel with some effort and restrictions.  The biggest argument for
> Rust over C++ in my view is that Rust is a much simpler, cleaner
> language.  C++ at this point is bewilderingly complex as a language, and
> has so many bells and whistles that it almost requires an effort to
> decide which subset of the language will be used.  This also tends to
> slow down efforts to expand the language over time.
>
> Rust by comparison is much leaner.  Its lifetime type system can be
> intimidating, but the benefits are well worth getting used to it.  To
> sum it up, C++ certainly isn't going anywhere, but I think Rust has a
> great deal more vitality as a language.
>
> == Go ==
>
> Go is not really a "systems language" in the sense that C or Rust are.
> It isn't a replacement for C so much as a replacement for python.
>
> What's more, it is almost universally disliked by people with a
> background in programming languages (I am one of those).  There's a
> couple of reasons for this.  First, it omits a number of features that
> modern languages *really* should include, parameterized types (generics)
> being one of the most glaring.  Beyond that, it completely ignores
> anything produced by modern PL research (unlike Rust, which benefits
> from that work).  Second, one of its main design parameters was "it
> should compile fast".  This as a design constraint basically guarantees
> that you can't do anything interesting in compilation, which rules out
> both advanced type systems as well as serious optimization.
>
> Unlike Rust, which offers substantial security and stability benefits
> through its type system, Go would do little more than just change the
> syntax of a bunch of code.  Moreover, it's likely that there would be
> performance consequences, given the design constraint of fast compilation.
>
> = Summary =
>
> Rust is a modern systems language with substantial benefits, and is
> well-poised to become the heir apparent to C as the dominant systems
> language going into the future.  For this reason, I think we should
> seriously consider it as a language for the base system at some point in
> the future, when the time is right and after sufficient exploration and
> experimentation.
>
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Wojciech Puchar-8
In reply to this post by Rozhuk Ivan-2
>> FreeBSD base components similar arguments would apply. You could ship
>> the C output for the platforms that don't have native Rust.
>
> Rust is modern geeks toy by mozilla.
> Just waste ot time, without any real profits.
>
> _______________________________________________

still it's quite fearful such discussion happens at all here.
I still need some real usable unix system and will need in coming years so
i'm feared.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Garance A Drosehn
In reply to this post by Alan Somers-2
On 31 Dec 2018, at 11:52, Alan Somers wrote:

> On Mon, Dec 31, 2018 at 12:17 AM Conrad Meyer <[hidden email]> wrote:
>>
>> Rust has many of the exact same problems C++ has.  Sure, it avoids
>> a few classes of issue.  But it doesn't fix the classic C++ problems
>> of extremely slow build times (if you think C++ is bad, Rust is
>> worse) and having a kitchen sink of language features that make Rust
>> and C++ code difficult to grok.
>
> Do you have any data to support the claim of slow build times?
> I know it's very hard to do an apples-to-apples comparison of build
> times between different languages, but I would be interested to see
> somebody try.

I know that it takes a very long time to build rust (the compiler
itself) on my mac, and FreeBSD's buildworld always starts out by
building the compilers that it needs.  So adding rust to the base
system is going to cause an immediate and noticeable increase in
the time used by buildworld.  (now maybe that means we should
come up with some different ideas for handling the compilers in
buildworld, but I digress...).


On 31 Dec 2018, at 8:18, Jan Knepper wrote:
>
> What about D?
> www.dlang.org
>
> ManiaC++
> Jan Knepper

For what it's worth, I'm also interested in D-lang, and I know that
the developers of D are paying attention to Rust, and trying to
figure out what ideas from Rust could be added to D-lang without
making a mess of things.  And it's nice that there are three
separate implementations of D-lang.  ... but no, I would not
include D-lang into the base system just yet.

But getting back to Rust:

On 31 Dec 2018, at 9:21, Igor Mozolevsky wrote:

>
> On Mon, 31 Dec 2018 at 05:42, Eric McCorkle <[hidden email]>
> wrote:
>>
>> Before I begin, I want to be clear that everything here is in the
>> realm
>> of speculative, long-term discussion.  My goal is to start a
>> conversation, not to propose anything concrete right now.
>
>
> Does the generated code size increase or decrease and does the
> data-path get longer or shorter with switch from C to Rust with
> all the "great features" of Rust?  Are there any metrics for
> these?  If a CPU cache gets trashed repeatedly or data takes
> massive detours that would be a significant downside, wouldn't it?

I've been somewhat interested in Rust for awhile, although I haven't
had the time to do much of anything with it.  Part of the answer to
your questions will depend on how good the programmers are.  One
advantage the BSD projects have by sticking with C is that we have
spent a lot of time becoming good at that one language.  We have a
lot of developers who can look at an update to C code, and quickly
notice some of the most-common problems which come up when writing C.

But there is evidence that once you have decent experience with Rust,
then you can write code which is as fast or even faster than what you
would write in C.

Lately Bryan Cantrill has been taking some deep dives into using Rust,
and it's interesting to listen to his experiences.  In particular his
summer-project for rewriting one of his own programs (called
"Statemaps")
into Rust, which went much better than he expected it to.  Here are a
few of his recent presentations where he's talked about Rust:

https://www.youtube.com/watch?v=aWbGPMxs0AM - Statemaps in Rust
https://www.youtube.com/watch?v=2wZ1pCpJUIM - Rust and other interesting
things
https://www.youtube.com/watch?v=LjFM8vw3pbU - The Summer of Rust
https://www.slideshare.net/bcantrill/is-it-time-to-rewrite-the-operating-system-in-rust
    -- Title: "Is it time to rewrite the operating system in Rust?"
    --        "A talk given at QConSF in 2018. Video to come"
    -- Spoiler: The answer is "No" for *rewriting* an operating
       system, but it could be a good choice for writing new
       modules for an existing kernel.

I think he's done a few other presentations wrt Rust, but I can't
seem to find them right now.  Maybe they're on Vimeo.

I expect Rust could be very interesting for systems-programming in
the future, but I also think we (FreeBSD developers) would need to
have more experience with it before pulling it into the base system.

--
Garance Alistair Drosehn                =     [hidden email]
Senior Systems Programmer               or   [hidden email]
Rensselaer Polytechnic Institute;             Troy, NY;  USA
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Garance A Drosehn
On 31 Dec 2018, at 17:44, Garance A Drosehn wrote:

>> On Mon, Dec 31, 2018 at 12:17 AM Conrad Meyer <[hidden email]> wrote:
>>
>> Do you have any data to support the claim of slow build times?
>> I know it's very hard to do an apples-to-apples comparison of build
>> times between different languages, but I would be interested to see
>> somebody try.
>
> I know that it takes a very long time to build rust (the compiler
> itself) on my mac, and FreeBSD's buildworld always starts out by
> building the compilers that it needs.  So adding rust to the base
> system is going to cause an immediate and noticeable increase in
> the time used by buildworld.


> On 31 Dec 2018, at 9:21, Igor Mozolevsky wrote:
>>
>> Does the generated code size increase or decrease and does the
>> data-path get longer or shorter with switch from C to Rust with
>> all the "great features" of Rust?  Are there any metrics for
>> these?  If a CPU cache gets trashed repeatedly or data takes
>> massive detours that would be a significant downside, wouldn't it?
>
> I've been somewhat interested in Rust for awhile, although I haven't
> had the time to do much of anything with it.  Part of the answer to
> your questions will depend on how good the programmers are.

There's two questions wrt performance.  One is how long it takes
to compile some program, and then the other is how fast the
code is which is produced by the compiler.  A note on both of
those:  Rust has the notion of "building for release", in which
case the compile-time step takes *much* longer, but all that
extra time goes into producing better code in the resulting
program.

So if you think the compiler is fast but that it's generating very
slow programs, then it might be that you're not doing "a release
build".

There's one program written in rust which I use a lot, which is
called 'ripgrep' (or 'rg' for short).  It's one of the few programs
which I install via pre-compiled pkgs, because I don't want to wait
for all the time it takes to compile a release-build.  But I'm very
happy with how fast the program runs.

--
Garance Alistair Drosehn                =     [hidden email]
Senior Systems Programmer               or   [hidden email]
Rensselaer Polytechnic Institute;             Troy, NY;  USA
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Igor Mozolevsky-2
On Tue, 1 Jan 2019 at 00:30, Garance A Drosehn wrote:

<snip>

> There's two questions wrt performance.  One is how long it takes
> to compile some program, and then the other is how fast the
> code is which is produced by the compiler.  A note on both of
> those:  Rust has the notion of "building for release", in which
> case the compile-time step takes *much* longer, but all that
> extra time goes into producing better code in the resulting
> program.

Quite frankly the compile time isn't really *that* important, and
longer (if not much longer) build times might push toward a better
modularisation and compartmentalisation of the OS and the kernel so a
small change in the kernel, for example, doesn't require the
recompilation of the whole damn thing when nothing else is affected.
*However,* in my experience various language "freebies" never truly
come "free," the cheese a mousetrap springs to mind. And some of the
"for"-reasons don't make much sense:- take the no need to alloc/free
memory, for example, the only way to guarantee that it is really free
is for some static analysis to envisage every plausible path the code
takes and insert the appropriate statements in the appropriate
places... *BUT* if you can do that in a static analyser, why not just
run the static analysed against c-code and let it flag up bugs
instead?


<snip>

> There's one program written in rust which I use a lot, which is
> called 'ripgrep' (or 'rg' for short).  It's one of the few programs
> which I install via pre-compiled pkgs, because I don't want to wait
> for all the time it takes to compile a release-build.  But I'm very
> happy with how fast the program runs.

And what metric is that against? I'm very happy with the speed jQuery
is executed in my browser, but that's no reasonable metric to argue
that JavaScript is a suitable language for the kernel...



--
Igor M.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

RE: Speculative: Rust for base system components

Cy Schubert-4
In reply to this post by Eric McCorkle-2
Unlikely I'll get to my computer, I'll reply more tersely.

Agreed.

Some thoughts/questions.

Do we need another language in base when the same exists in ports? Can this wait for pkg base?

Are we not trying to reduce base bloat and the maintenance of questionable software? Consider our current cull efforts.

---
Sent using a tiny phone keyboard.
Apologies for any typos and autocorrect.
Also, this old phone only supports top post. Apologies.

Cy Schubert
<[hidden email]> or <[hidden email]>
The need of the many outweighs the greed of the few.
---

-----Original Message-----
From: Wojciech Puchar
Sent: 31/12/2018 14:53
To: Eric McCorkle
Cc: [hidden email]
Subject: Re: Speculative: Rust for base system components

it's nothing wrong with C. no need for another modern language that
"fixes" problems that doesn't exist when properly programming

On Sun, 30 Dec 2018, Eric McCorkle wrote:

> Before I begin, I want to be clear that everything here is in the realm
> of speculative, long-term discussion.  My goal is to start a
> conversation, not to propose anything concrete right now.
>
>
>
> I've talked at several conferences about the possibility of bringing the
> Rust programming language into the base system, with the intent of
> making it a viable implementation language for parts of the base system.
> I believe the potential security and stability benefits of doing this
> are *substantial*.  This would obviously be a serious undertaking, but I
> think the benefits are worth the cost.  For the rest of this mail, I'll
> outline my position and present the important facts that support it.
>
> I of course welcome any discussion, criticism, or input.
>
> = Why Rust? =
>
> Rust is a true systems programming language that incorporates results
> from programming language research that took place in the late 90's and
> 2000's, and can be viewed as a culmination of this work.  (Specifically,
> it draws on work on region types as well as substructural types, but
> that's its own survey article).
>
> This manifests most clearly in Rust's lifetime type system, which is
> able to perform quite sophisticated reasoning about memory allocation
> and ownership.  Rust programs do not require you to explicitly manage
> memory, but neither do they use garbage collection.  Memory management
> is automatically tracked by the lifetime types and allocations/frees are
> added during compilation.  Moreover, threaded programs are able to use
> the same type system features to track ownership.  This effectively
> prevents a large class of memory- and concurrency-related errors *at the
> type system level*, resulting in dramatically safer programs.  Rust
> provides an "unsafe" language facility for accomplishing low-level
> tasks, so you're not prevented from getting things done; however, almost
> all programs can be implemented without relying on unsafe, and doing so
> eliminates what has probably been the two worst classes of bugs in the
> history of computing.
>
> Because this is done at the type system level, it has no cost at
> runtime.  This theme of "zero-cost abstractions" runs throughout the
> language, which makes it suitable as a replacement for C in the long
> term.  It also incorporates features befitting a modern programming
> language such as parameterized types (aka generics), pattern-matching,
> and trait-based polymorphism.  Finally, it can interface with C/C++
> binaries and libraries relatively easily, making it a suitable systems
> implementation language.
>
> Because of these benefits, several projects (Mozilla and Tor) have begun
> to reimplement their systems using Rust.  Additionally, there is a
> master's thesis about writing FreeBSD device drivers using Rust out
> there somewhere (forgive me for not chasing it down).
>
> A secondary benefit is that Rust's packaging system, cargo, is built
> around a more modern concept of modularity.  This is in line with
> FreeBSD's goals of modularizing the base system and reducing redundancy.
>
> = Downsides =
>
> There are a couple of downsides, which I'll outline briefly, and provide
> some counterargumentation for them.
>
> * It's a big component
>
> Adding the entire Rust platform would add more to the buildworld time,
> which is already quite long due to LLVM.  OTOH, LLVM itself is a big
> chunk of the rust build time, and we already build that as part of the
> base system now.  Moreover, it stands to reason that the compiler would
> end up being the majority of an open-source operating system's build
> time, with second place being the kernel itself.  Compilers are an
> active area of research with no signs of slowing down, and more hardware
> platforms means more backends, which means the compiler ends up growing
> over time, while most of the rest of the system doesn't.
>
> * The language is still evolving
>
> While this is true, the language has gotten *way* more mature in the
> past year and a half, and I think it's ready to hold up major codebases
> and production use, or at the very least it will be by the time we'd be
> seriously looking at bringing it in.
>
> * Rust binaries are large
>
> This issue can be avoided with the proper use of build flags.  The
> default settings use static linking for everything, and bring in the
> entire runtime library (which isn't necessary in most cases).  See this
> article:
> https://lifthrasiir.github.io/rustlog/why-is-a-rust-executable-large.html.
> With LTO, the size can be cut down to that of a C++ binary, and with
> dynamic linking, the executable size is comparable to a C program.  If
> we were seriously using rust, we would probably change the default flags
> to use dynamic linking/LTO by default, in fact.
>
>
> = Alternatives =
>
> There have been several alternatives that come up when I mention this.
> I'll discuss the two big ones: go and C++
>
> == C++ ==
>
> C++ is already in the base system, of course.  It could be brought into
> the kernel with some effort and restrictions.  The biggest argument for
> Rust over C++ in my view is that Rust is a much simpler, cleaner
> language.  C++ at this point is bewilderingly complex as a language, and
> has so many bells and whistles that it almost requires an effort to
> decide which subset of the language will be used.  This also tends to
> slow down efforts to expand the language over time.
>
> Rust by comparison is much leaner.  Its lifetime type system can be
> intimidating, but the benefits are well worth getting used to it.  To
> sum it up, C++ certainly isn't going anywhere, but I think Rust has a
> great deal more vitality as a language.
>
> == Go ==
>
> Go is not really a "systems language" in the sense that C or Rust are.
> It isn't a replacement for C so much as a replacement for python.
>
> What's more, it is almost universally disliked by people with a
> background in programming languages (I am one of those).  There's a
> couple of reasons for this.  First, it omits a number of features that
> modern languages *really* should include, parameterized types (generics)
> being one of the most glaring.  Beyond that, it completely ignores
> anything produced by modern PL research (unlike Rust, which benefits
> from that work).  Second, one of its main design parameters was "it
> should compile fast".  This as a design constraint basically guarantees
> that you can't do anything interesting in compilation, which rules out
> both advanced type systems as well as serious optimization.
>
> Unlike Rust, which offers substantial security and stability benefits
> through its type system, Go would do little more than just change the
> syntax of a bunch of code.  Moreover, it's likely that there would be
> performance consequences, given the design constraint of fast compilation.
>
> = Summary =
>
> Rust is a modern systems language with substantial benefits, and is
> well-poised to become the heir apparent to C as the dominant systems
> language going into the future.  For this reason, I think we should
> seriously consider it as a language for the base system at some point in
> the future, when the time is right and after sufficient exploration and
> experimentation.
>
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

RE: Speculative: Rust for base system components

Cy Schubert-4
In reply to this post by Eric McCorkle-2
If it requires rust it s/b in ports anyway.

---
Sent using a tiny phone keyboard.
Apologies for any typos and autocorrect.
Also, this old phone only supports top post. Apologies.

Cy Schubert
<[hidden email]> or <[hidden email]>
The need of the many outweighs the greed of the few.
---

-----Original Message-----
From: Warner Losh
Sent: 31/12/2018 07:48
To: Eric McCorkle
Cc: [hidden email]
Subject: Re: Speculative: Rust for base system components

On Mon, Dec 31, 2018 at 8:02 AM Warner Losh <[hidden email]> wrote:

>
>
> On Sun, Dec 30, 2018 at 10:41 PM Eric McCorkle <[hidden email]>
> wrote:
>
>> Before I begin, I want to be clear that everything here is in the realm
>> of speculative, long-term discussion.  My goal is to start a
>> conversation, not to propose anything concrete right now.
>>
>
> Today, this is a losing bid. The cost for rust is high (both in terms of
> people and added compile time), it's not well supported on all our
> architectures (and its robustness on the ones it does support has only been
> tested in limited scenarios), and there's 0 software it enables from day
> one. Plus, since it's a fast evolving language, we'll still need the ports
> to support those things that use it today since the likelihood of a version
> mismatch is high (and supporting 1 version would be a big stretch, multiple
> version is right out). So any sane cost / benefit analysis says: way more
> cost than benefit, forget about it. We simply don't have the man-power to
> maintain a high-cost, zero-benefit component in the tree. Lord knows we
> have a lot of non-zero-cost-with-almost-zero-benefit things in the tree
> today that we need to get rid of.
>
> In the future, when there are actual replacement things written, or there
> are new features written, that will shift the cost / benefit equation. And
> the circumstances about what makes up base will also have shifted, if we're
> lucky, and we'll be able to have a conversation. We imported perl and tcl
> on the speculative notion that people would build great things. That never
> really panned out, and they became a high-cost burden to keep modern for
> only minor benefit. And version skew in Perl was terrible by the end. Forth
> and Lua live in the tree because they have benefit (though Forth will be
> departing, most likely by 13, and definitely by 14). They are also small
> and easy to update to new versions.
>
> And we can't say, with certainty, that if a bunch of rust things show up
> we'll use them in base. We'll have to see what they provide to benefit the
> system.
>
> TBH, there's a stronger case for python than rust: there's actual python
> scripts in the tree today that we have to install a port to use. And there
> the benefit, while not zero, is small and the effort is large compared to
> just dragging it in as a port, so it hasn't been done. It's another fast
> evolving language that requires multiple versions as well...
>
> So write something that everybody wants, that must be in base, and that
> requires rust, and then we can have the conversation...
>

Just re-read this. It sounds a little more negative than I wanted to come
off. I'd only wanted to say today the needle is on 'nope' and I hope people
write enough cool stuff to justify moving the needle off 'nope' :) The last
part of that message seems more muted than I wanted.

Warner
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Bakul Shah
In reply to this post by Steffen Nurpmeso-2
On Mon, 31 Dec 2018 21:14:23 +0100 Steffen Nurpmeso <[hidden email]> wrote:

> Rozhuk Ivan wrote in <20181231221030.6471937e@rimwks>:
>  |On Mon, 31 Dec 2018 17:07:38 +0100
>  |Volker Lendecke <[hidden email]> wrote:
>  |
>  |> Not being involved in FreeBSD but in Samba, I could see a migration of
>  |> parts of Samba to Rust if it had a Rust->C translator. I think for
>  |> FreeBSD base components similar arguments would apply. You could ship
>  |> the C output for the platforms that don't have native Rust.
>  |
>  |Rust is modern geeks toy by mozilla.
>  |Just waste ot time, without any real profits.
>
> I vote for Nim (former Nimrod is no more).

So do I but it doesn't make sense to add any new language to
the base system without having a number of absolutely
essential programs written in that language that must also be
included in the base system. Even at that point, using any
such language for writing kernel code is madness. There are
millions of lines of C kernel code which are not going to
benefit in any way from the new language. Remember that except
for drivers the kernel is essentially a single binary. GC
becomes much less of an issue for user programs.

I'd rather see a much smaller base, with pkg install for
various typical user types.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
12345 ... 7