Speculative: Rust for base system components

classic Classic list List threaded Threaded
132 messages Options
1234567
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Peter Jeremy-8
On 2019-Jan-01 00:53:48 +0000, Igor Mozolevsky <[hidden email]> wrote:
>Quite frankly the compile time isn't really *that* important,

I disagree.  FreeBSD seems to be heading back to the batch days - you
submit your buildworld request and come back tomorrow to see if it worked.
That is a significant hinderance to development and, based on various
mailing list comments, is causing breakage because developers are cutting
corners due to the pain involved in running "make universe" before they
make a large change.

>longer (if not much longer) build times might push toward a better
>modularisation and compartmentalisation of the OS and the kernel so a
>small change in the kernel, for example, doesn't require the
>recompilation of the whole damn thing when nothing else is affected.

Two problems here:
1) make doesn't seem to be sufficient to correctly describe a dependency
tree to allow incremental/partial builds (at, everyone I'm aware of who
has a successful incremental build system has started by migrating off
make).  This means that a significant part of the work will be re-writing
the FreeBSD build system into something else like (eg) Bazel.
2) The bottleneck very quickly turns into the linker.

--
Peter Jeremy

signature.asc (981 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Brian Neal
In reply to this post by Eric McCorkle-2
This is a good read on some of the pitfalls of Rust:

https://hackernoon.com/why-im-dropping-rust-fd1c32986c88

In terms of performance, take a look at Rust on godbolt...a quick test I
just ran with a trivial program that sums up a sequence of odd numbers
is 29 instructions in C on clang 7.0 and a whopping 137 (!!!) on rustc
1.31.0!

Crazy...

On 12/30/2018 6:18 PM, Eric McCorkle wrote:

> Before I begin, I want to be clear that everything here is in the realm
> of speculative, long-term discussion.  My goal is to start a
> conversation, not to propose anything concrete right now.
>
>
>
> I've talked at several conferences about the possibility of bringing the
> Rust programming language into the base system, with the intent of
> making it a viable implementation language for parts of the base system.
>   I believe the potential security and stability benefits of doing this
> are *substantial*.  This would obviously be a serious undertaking, but I
> think the benefits are worth the cost.  For the rest of this mail, I'll
> outline my position and present the important facts that support it.
>
> I of course welcome any discussion, criticism, or input.
>
> = Why Rust? =
>
> Rust is a true systems programming language that incorporates results
> from programming language research that took place in the late 90's and
> 2000's, and can be viewed as a culmination of this work.  (Specifically,
> it draws on work on region types as well as substructural types, but
> that's its own survey article).
>
> This manifests most clearly in Rust's lifetime type system, which is
> able to perform quite sophisticated reasoning about memory allocation
> and ownership.  Rust programs do not require you to explicitly manage
> memory, but neither do they use garbage collection.  Memory management
> is automatically tracked by the lifetime types and allocations/frees are
> added during compilation.  Moreover, threaded programs are able to use
> the same type system features to track ownership.  This effectively
> prevents a large class of memory- and concurrency-related errors *at the
> type system level*, resulting in dramatically safer programs.  Rust
> provides an "unsafe" language facility for accomplishing low-level
> tasks, so you're not prevented from getting things done; however, almost
> all programs can be implemented without relying on unsafe, and doing so
> eliminates what has probably been the two worst classes of bugs in the
> history of computing.
>
> Because this is done at the type system level, it has no cost at
> runtime.  This theme of "zero-cost abstractions" runs throughout the
> language, which makes it suitable as a replacement for C in the long
> term.  It also incorporates features befitting a modern programming
> language such as parameterized types (aka generics), pattern-matching,
> and trait-based polymorphism.  Finally, it can interface with C/C++
> binaries and libraries relatively easily, making it a suitable systems
> implementation language.
>
> Because of these benefits, several projects (Mozilla and Tor) have begun
> to reimplement their systems using Rust.  Additionally, there is a
> master's thesis about writing FreeBSD device drivers using Rust out
> there somewhere (forgive me for not chasing it down).
>
> A secondary benefit is that Rust's packaging system, cargo, is built
> around a more modern concept of modularity.  This is in line with
> FreeBSD's goals of modularizing the base system and reducing redundancy.
>
> = Downsides =
>
> There are a couple of downsides, which I'll outline briefly, and provide
> some counterargumentation for them.
>
> * It's a big component
>
> Adding the entire Rust platform would add more to the buildworld time,
> which is already quite long due to LLVM.  OTOH, LLVM itself is a big
> chunk of the rust build time, and we already build that as part of the
> base system now.  Moreover, it stands to reason that the compiler would
> end up being the majority of an open-source operating system's build
> time, with second place being the kernel itself.  Compilers are an
> active area of research with no signs of slowing down, and more hardware
> platforms means more backends, which means the compiler ends up growing
> over time, while most of the rest of the system doesn't.
>
> * The language is still evolving
>
> While this is true, the language has gotten *way* more mature in the
> past year and a half, and I think it's ready to hold up major codebases
> and production use, or at the very least it will be by the time we'd be
> seriously looking at bringing it in.
>
> * Rust binaries are large
>
> This issue can be avoided with the proper use of build flags.  The
> default settings use static linking for everything, and bring in the
> entire runtime library (which isn't necessary in most cases).  See this
> article:
> https://lifthrasiir.github.io/rustlog/why-is-a-rust-executable-large.html.
>   With LTO, the size can be cut down to that of a C++ binary, and with
> dynamic linking, the executable size is comparable to a C program.  If
> we were seriously using rust, we would probably change the default flags
> to use dynamic linking/LTO by default, in fact.
>
>
> = Alternatives =
>
> There have been several alternatives that come up when I mention this.
> I'll discuss the two big ones: go and C++
>
> == C++ ==
>
> C++ is already in the base system, of course.  It could be brought into
> the kernel with some effort and restrictions.  The biggest argument for
> Rust over C++ in my view is that Rust is a much simpler, cleaner
> language.  C++ at this point is bewilderingly complex as a language, and
> has so many bells and whistles that it almost requires an effort to
> decide which subset of the language will be used.  This also tends to
> slow down efforts to expand the language over time.
>
> Rust by comparison is much leaner.  Its lifetime type system can be
> intimidating, but the benefits are well worth getting used to it.  To
> sum it up, C++ certainly isn't going anywhere, but I think Rust has a
> great deal more vitality as a language.
>
> == Go ==
>
> Go is not really a "systems language" in the sense that C or Rust are.
> It isn't a replacement for C so much as a replacement for python.
>
> What's more, it is almost universally disliked by people with a
> background in programming languages (I am one of those).  There's a
> couple of reasons for this.  First, it omits a number of features that
> modern languages *really* should include, parameterized types (generics)
> being one of the most glaring.  Beyond that, it completely ignores
> anything produced by modern PL research (unlike Rust, which benefits
> from that work).  Second, one of its main design parameters was "it
> should compile fast".  This as a design constraint basically guarantees
> that you can't do anything interesting in compilation, which rules out
> both advanced type systems as well as serious optimization.
>
> Unlike Rust, which offers substantial security and stability benefits
> through its type system, Go would do little more than just change the
> syntax of a bunch of code.  Moreover, it's likely that there would be
> performance consequences, given the design constraint of fast compilation.
>
> = Summary =
>
> Rust is a modern systems language with substantial benefits, and is
> well-poised to become the heir apparent to C as the dominant systems
> language going into the future.  For this reason, I think we should
> seriously consider it as a language for the base system at some point in
> the future, when the time is right and after sufficient exploration and
> experimentation.
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Igor Mozolevsky-2
In reply to this post by Peter Jeremy-8
On Tue, 1 Jan 2019 at 02:46, Peter Jeremy wrote:
>
> On 2019-Jan-01 00:53:48 +0000, Igor Mozolevsky <[hidden email]> wrote:
> >Quite frankly the compile time isn't really *that* important,
>
> I disagree.  FreeBSD seems to be heading back to the batch days - you
> submit your buildworld request and come back tomorrow to see if it worked.

<snip>

In my book, that makes the submitter think about what they write, and
not mindlessly write code hoping it would automagically compile.
Throwing some code at a compiler to "see if it worked" is the
equivalent of a builder throwing some bricks together to see if they
would hold up as a house... I hope you'd expect far better from the
people who build the place where you live, right, so why do not apply
a similarly rigorous process to coding? What you're blaming is the
messy architecture and the unnecessary re-parsing of the same files
over and over, but that's a different problem to a slow code
generation _after_ the source had been parsed, AFAIK any compiler's
bottleneck is lexing and parsing [1].

1. Joel on Software p13, AJ Spolsky, Apress, NYC (too late at night to
do a proper reference)

--
Igor M.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Garance A Drosehn-2
In reply to this post by Brian Neal
On 31 Dec 2018, at 23:04, Brian Neal wrote:

> This is a good read on some of the pitfalls of Rust:
>
> https://hackernoon.com/why-im-dropping-rust-fd1c32986c88
>
> In terms of performance, take a look at Rust on godbolt...a quick test
> I just ran with a trivial program that sums up a sequence of odd
> numbers is 29 instructions in C on clang 7.0 and a whopping 137 (!!!)
> on rustc 1.31.0!
>
> Crazy...

Did you compile your program as a "release build", or just as a
standard (quick) build?

  --
Garance Alistair Drosehn                =     [hidden email]
Senior Systems Programmer               or   [hidden email]
Rensselaer Polytechnic Institute;             Troy, NY;  USA
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Garance A Drosehn-2
In reply to this post by Igor Mozolevsky-2
On 31 Dec 2018, at 19:53, Igor Mozolevsky wrote:

> On Tue, 1 Jan 2019 at 00:30, Garance A Drosehn wrote:
>
>> There's one program written in rust which I use a lot, which is
>> called 'ripgrep' (or 'rg' for short).  It's one of the few programs
>> which I install via pre-compiled pkgs, because I don't want to wait
>> for all the time it takes to compile a release-build.  But I'm very
>> happy with how fast the program runs.
>
> And what metric is that against? I'm very happy with the speed jQuery
> is executed in my browser, but that's no reasonable metric to argue
> that JavaScript is a suitable language for the kernel...

I mentioned that program so each person could test it for themselves,
using whatever metric that they are comfortable with.

It is a self-contained alternative for 'grep', so you can test this
one program without pulling in a bunch of other dependencies (as
long as you install the pre-compiled package).  And it has some nice
features which people might want to play with.

So if someone *wanted* to see a useful program which happens to be
written in rust, then this might be a good one to try.  And it seems
to me that 'grep' is a good example of the kind of base-system
component which could be rewritten in rust.  Note that 'grep' is
not part of the kernel.  I'm sure we'd need to have a *lot* more
experience with rust (like, many years of experience) before we'd
talk about rewriting the kernel in rust.  (or in any other language).

In case it isn't obvious, my position is that we need more developers
who have real-world experience writing real programs in rust before
we add it to the base system.  IMO, we should not add rust and then
"hope" someone will find a glorious use for it.  However, I also think
that rust is interesting enough that it would be good if some of us
could spend some time (out of our copious spare time) to gain that
experience.

And a happy new year to all, with or without rust.

--
Garance Alistair Drosehn                =     [hidden email]
Senior Systems Programmer               or   [hidden email]
Rensselaer Polytechnic Institute;             Troy, NY;  USA
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Igor Mozolevsky-2
On Tue, 1 Jan 2019 at 03:40, Garance A Drosehn wrote:

>
> On 31 Dec 2018, at 19:53, Igor Mozolevsky wrote:
>
> On Tue, 1 Jan 2019 at 00:30, Garance A Drosehn wrote:
>
> There's one program written in rust which I use a lot, which is
> called 'ripgrep' (or 'rg' for short). It's one of the few programs
> which I install via pre-compiled pkgs, because I don't want to wait
> for all the time it takes to compile a release-build. But I'm very
> happy with how fast the program runs.
>
> And what metric is that against? I'm very happy with the speed jQuery
> is executed in my browser, but that's no reasonable metric to argue
> that JavaScript is a suitable language for the kernel...
>
> I mentioned that program so each person could test it for themselves,
> using whatever metric that they are comfortable with.


<snip>


I think you missed the point I was trying to make: you're comparing
apples and oranges---sure, they are (for most part) round, but they're
different! What you need is the implementation of the same algorithm
in both C and Rust and then we can talk...


Happy New Year!

--
Igor M.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

RE: Speculative: Rust for base system components

Cy Schubert-4
In reply to this post by Eric McCorkle-2
What would having another language in base buy us? This reminds me of a couple of months ago at OpenHack Victoria someone was trying to convince me that the kernel needed a JavaVM. (Sure we each had a few beers) but the similarity of this discussion doesn't escape me. Kernel modules and functions written in java^H^H^H^H rust: why?

---
Sent using a tiny phone keyboard.
Apologies for any typos and autocorrect.
Also, this old phone only supports top post. Apologies.

Cy Schubert
<[hidden email]> or <[hidden email]>
The need of the many outweighs the greed of the few.
---

-----Original Message-----
From: Peter Jeremy
Sent: 31/12/2018 18:48
To: Igor Mozolevsky
Cc: Hackers freeBSD
Subject: Re: Speculative: Rust for base system components

On 2019-Jan-01 00:53:48 +0000, Igor Mozolevsky <[hidden email]> wrote:
>Quite frankly the compile time isn't really *that* important,

I disagree.  FreeBSD seems to be heading back to the batch days - you
submit your buildworld request and come back tomorrow to see if it worked.
That is a significant hinderance to development and, based on various
mailing list comments, is causing breakage because developers are cutting
corners due to the pain involved in running "make universe" before they
make a large change.

>longer (if not much longer) build times might push toward a better
>modularisation and compartmentalisation of the OS and the kernel so a
>small change in the kernel, for example, doesn't require the
>recompilation of the whole damn thing when nothing else is affected.

Two problems here:
1) make doesn't seem to be sufficient to correctly describe a dependency
tree to allow incremental/partial builds (at, everyone I'm aware of who
has a successful incremental build system has started by migrating off
make).  This means that a significant part of the work will be re-writing
the FreeBSD build system into something else like (eg) Bazel.
2) The bottleneck very quickly turns into the linker.

--
Peter Jeremy
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Damjan Jovanovic
Kernel modules in Rust might benefit from the better type checking and thus
less memory leaks, less memory corruption, etc. but they are usually small
anyway so it's probably not worth it.

The bigger benefit of running Java applications (not device drivers) in the
kernel is performance. If applications run in kernel mode, context
switching between kernel threads is much faster than user threads, system
calls just become function calls, etc. Unlike C, Java can be sandboxed (at
least in theory), and lack of native pointers and memory safety ensure it
cannot access hardware directly anyway, so the rest of the system is still
protected. Around 2005, Microsoft Research made Singularity OS, which used
such techniques to run safely C# applications in the kernel, and it was
about 20% faster than C in user space.

On Tue, Jan 1, 2019 at 6:59 AM Cy Schubert <[hidden email]>
wrote:

> What would having another language in base buy us? This reminds me of a
> couple of months ago at OpenHack Victoria someone was trying to convince me
> that the kernel needed a JavaVM. (Sure we each had a few beers) but the
> similarity of this discussion doesn't escape me. Kernel modules and
> functions written in java^H^H^H^H rust: why?
>
> ---
> Sent using a tiny phone keyboard.
> Apologies for any typos and autocorrect.
> Also, this old phone only supports top post. Apologies.
>
> Cy Schubert
> <[hidden email]> or <[hidden email]>
> The need of the many outweighs the greed of the few.
> ---
>
> -----Original Message-----
> From: Peter Jeremy
> Sent: 31/12/2018 18:48
> To: Igor Mozolevsky
> Cc: Hackers freeBSD
> Subject: Re: Speculative: Rust for base system components
>
> On 2019-Jan-01 00:53:48 +0000, Igor Mozolevsky <[hidden email]>
> wrote:
> >Quite frankly the compile time isn't really *that* important,
>
> I disagree.  FreeBSD seems to be heading back to the batch days - you
> submit your buildworld request and come back tomorrow to see if it worked.
> That is a significant hinderance to development and, based on various
> mailing list comments, is causing breakage because developers are cutting
> corners due to the pain involved in running "make universe" before they
> make a large change.
>
> >longer (if not much longer) build times might push toward a better
> >modularisation and compartmentalisation of the OS and the kernel so a
> >small change in the kernel, for example, doesn't require the
> >recompilation of the whole damn thing when nothing else is affected.
>
> Two problems here:
> 1) make doesn't seem to be sufficient to correctly describe a dependency
> tree to allow incremental/partial builds (at, everyone I'm aware of who
> has a successful incremental build system has started by migrating off
> make).  This means that a significant part of the work will be re-writing
> the FreeBSD build system into something else like (eg) Bazel.
> 2) The bottleneck very quickly turns into the linker.
>
> --
> Peter Jeremy
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "[hidden email]"
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Brian Neal
In reply to this post by Garance A Drosehn-2
It was a debug build with no optimization for either compiler. But we
can easily run a variety of settings for comparison:

Compiler              Flags                 Inst. Count Build Time
======================================================================
clang 7.0.0           none                           33 296ms
                       -O3                            23 341ms
rustc 1.31.0          none                          110 606ms
                       -C opt-level=3                 67 643ms
gcc 8.2               none                           37 211ms
                       -O2                            24 249ms
                       -O3                          119* 206ms

* With -O3, gcc unrolled and vectorized the loop. The other compilers
did not emit vectorized code at any of the standard optimization levels.

So, essentially, double the build time and ~3 times the code for the
same logic.

On 12/31/2018 7:23 PM, Garance A Drosehn wrote:

>
> On 31 Dec 2018, at 23:04, Brian Neal wrote:
>
>     This is a good read on some of the pitfalls of Rust:
>
>     https://hackernoon.com/why-im-dropping-rust-fd1c32986c88
>
>     In terms of performance, take a look at Rust on godbolt...a quick
>     test I just ran with a trivial program that sums up a sequence of
>     odd numbers is 29 instructions in C on clang 7.0 and a whopping
>     137 (!!!) on rustc 1.31.0!
>
>     Crazy...
>
> Did you compile your program as a "release build", or just as a
> standard (quick) build?
>
> --
> Garance Alistair Drosehn = [hidden email]
> Senior Systems Programmer or [hidden email]
> Rensselaer Polytechnic Institute; Troy, NY; USA
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Cy Schubert-4
In reply to this post by Eric McCorkle-2
In message <CAJm2B-=o8xtCDcN38mcwDTToxS8zw6twyfgQQEWqdj3eLwcB=[hidden email]
il.com>
, Damjan Jovanovic writes:

> --000000000000caa538057e5f7945
> Content-Type: text/plain; charset="UTF-8"
>
> Kernel modules in Rust might benefit from the better type checking and thus
> less memory leaks, less memory corruption, etc. but they are usually small
> anyway so it's probably not worth it.
>
> The bigger benefit of running Java applications (not device drivers) in the
> kernel is performance. If applications run in kernel mode, context
> switching between kernel threads is much faster than user threads, system
> calls just become function calls, etc. Unlike C, Java can be sandboxed (at
> least in theory), and lack of native pointers and memory safety ensure it
> cannot access hardware directly anyway, so the rest of the system is still
> protected. Around 2005, Microsoft Research made Singularity OS, which used
> such techniques to run safely C# applications in the kernel, and it was
> about 20% faster than C in user space.

Microsoft renders fonts in the kernel too. Look at the (security)
trouble that that's caused them.

C++ exception handling (which can be "muted" and not recommended) isn't
best practice for kernel programming. Are you sure you want C++, C#,
Java, Javascript (also discussed at OpenHack), or rust in the kernel.
Really?

Not being rhetorical here but Microsoft isn't a great a example of we
should strive for.

I'm with Warner. No. Rust should remain in ports. Apps that depend on
rust should be in ports. Optional kernel modules should also be in
ports.


--
Cheers,
Cy Schubert <[hidden email]>
FreeBSD UNIX:  <[hidden email]>   Web:  http://www.FreeBSD.org

        The need of the many outweighs the greed of the few.


> On Tue, Jan 1, 2019 at 6:59 AM Cy Schubert <[hidden email]>
> wrote:
>
> > What would having another language in base buy us? This reminds me of a
> > couple of months ago at OpenHack Victoria someone was trying to convince me
> > that the kernel needed a JavaVM. (Sure we each had a few beers) but the
> > similarity of this discussion doesn't escape me. Kernel modules and
> > functions written in java^H^H^H^H rust: why?
> >
> > ---
> > Sent using a tiny phone keyboard.
> > Apologies for any typos and autocorrect.
> > Also, this old phone only supports top post. Apologies.
> >
> > Cy Schubert
> > <[hidden email]> or <[hidden email]>
> > The need of the many outweighs the greed of the few.
> > ---
> >
> > -----Original Message-----
> > From: Peter Jeremy
> > Sent: 31/12/2018 18:48
> > To: Igor Mozolevsky
> > Cc: Hackers freeBSD
> > Subject: Re: Speculative: Rust for base system components
> >
> > On 2019-Jan-01 00:53:48 +0000, Igor Mozolevsky <[hidden email]>
> > wrote:
> > >Quite frankly the compile time isn't really *that* important,
> >
> > I disagree.  FreeBSD seems to be heading back to the batch days - you
> > submit your buildworld request and come back tomorrow to see if it worked.
> > That is a significant hinderance to development and, based on various
> > mailing list comments, is causing breakage because developers are cutting
> > corners due to the pain involved in running "make universe" before they
> > make a large change.
> >
> > >longer (if not much longer) build times might push toward a better
> > >modularisation and compartmentalisation of the OS and the kernel so a
> > >small change in the kernel, for example, doesn't require the
> > >recompilation of the whole damn thing when nothing else is affected.
> >
> > Two problems here:
> > 1) make doesn't seem to be sufficient to correctly describe a dependency
> > tree to allow incremental/partial builds (at, everyone I'm aware of who
> > has a successful incremental build system has started by migrating off
> > make).  This means that a significant part of the work will be re-writing
> > the FreeBSD build system into something else like (eg) Bazel.
> > 2) The bottleneck very quickly turns into the linker.
> >
> > --
> > Peter Jeremy
> > _______________________________________________
> > [hidden email] mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> > To unsubscribe, send any mail to "[hidden email]"
> >
>
> --000000000000caa538057e5f7945
> Content-Type: text/html; charset="UTF-8"
> Content-Transfer-Encoding: quoted-printable
>
> <div dir=3D"ltr"><div>Kernel modules in Rust might benefit from the better =
> type checking and thus less memory leaks, less memory corruption, etc. but =
> they are usually small anyway so it&#39;s probably not worth it.<br></div><=
> div><br></div><div>The bigger benefit of running Java applications (not dev=
> ice drivers) in the kernel is performance. If applications run in kernel mo=
> de, context switching between kernel threads is much faster than user threa=
> ds, system calls just become function calls, etc. Unlike C, Java can be san=
> dboxed (at least in theory), and lack of native pointers and memory safety =
> ensure it cannot access hardware directly anyway, so the rest of the system=
>  is still protected. Around 2005, Microsoft Research made Singularity OS, w=
> hich used such techniques to run safely C# applications in the kernel, and =
> it was about 20% faster than C in user space.<br></div><br><div class=3D"gm=
> ail_quote"><div dir=3D"ltr">On Tue, Jan 1, 2019 at 6:59 AM Cy Schubert &lt;=
> <a href=3D"mailto:[hidden email]">[hidden email]</a>&=
> gt; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0=
> px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">What =
> would having another language in base buy us? This reminds me of a couple o=
> f months ago at OpenHack Victoria someone was trying to convince me that th=
> e kernel needed a JavaVM. (Sure we each had a few beers) but the similarity=
>  of this discussion doesn&#39;t escape me. Kernel modules and functions wri=
> tten in java^H^H^H^H rust: why?<br>
> <br>
> ---<br>
> Sent using a tiny phone keyboard.<br>
> Apologies for any typos and autocorrect.<br>
> Also, this old phone only supports top post. Apologies.<br>
> <br>
> Cy Schubert<br>
> &lt;<a href=3D"mailto:[hidden email]" target=3D"_blank">Cy.Schub=
> [hidden email]</a>&gt; or &lt;<a href=3D"mailto:[hidden email]" target=
> =3D"_blank">[hidden email]</a>&gt;<br>
> The need of the many outweighs the greed of the few.<br>
> ---<br>
> <br>
> -----Original Message-----<br>
> From: Peter Jeremy<br>
> Sent: 31/12/2018 18:48<br>
> To: Igor Mozolevsky<br>
> Cc: Hackers freeBSD<br>
> Subject: Re: Speculative: Rust for base system components<br>
> <br>
> On 2019-Jan-01 00:53:48 +0000, Igor Mozolevsky &lt;<a href=3D"mailto:igor@h=
> ybrid-lab.co.uk" target=3D"_blank">[hidden email]</a>&gt; wrote:<br>
> &gt;Quite frankly the compile time isn&#39;t really *that* important,<br>
> <br>
> I disagree.=C2=A0 FreeBSD seems to be heading back to the batch days - you<=
> br>
> submit your buildworld request and come back tomorrow to see if it worked.<=
> br>
> That is a significant hinderance to development and, based on various<br>
> mailing list comments, is causing breakage because developers are cutting<b=
> r>
> corners due to the pain involved in running &quot;make universe&quot; befor=
> e they<br>
> make a large change.<br>
> <br>
> &gt;longer (if not much longer) build times might push toward a better<br>
> &gt;modularisation and compartmentalisation of the OS and the kernel so a<b=
> r>
> &gt;small change in the kernel, for example, doesn&#39;t require the<br>
> &gt;recompilation of the whole damn thing when nothing else is affected.<br=
> >
> <br>
> Two problems here:<br>
> 1) make doesn&#39;t seem to be sufficient to correctly describe a dependenc=
> y<br>
> tree to allow incremental/partial builds (at, everyone I&#39;m aware of who=
> <br>
> has a successful incremental build system has started by migrating off<br>
> make).=C2=A0 This means that a significant part of the work will be re-writ=
> ing<br>
> the FreeBSD build system into something else like (eg) Bazel.<br>
> 2) The bottleneck very quickly turns into the linker.<br>
> <br>
> -- <br>
> Peter Jeremy<br>
> _______________________________________________<br>
> <a href=3D"mailto:[hidden email]" target=3D"_blank">freebsd-ha=
> [hidden email]</a> mailing list<br>
> <a href=3D"https://lists.freebsd.org/mailman/listinfo/freebsd-hackers" rel=
> =3D"noreferrer" target=3D"_blank">https://lists.freebsd.org/mailman/listinf=
> o/freebsd-hackers</a><br>
> To unsubscribe, send any mail to &quot;<a href=3D"mailto:freebsd-hackers-un=
> [hidden email]" target=3D"_blank">freebsd-hackers-unsubscribe@freebs=
> d.org</a>&quot;<br>
> </blockquote></div></div>
>
> --000000000000caa538057e5f7945--


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Poul-Henning Kamp
--------
In message <[hidden email]>, Cy Schubert writes:

>I'm with Warner. No. Rust should remain in ports. Apps that depend on
>rust should be in ports. Optional kernel modules should also be in
>ports.

And we should get pkgbase pushed across the finish-line and split the most
rarely used parts of src into ports as well.

--
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[hidden email]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Jonathan Anderson-2
In reply to this post by Peter Jeremy-8
On 31 Dec 2018, at 23:16, Peter Jeremy wrote:

> 1) make doesn't seem to be sufficient to correctly describe a
> dependency
> tree to allow incremental/partial builds (at, everyone I'm aware of
> who
> has a successful incremental build system has started by migrating off
> make).  This means that a significant part of the work will be
> re-writing
> the FreeBSD build system into something else like (eg) Bazel.

Or Fabrique? :)

https://fabriquer.github.io

That page is a bit out of date with respect to the language as it
currently stands, but the essentials (and the aspirations!) are still
the same.


Jon
--
Jonathan Anderson
[hidden email]
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Eric McCorkle-2
In reply to this post by Cy Schubert-4
On 12/31/18 11:56 PM, Cy Schubert wrote:
> What would having another language in base buy us? This reminds me of a couple of months ago at OpenHack Victoria someone was trying to convince me that the kernel needed a JavaVM. (Sure we each had a few beers) but the similarity of this discussion doesn't escape me. Kernel modules and functions written in java^H^H^H^H rust: why?

I don't think that's a fair comparison at all.  Rust is a systems
language built around zero-cost abstractions that is usable for
developing real embedded code.  Java is a completely different animal,
and there is no reasonable case for a Java VM in the kernel/loader.

I'm all for discussion and criticism of this, that's why I posted it,
but I don't think these kinds of false equivalences are helpful.


signature.asc (235 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Igor Mozolevsky-2
On Tue, 1 Jan 2019 at 15:54, Eric McCorkle wrote:

<snip>

&gt; I don't think that's a fair comparison at all.  Rust is a systems
&gt; language built around zero-cost abstractions that is usable for
&gt; developing real embedded code

<snip>

Brian's simple experiment [1] demonstrates that "zero-cost" is more of
an aspiration (and a very long term one, perhaps) than a hard fact ;-)

1. https://lists.freebsd.org/pipermail/freebsd-hackers/2019-January/053837.html
</snip></snip>

--
Igor M.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Steffen Nurpmeso-2
In reply to this post by Bakul Shah
Bakul Shah wrote in <[hidden email]>:
 |On Mon, 31 Dec 2018 21:14:23 +0100 Steffen Nurpmeso <[hidden email]> \
 |wrote:
 |> Rozhuk Ivan wrote in <20181231221030.6471937e@rimwks>:
 |>|On Mon, 31 Dec 2018 17:07:38 +0100
 |>|Volker Lendecke <[hidden email]> wrote:
 |>|
 |>|> Not being involved in FreeBSD but in Samba, I could see a migration of
 |>|> parts of Samba to Rust if it had a Rust->C translator. I think for
 |>|> FreeBSD base components similar arguments would apply. You could ship
 |>|> the C output for the platforms that don't have native Rust.
 |>|
 |>|Rust is modern geeks toy by mozilla.
 |>|Just waste ot time, without any real profits.
 |>
 |> I vote for Nim (former Nimrod is no more).
 |
 |So do I but it doesn't make sense to add any new language to
 |the base system without having a number of absolutely
 |essential programs written in that language that must also be
 |included in the base system. Even at that point, using any
 |such language for writing kernel code is madness. There are
 |millions of lines of C kernel code which are not going to
 |benefit in any way from the new language. Remember that except
 |for drivers the kernel is essentially a single binary. GC
 |becomes much less of an issue for user programs.

Of course not (imho), why not?, kernel-me? no!, and bloats of
undocumented sysctl knobs there too!, and a very large one!  Yes,
nim uses gc.  It is converted to native C[/C++/JavaScript] and
then compiled.  Oh - i for one would wish that C++ would be
nothing than a layer on top of the current ISO C standard (weird
enough), and could be dispatched in a C++-preprocessed form, so
that TinyCC, pcc and all the rest could be used to compile it.
(Not on FreeBSD of course, where those are broken, or were once
i last tried.)

All i was saying is that Julia and Nim are good languages.
I like the latter the most.  It is for real-life programmers.

 |I'd rather see a much smaller base, with pkg install for
 |various typical user types.

Things get damaged.  Things get broken.
And i am almost away.  (I have no Phabricator account.)
I shake my head often.  Unforgotten for example the claim
"multi-column layout seems to have never worked" from a core team
member when some old but great document by some "danish dynamite"
has been thrown away (to packages).  I seem to remember it that
way (sorry if false).

I am not an administrator, and i am almost not using perl anymore
for new things, there are not many of those, and if i try to go
sh -> awk -> perl and seldom have the needs to reach the end of
that pipeline.  But perl is a text processing language with full
Unicode support and no doubts is a helpful hand to many, take
email and you find git send-email (not me however), spam checking,
dkim and other such server side stuff.  Log parsing.  An admin
needs something, you hack it out of the box, from within the box.
Now as a package, i seemed to have understood the reasoning.
About this thread: i would have understood on April 1st.

Ciao,
require the last for the
things which i need to do



Aren't there 100000 packages already?

 --End of <[hidden email]>

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Julian H. Stacey-3
In reply to this post by Eric McCorkle-2
I'm against rust being imported into src/, wasting hard bought
space & build time. Src/ is still too big, (& obj/ monstrous compared
to old days (clang I spose, shrug)), even though we've suffered
much disruption, things being ripped out of src/ supposedly to save
[sometime little] space.  Let's not squander that hard bought space & time.

Cheers,
Julian
--
Julian Stacey, Computer Consultant Sys.Eng. BSD Linux Unix, Munich Aachen Kent
 First referendum stole 700,000 votes from Brits in EU;  3,700,000 globaly.
 Lies criminal funded; jobs pound & markets down. 1.9M new voters 1.3M dead.
 Email MP: "A new referendum will buy UK & EU more time (Art 50.3), to avoid
       a hard crash, & consider all options."  http://berklix.org/brexit/#mp
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Steffen Nurpmeso-2
Julian H. Stacey wrote in <[hidden email]>:
 |I'm against rust being imported into src/, wasting hard bought
 |space & build time. Src/ is still too big, (& obj/ monstrous compared
 |to old days (clang I spose, shrug)), even though we've suffered
 |much disruption, things being ripped out of src/ supposedly to save
 |[sometime little] space.  Let's not squander that hard bought space & time.

And not to forget the base system as such.
Just last week (or the end of the week before that) i have tried
to use the address sanitizer on a freshly installed 11.2 i386
FreeBSD.  No (result unbelievable).

Because of my very limited download budget i had to choose what
i could download next to get a working installation, and have
chosen to go for ArchLinux (even though i never had a workign
memory sanitizers there, but address sanitizer i have used there
many times) December/2018 and do the upgrade with
all the stuff (still one GB+).  I was right (in that it was not my
fault).

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Pedro Giffuni-4
In reply to this post by Eric McCorkle-2
> Before I begin, I want to be clear that everything here is in the realm
> of speculative, long-term discussion.  My goal is to start a
> conversation, not to propose anything concrete right now.
>
>
>
> I've talked at several conferences about the possibility of bringing the
> Rust programming language into the base system, with the intent of
> making it a viable implementation language for parts of the base system.
>   I believe the potential security and stability benefits of doing this
> are *substantial*.  This would obviously be a serious undertaking, but I
> think the benefits are worth the cost.  For the rest of this mail, I'll
> outline my position and present the important facts that support it.
>

FWIW, my take around this and similar proposals is ...

Just go ahead and fork it on github and see how far it gets. Nowadays it
is very easy to start a new project and explore the possibilities.

If it seems to have a real future eventually people will ask for it to
be merged, if it doesn't .. well at least you really explored the idea.
Perhaps you may even end up merging it with redux-os :).

Cheers,

Pedro.

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Garance A Drosehn-2
In reply to this post by Igor Mozolevsky-2
On 2019-01-01 11:00, Igor Mozolevsky wrote:

> On Tue, 1 Jan 2019 at 15:54, Eric McCorkle wrote:
>
> <snip>
>
> &gt; I don't think that's a fair comparison at all.  Rust is a systems
> &gt; language built around zero-cost abstractions that is usable for
> &gt; developing real embedded code
>
> <snip>
>
> Brian's simple experiment [1] demonstrates that "zero-cost" is
> more of an aspiration (and a very long term one, perhaps) than
> a hard fact ;-)

Brian's simple experiment is a simple experiment.  It's interesting,
but hardly the definitive word in evaluating a language.  We need
more real-world experience with serious programs before dismissing
rust.  But at the same time, we need much more real-world experience
with writing rust before we can consider bringing it into the base
system.  We can't bring it into the base system and then hope that
"magic happens" because it's sitting there.

I thought the 'ripgrep' program seemed like an interesting one
to look at.  Compare how fast it works to how fast our grep works.
(I have not done that comparison!  I just take advantage of the
extra features that 'ripgrep' has.)  And 'ripgrep' works fine as
a port.  It does not need to be in the base system.  It's just an
example to look at, for those who might be interested in rust.

Maybe other people know of other real-world useful programs which
are written in rust, and which might be interesting to look at.

So consider me a cheerleader for "Let's get some more experience!".
(not that all of us can do that, but at least *some* of us!)

     -- garance alistair drosehn  aka [hidden email]
     -- senior systems programmer @rpi

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Speculative: Rust for base system components

Alan Somers-2
On Tue, Jan 1, 2019 at 4:17 PM drosih <[hidden email]> wrote:

>
> On 2019-01-01 11:00, Igor Mozolevsky wrote:
> > On Tue, 1 Jan 2019 at 15:54, Eric McCorkle wrote:
> >
> > <snip>
> >
> > &gt; I don't think that's a fair comparison at all.  Rust is a systems
> > &gt; language built around zero-cost abstractions that is usable for
> > &gt; developing real embedded code
> >
> > <snip>
> >
> > Brian's simple experiment [1] demonstrates that "zero-cost" is
> > more of an aspiration (and a very long term one, perhaps) than
> > a hard fact ;-)
>
> Brian's simple experiment is a simple experiment.  It's interesting,
> but hardly the definitive word in evaluating a language.  We need
> more real-world experience with serious programs before dismissing
> rust.  But at the same time, we need much more real-world experience
> with writing rust before we can consider bringing it into the base
> system.  We can't bring it into the base system and then hope that
> "magic happens" because it's sitting there.
>
> I thought the 'ripgrep' program seemed like an interesting one
> to look at.  Compare how fast it works to how fast our grep works.
> (I have not done that comparison!  I just take advantage of the
> extra features that 'ripgrep' has.)  And 'ripgrep' works fine as
> a port.  It does not need to be in the base system.  It's just an
> example to look at, for those who might be interested in rust.
>
> Maybe other people know of other real-world useful programs which
> are written in rust, and which might be interesting to look at.

Freshmeat knows of 38.  Just look at the reverse dependencies for
rust: https://www.freshports.org/lang/rust/ .  Some interesting ones
are alacritty (a fast terminal emulator), exa (ls replacement), fd
(find replacement), suricata (IDS), xi (text editor), tokei (cloc
replacement), sccache (like ccache but networked), and of course our
favorite web browser.  Also notable is libjail-rs, a jail(3)
replacement written in Rust.  https://github.com/fubarnetes/libjail-rs
.

-Alan

>
> So consider me a cheerleader for "Let's get some more experience!".
> (not that all of us can do that, but at least *some* of us!)
>
>      -- garance alistair drosehn  aka [hidden email]
>      -- senior systems programmer @rpi
>
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "[hidden email]"
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
1234567