libthr shared locks

classic Classic list List threaded Threaded
46 messages Options
123
Reply | Threaded
Open this post in threaded view
|

libthr shared locks

Konstantin Belousov
A well-known limitation of the FreeBSD libthr implementation of the
pthread locking objects is the missed support for the process-shared
locks.  The hardest part of implementing the support is the neccessity
of providing ABI compatibility for the current implementation.

Right now, the ABI-visible handle to the locks is a single pointer
word.  As an example which allows to make the description less vague,
let consider pthread_mutex_t.  It is defined in
sys/sys/_pthreadtypes.h as
typedef struct pthread_mutex *pthread_mutex_t;
The pointer points to the following structure, after the
pthread_mutex_init(3) is called
struct pthread_mutex {
        struct umutex m_lock;
        int m_flags;
        struct pthread *m_owner;
        int m_count;
        int m_spinloops;
        int m_yieldloops;
        TAILQ_ENTRY(pthread_mutex) m_qe;
};
struct umutex {
        volatile __lwpid_t m_owner; /* Owner of the mutex */
        __uint32_t m_flags; /* Flags of the mutex */
        __uint32_t m_ceilings[2]; /* Priority protect ceiling */
        __uint32_t m_spare[4];
};

Would the ABI modified to make the pthread_mutex_t large enough to
hold struct pthread_mutex, the rest of the implementation of the
shared mutex is relatively trivial, if not already done.

Changing this ABI is very hard.  libthr provides the symbol
versioning, which allows to provide compatible shims for the previous
ABI variant.  But since userspace tends to use the pthread objects in
the layouts of the library objects, this causes serious ABI issues
when mixing libraries built against different default versions of
libthr.

My idea to provide the shared locks, while not changing the ABI for
libthr, is to use marker pointers to indicate the shared objects.  The
real struct pthread_mutex, which carries the locking information, is
allocated by at the off-page from some anonymous posix shared memory
object.

The marker is defined as
#define THR_PSHARED_PTR \
    ((void *)(uintptr_t)((1ULL << (NBBY * sizeof(long) - 1)) | 1))
The bit-pattern is 1000....00001.  There are two tricks used:
1. All correctly allocated objects in all supported ABIs are at least
   word-aligned, so the least-significant bit cannot be set.  This
   should made the THR_PSHARED_PTR pattern unique against non-shared
   allocations.
2. The high bit is set, which makes the address non-canonical on
   amd64, causing attempts to dereference the pointer guaranteed to
   segfault, instead of relying of not having the corresponding page
   not mapped on the weakly-aligned arches.

The majority of the libthr modifications follow the easy pattern where
the library must store the THR_PSHARED_PTR upon the initialization of
the shared objects, allocate the off-page and initialize the lock
there.  If a call assumes that the object is already initialized, then
the we must not instantiate the off-page.  To speed-up the lookup, a
cache is kept at the userspace which translates address of locks to
the off-page.  Note that we can safely ignore possible unmapping of
the locks, since correct pthread_* API use assumes the call to
pthread_*_destroy() on the end of the object lifecycle.  If the lock
is remapped in the usermode, then userspace off-page translation cache
fails, but kernel returns the same shm for lookup, and we end with two
off-page mappings, which is acceptable.

Kernel holds a lookup table which translates the (vm_object, offset)
pair, obtained by the dereference of the user-space address, into the
posix shared memory object.  The lifecycle of the shm objects is bound
to the existence of the corresponding vm object.

Note that lifecycle of the kernel objects does not correspond well to
the lifecycle of the vnode vm object.  Closed vnode could be recycled
by VFS for whatever reasons, and then we would loose the entry in the
registry.  I am not sure if this is very serious issue, since I
suppose that typical use case assumes the anonymous shared memory
backing.  Right now kernel drops the off-page shm object on the last
vnode unmap.

Due to backing by the kernel objects, the implementation imposes
per-uid limits on the amount of the shared objects created.  An issue
is that there are no such limits in other implementations.

Overhead of the implementation, comparing with the non-process shared
locks, is due to the mandatory off-page lookup, which is mostly
ammortized by the (read-locked) userspace cache.  Also, for each
shared lock we get an additional page of memory, which works fine
assuming the applications use limited amount of the shared locks.
Cost for the non-shared locks is a single memory load for each
pthread_* call.

Below are the timing results of my implementation on the 4-core sandy
against the Fedora 22 glibc, done with the same program on the same
hardware (https://www.kib.kiev.ua/kib/pshared/pthread_shared_mtx1.c).

[FreeBSD]
# time /root/pthread_shared_mtx1-64
iter1 10000000 aiter1 10000000 iter2 10000000 aiter2 10000000
./pthread_shared_mtx1-64  2.47s user 3.27s system 166% cpu 3.443 total

[Fedora]
[kostik@sandy tests]$ /usr/bin/time ./pthread_shared_mtx1-linux64
iter1 10000000 aiter1 10000000 iter2 10000000 aiter2 10000000
1.38user 2.46system 0:01.95elapsed 196%CPU (0avgtext+0avgdata 1576maxresident)k
0inputs+0outputs (0major+142minor)pagefaults 0swaps

The implementation in the patch
https://www.kib.kiev.ua/kib/pshared/pshared.1.patch
gives shared mutexes, condvars, rwlocks and barriers. I did some
smoke-testing, only on amd64. Not implementated are the robust mutexes.
I want to finalize this part of work before implementing robustness,
but some restructuring in the patch, which seems to be arbitrary, like
the normal/pp queues rework to live in arrays, is a preparation to the
robustness feature.

The work was sponsored by The FreeBSD Foundation, previous and current
versions of idea and previous patch were discussed with John Baldwin
and Ed Maste.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: libthr shared locks

Daniel Eischen
On Wed, 23 Dec 2015, Konstantin Belousov wrote:

> A well-known limitation of the FreeBSD libthr implementation of the
> pthread locking objects is the missed support for the process-shared
> locks.  The hardest part of implementing the support is the neccessity
> of providing ABI compatibility for the current implementation.
>
> Right now, the ABI-visible handle to the locks is a single pointer
> word.  As an example which allows to make the description less vague,
> let consider pthread_mutex_t.  It is defined in
> sys/sys/_pthreadtypes.h as
> typedef struct pthread_mutex *pthread_mutex_t;
> The pointer points to the following structure, after the
> pthread_mutex_init(3) is called
> struct pthread_mutex {
> struct umutex m_lock;
> int m_flags;
> struct pthread *m_owner;
> int m_count;
> int m_spinloops;
> int m_yieldloops;
> TAILQ_ENTRY(pthread_mutex) m_qe;
> };
> struct umutex {
> volatile __lwpid_t m_owner; /* Owner of the mutex */
> __uint32_t m_flags; /* Flags of the mutex */
> __uint32_t m_ceilings[2]; /* Priority protect ceiling */
> __uint32_t m_spare[4];
> };
>
> Would the ABI modified to make the pthread_mutex_t large enough to
> hold struct pthread_mutex, the rest of the implementation of the
> shared mutex is relatively trivial, if not already done.
>
> Changing this ABI is very hard.  libthr provides the symbol
> versioning, which allows to provide compatible shims for the previous
> ABI variant.  But since userspace tends to use the pthread objects in
> the layouts of the library objects, this causes serious ABI issues
> when mixing libraries built against different default versions of
> libthr.
>
> My idea to provide the shared locks, while not changing the ABI for
> libthr, is to use marker pointers to indicate the shared objects.  The
> real struct pthread_mutex, which carries the locking information, is
> allocated by at the off-page from some anonymous posix shared memory
> object.

I'd rather just change the pthread_foo lock types to include
the space.  I really don't like the way libc, rtld, etc have
to jump through hoops to initialize the locks.  If the lock
types included the storage, then they wouldn't have to.

My idea was to keep the first storage unit of the lock as
a pointer/self-reference, so that it is easy to keep the ABI.
You can set the 0 bit in the pointer to indicate whether the
lock was the old ABI, then just clear it before using it.
And we could hide old ABI implementation compilation behind
WITH_FOO to avoid overhead of checking (ptr & 0x1) != 0.
I think this is similar to what you've done below.

>
> The marker is defined as
> #define THR_PSHARED_PTR \
>    ((void *)(uintptr_t)((1ULL << (NBBY * sizeof(long) - 1)) | 1))
> The bit-pattern is 1000....00001.  There are two tricks used:

[ ... ]

I also don't like to get the kernel involved if it isn't
necessary.

--
DE
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: libthr shared locks

Daniel Eischen
In reply to this post by Konstantin Belousov
On Wed, 23 Dec 2015, Konstantin Belousov wrote:

[ ... ]

> Would the ABI modified to make the pthread_mutex_t large enough to
> hold struct pthread_mutex, the rest of the implementation of the
> shared mutex is relatively trivial, if not already done.
>
> Changing this ABI is very hard.  libthr provides the symbol
> versioning, which allows to provide compatible shims for the previous
> ABI variant.  But since userspace tends to use the pthread objects in
> the layouts of the library objects, this causes serious ABI issues
> when mixing libraries built against different default versions of
> libthr.

I think this is only if the libraries (or apps) pass pthread
lock types between them, that one has initialized with one ABI
but the other library uses another ABI.  We should really
exclude this as part of our ABI compatibility.

Mixing libraries built with different pthread ABIs should not
be a problem as long as they don't directly operate on the
other's pthread lock types.

There is also our ability to do a library version bump as a last
resort.

--
DE
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: libthr shared locks

Konstantin Belousov
On Wed, Dec 23, 2015 at 01:27:35PM -0500, Daniel Eischen wrote:

> On Wed, 23 Dec 2015, Konstantin Belousov wrote:
>
> [ ... ]
> > Would the ABI modified to make the pthread_mutex_t large enough to
> > hold struct pthread_mutex, the rest of the implementation of the
> > shared mutex is relatively trivial, if not already done.
> >
> > Changing this ABI is very hard.  libthr provides the symbol
> > versioning, which allows to provide compatible shims for the previous
> > ABI variant.  But since userspace tends to use the pthread objects in
> > the layouts of the library objects, this causes serious ABI issues
> > when mixing libraries built against different default versions of
> > libthr.
>
> I think this is only if the libraries (or apps) pass pthread
> lock types between them, that one has initialized with one ABI
> but the other library uses another ABI.  We should really
> exclude this as part of our ABI compatibility.
Applications commonly embed pthread locks into their objects, including
the exposed ABI in the third-party libraries.

struct my_object {
        pthread_mutex_t obj_lock;
        ...
};

Changing the size of the pthread locks causes uncontrolled breakage of
the ABI for significant number of third-party code.

The issue is similar to the effect of the attempted ino_t expansion
to the 64 bit, which changes the struct stat layout and then triggers
cascade of the ABI issues. With struct stat, most of it is localized in
the src/, which makes handling easier because all of the changed staff
is under our control.

For libpthread, the approach would cause *big* blow up for ports and for
software compiled locally, and we cannot do anything. It would become
the flag day, with subtle bugs all over there after the transition on a
machine where ABIs are mixed. This is not an issue for e.g. appliance
vendors, but IMO is unnaceptable for general-purpose OS.

And, why pay the cost of the flag day for the feature that was not present
up to today ?

>
> Mixing libraries built with different pthread ABIs should not
> be a problem as long as they don't directly operate on the
> other's pthread lock types.
>
> There is also our ability to do a library version bump as a last
> resort.

Bumping libthr version would be not enough. It is very much possible to
get both libthr.so.3 and libthr.so.4 into the same process. Versions
for the symbols must be adjusted to properly bind new and old ABI'
consumers.

I did evaluated this route, and I am between being very skeptical that
it is workable and completely denying the viability.  As I pointed out
earlier, this causes enormous amount of bugs due to the ABI change
in third-party, you cannot mix.  Clean recompilation for the new ABI
would solve it, but it is not acceptable for FreeBSD pretending to
support upgrades.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: libthr shared locks

Daniel Eischen
On Wed, 23 Dec 2015, Konstantin Belousov wrote:

> On Wed, Dec 23, 2015 at 01:27:35PM -0500, Daniel Eischen wrote:
>> On Wed, 23 Dec 2015, Konstantin Belousov wrote:
>>
>> [ ... ]
>>> Would the ABI modified to make the pthread_mutex_t large enough to
>>> hold struct pthread_mutex, the rest of the implementation of the
>>> shared mutex is relatively trivial, if not already done.
>>>
>>> Changing this ABI is very hard.  libthr provides the symbol
>>> versioning, which allows to provide compatible shims for the previous
>>> ABI variant.  But since userspace tends to use the pthread objects in
>>> the layouts of the library objects, this causes serious ABI issues
>>> when mixing libraries built against different default versions of
>>> libthr.
>>
>> I think this is only if the libraries (or apps) pass pthread
>> lock types between them, that one has initialized with one ABI
>> but the other library uses another ABI.  We should really
>> exclude this as part of our ABI compatibility.
> Applications commonly embed pthread locks into their objects, including
> the exposed ABI in the third-party libraries.
>
> struct my_object {
> pthread_mutex_t obj_lock;
> ...
> };
>
> Changing the size of the pthread locks causes uncontrolled breakage of
> the ABI for significant number of third-party code.

If the application creates the object itself or allocates storage
for it, basically, if it isn't opaque, yes.  But we can bump port
revisions for affected libraries (probably just searching
/usr/local/include/... for pthread_mutex, pthread_cond, etc
types to see possible problems.  I did a search for the installed
ports on my system and found a few that might cause problems.

I think we're just putting off the inevitable.  Do we not want
to change our pthread sync types anyway, to get rid of an extra
dereference per lock?  To get rid of the hacks in libc, rtld,
etc?

If the answer is no, we never want to do that, then ok.

--
DE
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: libthr shared locks

Konstantin Belousov
On Wed, Dec 23, 2015 at 02:48:56PM -0500, Daniel Eischen wrote:

> On Wed, 23 Dec 2015, Konstantin Belousov wrote:
>
> > On Wed, Dec 23, 2015 at 01:27:35PM -0500, Daniel Eischen wrote:
> >> On Wed, 23 Dec 2015, Konstantin Belousov wrote:
> >>
> >> [ ... ]
> >>> Would the ABI modified to make the pthread_mutex_t large enough to
> >>> hold struct pthread_mutex, the rest of the implementation of the
> >>> shared mutex is relatively trivial, if not already done.
> >>>
> >>> Changing this ABI is very hard.  libthr provides the symbol
> >>> versioning, which allows to provide compatible shims for the previous
> >>> ABI variant.  But since userspace tends to use the pthread objects in
> >>> the layouts of the library objects, this causes serious ABI issues
> >>> when mixing libraries built against different default versions of
> >>> libthr.
> >>
> >> I think this is only if the libraries (or apps) pass pthread
> >> lock types between them, that one has initialized with one ABI
> >> but the other library uses another ABI.  We should really
> >> exclude this as part of our ABI compatibility.
> > Applications commonly embed pthread locks into their objects, including
> > the exposed ABI in the third-party libraries.
> >
> > struct my_object {
> > pthread_mutex_t obj_lock;
> > ...
> > };
> >
> > Changing the size of the pthread locks causes uncontrolled breakage of
> > the ABI for significant number of third-party code.
>
> If the application creates the object itself or allocates storage
> for it, basically, if it isn't opaque, yes.  But we can bump port
> revisions for affected libraries (probably just searching
> /usr/local/include/... for pthread_mutex, pthread_cond, etc
> types to see possible problems.  I did a search for the installed
> ports on my system and found a few that might cause problems.
This relegates the issue to an attempt to do the full rebuild.  But I
do not see how the port bump would fix it, assume that you are updating
from the 10.x to 11.x and have the mix of the libraries, some of which
were built during the 10.x lifetime but with the bumped ports version.

It is not feasible to do a reliable audit of the 24+ Kports.

>
> I think we're just putting off the inevitable.  Do we not want
> to change our pthread sync types anyway, to get rid of an extra
> dereference per lock?  To get rid of the hacks in libc, rtld,
> etc?
>
> If the answer is no, we never want to do that, then ok.
An answer to this question requires a) consensus b) a workforce that
would do the decided transition.  I evaluated my opinion, potential
consequences and efforts required, and ended up with the posted patch.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: libthr shared locks

John Baldwin
On Wednesday, December 23, 2015 10:18:37 PM Konstantin Belousov wrote:

> On Wed, Dec 23, 2015 at 02:48:56PM -0500, Daniel Eischen wrote:
> > On Wed, 23 Dec 2015, Konstantin Belousov wrote:
> >
> > > On Wed, Dec 23, 2015 at 01:27:35PM -0500, Daniel Eischen wrote:
> > >> On Wed, 23 Dec 2015, Konstantin Belousov wrote:
> > >>
> > >> [ ... ]
> > >>> Would the ABI modified to make the pthread_mutex_t large enough to
> > >>> hold struct pthread_mutex, the rest of the implementation of the
> > >>> shared mutex is relatively trivial, if not already done.
> > >>>
> > >>> Changing this ABI is very hard.  libthr provides the symbol
> > >>> versioning, which allows to provide compatible shims for the previous
> > >>> ABI variant.  But since userspace tends to use the pthread objects in
> > >>> the layouts of the library objects, this causes serious ABI issues
> > >>> when mixing libraries built against different default versions of
> > >>> libthr.
> > >>
> > >> I think this is only if the libraries (or apps) pass pthread
> > >> lock types between them, that one has initialized with one ABI
> > >> but the other library uses another ABI.  We should really
> > >> exclude this as part of our ABI compatibility.
> > > Applications commonly embed pthread locks into their objects, including
> > > the exposed ABI in the third-party libraries.
> > >
> > > struct my_object {
> > > pthread_mutex_t obj_lock;
> > > ...
> > > };
> > >
> > > Changing the size of the pthread locks causes uncontrolled breakage of
> > > the ABI for significant number of third-party code.
> >
> > If the application creates the object itself or allocates storage
> > for it, basically, if it isn't opaque, yes.  But we can bump port
> > revisions for affected libraries (probably just searching
> > /usr/local/include/... for pthread_mutex, pthread_cond, etc
> > types to see possible problems.  I did a search for the installed
> > ports on my system and found a few that might cause problems.
> This relegates the issue to an attempt to do the full rebuild.  But I
> do not see how the port bump would fix it, assume that you are updating
> from the 10.x to 11.x and have the mix of the libraries, some of which
> were built during the 10.x lifetime but with the bumped ports version.
>
> It is not feasible to do a reliable audit of the 24+ Kports.

As a bit of a devil's advocate, I think the 64-bit ino_t change will in
fact require this for 11.  I suspect 3rd pary apps embed struct stat in
various structures as well and that that ABI change will require not
mixing old and new libraries.

One other point in favor of Konstantin's approach (IMO) is that keeping
the structures private prevents having to maintain the ABI of those
structures in the future.  I'm already keenly aware of how painful a
problem that can be with our non-opaque FILE (and which we cannot now
make opaque even though the standard APIs would work fine with an opaque
object).

--
John Baldwin
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: libthr shared locks

Daniel Eischen
In reply to this post by Konstantin Belousov
On Wed, 23 Dec 2015, Konstantin Belousov wrote:

> On Wed, Dec 23, 2015 at 02:48:56PM -0500, Daniel Eischen wrote:
>>
>> If the application creates the object itself or allocates storage
>> for it, basically, if it isn't opaque, yes.  But we can bump port
>> revisions for affected libraries (probably just searching
>> /usr/local/include/... for pthread_mutex, pthread_cond, etc
>> types to see possible problems.  I did a search for the installed
>> ports on my system and found a few that might cause problems.
>
> This relegates the issue to an attempt to do the full rebuild.  But I
> do not see how the port bump would fix it, assume that you are updating
> from the 10.x to 11.x and have the mix of the libraries, some of which
> were built during the 10.x lifetime but with the bumped ports version.
>
> It is not feasible to do a reliable audit of the 24+ Kports.

Is it really that hard to do a port run and insert a grep
for pthread_{mutex,cond,rwlock}_t in a ports installed
header files?  Then just blindly bumping portrevisions
for those ports and those depending on it?

Other than errors caused by storage layouts, libthr can
easily be instrumented to emit a warning when a sync type
is used with the wrong versioned function.

>> I think we're just putting off the inevitable.  Do we not want
>> to change our pthread sync types anyway, to get rid of an extra
>> dereference per lock?  To get rid of the hacks in libc, rtld,
>> etc?
>>
>> If the answer is no, we never want to do that, then ok.
>
> An answer to this question requires a) consensus b) a workforce that
> would do the decided transition.  I evaluated my opinion, potential
> consequences and efforts required, and ended up with the posted patch.

There's an old patch here:

   http://people.freebsd.org/~davidxu/pshared/patch6.diff

--
DE
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: libthr shared locks

Konstantin Belousov
In reply to this post by John Baldwin
On Wed, Dec 23, 2015 at 01:22:16PM -0800, John Baldwin wrote:

> On Wednesday, December 23, 2015 10:18:37 PM Konstantin Belousov wrote:
> > On Wed, Dec 23, 2015 at 02:48:56PM -0500, Daniel Eischen wrote:
> > > On Wed, 23 Dec 2015, Konstantin Belousov wrote:
> > >
> > > > On Wed, Dec 23, 2015 at 01:27:35PM -0500, Daniel Eischen wrote:
> > > >> On Wed, 23 Dec 2015, Konstantin Belousov wrote:
> > > >>
> > > >> [ ... ]
> > > >>> Would the ABI modified to make the pthread_mutex_t large enough to
> > > >>> hold struct pthread_mutex, the rest of the implementation of the
> > > >>> shared mutex is relatively trivial, if not already done.
> > > >>>
> > > >>> Changing this ABI is very hard.  libthr provides the symbol
> > > >>> versioning, which allows to provide compatible shims for the previous
> > > >>> ABI variant.  But since userspace tends to use the pthread objects in
> > > >>> the layouts of the library objects, this causes serious ABI issues
> > > >>> when mixing libraries built against different default versions of
> > > >>> libthr.
> > > >>
> > > >> I think this is only if the libraries (or apps) pass pthread
> > > >> lock types between them, that one has initialized with one ABI
> > > >> but the other library uses another ABI.  We should really
> > > >> exclude this as part of our ABI compatibility.
> > > > Applications commonly embed pthread locks into their objects, including
> > > > the exposed ABI in the third-party libraries.
> > > >
> > > > struct my_object {
> > > > pthread_mutex_t obj_lock;
> > > > ...
> > > > };
> > > >
> > > > Changing the size of the pthread locks causes uncontrolled breakage of
> > > > the ABI for significant number of third-party code.
> > >
> > > If the application creates the object itself or allocates storage
> > > for it, basically, if it isn't opaque, yes.  But we can bump port
> > > revisions for affected libraries (probably just searching
> > > /usr/local/include/... for pthread_mutex, pthread_cond, etc
> > > types to see possible problems.  I did a search for the installed
> > > ports on my system and found a few that might cause problems.
> > This relegates the issue to an attempt to do the full rebuild.  But I
> > do not see how the port bump would fix it, assume that you are updating
> > from the 10.x to 11.x and have the mix of the libraries, some of which
> > were built during the 10.x lifetime but with the bumped ports version.
> >
> > It is not feasible to do a reliable audit of the 24+ Kports.
>
> As a bit of a devil's advocate, I think the 64-bit ino_t change will in
> fact require this for 11.  I suspect 3rd pary apps embed struct stat in
> various structures as well and that that ABI change will require not
> mixing old and new libraries.
The stat change is different in this.  I do think (and quick overview
of includes seems to support the opinion) that the struct stat is rarely
embedded into the user objects.  I already wrote about this, when I said
that 'struct stat' changes are mostly confined to the src/ tree, in one
of the previous messages.  When I thought about the ino_t 64bit change,
the only example of affected app that I could remember, without looking
at the code, was rogue/nethack.

>
> One other point in favor of Konstantin's approach (IMO) is that keeping
> the structures private prevents having to maintain the ABI of those
> structures in the future.  I'm already keenly aware of how painful a
> problem that can be with our non-opaque FILE (and which we cannot now
> make opaque even though the standard APIs would work fine with an opaque
> object).

My patch does not prevent future change to use in-place pthread_mutex_t,
since the patch does not change current ABI. If somebody is willing to
go that route, well, fine. Nobody productized David' work for ten (?)
years. And the reason is, I believe, that the ABI change required to the
fundamental facility (threads did become ubiquios) at that late point in
the OS lifecycle is destructive.

The issue we have after the embedded structures layout modification is
exactly the issue which cannot be handled by symbol versioning.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: libthr shared locks

Daniel Eischen
In reply to this post by John Baldwin
On Wed, 23 Dec 2015, John Baldwin wrote:

> On Wednesday, December 23, 2015 10:18:37 PM Konstantin Belousov wrote:
>> On Wed, Dec 23, 2015 at 02:48:56PM -0500, Daniel Eischen wrote:
>>> On Wed, 23 Dec 2015, Konstantin Belousov wrote:
>>>
>>>> On Wed, Dec 23, 2015 at 01:27:35PM -0500, Daniel Eischen wrote:
>>>>> On Wed, 23 Dec 2015, Konstantin Belousov wrote:
>>>>>
>>>>> [ ... ]
>>>>>> Would the ABI modified to make the pthread_mutex_t large enough to
>>>>>> hold struct pthread_mutex, the rest of the implementation of the
>>>>>> shared mutex is relatively trivial, if not already done.
>>>>>>
>>>>>> Changing this ABI is very hard.  libthr provides the symbol
>>>>>> versioning, which allows to provide compatible shims for the previous
>>>>>> ABI variant.  But since userspace tends to use the pthread objects in
>>>>>> the layouts of the library objects, this causes serious ABI issues
>>>>>> when mixing libraries built against different default versions of
>>>>>> libthr.
>>>>>
>>>>> I think this is only if the libraries (or apps) pass pthread
>>>>> lock types between them, that one has initialized with one ABI
>>>>> but the other library uses another ABI.  We should really
>>>>> exclude this as part of our ABI compatibility.
>>>> Applications commonly embed pthread locks into their objects, including
>>>> the exposed ABI in the third-party libraries.
>>>>
>>>> struct my_object {
>>>> pthread_mutex_t obj_lock;
>>>> ...
>>>> };
>>>>
>>>> Changing the size of the pthread locks causes uncontrolled breakage of
>>>> the ABI for significant number of third-party code.
>>>
>>> If the application creates the object itself or allocates storage
>>> for it, basically, if it isn't opaque, yes.  But we can bump port
>>> revisions for affected libraries (probably just searching
>>> /usr/local/include/... for pthread_mutex, pthread_cond, etc
>>> types to see possible problems.  I did a search for the installed
>>> ports on my system and found a few that might cause problems.
>> This relegates the issue to an attempt to do the full rebuild.  But I
>> do not see how the port bump would fix it, assume that you are updating
>> from the 10.x to 11.x and have the mix of the libraries, some of which
>> were built during the 10.x lifetime but with the bumped ports version.
>>
>> It is not feasible to do a reliable audit of the 24+ Kports.
>
> As a bit of a devil's advocate, I think the 64-bit ino_t change will in
> fact require this for 11.  I suspect 3rd pary apps embed struct stat in
> various structures as well and that that ABI change will require not
> mixing old and new libraries.
>
> One other point in favor of Konstantin's approach (IMO) is that keeping
> the structures private prevents having to maintain the ABI of those
> structures in the future.  I'm already keenly aware of how painful a
> problem that can be with our non-opaque FILE (and which we cannot now
> make opaque even though the standard APIs would work fine with an opaque
> object).

We would include extra/spare words in the struct from day 1.
The public struct should just consist of an array of storage
units or similar, so that nothing can be gleaned from the
elements of the struct.  This is what Solaris does (or at least
used to).

Going forward, I think the sync structures just need to be able
to be properly initialized with PTHREAD_FOO_INITIALIZER.  If
10 years out the spare words are not enough, we could still
allocate remaining storage on first use like we do now.

--
DE
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: libthr shared locks

Daniel Eischen
In reply to this post by John Baldwin
On Wed, 23 Dec 2015, John Baldwin wrote:

> On Wednesday, December 23, 2015 10:18:37 PM Konstantin Belousov wrote:
[ much snipped for brevity ]

>>
>> It is not feasible to do a reliable audit of the 24+ Kports.
>
> As a bit of a devil's advocate, I think the 64-bit ino_t change will in
> fact require this for 11.  I suspect 3rd pary apps embed struct stat in
> various structures as well and that that ABI change will require not
> mixing old and new libraries.
>
> One other point in favor of Konstantin's approach (IMO) is that keeping
> the structures private prevents having to maintain the ABI of those
> structures in the future.  I'm already keenly aware of how painful a
> problem that can be with our non-opaque FILE (and which we cannot now
> make opaque even though the standard APIs would work fine with an opaque
> object).

This seems to be David's latest patch:

   http://people.freebsd.org/~davidxu/patch/pshared0607.diff

It is only 3 years old (2012).  I have email from David that says
he got 8-10% speedup in mysql OLTP from making the synch types
structures.

The patch also implements robust and priority inheritence mutexes
bumps shared library versions.

--
DE
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: libthr shared locks

John Baldwin
In reply to this post by Daniel Eischen
On Thursday, December 24, 2015 08:45:38 AM Daniel Eischen wrote:

> On Wed, 23 Dec 2015, John Baldwin wrote:
>
> > On Wednesday, December 23, 2015 10:18:37 PM Konstantin Belousov wrote:
> >> On Wed, Dec 23, 2015 at 02:48:56PM -0500, Daniel Eischen wrote:
> >>> On Wed, 23 Dec 2015, Konstantin Belousov wrote:
> >>>
> >>>> On Wed, Dec 23, 2015 at 01:27:35PM -0500, Daniel Eischen wrote:
> >>>>> On Wed, 23 Dec 2015, Konstantin Belousov wrote:
> >>>>>
> >>>>> [ ... ]
> >>>>>> Would the ABI modified to make the pthread_mutex_t large enough to
> >>>>>> hold struct pthread_mutex, the rest of the implementation of the
> >>>>>> shared mutex is relatively trivial, if not already done.
> >>>>>>
> >>>>>> Changing this ABI is very hard.  libthr provides the symbol
> >>>>>> versioning, which allows to provide compatible shims for the previous
> >>>>>> ABI variant.  But since userspace tends to use the pthread objects in
> >>>>>> the layouts of the library objects, this causes serious ABI issues
> >>>>>> when mixing libraries built against different default versions of
> >>>>>> libthr.
> >>>>>
> >>>>> I think this is only if the libraries (or apps) pass pthread
> >>>>> lock types between them, that one has initialized with one ABI
> >>>>> but the other library uses another ABI.  We should really
> >>>>> exclude this as part of our ABI compatibility.
> >>>> Applications commonly embed pthread locks into their objects, including
> >>>> the exposed ABI in the third-party libraries.
> >>>>
> >>>> struct my_object {
> >>>> pthread_mutex_t obj_lock;
> >>>> ...
> >>>> };
> >>>>
> >>>> Changing the size of the pthread locks causes uncontrolled breakage of
> >>>> the ABI for significant number of third-party code.
> >>>
> >>> If the application creates the object itself or allocates storage
> >>> for it, basically, if it isn't opaque, yes.  But we can bump port
> >>> revisions for affected libraries (probably just searching
> >>> /usr/local/include/... for pthread_mutex, pthread_cond, etc
> >>> types to see possible problems.  I did a search for the installed
> >>> ports on my system and found a few that might cause problems.
> >> This relegates the issue to an attempt to do the full rebuild.  But I
> >> do not see how the port bump would fix it, assume that you are updating
> >> from the 10.x to 11.x and have the mix of the libraries, some of which
> >> were built during the 10.x lifetime but with the bumped ports version.
> >>
> >> It is not feasible to do a reliable audit of the 24+ Kports.
> >
> > As a bit of a devil's advocate, I think the 64-bit ino_t change will in
> > fact require this for 11.  I suspect 3rd pary apps embed struct stat in
> > various structures as well and that that ABI change will require not
> > mixing old and new libraries.
> >
> > One other point in favor of Konstantin's approach (IMO) is that keeping
> > the structures private prevents having to maintain the ABI of those
> > structures in the future.  I'm already keenly aware of how painful a
> > problem that can be with our non-opaque FILE (and which we cannot now
> > make opaque even though the standard APIs would work fine with an opaque
> > object).
>
> We would include extra/spare words in the struct from day 1.
> The public struct should just consist of an array of storage
> units or similar, so that nothing can be gleaned from the
> elements of the struct.  This is what Solaris does (or at least
> used to).
>
> Going forward, I think the sync structures just need to be able
> to be properly initialized with PTHREAD_FOO_INITIALIZER.  If
> 10 years out the spare words are not enough, we could still
> allocate remaining storage on first use like we do now.

You can't allocate extra storage for the PSHARED case.  Any changes
to PSHARED primitives that require altering the layout are full-blown
ABI breakages the same as the one being contemplated here.

--
John Baldwin
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: libthr shared locks

Daniel Eischen
On Thu, 24 Dec 2015, John Baldwin wrote:

> On Thursday, December 24, 2015 08:45:38 AM Daniel Eischen wrote:
>> We would include extra/spare words in the struct from day 1.
>> The public struct should just consist of an array of storage
>> units or similar, so that nothing can be gleaned from the
>> elements of the struct.  This is what Solaris does (or at least
>> used to).
>>
>> Going forward, I think the sync structures just need to be able
>> to be properly initialized with PTHREAD_FOO_INITIALIZER.  If
>> 10 years out the spare words are not enough, we could still
>> allocate remaining storage on first use like we do now.
>
> You can't allocate extra storage for the PSHARED case.  Any changes
> to PSHARED primitives that require altering the layout are full-blown
> ABI breakages the same as the one being contemplated here.

Yes, I know.  With spare slots and being able to move anything required
for shared to near the beginning of the struct, I think we'd be good
for quite a while anyway.

Can we still implement things like robust and priority mutexes in a
pshared mutex with Konstantin's implementation?  The rest of the world
(Linux, Solaris anyway) use structs and get by just fine ABI-wise.

--
DE
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: libthr shared locks

Konstantin Belousov
On Thu, Dec 24, 2015 at 01:46:25PM -0500, Daniel Eischen wrote:
> Can we still implement things like robust and priority mutexes in a
> pshared mutex with Konstantin's implementation?  The rest of the world
> (Linux, Solaris anyway) use structs and get by just fine ABI-wise.
Yes, we do. I intend to do this as the next stage, planned the
implementation and did some preparations for it in the current patch.
This raises the question, why do you suppose that my approach with
the off-page would not allow it ? I am asking not to argument, but to
understand a possible shortcoming in the approach, which I missed.

From the implementation PoV, off-page is completely equivalent to the
in-line structs approach, and the difference is only in the initial
translation of say pthread_mutex_t to the structure (off-page lookup vs.
identity). The issues with maintaining the lists of the owned robust of
pi mutexes are same, and in fact you could see some non-trivial traces
of this in the _mutex_fork() reimplementation in my patch.

Linux and Solaris get (large enough) structs early enough to avoid the ABI
issues.  I remember Linux changing FILE layout in early days of glibc 2.0
and resulting pain, while they already had the GNU symbol versioning
working and used.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: libthr shared locks

Daniel Eischen
On Thu, 24 Dec 2015, Konstantin Belousov wrote:

> On Thu, Dec 24, 2015 at 01:46:25PM -0500, Daniel Eischen wrote:
>> Can we still implement things like robust and priority mutexes in a
>> pshared mutex with Konstantin's implementation?  The rest of the world
>> (Linux, Solaris anyway) use structs and get by just fine ABI-wise.
>
> Yes, we do. I intend to do this as the next stage, planned the
> implementation and did some preparations for it in the current patch.
> This raises the question, why do you suppose that my approach with
> the off-page would not allow it ? I am asking not to argument, but to
> understand a possible shortcoming in the approach, which I missed.

No, I was merely inquiring, if this is on your agenda, that's
good.

> From the implementation PoV, off-page is completely equivalent to the
> in-line structs approach, and the difference is only in the initial
> translation of say pthread_mutex_t to the structure (off-page lookup vs.
> identity). The issues with maintaining the lists of the owned robust of
> pi mutexes are same, and in fact you could see some non-trivial traces
> of this in the _mutex_fork() reimplementation in my patch.
>
> Linux and Solaris get (large enough) structs early enough to avoid the ABI
> issues.  I remember Linux changing FILE layout in early days of glibc 2.0
> and resulting pain, while they already had the GNU symbol versioning
> working and used.

I guess the issue now is the apparent speedups without having the
extra de-reference.  8-10% for things like mysql might be worth
considering the userland structs.

--
DE
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: libthr shared locks

Julian Elischer-5
In reply to this post by Konstantin Belousov
On 24/12/2015 1:25 AM, Konstantin Belousov wrote:
> A well-known limitation of the FreeBSD libthr implementation of the
> pthread locking objects is the missed support for the process-shared
> locks.  The hardest part of implementing the support is the neccessity
> of providing ABI compatibility for the current implementation.

> [...]

>
> Changing this ABI is very hard.  libthr provides the symbol
> versioning, which allows to provide compatible shims for the previous
> ABI variant.  But since userspace tends to use the pthread objects in
> the layouts of the library objects, this causes serious ABI issues
> when mixing libraries built against different default versions of
> libthr.

which is one reason I think symbol versioning is over-rated..
just make a new library version.
>

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: libthr shared locks

Konstantin Belousov
In reply to this post by Daniel Eischen
On Fri, Dec 25, 2015 at 01:18:14PM -0500, Daniel Eischen wrote:

> On Thu, 24 Dec 2015, Konstantin Belousov wrote:
>
> > On Thu, Dec 24, 2015 at 01:46:25PM -0500, Daniel Eischen wrote:
> >> Can we still implement things like robust and priority mutexes in a
> >> pshared mutex with Konstantin's implementation?  The rest of the world
> >> (Linux, Solaris anyway) use structs and get by just fine ABI-wise.
> >
> > Yes, we do. I intend to do this as the next stage, planned the
> > implementation and did some preparations for it in the current patch.
> > This raises the question, why do you suppose that my approach with
> > the off-page would not allow it ? I am asking not to argument, but to
> > understand a possible shortcoming in the approach, which I missed.
>
> No, I was merely inquiring, if this is on your agenda, that's
> good.
>
> > From the implementation PoV, off-page is completely equivalent to the
> > in-line structs approach, and the difference is only in the initial
> > translation of say pthread_mutex_t to the structure (off-page lookup vs.
> > identity). The issues with maintaining the lists of the owned robust of
> > pi mutexes are same, and in fact you could see some non-trivial traces
> > of this in the _mutex_fork() reimplementation in my patch.
> >
> > Linux and Solaris get (large enough) structs early enough to avoid the ABI
> > issues.  I remember Linux changing FILE layout in early days of glibc 2.0
> > and resulting pain, while they already had the GNU symbol versioning
> > working and used.
>
> I guess the issue now is the apparent speedups without having the
> extra de-reference.  8-10% for things like mysql might be worth
> considering the userland structs.

I am sorry for the delay in answering, I read the David' patch you pointed
out, and it took time to get through 9KLOCs patch which is not applicable
to the current sources.

I should note that the patch is not immediately useable, not due to
the source code drift, but because the patch combines many unrelated
things, and I do not agree with everything in it.

E.g., I found quite interesting the notion that our libthr fork() does
not work when called from the signal handler. I do not quite understand
how the detection of the situation is done in the patch, and I do not
agree with the cleanup of the 'in sighandler' state on longjmp. I
think that this part of the patch is obsoleted by libthr intercepting
signals and postponing signal delivery until a critical section is
exited.

Patch adds yet another private malloc(), now to libthr. I think that
cooperating with rtld and share the same allocator would be more
reasonable.

Notes above are to explain why I think that productizing the David' patch
is huge work, IMO significantly more than just 'test and commit'.

Returning to the point of potential gain in performance due to the
ABI change. As I already stated in my previous replies, I am quite
worried about the impact of the ABI change, and I think that potential
destabilizations, which would manifest itself in the subtle and hard to
diagnose ways (i.e. it is not a sigsegv outright on the start) is huge
setback for the change.

OTOH, we should be able to plan such change, which requires much more
drastic measures to be implementable. I started thinking about it, and
I noted that what is needed is solved by renaming libthr to something
else, e.g. libthr2. One of the issue which is not solved by the dso
version bump (not to even mention versioned symbols bump) is the
static linker bringing both libthr.so.3 and libthr.so.4 into the same
process, by the mere use of -lpthread by different objects at different
times. Then, libthr and hypothetical libthr2 should check and prevent
activation of image which loads both.

But these are longtime to evaluate and implement. The feature at hand
does not require ABI change, as my patch demostrated. Yes, shared
mutexes would be more naturally implemented with the inline locks, and
avoidance of an indirection in the libthr is also good, but lets not mix
significant ABI change and some less ambitious feature. My approach does
not lock out future strategies.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: libthr shared locks

Daniel Eischen
On Sat, 26 Dec 2015, Konstantin Belousov wrote:

> On Fri, Dec 25, 2015 at 01:18:14PM -0500, Daniel Eischen wrote:
>> On Thu, 24 Dec 2015, Konstantin Belousov wrote:
>>
>>> On Thu, Dec 24, 2015 at 01:46:25PM -0500, Daniel Eischen wrote:
>>>> Can we still implement things like robust and priority mutexes in a
>>>> pshared mutex with Konstantin's implementation?  The rest of the world
>>>> (Linux, Solaris anyway) use structs and get by just fine ABI-wise.
>>>
>>> Yes, we do. I intend to do this as the next stage, planned the
>>> implementation and did some preparations for it in the current patch.
>>> This raises the question, why do you suppose that my approach with
>>> the off-page would not allow it ? I am asking not to argument, but to
>>> understand a possible shortcoming in the approach, which I missed.
>>
>> No, I was merely inquiring, if this is on your agenda, that's
>> good.
>>
>>> From the implementation PoV, off-page is completely equivalent to the
>>> in-line structs approach, and the difference is only in the initial
>>> translation of say pthread_mutex_t to the structure (off-page lookup vs.
>>> identity). The issues with maintaining the lists of the owned robust of
>>> pi mutexes are same, and in fact you could see some non-trivial traces
>>> of this in the _mutex_fork() reimplementation in my patch.
>>>
>>> Linux and Solaris get (large enough) structs early enough to avoid the ABI
>>> issues.  I remember Linux changing FILE layout in early days of glibc 2.0
>>> and resulting pain, while they already had the GNU symbol versioning
>>> working and used.
>>
>> I guess the issue now is the apparent speedups without having the
>> extra de-reference.  8-10% for things like mysql might be worth
>> considering the userland structs.
>
> I am sorry for the delay in answering, I read the David' patch you pointed
> out, and it took time to get through 9KLOCs patch which is not applicable
> to the current sources.

It is the holidays, I'm sure everyone is busy with other things.
I am in the same camp :-)

> I should note that the patch is not immediately useable, not due to
> the source code drift, but because the patch combines many unrelated
> things, and I do not agree with everything in it.

Yeah, I would advocate only committing part of David's patch,
in stages.  Just the part that implements the struct change and
shared synch primitives.  Other changes may no longer apply or
can be done later.

> E.g., I found quite interesting the notion that our libthr fork() does
> not work when called from the signal handler. I do not quite understand
> how the detection of the situation is done in the patch, and I do not
> agree with the cleanup of the 'in sighandler' state on longjmp. I
> think that this part of the patch is obsoleted by libthr intercepting
> signals and postponing signal delivery until a critical section is
> exited.

I think libthr has always done that (postpoining until critical
section is exited).  But there are libc (and perhaps other)
locks that may be left locked on a fork().  Really (according
to POSIX), a fork() from a threaded program can only be used
to exec() something.

> Patch adds yet another private malloc(), now to libthr. I think that
> cooperating with rtld and share the same allocator would be more
> reasonable.

I tend to agree.

> Notes above are to explain why I think that productizing the David' patch
> is huge work, IMO significantly more than just 'test and commit'.
>
> Returning to the point of potential gain in performance due to the
> ABI change. As I already stated in my previous replies, I am quite
> worried about the impact of the ABI change, and I think that potential
> destabilizations, which would manifest itself in the subtle and hard to
> diagnose ways (i.e. it is not a sigsegv outright on the start) is huge
> setback for the change.

I guess I am just not that worried about the ABI change when combined
with a library version bump.  We use to do this for every new major
release, it isn't anything that hasn't been done before.  We never
supported 2 thread libraries in the same executable, and it is
obvious when that happens (I think we even instrumented libc_r,
or was it rtld?, to emit a warning when that happened.  My memory
is fuzzy).  With poudriere, our new package system, I would think
we would be better off now than 10 years ago?

I do note that I was initially wrong in thinking that port revision
bumps could help alleviate any pain.  A port could be built in
-stable with the revision bump just as well as in -current (or 11).
What would be needed is for pkg to understand that any port built
for 10 and previous could not run on 11+ (just to be on the safe
side).

> OTOH, we should be able to plan such change, which requires much more
> drastic measures to be implementable. I started thinking about it, and
> I noted that what is needed is solved by renaming libthr to something
> else, e.g. libthr2. One of the issue which is not solved by the dso
> version bump (not to even mention versioned symbols bump) is the
> static linker bringing both libthr.so.3 and libthr.so.4 into the same
> process, by the mere use of -lpthread by different objects at different
> times. Then, libthr and hypothetical libthr2 should check and prevent
> activation of image which loads both.
>
> But these are longtime to evaluate and implement. The feature at hand
> does not require ABI change, as my patch demostrated. Yes, shared
> mutexes would be more naturally implemented with the inline locks, and
> avoidance of an indirection in the libthr is also good, but lets not mix
> significant ABI change and some less ambitious feature. My approach does
> not lock out future strategies.

I agree, but the work that you are doing now would be basically
thrown out later on.  I will not stand in your way and appreciate
any work you do.  I would just rather that the struct change be
made now for 11, even without any pshared or other changes.  For
once the struct change is made, pshared or other additions can
be made afterward, even in the 11 branch because they would not
break the ABI.

--
DE
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: libthr shared locks

Simon J. Gerraty
In reply to this post by Julian Elischer-5
Julian Elischer <[hidden email]> wrote:
> > ABI variant.  But since userspace tends to use the pthread objects in
> > the layouts of the library objects, this causes serious ABI issues
> > when mixing libraries built against different default versions of
> > libthr.
>
> which is one reason I think symbol versioning is over-rated..

symbol versioning works fine - provided you have opaque structs.

> just make a new library version.

as previously noted, this rarely solves anything since to avoid problems
you need to bump the major version of all shared libs, and you cannot do
that for 3rd party code you don't even know exists.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: libthr shared locks

Konstantin Belousov
In reply to this post by Daniel Eischen
On Sat, Dec 26, 2015 at 12:15:43PM -0500, Daniel Eischen wrote:
> I think libthr has always done that (postpoining until critical
> section is exited).  But there are libc (and perhaps other)
> locks that may be left locked on a fork().  Really (according
> to POSIX), a fork() from a threaded program can only be used
> to exec() something.
libthr only started to do the interception in r212076, which was
committed by David on Sep 1, 2010.

Posix left the behaviour undefined after the fork, but real world
rejects implementations which make C runtime broken after fork in mt
process. Rtld must work in the child anyway (and in sighandler as well).
We were forced to ensure also that malloc(3) and pthread_create(3) work
in the child after the fork, and any transient regressions in this area
where immediately reported. One of the biggest abusers of that are
Qt/KDE.

> I guess I am just not that worried about the ABI change when combined
> with a library version bump.  We use to do this for every new major
> release, it isn't anything that hasn't been done before.  We never
> supported 2 thread libraries in the same executable, and it is
> obvious when that happens (I think we even instrumented libc_r,
> or was it rtld?, to emit a warning when that happened.  My memory
> is fuzzy).  With poudriere, our new package system, I would think
> we would be better off now than 10 years ago?

We did significant work to avoid requiring complete recompilation of all
user binaries on the major version updates. Part of it is the symbol
versioning, another part is the stronger discipline and future planning
when doing ABI-sensitive changes. At least starting with stable/8, when
you binary depends only on the basic C runtime (i.e. rtld + libc + libm
+ libthr), you are no longer required to recompile it, and you do not
need to install any compat libraries. Things just work.

We are not there yet, since some libraries are not managed that good,
e.g. libutil is not symver-ed but at least usually bumped correctly.
Some libraries are handled much worse, e.g. libcam _interface_ depends
on volatile kernel parts and libcam version is not always bumped when
needed.  Unfortunately, libcam is the dependency of high-profile user
applications, from Gnome and KDE.  This is what prevents us from
stopping declaring 'recompile ports on major upgrade', but it is a
goal.

Still, basic C runtime is in much better shape WRT ABI stability
than it was, say, in the 6.x-7.x time. This is why ino_t 64bit is moved
with so much caution, and should also explain why I am so nervous to
lock inlining.  Doing libthr bump would break things.

>
> I do note that I was initially wrong in thinking that port revision
> bumps could help alleviate any pain.  A port could be built in
> -stable with the revision bump just as well as in -current (or 11).
> What would be needed is for pkg to understand that any port built
> for 10 and previous could not run on 11+ (just to be on the safe
> side).
BTW, I tried to explain exactly this scenario in one of the previous
replies.

>
> > OTOH, we should be able to plan such change, which requires much more
> > drastic measures to be implementable. I started thinking about it, and
> > I noted that what is needed is solved by renaming libthr to something
> > else, e.g. libthr2. One of the issue which is not solved by the dso
> > version bump (not to even mention versioned symbols bump) is the
> > static linker bringing both libthr.so.3 and libthr.so.4 into the same
> > process, by the mere use of -lpthread by different objects at different
> > times. Then, libthr and hypothetical libthr2 should check and prevent
> > activation of image which loads both.
> >
> > But these are longtime to evaluate and implement. The feature at hand
> > does not require ABI change, as my patch demostrated. Yes, shared
> > mutexes would be more naturally implemented with the inline locks, and
> > avoidance of an indirection in the libthr is also good, but lets not mix
> > significant ABI change and some less ambitious feature. My approach does
> > not lock out future strategies.
>
> I agree, but the work that you are doing now would be basically
> thrown out later on.  I will not stand in your way and appreciate
> any work you do.  I would just rather that the struct change be
> made now for 11, even without any pshared or other changes.  For
> once the struct change is made, pshared or other additions can
> be made afterward, even in the 11 branch because they would not
> break the ABI.

Lock inlining was not done for ten years, now cost of doing it is
extremely high, as discussed above.  Who would drive the change, and
with what time frame ?  If me, I seriosly consider renaming libthr
to libthr2, but I had no time to think much about it.

Right now, I think that I want to commit my current patch and implement
robust mutexes as the next step, without ABI breakage. At least, this
seems to have fixed time-frame and can be made ready for 11.x. Lock
inlining might be not. Are there serious objections against the plan,
except that (lock inlining + pshared) is ideal situation, while the plan
is not (but more practical) ?
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
123