Re: [HEADS UP] - OFED/RDMA stack update

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: [HEADS UP] - OFED/RDMA stack update

Navdeep Parhar-2
+freebsd-arch@

Hi Meny,

Can you please post the KPI/KBI analysis that you generated to some
public location and provide a link here?  A straight MFC would be a
major break of KPI/KBI in -STABLE and the options we're looking at are:

a) Ignore the breakage and let downstream consumers deal with the
fallout.  This obviously isn't ideal in a -STABLE branch.

b) Provide compat shims to at least preserve the KPI.  One challenge is
that the changes include functions with the same name but different
signature/behavior.  See, for example, ib_create_cq in Meny's list once
he publishes it.

c) Have two versions of the OFED interfaces in 11-STABLE and not break
existing downstream consumers at all.

I've reached out to users that I know of and know will be affected.
If you use OFED and FreeBSD 11 this would be a good time to weigh
in with your thoughts, ideas, concerns etc..

Regards,
Navdeep

On Mon, 2018-02-26 at 14:49 +0000, Meny Yossefi wrote:

> Hi,
>
> We are currently working on MFC'ing the OFED/RDMA stack update
> mentioned below to FreeBSD11-STABLE.
>
> We already have working version in 'projects/bsd_rdma_4_9_stable_11'
> (pending iWARP updates) and currently working on ULP integration.
>
> Again, as always, for any concern/comments you might have, please
> don't hesitate contacting us.
>
> [hidden email]
>
>
> Regards,
>
> Meny Yossefi,
> Mellanox technologies
>
>
> -----Original Message-----
> From: Meny Yossefi
> Sent: Monday, November 13, 2017 11:09 AM
> To: '[hidden email]' <[hidden email]>;
> '[hidden email]' <[hidden email]>
> Cc: freebsd-drivers <[hidden email]>
> Subject: [HEADS UP] - OFED/RDMA stack update
>
> Hi,
>
> This is to inform you that by end of this week we plan to merge the
> OFED/RDMA stack update from the project - 'bsd_rdma_4_9' into 12-
> CURRENT.
> The update aligns the OFED common code and RDMA vendor drivers with
> Linux v4.9.
>
> We are still working on final modifications and build testing it prior
> to submission.
>
>
> For any concern/comments you might have, please don't hesitate
> contacting us.
>
> [hidden email]
>
>
> Regards,
>
> Meny Yossefi,
> Mellanox technologies
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-infiniband
> To unsubscribe, send any mail to "freebsd-infiniband-unsubscribe@freeb
> sd.org"
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [HEADS UP] - OFED/RDMA stack update

Konstantin Belousov-3
On Mon, Feb 26, 2018 at 02:21:39PM -0800, Navdeep Parhar wrote:
> +freebsd-arch@
>
> Hi Meny,
>
> Can you please post the KPI/KBI analysis that you generated to some
> public location and provide a link here?  A straight MFC would be a
> major break of KPI/KBI in -STABLE and the options we're looking at are:
I put the report at
https://kib.kiev.ua/kib/ibcore_11_to_11_merged_compat_report.html

>
> a) Ignore the breakage and let downstream consumers deal with the
> fallout.  This obviously isn't ideal in a -STABLE branch.
>
> b) Provide compat shims to at least preserve the KPI.  One challenge is
> that the changes include functions with the same name but different
> signature/behavior.  See, for example, ib_create_cq in Meny's list once
> he publishes it.
Project did handled similar issues already.  One of the approaches is to
renname the ib_create_cq with the new signature to ib_create_cq_n12 and
check for (e.g.) _WANT_NEW_OFED symbol and to select one or another:
#ifdef _WANT_NEW_OFED
#define ib_create_cq(new args there) ib_create_cq_n21(new args there)
#else
#define ib_create_cq (ib_create_cq)
#endif

Then ULP that wants new KPI defines _WANT_NEW_OFED.

>
> c) Have two versions of the OFED interfaces in 11-STABLE and not break
> existing downstream consumers at all.
It is possible to make them loadable simultaneously as modules, but it
is quite confusing to users, because Mellanox clearly wants mlx5_ib and
mlx4_ib to work only with new OFED, while cxgbe would use old OFED ?

Also, either we would need to mess with the ibcore.ko module name, or
with version.  I am not sure that our module handling is robust enough
to make the version trick possible.

>
> I've reached out to users that I know of and know will be affected.
> If you use OFED and FreeBSD 11 this would be a good time to weigh
> in with your thoughts, ideas, concerns etc..
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [HEADS UP] - OFED/RDMA stack update

Hans Petter Selasky-6
Hi,

The bsd_rdma_4_9_stable_11 projects branch is close to being merged into
FreeBSD 11-stable. Mellanox plans to merge no later than 12:00 CEST TUE
20th of March 2018, unless objections are received.

A compatibility header file has been created, ib_verbs_compat.h, which
offers full source compatibility to existing OFED kernel applications,
as a response to the raised conserns. User-space compatibility is
maintained through library symbol versioning.

https://svnweb.freebsd.org/base/projects/bsd_rdma_4_9_stable_11/sys/ofed/include/rdma/ib_verbs_compat.h

An example client for this header file can be found here:

https://svnweb.freebsd.org/base/projects/bsd_rdma_4_9_stable_11/sys/contrib/rdma/krping_compat/

Currently the cxgb and cxgbe i-Warp drivers are not building, and will
be stubbed from the kernel build before the branch is merged, unless
Chelsio can add patches for these.

Here is a quick and dirty patch to make the bsd_rdma_4_9_stable_11
branch build:

> diff --git a/sys/modules/Makefile b/sys/modules/Makefile
> index 6b005c854d7..b918a208f21 100644
> --- a/sys/modules/Makefile
> +++ b/sys/modules/Makefile
> @@ -530,7 +530,7 @@ _txp=               txp
>  .if ${MK_SOURCELESS_UCODE} != "no" && ${MACHINE_CPUARCH} != "arm" && \
>         ${MACHINE_ARCH:C/mips(el)?/mips/} != "mips" && \
>         ${MACHINE_ARCH} != "powerpc" && ${MACHINE_CPUARCH} != "riscv"
> -_cxgbe=                cxgbe
> +#_cxgbe=               cxgbe
>  .endif
>  
>  .if ${MK_TESTS} != "no" || defined(ALL_MODULES)
> @@ -554,7 +554,7 @@ _vpo=               vpo
>  _sym=          sym
>  # intr_disable() is a macro, causes problems
>  .if ${MK_SOURCELESS_UCODE} != "no"
> -_cxgb=         cxgb
> +#_cxgb=                cxgb
>  .endif
>  .endif

--HPS
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [HEADS UP] - OFED/RDMA stack update

Navdeep Parhar-2
Hold your horses.  Do you have confirmation from the affected party that
the shims are adequate for them?  I have been waiting for that before
looking at this branch.

Is the iw_cxgbe breakage a simple merge conflict as previously discussed
or do the shims require driver changes?  If they don't then it should be
possible to drop the iw_cxgbe from head into this branch and it should
just work, is that correct?

Regards,
Navdeep

On Fri, Mar 16, 2018 at 05:44:41PM +0100, Hans Petter Selasky wrote:

> Hi,
>
> The bsd_rdma_4_9_stable_11 projects branch is close to being merged into
> FreeBSD 11-stable. Mellanox plans to merge no later than 12:00 CEST TUE 20th
> of March 2018, unless objections are received.
>
> A compatibility header file has been created, ib_verbs_compat.h, which
> offers full source compatibility to existing OFED kernel applications, as a
> response to the raised conserns. User-space compatibility is maintained
> through library symbol versioning.
>
> https://svnweb.freebsd.org/base/projects/bsd_rdma_4_9_stable_11/sys/ofed/include/rdma/ib_verbs_compat.h
>
> An example client for this header file can be found here:
>
> https://svnweb.freebsd.org/base/projects/bsd_rdma_4_9_stable_11/sys/contrib/rdma/krping_compat/
>
> Currently the cxgb and cxgbe i-Warp drivers are not building, and will be
> stubbed from the kernel build before the branch is merged, unless Chelsio
> can add patches for these.
>
> Here is a quick and dirty patch to make the bsd_rdma_4_9_stable_11 branch
> build:
>
> >diff --git a/sys/modules/Makefile b/sys/modules/Makefile
> >index 6b005c854d7..b918a208f21 100644
> >--- a/sys/modules/Makefile
> >+++ b/sys/modules/Makefile
> >@@ -530,7 +530,7 @@ _txp=               txp
> > .if ${MK_SOURCELESS_UCODE} != "no" && ${MACHINE_CPUARCH} != "arm" && \
> >        ${MACHINE_ARCH:C/mips(el)?/mips/} != "mips" && \
> >        ${MACHINE_ARCH} != "powerpc" && ${MACHINE_CPUARCH} != "riscv"
> >-_cxgbe=                cxgbe
> >+#_cxgbe=               cxgbe
> > .endif
> > .if ${MK_TESTS} != "no" || defined(ALL_MODULES)
> >@@ -554,7 +554,7 @@ _vpo=               vpo
> > _sym=          sym
> > # intr_disable() is a macro, causes problems
> > .if ${MK_SOURCELESS_UCODE} != "no"
> >-_cxgb=         cxgb
> >+#_cxgb=                cxgb
> > .endif
> > .endif
>
> --HPS
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [HEADS UP] - OFED/RDMA stack update

Hans Petter Selasky-6
On 03/17/18 20:52, Navdeep Parhar wrote:
> Hold your horses.  Do you have confirmation from the affected party that
> the shims are adequate for them?  I have been waiting for that before
> looking at this branch.

Hi Navdeep,

Mellanox has received an API list from at least one party, and has taken
the action to support all the required APIs.

> Is the iw_cxgbe breakage a simple merge conflict as previously discussed
> or do the shims require driver changes?  

It is a merge conflict. The code already compiles in 12-current.

> If they don't then it should be
> possible to drop the iw_cxgbe from head into this branch and it should
> just work, is that correct?

Yes, it shouldn't be too hard to do and I would appreciate if Chelsio
could throw some resources at this ASAP. I believe the new code will
benefit everyone using Infiniband/RoCE and iWarp with FreeBSD.

--HPS
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [HEADS UP] - OFED/RDMA stack update

Navdeep Parhar-2
On 03/17/2018 13:03, Hans Petter Selasky wrote:

> On 03/17/18 20:52, Navdeep Parhar wrote:
>> Hold your horses.  Do you have confirmation from the affected party that
>> the shims are adequate for them?  I have been waiting for that before
>> looking at this branch.
>
> Hi Navdeep,
>
> Mellanox has received an API list from at least one party, and has taken
> the action to support all the required APIs.
>
>> Is the iw_cxgbe breakage a simple merge conflict as previously discussed
>> or do the shims require driver changes? 
>
> It is a merge conflict. The code already compiles in 12-current.

I tried backing out r329391 and r329017 in a local copy and then was
able to merge r320418, r323082, and r326169 in that order without any
conflicts.  But iw_cxgbe/cm.c still doesn't compile in the projects
branch because it has some socket code that relies on some of glebius's
changes available only in head (I checked with him and they aren't
MFC'able).  I'm trying to figure out what to do about those.

And what about the cxgb breakage?  Is there any simple way to make an
old style driver work with the new stack?  T3 iw_cxgb in head was
retired before the ofed overhaul.

Regards,
Navdeep

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [HEADS UP] - OFED/RDMA stack update

Hans Petter Selasky-6
On 03/20/18 21:07, Navdeep Parhar wrote:
> And what about the cxgb breakage?  Is there any simple way to make an
> old style driver work with the new stack?  T3 iw_cxgb in head was
> retired before the ofed overhaul.

Hi,

The most simple way is to pull this driver from Linux 4.9, including
libcxgb for user-space, if possible, though I'm not sure about the
FreeBSD specific areas of iWarp and socket handling. I think you know
that part better than I do.

After pulling from Linux 4.9 you need to extend this change to cover the
iw_cxgb:

https://svnweb.freebsd.org/changeset/base/330508

Or can you retire iw_cxgb?

--HPS
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

RE: [HEADS UP] - OFED/RDMA stack update

Meny Yossefi
In reply to this post by Navdeep Parhar-2
Hi Navdeep,

What's the current iWARP status on the project branch?
If it's ready we'd like to merge it to FreeBSD11-STABLE immediately.

-Meny


-----Original Message-----
From: Navdeep Parhar [mailto:[hidden email]] On Behalf Of Navdeep Parhar
Sent: Tuesday, March 20, 2018 10:08 PM
To: Hans Petter Selasky <[hidden email]>; Konstantin Belousov <[hidden email]>; '[hidden email]' <[hidden email]>; freebsd-drivers <[hidden email]>; Meny Yossefi <[hidden email]>; '[hidden email]' <[hidden email]>; freebsd-arch <[hidden email]>
Subject: Re: [HEADS UP] - OFED/RDMA stack update

On 03/17/2018 13:03, Hans Petter Selasky wrote:

> On 03/17/18 20:52, Navdeep Parhar wrote:
>> Hold your horses.  Do you have confirmation from the affected party
>> that the shims are adequate for them?  I have been waiting for that
>> before looking at this branch.
>
> Hi Navdeep,
>
> Mellanox has received an API list from at least one party, and has
> taken the action to support all the required APIs.
>
>> Is the iw_cxgbe breakage a simple merge conflict as previously
>> discussed or do the shims require driver changes?
>
> It is a merge conflict. The code already compiles in 12-current.

I tried backing out r329391 and r329017 in a local copy and then was able to merge r320418, r323082, and r326169 in that order without any conflicts.  But iw_cxgbe/cm.c still doesn't compile in the projects branch because it has some socket code that relies on some of glebius's changes available only in head (I checked with him and they aren't MFC'able).  I'm trying to figure out what to do about those.

And what about the cxgb breakage?  Is there any simple way to make an old style driver work with the new stack?  T3 iw_cxgb in head was retired before the ofed overhaul.

Regards,
Navdeep

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"