PCIe NVME drives not detected on Dell R6515

classic Classic list List threaded Threaded
28 messages Options
12
Reply | Threaded
Open this post in threaded view
|

PCIe NVME drives not detected on Dell R6515

Miroslav Lachman
I already asked on stable@ but as I tried it on 13-CURRENT with the same
result I am trying to ask for help here.

I have rented dedicated server Dell PowerEdge R6515 with iDRAC access only.
There are 2 NVME drives which I wanted to use as ZFS root pool.

They are shown in iDRAC

Device Description:   PCIe SSD in Slot 1 in Bay 1
Device Protocol:      NVMe-MI1.0
Model:                Dell Express Flash NVMe P4510 1TB SFF
Bus:                  130
Manufacturer:         INTEL
Product ID:           a54
Revision:             VDV1DP23
Enclosure:            PCIe SSD Backplane 1


pciconf -l show many things, some of them are named "noneN@pci..." but
none "nvme"

The is printscreen (12.1 but 13-CURRENT is the same)

https://ibb.co/tPnymL7

But I booted Linux SystemRescueCd and nvme devices are there visible in
/dev/
https://ibb.co/sj22Nwg

There is verbose output of Linux lspci https://ibb.co/dPZTwV1

Linux dmesg contains:
nvme nvme0: pci function 0000:81:00.0
nvme nvme1: pci function 0000:82:00.0
nvme nvme0: 32/0/0 default/read/poll queues
nvme nvme1: 32/0/0 default/read/poll queues


The machine is Dell PowerEdge R6515 with AMD EPYC 7302P


More details extracted from iDRAC web interface

I found this informations

PCIe SSD in Slot 1 in Bay 1
Bus:   82
BusProtocol:  PCIE
Device:  0
DeviceDescription:  PCIe SSD in Slot 1 in Bay 1
DeviceProtocol:  NVMe-MI1.0
DeviceType:  PCIeSSD
DriveFormFactor:  2.5 inch
FailurePredicted:  NO
FQDD:  Disk.Bay.1:Enclosure.Internal.0-1
FreeSizeInBytes:  Information Not Available
Function:  0
HotSpareStatus:  Information Not Available
InstanceID:  Disk.Bay.1:Enclosure.Internal.0-1
Manufacturer:  INTEL
MaximumCapableSpeed:  8 GT/s
MediaType:  Solid State Drive
Model:  Dell Express Flash NVMe P4510 1TB SFF
NegotiatedSpeed:  8 GT/s
PCIeCapableLinkWidth:  x4
PCIeNegotiatedLinkWidth:  x4
PrimaryStatus:  Ok
ProductID:  a54
RaidStatus:  Information Not Available
RAIDType:  Unknown
RemainingRatedWriteEndurance:  100 %
Revision:  VDV1DP23
SerialNumber:  PHLJxxxxxxWF1PxxxxN
SizeInBytes:  1000204886016
Slot:  1
State:  Ready
SystemEraseCapability:  CryptographicErasePD

PCIe SSD in Slot 1 in Bay 1 - PCI Device
BusNumber:  130
DataBusWidth:  4x or x4
Description:  Express Flash NVMe 1.0 TB 2.5" U.2 (P4510)
DeviceDescription:  PCIe SSD in Slot 1 in Bay 1
DeviceNumber:  0
DeviceType:  PCIDevice
FQDD:  Disk.Bay.1:Enclosure.Internal.0-1
FunctionNumber:  0
InstanceID:  Disk.Bay.1:Enclosure.Internal.0-1
LastSystemInventoryTime:  2020-04-17T03:21:39
LastUpdateTime:  2020-03-31T13:55:06
Manufacturer:  Intel Corporation
PCIDeviceID:  0A54
PCISubDeviceID:  2003
PCISubVendorID:  1028
PCIVendorID:  8086
SlotLength:  2.5 Inch Drive Form Factor
SlotType:  PCI Express Gen 3 SFF-8639


Can anybody shed some light what the real problem is?

Is the hardware not properly detected or is the driver completely missing?

NVME PCIe architecture is out of my knowledge.

I really appreciate any help.

Kind regards
Miroslav Lachman
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PCIe NVME drives not detected on Dell R6515

Scott Long-2
Can you send me the output of ‘pciconf -llv’, either in 12-STABLE or 13-CURRENT?  Also, can you send me the output of ‘dmesg’?

Thanks,
Scott


> On Apr 17, 2020, at 5:23 AM, Miroslav Lachman <[hidden email]> wrote:
>
> I already asked on stable@ but as I tried it on 13-CURRENT with the same result I am trying to ask for help here.
>
> I have rented dedicated server Dell PowerEdge R6515 with iDRAC access only.
> There are 2 NVME drives which I wanted to use as ZFS root pool.
>
> They are shown in iDRAC
>
> Device Description:   PCIe SSD in Slot 1 in Bay 1
> Device Protocol:      NVMe-MI1.0
> Model:                Dell Express Flash NVMe P4510 1TB SFF
> Bus:                  130
> Manufacturer:         INTEL
> Product ID:           a54
> Revision:             VDV1DP23
> Enclosure:            PCIe SSD Backplane 1
>
>
> pciconf -l show many things, some of them are named "noneN@pci..." but none "nvme"
>
> The is printscreen (12.1 but 13-CURRENT is the same)
>
> https://ibb.co/tPnymL7
>
> But I booted Linux SystemRescueCd and nvme devices are there visible in /dev/
> https://ibb.co/sj22Nwg
>
> There is verbose output of Linux lspci https://ibb.co/dPZTwV1
>
> Linux dmesg contains:
> nvme nvme0: pci function 0000:81:00.0
> nvme nvme1: pci function 0000:82:00.0
> nvme nvme0: 32/0/0 default/read/poll queues
> nvme nvme1: 32/0/0 default/read/poll queues
>
>
> The machine is Dell PowerEdge R6515 with AMD EPYC 7302P
>
>
> More details extracted from iDRAC web interface
>
> I found this informations
>
> PCIe SSD in Slot 1 in Bay 1
> Bus:   82
> BusProtocol:  PCIE
> Device:  0
> DeviceDescription:  PCIe SSD in Slot 1 in Bay 1
> DeviceProtocol:  NVMe-MI1.0
> DeviceType:  PCIeSSD
> DriveFormFactor:  2.5 inch
> FailurePredicted:  NO
> FQDD:  Disk.Bay.1:Enclosure.Internal.0-1
> FreeSizeInBytes:  Information Not Available
> Function:  0
> HotSpareStatus:  Information Not Available
> InstanceID:  Disk.Bay.1:Enclosure.Internal.0-1
> Manufacturer:  INTEL
> MaximumCapableSpeed:  8 GT/s
> MediaType:  Solid State Drive
> Model:  Dell Express Flash NVMe P4510 1TB SFF
> NegotiatedSpeed:  8 GT/s
> PCIeCapableLinkWidth:  x4
> PCIeNegotiatedLinkWidth:  x4
> PrimaryStatus:  Ok
> ProductID:  a54
> RaidStatus:  Information Not Available
> RAIDType:  Unknown
> RemainingRatedWriteEndurance:  100 %
> Revision:  VDV1DP23
> SerialNumber:  PHLJxxxxxxWF1PxxxxN
> SizeInBytes:  1000204886016
> Slot:  1
> State:  Ready
> SystemEraseCapability:  CryptographicErasePD
>
> PCIe SSD in Slot 1 in Bay 1 - PCI Device
> BusNumber:  130
> DataBusWidth:  4x or x4
> Description:  Express Flash NVMe 1.0 TB 2.5" U.2 (P4510)
> DeviceDescription:  PCIe SSD in Slot 1 in Bay 1
> DeviceNumber:  0
> DeviceType:  PCIDevice
> FQDD:  Disk.Bay.1:Enclosure.Internal.0-1
> FunctionNumber:  0
> InstanceID:  Disk.Bay.1:Enclosure.Internal.0-1
> LastSystemInventoryTime:  2020-04-17T03:21:39
> LastUpdateTime:  2020-03-31T13:55:06
> Manufacturer:  Intel Corporation
> PCIDeviceID:  0A54
> PCISubDeviceID:  2003
> PCISubVendorID:  1028
> PCIVendorID:  8086
> SlotLength:  2.5 Inch Drive Form Factor
> SlotType:  PCI Express Gen 3 SFF-8639
>
>
> Can anybody shed some light what the real problem is?
>
> Is the hardware not properly detected or is the driver completely missing?
>
> NVME PCIe architecture is out of my knowledge.
>
> I really appreciate any help.
>
> Kind regards
> Miroslav Lachman
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "[hidden email]"

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PCIe NVME drives not detected on Dell R6515

Miroslav Lachman
Scott Long wrote on 04/17/2020 17:38:
> Can you send me the output of ‘pciconf -llv’, either in 12-STABLE or 13-CURRENT?  Also, can you send me the output of ‘dmesg’?

I have only iDRAC access to this machine, I cannot copy or export text.
Can I send you printscreen images?

Thank you!

Miroslav Lachman

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PCIe NVME drives not detected on Dell R6515

Warner Losh
In reply to this post by Scott Long-2
On Fri, Apr 17, 2020 at 9:39 AM Scott Long <[hidden email]> wrote:

> Can you send me the output of ‘pciconf -llv’, either in 12-STABLE or
> 13-CURRENT?  Also, can you send me the output of ‘dmesg’?
>

There was another thread that said there was a raid card in the way... It
would be cool to find a way to get it out of the way... :)

Warner


> Thanks,
> Scott
>
>
> > On Apr 17, 2020, at 5:23 AM, Miroslav Lachman <[hidden email]> wrote:
> >
> > I already asked on stable@ but as I tried it on 13-CURRENT with the
> same result I am trying to ask for help here.
> >
> > I have rented dedicated server Dell PowerEdge R6515 with iDRAC access
> only.
> > There are 2 NVME drives which I wanted to use as ZFS root pool.
> >
> > They are shown in iDRAC
> >
> > Device Description:   PCIe SSD in Slot 1 in Bay 1
> > Device Protocol:      NVMe-MI1.0
> > Model:                Dell Express Flash NVMe P4510 1TB SFF
> > Bus:                  130
> > Manufacturer:         INTEL
> > Product ID:           a54
> > Revision:             VDV1DP23
> > Enclosure:            PCIe SSD Backplane 1
> >
> >
> > pciconf -l show many things, some of them are named "noneN@pci..." but
> none "nvme"
> >
> > The is printscreen (12.1 but 13-CURRENT is the same)
> >
> > https://ibb.co/tPnymL7
> >
> > But I booted Linux SystemRescueCd and nvme devices are there visible in
> /dev/
> > https://ibb.co/sj22Nwg
> >
> > There is verbose output of Linux lspci https://ibb.co/dPZTwV1
> >
> > Linux dmesg contains:
> > nvme nvme0: pci function 0000:81:00.0
> > nvme nvme1: pci function 0000:82:00.0
> > nvme nvme0: 32/0/0 default/read/poll queues
> > nvme nvme1: 32/0/0 default/read/poll queues
> >
> >
> > The machine is Dell PowerEdge R6515 with AMD EPYC 7302P
> >
> >
> > More details extracted from iDRAC web interface
> >
> > I found this informations
> >
> > PCIe SSD in Slot 1 in Bay 1
> > Bus:   82
> > BusProtocol:  PCIE
> > Device:  0
> > DeviceDescription:  PCIe SSD in Slot 1 in Bay 1
> > DeviceProtocol:  NVMe-MI1.0
> > DeviceType:  PCIeSSD
> > DriveFormFactor:  2.5 inch
> > FailurePredicted:  NO
> > FQDD:  Disk.Bay.1:Enclosure.Internal.0-1
> > FreeSizeInBytes:  Information Not Available
> > Function:  0
> > HotSpareStatus:  Information Not Available
> > InstanceID:  Disk.Bay.1:Enclosure.Internal.0-1
> > Manufacturer:  INTEL
> > MaximumCapableSpeed:  8 GT/s
> > MediaType:  Solid State Drive
> > Model:  Dell Express Flash NVMe P4510 1TB SFF
> > NegotiatedSpeed:  8 GT/s
> > PCIeCapableLinkWidth:  x4
> > PCIeNegotiatedLinkWidth:  x4
> > PrimaryStatus:  Ok
> > ProductID:  a54
> > RaidStatus:  Information Not Available
> > RAIDType:  Unknown
> > RemainingRatedWriteEndurance:  100 %
> > Revision:  VDV1DP23
> > SerialNumber:  PHLJxxxxxxWF1PxxxxN
> > SizeInBytes:  1000204886016
> > Slot:  1
> > State:  Ready
> > SystemEraseCapability:  CryptographicErasePD
> >
> > PCIe SSD in Slot 1 in Bay 1 - PCI Device
> > BusNumber:  130
> > DataBusWidth:  4x or x4
> > Description:  Express Flash NVMe 1.0 TB 2.5" U.2 (P4510)
> > DeviceDescription:  PCIe SSD in Slot 1 in Bay 1
> > DeviceNumber:  0
> > DeviceType:  PCIDevice
> > FQDD:  Disk.Bay.1:Enclosure.Internal.0-1
> > FunctionNumber:  0
> > InstanceID:  Disk.Bay.1:Enclosure.Internal.0-1
> > LastSystemInventoryTime:  2020-04-17T03:21:39
> > LastUpdateTime:  2020-03-31T13:55:06
> > Manufacturer:  Intel Corporation
> > PCIDeviceID:  0A54
> > PCISubDeviceID:  2003
> > PCISubVendorID:  1028
> > PCIVendorID:  8086
> > SlotLength:  2.5 Inch Drive Form Factor
> > SlotType:  PCI Express Gen 3 SFF-8639
> >
> >
> > Can anybody shed some light what the real problem is?
> >
> > Is the hardware not properly detected or is the driver completely
> missing?
> >
> > NVME PCIe architecture is out of my knowledge.
> >
> > I really appreciate any help.
> >
> > Kind regards
> > Miroslav Lachman
> > _______________________________________________
> > [hidden email] mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-current
> > To unsubscribe, send any mail to "
> [hidden email]"
>
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "[hidden email]"
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PCIe NVME drives not detected on Dell R6515

Scott Long-2
Would that be the Intel VMD/VROC stuff?  If so, there’s a driver for FreeBSD, but it’s not well tested yet.  Will have to dig in further.

Scott


> On Apr 17, 2020, at 9:50 AM, Warner Losh <[hidden email]> wrote:
>
>
>
> On Fri, Apr 17, 2020 at 9:39 AM Scott Long <[hidden email]> wrote:
> Can you send me the output of ‘pciconf -llv’, either in 12-STABLE or 13-CURRENT?  Also, can you send me the output of ‘dmesg’?
>
> There was another thread that said there was a raid card in the way... It would be cool to find a way to get it out of the way... :)
>
> Warner
>  
> Thanks,
> Scott
>
>
> > On Apr 17, 2020, at 5:23 AM, Miroslav Lachman <[hidden email]> wrote:
> >
> > I already asked on stable@ but as I tried it on 13-CURRENT with the same result I am trying to ask for help here.
> >
> > I have rented dedicated server Dell PowerEdge R6515 with iDRAC access only.
> > There are 2 NVME drives which I wanted to use as ZFS root pool.
> >
> > They are shown in iDRAC
> >
> > Device Description:   PCIe SSD in Slot 1 in Bay 1
> > Device Protocol:      NVMe-MI1.0
> > Model:                Dell Express Flash NVMe P4510 1TB SFF
> > Bus:                  130
> > Manufacturer:         INTEL
> > Product ID:           a54
> > Revision:             VDV1DP23
> > Enclosure:            PCIe SSD Backplane 1
> >
> >
> > pciconf -l show many things, some of them are named "noneN@pci..." but none "nvme"
> >
> > The is printscreen (12.1 but 13-CURRENT is the same)
> >
> > https://ibb.co/tPnymL7
> >
> > But I booted Linux SystemRescueCd and nvme devices are there visible in /dev/
> > https://ibb.co/sj22Nwg
> >
> > There is verbose output of Linux lspci https://ibb.co/dPZTwV1
> >
> > Linux dmesg contains:
> > nvme nvme0: pci function 0000:81:00.0
> > nvme nvme1: pci function 0000:82:00.0
> > nvme nvme0: 32/0/0 default/read/poll queues
> > nvme nvme1: 32/0/0 default/read/poll queues
> >
> >
> > The machine is Dell PowerEdge R6515 with AMD EPYC 7302P
> >
> >
> > More details extracted from iDRAC web interface
> >
> > I found this informations
> >
> > PCIe SSD in Slot 1 in Bay 1
> > Bus:   82
> > BusProtocol:  PCIE
> > Device:  0
> > DeviceDescription:  PCIe SSD in Slot 1 in Bay 1
> > DeviceProtocol:  NVMe-MI1.0
> > DeviceType:  PCIeSSD
> > DriveFormFactor:  2.5 inch
> > FailurePredicted:  NO
> > FQDD:  Disk.Bay.1:Enclosure.Internal.0-1
> > FreeSizeInBytes:  Information Not Available
> > Function:  0
> > HotSpareStatus:  Information Not Available
> > InstanceID:  Disk.Bay.1:Enclosure.Internal.0-1
> > Manufacturer:  INTEL
> > MaximumCapableSpeed:  8 GT/s
> > MediaType:  Solid State Drive
> > Model:  Dell Express Flash NVMe P4510 1TB SFF
> > NegotiatedSpeed:  8 GT/s
> > PCIeCapableLinkWidth:  x4
> > PCIeNegotiatedLinkWidth:  x4
> > PrimaryStatus:  Ok
> > ProductID:  a54
> > RaidStatus:  Information Not Available
> > RAIDType:  Unknown
> > RemainingRatedWriteEndurance:  100 %
> > Revision:  VDV1DP23
> > SerialNumber:  PHLJxxxxxxWF1PxxxxN
> > SizeInBytes:  1000204886016
> > Slot:  1
> > State:  Ready
> > SystemEraseCapability:  CryptographicErasePD
> >
> > PCIe SSD in Slot 1 in Bay 1 - PCI Device
> > BusNumber:  130
> > DataBusWidth:  4x or x4
> > Description:  Express Flash NVMe 1.0 TB 2.5" U.2 (P4510)
> > DeviceDescription:  PCIe SSD in Slot 1 in Bay 1
> > DeviceNumber:  0
> > DeviceType:  PCIDevice
> > FQDD:  Disk.Bay.1:Enclosure.Internal.0-1
> > FunctionNumber:  0
> > InstanceID:  Disk.Bay.1:Enclosure.Internal.0-1
> > LastSystemInventoryTime:  2020-04-17T03:21:39
> > LastUpdateTime:  2020-03-31T13:55:06
> > Manufacturer:  Intel Corporation
> > PCIDeviceID:  0A54
> > PCISubDeviceID:  2003
> > PCISubVendorID:  1028
> > PCIVendorID:  8086
> > SlotLength:  2.5 Inch Drive Form Factor
> > SlotType:  PCI Express Gen 3 SFF-8639
> >
> >
> > Can anybody shed some light what the real problem is?
> >
> > Is the hardware not properly detected or is the driver completely missing?
> >
> > NVME PCIe architecture is out of my knowledge.
> >
> > I really appreciate any help.
> >
> > Kind regards
> > Miroslav Lachman
> > _______________________________________________
> > [hidden email] mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-current
> > To unsubscribe, send any mail to "[hidden email]"
>
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "[hidden email]"

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PCIe NVME drives not detected on Dell R6515

Miroslav Lachman
Scott Long wrote on 04/17/2020 17:54:
> Would that be the Intel VMD/VROC stuff?  If so, there’s a driver for FreeBSD, but it’s not well tested yet.  Will have to dig in further.

This is AMD EPYC machine. Isn't VROC Intel only thing?

>> On Apr 17, 2020, at 9:50 AM, Warner Losh <[hidden email]> wrote:

>> On Fri, Apr 17, 2020 at 9:39 AM Scott Long <[hidden email]> wrote:
>> Can you send me the output of ‘pciconf -llv’, either in 12-STABLE or 13-CURRENT?  Also, can you send me the output of ‘dmesg’?
>>
>> There was another thread that said there was a raid card in the way... It would be cool to find a way to get it out of the way... :)

I didn't find any evidence of RAID card but maybe I cannot look for it.


Miroslav Lachman

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PCIe NVME drives not detected on Dell R6515

Warner Losh
In reply to this post by Scott Long-2
No. It was some kind of extra card (Perteq?) managing things in a Dell
server. It may be the VMD/VROC stuff, and that driver may be worth a shot,
but I'm thinking not.  It looks like Doug's code for that is in the tree,
though https://reviews.freebsd.org/rS353380. Or are other changes needed?
I've been blessed with either being able to turn this off, or  not having
to deal...

Warner

On Fri, Apr 17, 2020 at 9:54 AM Scott Long <[hidden email]> wrote:

> Would that be the Intel VMD/VROC stuff?  If so, there’s a driver for
> FreeBSD, but it’s not well tested yet.  Will have to dig in further.
>
> Scott
>
>
> > On Apr 17, 2020, at 9:50 AM, Warner Losh <[hidden email]> wrote:
> >
> >
> >
> > On Fri, Apr 17, 2020 at 9:39 AM Scott Long <[hidden email]> wrote:
> > Can you send me the output of ‘pciconf -llv’, either in 12-STABLE or
> 13-CURRENT?  Also, can you send me the output of ‘dmesg’?
> >
> > There was another thread that said there was a raid card in the way...
> It would be cool to find a way to get it out of the way... :)
> >
> > Warner
> >
> > Thanks,
> > Scott
> >
> >
> > > On Apr 17, 2020, at 5:23 AM, Miroslav Lachman <[hidden email]>
> wrote:
> > >
> > > I already asked on stable@ but as I tried it on 13-CURRENT with the
> same result I am trying to ask for help here.
> > >
> > > I have rented dedicated server Dell PowerEdge R6515 with iDRAC access
> only.
> > > There are 2 NVME drives which I wanted to use as ZFS root pool.
> > >
> > > They are shown in iDRAC
> > >
> > > Device Description:   PCIe SSD in Slot 1 in Bay 1
> > > Device Protocol:      NVMe-MI1.0
> > > Model:                Dell Express Flash NVMe P4510 1TB SFF
> > > Bus:                  130
> > > Manufacturer:         INTEL
> > > Product ID:           a54
> > > Revision:             VDV1DP23
> > > Enclosure:            PCIe SSD Backplane 1
> > >
> > >
> > > pciconf -l show many things, some of them are named "noneN@pci..."
> but none "nvme"
> > >
> > > The is printscreen (12.1 but 13-CURRENT is the same)
> > >
> > > https://ibb.co/tPnymL7
> > >
> > > But I booted Linux SystemRescueCd and nvme devices are there visible
> in /dev/
> > > https://ibb.co/sj22Nwg
> > >
> > > There is verbose output of Linux lspci https://ibb.co/dPZTwV1
> > >
> > > Linux dmesg contains:
> > > nvme nvme0: pci function 0000:81:00.0
> > > nvme nvme1: pci function 0000:82:00.0
> > > nvme nvme0: 32/0/0 default/read/poll queues
> > > nvme nvme1: 32/0/0 default/read/poll queues
> > >
> > >
> > > The machine is Dell PowerEdge R6515 with AMD EPYC 7302P
> > >
> > >
> > > More details extracted from iDRAC web interface
> > >
> > > I found this informations
> > >
> > > PCIe SSD in Slot 1 in Bay 1
> > > Bus:   82
> > > BusProtocol:  PCIE
> > > Device:  0
> > > DeviceDescription:  PCIe SSD in Slot 1 in Bay 1
> > > DeviceProtocol:  NVMe-MI1.0
> > > DeviceType:  PCIeSSD
> > > DriveFormFactor:  2.5 inch
> > > FailurePredicted:  NO
> > > FQDD:  Disk.Bay.1:Enclosure.Internal.0-1
> > > FreeSizeInBytes:  Information Not Available
> > > Function:  0
> > > HotSpareStatus:  Information Not Available
> > > InstanceID:  Disk.Bay.1:Enclosure.Internal.0-1
> > > Manufacturer:  INTEL
> > > MaximumCapableSpeed:  8 GT/s
> > > MediaType:  Solid State Drive
> > > Model:  Dell Express Flash NVMe P4510 1TB SFF
> > > NegotiatedSpeed:  8 GT/s
> > > PCIeCapableLinkWidth:  x4
> > > PCIeNegotiatedLinkWidth:  x4
> > > PrimaryStatus:  Ok
> > > ProductID:  a54
> > > RaidStatus:  Information Not Available
> > > RAIDType:  Unknown
> > > RemainingRatedWriteEndurance:  100 %
> > > Revision:  VDV1DP23
> > > SerialNumber:  PHLJxxxxxxWF1PxxxxN
> > > SizeInBytes:  1000204886016
> > > Slot:  1
> > > State:  Ready
> > > SystemEraseCapability:  CryptographicErasePD
> > >
> > > PCIe SSD in Slot 1 in Bay 1 - PCI Device
> > > BusNumber:  130
> > > DataBusWidth:  4x or x4
> > > Description:  Express Flash NVMe 1.0 TB 2.5" U.2 (P4510)
> > > DeviceDescription:  PCIe SSD in Slot 1 in Bay 1
> > > DeviceNumber:  0
> > > DeviceType:  PCIDevice
> > > FQDD:  Disk.Bay.1:Enclosure.Internal.0-1
> > > FunctionNumber:  0
> > > InstanceID:  Disk.Bay.1:Enclosure.Internal.0-1
> > > LastSystemInventoryTime:  2020-04-17T03:21:39
> > > LastUpdateTime:  2020-03-31T13:55:06
> > > Manufacturer:  Intel Corporation
> > > PCIDeviceID:  0A54
> > > PCISubDeviceID:  2003
> > > PCISubVendorID:  1028
> > > PCIVendorID:  8086
> > > SlotLength:  2.5 Inch Drive Form Factor
> > > SlotType:  PCI Express Gen 3 SFF-8639
> > >
> > >
> > > Can anybody shed some light what the real problem is?
> > >
> > > Is the hardware not properly detected or is the driver completely
> missing?
> > >
> > > NVME PCIe architecture is out of my knowledge.
> > >
> > > I really appreciate any help.
> > >
> > > Kind regards
> > > Miroslav Lachman
> > > _______________________________________________
> > > [hidden email] mailing list
> > > https://lists.freebsd.org/mailman/listinfo/freebsd-current
> > > To unsubscribe, send any mail to "
> [hidden email]"
> >
> > _______________________________________________
> > [hidden email] mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-current
> > To unsubscribe, send any mail to "
> [hidden email]"
>
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PCIe NVME drives not detected on Dell R6515

Scott Long-2
In reply to this post by Scott Long-2
You are correct about Intel vs AMD.  Comparing the full output of pciconf from FreeBSD with the fragment of lspci from Linux suggests that there’s at least one set of a PCIe switch and child devices that is not being enumerated by FreeBSD.  Can you send the full output of `lspci -tvv` from linux?

Thanks,
Scott


> On Apr 17, 2020, at 9:54 AM, Scott Long <[hidden email]> wrote:
>
> Would that be the Intel VMD/VROC stuff?  If so, there’s a driver for FreeBSD, but it’s not well tested yet.  Will have to dig in further.
>
> Scott
>
>
>> On Apr 17, 2020, at 9:50 AM, Warner Losh <[hidden email]> wrote:
>>
>>
>>
>> On Fri, Apr 17, 2020 at 9:39 AM Scott Long <[hidden email]> wrote:
>> Can you send me the output of ‘pciconf -llv’, either in 12-STABLE or 13-CURRENT?  Also, can you send me the output of ‘dmesg’?
>>
>> There was another thread that said there was a raid card in the way... It would be cool to find a way to get it out of the way... :)
>>
>> Warner
>>
>> Thanks,
>> Scott
>>
>>
>>> On Apr 17, 2020, at 5:23 AM, Miroslav Lachman <[hidden email]> wrote:
>>>
>>> I already asked on stable@ but as I tried it on 13-CURRENT with the same result I am trying to ask for help here.
>>>
>>> I have rented dedicated server Dell PowerEdge R6515 with iDRAC access only.
>>> There are 2 NVME drives which I wanted to use as ZFS root pool.
>>>
>>> They are shown in iDRAC
>>>
>>> Device Description:   PCIe SSD in Slot 1 in Bay 1
>>> Device Protocol:      NVMe-MI1.0
>>> Model:                Dell Express Flash NVMe P4510 1TB SFF
>>> Bus:                  130
>>> Manufacturer:         INTEL
>>> Product ID:           a54
>>> Revision:             VDV1DP23
>>> Enclosure:            PCIe SSD Backplane 1
>>>
>>>
>>> pciconf -l show many things, some of them are named "noneN@pci..." but none "nvme"
>>>
>>> The is printscreen (12.1 but 13-CURRENT is the same)
>>>
>>> https://ibb.co/tPnymL7
>>>
>>> But I booted Linux SystemRescueCd and nvme devices are there visible in /dev/
>>> https://ibb.co/sj22Nwg
>>>
>>> There is verbose output of Linux lspci https://ibb.co/dPZTwV1
>>>
>>> Linux dmesg contains:
>>> nvme nvme0: pci function 0000:81:00.0
>>> nvme nvme1: pci function 0000:82:00.0
>>> nvme nvme0: 32/0/0 default/read/poll queues
>>> nvme nvme1: 32/0/0 default/read/poll queues
>>>
>>>
>>> The machine is Dell PowerEdge R6515 with AMD EPYC 7302P
>>>
>>>
>>> More details extracted from iDRAC web interface
>>>
>>> I found this informations
>>>
>>> PCIe SSD in Slot 1 in Bay 1
>>> Bus:   82
>>> BusProtocol:  PCIE
>>> Device:  0
>>> DeviceDescription:  PCIe SSD in Slot 1 in Bay 1
>>> DeviceProtocol:  NVMe-MI1.0
>>> DeviceType:  PCIeSSD
>>> DriveFormFactor:  2.5 inch
>>> FailurePredicted:  NO
>>> FQDD:  Disk.Bay.1:Enclosure.Internal.0-1
>>> FreeSizeInBytes:  Information Not Available
>>> Function:  0
>>> HotSpareStatus:  Information Not Available
>>> InstanceID:  Disk.Bay.1:Enclosure.Internal.0-1
>>> Manufacturer:  INTEL
>>> MaximumCapableSpeed:  8 GT/s
>>> MediaType:  Solid State Drive
>>> Model:  Dell Express Flash NVMe P4510 1TB SFF
>>> NegotiatedSpeed:  8 GT/s
>>> PCIeCapableLinkWidth:  x4
>>> PCIeNegotiatedLinkWidth:  x4
>>> PrimaryStatus:  Ok
>>> ProductID:  a54
>>> RaidStatus:  Information Not Available
>>> RAIDType:  Unknown
>>> RemainingRatedWriteEndurance:  100 %
>>> Revision:  VDV1DP23
>>> SerialNumber:  PHLJxxxxxxWF1PxxxxN
>>> SizeInBytes:  1000204886016
>>> Slot:  1
>>> State:  Ready
>>> SystemEraseCapability:  CryptographicErasePD
>>>
>>> PCIe SSD in Slot 1 in Bay 1 - PCI Device
>>> BusNumber:  130
>>> DataBusWidth:  4x or x4
>>> Description:  Express Flash NVMe 1.0 TB 2.5" U.2 (P4510)
>>> DeviceDescription:  PCIe SSD in Slot 1 in Bay 1
>>> DeviceNumber:  0
>>> DeviceType:  PCIDevice
>>> FQDD:  Disk.Bay.1:Enclosure.Internal.0-1
>>> FunctionNumber:  0
>>> InstanceID:  Disk.Bay.1:Enclosure.Internal.0-1
>>> LastSystemInventoryTime:  2020-04-17T03:21:39
>>> LastUpdateTime:  2020-03-31T13:55:06
>>> Manufacturer:  Intel Corporation
>>> PCIDeviceID:  0A54
>>> PCISubDeviceID:  2003
>>> PCISubVendorID:  1028
>>> PCIVendorID:  8086
>>> SlotLength:  2.5 Inch Drive Form Factor
>>> SlotType:  PCI Express Gen 3 SFF-8639
>>>
>>>
>>> Can anybody shed some light what the real problem is?
>>>
>>> Is the hardware not properly detected or is the driver completely missing?
>>>
>>> NVME PCIe architecture is out of my knowledge.
>>>
>>> I really appreciate any help.
>>>
>>> Kind regards
>>> Miroslav Lachman
>>> _______________________________________________
>>> [hidden email] mailing list
>>> https://lists.freebsd.org/mailman/listinfo/freebsd-current
>>> To unsubscribe, send any mail to "[hidden email]"
>>
>> _______________________________________________
>> [hidden email] mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-current
>> To unsubscribe, send any mail to "[hidden email]"
>
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "[hidden email]"

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PCIe NVME drives not detected on Dell R6515

Miroslav Lachman
In reply to this post by Warner Losh
Warner Losh wrote on 04/17/2020 18:15:
> No. It was some kind of extra card (Perteq?) managing things in a Dell
> server. It may be the VMD/VROC stuff, and that driver may be worth a shot,
> but I'm thinking not.  It looks like Doug's code for that is in the tree,
> though https://reviews.freebsd.org/rS353380. Or are other changes needed?
> I've been blessed with either being able to turn this off, or  not having
> to deal...

vmd is in the kernel of 13-CURRENT installer I tried but I don't see any
vmd device.

Miroslav Lachman

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PCIe NVME drives not detected on Dell R6515

Gary Palmer-2
In reply to this post by Miroslav Lachman
On Fri, Apr 17, 2020 at 05:40:55PM +0200, Miroslav Lachman wrote:
> Scott Long wrote on 04/17/2020 17:38:
> > Can you send me the output of ???pciconf -llv???, either in 12-STABLE or 13-CURRENT?  Also, can you send me the output of ???dmesg????
>
> I have only iDRAC access to this machine, I cannot copy or export text. Can
> I send you printscreen images?

You used to be able to ssh to the iDRAC IP and there was a magic command
that let you get to the serial console via SSH.  I don't have access to Dell
hardware any more to dig out the magic command, sorry

Regards,

Gary
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PCIe NVME drives not detected on Dell R6515

Miroslav Lachman
In reply to this post by Scott Long-2
Scott Long wrote on 04/17/2020 18:17:
> You are correct about Intel vs AMD.  Comparing the full output of pciconf from FreeBSD with the fragment of lspci from Linux suggests that there’s at least one set of a PCIe switch and child devices that is not being enumerated by FreeBSD.  Can you send the full output of `lspci -tvv` from linux?

Sorry for my late reply. Booting the Linux SystemRescueCd is too slow.

lspci -tvv output is attached

I tried to connect to SOL by SSH but it shows black screen only.

lspci shows drives:

Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller]

Kind regards
Miroslav Lachman


>> On Apr 17, 2020, at 9:54 AM, Scott Long <[hidden email]> wrote:
>>
>> Would that be the Intel VMD/VROC stuff?  If so, there’s a driver for FreeBSD, but it’s not well tested yet.  Will have to dig in further.
>>
>> Scott
>>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PCIe NVME drives not detected on Dell R6515

Scott Long-2


> On Apr 17, 2020, at 11:46 AM, Miroslav Lachman <[hidden email]> wrote:
>
> Scott Long wrote on 04/17/2020 18:17:
>> You are correct about Intel vs AMD.  Comparing the full output of pciconf from FreeBSD with the fragment of lspci from Linux suggests that there’s at least one set of a PCIe switch and child devices that is not being enumerated by FreeBSD.  Can you send the full output of `lspci -tvv` from linux?
>
> Sorry for my late reply. Booting the Linux SystemRescueCd is too slow.
>
> lspci -tvv output is attached
>
> I tried to connect to SOL by SSH but it shows black screen only.
>
> lspci shows drives:
>
> Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller]
>
> Kind regards
> Miroslav Lachman
>

It looks like pcib12 and pcib13 in FreeBSD should be the bridges that have the NVMe devices behind them, but those devices aren’t showing up.  I don’t see any obvious code in linux to handle those bridges, so there must be something subtle that we’re missing in FreeBSD.  Maybe we’re having trouble enumerating above PCI bus 128?  I think that was a problem a few years ago but I also thought it was fixed.  Can you do the following in FreeBSD:

pciconf -lBc pcib12
pciconf -lBc pcib13

Thanks,
Scott

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PCIe NVME drives not detected on Dell R6515

Miroslav Lachman
Scott Long wrote on 04/17/2020 21:18:

>
>
>> On Apr 17, 2020, at 11:46 AM, Miroslav Lachman <[hidden email]> wrote:
>>
>> Scott Long wrote on 04/17/2020 18:17:
>>> You are correct about Intel vs AMD.  Comparing the full output of pciconf from FreeBSD with the fragment of lspci from Linux suggests that there’s at least one set of a PCIe switch and child devices that is not being enumerated by FreeBSD.  Can you send the full output of `lspci -tvv` from linux?
>>
>> Sorry for my late reply. Booting the Linux SystemRescueCd is too slow.
>>
>> lspci -tvv output is attached
>>
>> I tried to connect to SOL by SSH but it shows black screen only.
>>
>> lspci shows drives:
>>
>> Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller]
>>
>> Kind regards
>> Miroslav Lachman
>>
>
> It looks like pcib12 and pcib13 in FreeBSD should be the bridges that have the NVMe devices behind them, but those devices aren’t showing up.  I don’t see any obvious code in linux to handle those bridges, so there must be something subtle that we’re missing in FreeBSD.  Maybe we’re having trouble enumerating above PCI bus 128?  I think that was a problem a few years ago but I also thought it was fixed.  Can you do the following in FreeBSD:
>
> pciconf -lBc pcib12
> pciconf -lBc pcib13

Printscreen attached.

Anything else I can provide?

Thank you
Miroslav Lachman
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PCIe NVME drives not detected on Dell R6515

Kurt Jaeger-6
Hi!

> > pciconf -lBc pcib12
> > pciconf -lBc pcib13
>
> Printscreen attached.

Attachments are stripped from the list -- can you put them somewhere
online ?

--
[hidden email]            +49 171 3101372                    Now what ?
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PCIe NVME drives not detected on Dell R6515

Miroslav Lachman
Kurt Jaeger wrote on 04/17/2020 21:44:
> Hi!
>
>>> pciconf -lBc pcib12
>>> pciconf -lBc pcib13
>>
>> Printscreen attached.
>
> Attachments are stripped from the list -- can you put them somewhere
> online ?

Here it is https://ibb.co/c1dZrTf

Miroslav Lachman

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PCIe NVME drives not detected on Dell R6515

Scott Long-2


> On Apr 17, 2020, at 1:47 PM, Miroslav Lachman <[hidden email]> wrote:
>
> Kurt Jaeger wrote on 04/17/2020 21:44:
>> Hi!
>>>> pciconf -lBc pcib12
>>>> pciconf -lBc pcib13
>>>
>>> Printscreen attached.
>> Attachments are stripped from the list -- can you put them somewhere
>> online ?
>
> Here it is https://ibb.co/c1dZrTf
>
> Miroslav Lachman
>

Ok, the bridges know about their downstream bus numbers, but I see nothing that suggests that they’re being probed.  The next step would be bootverbose, but that’s going to be a lot of output to collect in screen captures.

Scott

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PCIe NVME drives not detected on Dell R6515

Miroslav Lachman
Scott Long wrote on 04/17/2020 22:17:

>
>
>> On Apr 17, 2020, at 1:47 PM, Miroslav Lachman <[hidden email]> wrote:
>>
>> Kurt Jaeger wrote on 04/17/2020 21:44:
>>> Hi!
>>>>> pciconf -lBc pcib12
>>>>> pciconf -lBc pcib13
>>>>
>>>> Printscreen attached.
>>> Attachments are stripped from the list -- can you put them somewhere
>>> online ?
>>
>> Here it is https://ibb.co/c1dZrTf
>>
>> Miroslav Lachman
>>
>
> Ok, the bridges know about their downstream bus numbers, but I see nothing that suggests that they’re being probed.  The next step would be bootverbose, but that’s going to be a lot of output to collect in screen captures.

Over 3000 lines long but I finally managed to make SOL work so I have it
as text!

https://pastebin.pl/view/90fdaafb

Miroslav Lachman


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PCIe NVME drives not detected on Dell R6515

Scott Long-2


> On Apr 17, 2020, at 2:45 PM, Miroslav Lachman <[hidden email]> wrote:
>
> Scott Long wrote on 04/17/2020 22:17:
>>> On Apr 17, 2020, at 1:47 PM, Miroslav Lachman <[hidden email]> wrote:
>>>
>>> Kurt Jaeger wrote on 04/17/2020 21:44:
>>>> Hi!
>>>>>> pciconf -lBc pcib12
>>>>>> pciconf -lBc pcib13
>>>>>
>>>>> Printscreen attached.
>>>> Attachments are stripped from the list -- can you put them somewhere
>>>> online ?
>>>
>>> Here it is https://ibb.co/c1dZrTf
>>>
>>> Miroslav Lachman
>>>
>> Ok, the bridges know about their downstream bus numbers, but I see nothing that suggests that they’re being probed.  The next step would be bootverbose, but that’s going to be a lot of output to collect in screen captures.
>
> Over 3000 lines long but I finally managed to make SOL work so I have it as text!
>
> https://pastebin.pl/view/90fdaafb
>

This helped a lot, thanks.  It looks like these PCIe buses are marked as being hotplug, and for some reason we’re not probing them.  At this point, I’d need to feed you some kernel patches that will dump out more info, but you’d have to compile them and get them onto your boot media.  Is that a possibility?

Thanks,
Scott


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PCIe NVME drives not detected on Dell R6515

Miroslav Lachman
Scott Long wrote on 04/17/2020 23:04:

>
>
>> On Apr 17, 2020, at 2:45 PM, Miroslav Lachman <[hidden email]> wrote:
>>
>> Scott Long wrote on 04/17/2020 22:17:
>>>> On Apr 17, 2020, at 1:47 PM, Miroslav Lachman <[hidden email]> wrote:
>>>>
>>>> Kurt Jaeger wrote on 04/17/2020 21:44:
>>>>> Hi!
>>>>>>> pciconf -lBc pcib12
>>>>>>> pciconf -lBc pcib13
>>>>>>
>>>>>> Printscreen attached.
>>>>> Attachments are stripped from the list -- can you put them somewhere
>>>>> online ?
>>>>
>>>> Here it is https://ibb.co/c1dZrTf
>>>>
>>>> Miroslav Lachman
>>>>
>>> Ok, the bridges know about their downstream bus numbers, but I see nothing that suggests that they’re being probed.  The next step would be bootverbose, but that’s going to be a lot of output to collect in screen captures.
>>
>> Over 3000 lines long but I finally managed to make SOL work so I have it as text!
>>
>> https://pastebin.pl/view/90fdaafb
>>
>
> This helped a lot, thanks.  It looks like these PCIe buses are marked as being hotplug, and for some reason we’re not probing them.  At this point, I’d need to feed you some kernel patches that will dump out more info, but you’d have to compile them and get them onto your boot media.  Is that a possibility?

Currently I have all machines on 11.3 (where I can rebuild kernel
without problem)
If CURRENT is required I would need to setup some CURRENT VM in VirtualBox.

Can you send me some link to documentation who should I create new ISO
after rebuild?

Miroslav Lachman
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PCIe NVME drives not detected on Dell R6515

Mel Pilgrim
In reply to this post by Warner Losh
On 2020-04-17 8:50, Warner Losh wrote:
> On Fri, Apr 17, 2020 at 9:39 AM Scott Long <[hidden email]> wrote:
>
>> Can you send me the output of ‘pciconf -llv’, either in 12-STABLE or
>> 13-CURRENT?  Also, can you send me the output of ‘dmesg’?
>>
>
> There was another thread that said there was a raid card in the way... It
> would be cool to find a way to get it out of the way... :)
On the R6515, NVME drives are supported through a PERC S150 mini card.
There's no m.2 slots or U.2 ports on the mainboard itself.  The S150
supports non-RAID mode, but that still leaves supporting a
software-based PERC card as a PCIe switch for the 8 or 10 NVME bays
behind it.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
12