Options for FBSD support with LCD device - new project

classic Classic list List threaded Threaded
67 messages Options
1234
Reply | Threaded
Open this post in threaded view
|

Re: I2c producing crazy console messages [[Re: insanely-high interrupt rates -- PARTIAL resolution (Pi2)]]

Ian Lepore-3
On Fri, 2019-04-19 at 13:32 +0200, Per Hedeland wrote:
> But anyway I would be *extremely* surprised if I saw them, since
> AFAIU
> the i2c bus per se has no concept of interrupts - you need to connect
> some other wire from the device to e.g. a gpio pin (with appropriate
> config) in order to generate interrupts - and I haven't done that.
> (The
> ads1015 does have an ALERT/RDY pin that could potentially be used for
> it, but since FreeBSD AFAIK doesn't have a way to deliver the
> interrupts to userland code, I had no interest in it.)

You're thinking about this all wrong.  The interrupts have nothing to
do with the i2c bus, but the i2c controller still uses interrupts to
signal things like "trasnfer done" or "fifo empty".  If there's nothing
on the bus, you don't end up doing any transfers, so you don't get any
spurious interrupts.

I can't reproduce this using RTC or EEPROM chips.  We use ADS1013 parts
at $work, I may have some around here somewhere.  If so, I'll see if I
can reproduce the problem using them.

-- Ian

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arm
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: I2c producing crazy console messages [[Re: insanely-high interrupt rates -- PARTIAL resolution (Pi2)]]

Per Hedeland
On 2019-04-19 18:04, Ian Lepore wrote:

> On Fri, 2019-04-19 at 13:32 +0200, Per Hedeland wrote:
>> But anyway I would be *extremely* surprised if I saw them, since
>> AFAIU
>> the i2c bus per se has no concept of interrupts - you need to connect
>> some other wire from the device to e.g. a gpio pin (with appropriate
>> config) in order to generate interrupts - and I haven't done that.
>> (The
>> ads1015 does have an ALERT/RDY pin that could potentially be used for
>> it, but since FreeBSD AFAIK doesn't have a way to deliver the
>> interrupts to userland code, I had no interest in it.)
>
> You're thinking about this all wrong.  The interrupts have nothing to
> do with the i2c bus, but the i2c controller still uses interrupts to
> signal things like "trasnfer done" or "fifo empty".

OK...

> If there's nothing
> on the bus, you don't end up doing any transfers, so you don't get any
> spurious interrupts.

That much is clear even to me, but I'm doing 128 transfers/second
without seeing any spurious interrupts. But I guess my observation
matches the second alternative in one of your earlier messages (where
the first alternative, as far as I understood, was about interrupts
from the i2c device itself...) - i.e. interrupts being dispatched to
multiple cores. This obviously can't happen on the RPi 0/1 that I've
tried, since they're single-core.

--Per

> I can't reproduce this using RTC or EEPROM chips.  We use ADS1013 parts
> at $work, I may have some around here somewhere.  If so, I'll see if I
> can reproduce the problem using them.
>
> -- Ian
>

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arm
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: I2c producing crazy console messages [[Re: insanely-high interrupt rates -- PARTIAL resolution (Pi2)]]

Ian Lepore-3
On Sat, 2019-04-20 at 12:39 +0200, Per Hedeland wrote:

> On 2019-04-19 18:04, Ian Lepore wrote:
> > On Fri, 2019-04-19 at 13:32 +0200, Per Hedeland wrote:
> > > But anyway I would be *extremely* surprised if I saw them, since
> > > AFAIU
> > > the i2c bus per se has no concept of interrupts - you need to
> > > connect
> > > some other wire from the device to e.g. a gpio pin (with
> > > appropriate
> > > config) in order to generate interrupts - and I haven't done
> > > that.
> > > (The
> > > ads1015 does have an ALERT/RDY pin that could potentially be used
> > > for
> > > it, but since FreeBSD AFAIK doesn't have a way to deliver the
> > > interrupts to userland code, I had no interest in it.)
> >
> > You're thinking about this all wrong.  The interrupts have nothing
> > to
> > do with the i2c bus, but the i2c controller still uses interrupts
> > to
> > signal things like "trasnfer done" or "fifo empty".
>
> OK...
>
> > If there's nothing
> > on the bus, you don't end up doing any transfers, so you don't get
> > any
> > spurious interrupts.
>
> That much is clear even to me, but I'm doing 128 transfers/second
> without seeing any spurious interrupts. But I guess my observation
> matches the second alternative in one of your earlier messages (where
> the first alternative, as far as I understood, was about interrupts
> from the i2c device itself...) - i.e. interrupts being dispatched to
> multiple cores. This obviously can't happen on the RPi 0/1 that I've
> tried, since they're single-core.
>
>

Exactly so.  I did all my recent rpi i2c driver work using an old rpi1
so I never saw the problem.  Now I've got an rpi2, and oddly enough,
last night I finally saw a few spurious interrupts while writing to an
RTC chip (but never while reading from one).  I also now have an
ADS1015 in hand (gotta love amazon free same-day delivery).  So
hopefully today I can track this down, although without something like
jtag debugging, the only way to know if it's the "multiple cores
signalled at once" situation is to eliminate all other possibilities.

-- Ian

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arm
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: insanely-high interrupt rates -- PARTIAL resolution (Pi2)

Ian Lepore-3
In reply to this post by Karl Denninger
On Wed, 2019-04-17 at 14:56 -0500, Karl Denninger wrote:

> On 4/9/2019 19:25, Ian Lepore wrote:
> > On Tue, 2019-04-09 at 09:55 -0500, Karl Denninger wrote:
> > > On 4/3/2019 11:48, Andrew Gierth wrote:
> > > > [...]
> >
> > I've just posted https://reviews.freebsd.org/D19871 for this.
> > Hopefully I'll get it committed in a day or so and merged to 12-
> > stable
> > a few days after that.
> >
> > -- Ian
>
> I am running that now on a Pi2 and so far the load problem is gone
> but
> the spurious interrupt warnings are not....
>
>
[...]

>
> On my bench without the I2c inputs connected (which do analog reads) I
> do NOT get the spurious interrupt prints.  With it connected I do.  The
> process that reads them is code that is running in both cases, but if it
> cannot find the I2c devices it logs the error but continues, so all it
> gets to is trying to open the unit, doesn't see it when probed, and
> gives up.
>
> It appears that I2c is an inherent part of the spurious interrupt thing
> still and while the timer issue appears to be fixed that doesn't resolve
> the other problem.
>
> Any ideas on how to track down exactly what is generating those warnings?
>
>

After spending the whole day yesterday trying all the usual driver
techniques for eliminating spurious interrupts, I was unable to make
them go away completely, but I also convinced myself they're harmless.

I was a little surprised that the "read after write" technique didn't
work.  That is, after writing to the i2c control register to clear all
the interrupt-enable bits, read back that register.  In theory, at
least on normal arm chips, that ensures that the prior write has
reached the hardware before the read can procede, so it's a way to
guarantee that the write has taken effect and the interrupt can no
longer be asserted, before returning from the interrupt handler.  But,
on the rpi chips even that doesn't work... you can read back the
register and verify the interrupt-enable bits are cleared, and still
after returning from the handler, it re-interrupts immediately.

If you stick in a nice long DELAY() after clearing the control
register, the spurious interrupts go away, but that's a horrible fix.
It would be especially horrible for i2c devices that do a lot of
transfers, you'd end up with the delay time overwhelming the time to do
the actual transfers themselves.

So, in r346489, I moved the reporting of the spurious transfers under
the bootverbose flag, so that normally you just won't see them anymore,
but we can still enable the reporting if we suspect some device driver
is behaving badly.  I'll mfc that change to 12-stable after a few days.

-- Ian

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arm
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: insanely-high interrupt rates -- PARTIAL resolution (Pi2)

Karl Denninger

On 4/21/2019 12:57, Ian Lepore wrote:

> On Wed, 2019-04-17 at 14:56 -0500, Karl Denninger wrote:
>> On 4/9/2019 19:25, Ian Lepore wrote:
>>> On Tue, 2019-04-09 at 09:55 -0500, Karl Denninger wrote:
>>>> On 4/3/2019 11:48, Andrew Gierth wrote:
>>>>> [...]
>>> I've just posted https://reviews.freebsd.org/D19871 for this.
>>> Hopefully I'll get it committed in a day or so and merged to 12-
>>> stable
>>> a few days after that.
>>>
>>> -- Ian
>> I am running that now on a Pi2 and so far the load problem is gone
>> but
>> the spurious interrupt warnings are not....
>>
>>
> [...]
>> On my bench without the I2c inputs connected (which do analog reads) I
>> do NOT get the spurious interrupt prints.  With it connected I do.  The
>> process that reads them is code that is running in both cases, but if it
>> cannot find the I2c devices it logs the error but continues, so all it
>> gets to is trying to open the unit, doesn't see it when probed, and
>> gives up.
>>
>> It appears that I2c is an inherent part of the spurious interrupt thing
>> still and while the timer issue appears to be fixed that doesn't resolve
>> the other problem.
>>
>> Any ideas on how to track down exactly what is generating those warnings?
>>
>>
> After spending the whole day yesterday trying all the usual driver
> techniques for eliminating spurious interrupts, I was unable to make
> them go away completely, but I also convinced myself they're harmless.
>
> I was a little surprised that the "read after write" technique didn't
> work.  That is, after writing to the i2c control register to clear all
> the interrupt-enable bits, read back that register.  In theory, at
> least on normal arm chips, that ensures that the prior write has
> reached the hardware before the read can procede, so it's a way to
> guarantee that the write has taken effect and the interrupt can no
> longer be asserted, before returning from the interrupt handler.  But,
> on the rpi chips even that doesn't work... you can read back the
> register and verify the interrupt-enable bits are cleared, and still
> after returning from the handler, it re-interrupts immediately.
>
> If you stick in a nice long DELAY() after clearing the control
> register, the spurious interrupts go away, but that's a horrible fix.
> It would be especially horrible for i2c devices that do a lot of
> transfers, you'd end up with the delay time overwhelming the time to do
> the actual transfers themselves.
>
> So, in r346489, I moved the reporting of the spurious transfers under
> the bootverbose flag, so that normally you just won't see them anymore,
> but we can still enable the reporting if we suspect some device driver
> is behaving badly.  I'll mfc that change to 12-stable after a few days.
>
> -- Ian
Thanks...


--
Karl Denninger
[hidden email] <mailto:[hidden email]>
/The Market Ticker/
/[S/MIME encrypted email preferred]/

smime.p7s (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: insanely-high interrupt rates -- PARTIAL resolution (Pi2)

Warner Losh
In reply to this post by Ian Lepore-3
On Sun, Apr 21, 2019 at 11:58 AM Ian Lepore <[hidden email]> wrote:

> On Wed, 2019-04-17 at 14:56 -0500, Karl Denninger wrote:
> > On 4/9/2019 19:25, Ian Lepore wrote:
> > > On Tue, 2019-04-09 at 09:55 -0500, Karl Denninger wrote:
> > > > On 4/3/2019 11:48, Andrew Gierth wrote:
> > > > > [...]
> > >
> > > I've just posted https://reviews.freebsd.org/D19871 for this.
> > > Hopefully I'll get it committed in a day or so and merged to 12-
> > > stable
> > > a few days after that.
> > >
> > > -- Ian
> >
> > I am running that now on a Pi2 and so far the load problem is gone
> > but
> > the spurious interrupt warnings are not....
> >
> >
> [...]
> >
> > On my bench without the I2c inputs connected (which do analog reads) I
> > do NOT get the spurious interrupt prints.  With it connected I do.  The
> > process that reads them is code that is running in both cases, but if it
> > cannot find the I2c devices it logs the error but continues, so all it
> > gets to is trying to open the unit, doesn't see it when probed, and
> > gives up.
> >
> > It appears that I2c is an inherent part of the spurious interrupt thing
> > still and while the timer issue appears to be fixed that doesn't resolve
> > the other problem.
> >
> > Any ideas on how to track down exactly what is generating those warnings?
> >
> >
>
> After spending the whole day yesterday trying all the usual driver
> techniques for eliminating spurious interrupts, I was unable to make
> them go away completely, but I also convinced myself they're harmless.
>
> I was a little surprised that the "read after write" technique didn't
> work.  That is, after writing to the i2c control register to clear all
> the interrupt-enable bits, read back that register.  In theory, at
> least on normal arm chips, that ensures that the prior write has
> reached the hardware before the read can procede, so it's a way to
> guarantee that the write has taken effect and the interrupt can no
> longer be asserted, before returning from the interrupt handler.  But,
> on the rpi chips even that doesn't work... you can read back the
> register and verify the interrupt-enable bits are cleared, and still
> after returning from the handler, it re-interrupts immediately.
>
> If you stick in a nice long DELAY() after clearing the control
> register, the spurious interrupts go away, but that's a horrible fix.
> It would be especially horrible for i2c devices that do a lot of
> transfers, you'd end up with the delay time overwhelming the time to do
> the actual transfers themselves.
>
> So, in r346489, I moved the reporting of the spurious transfers under
> the bootverbose flag, so that normally you just won't see them anymore,
> but we can still enable the reporting if we suspect some device driver
> is behaving badly.  I'll mfc that change to 12-stable after a few days.
>

vmstat -i will also show if you're system has an unusually high interrupt
rate in general as well, and is preferable to spamming the console with
printfs :)

Warner
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arm
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: insanely-high interrupt rates -- PARTIAL resolution (Pi2)

Ian Lepore-3
On Sun, 2019-04-21 at 12:00 -0600, Warner Losh wrote:

> On Sun, Apr 21, 2019 at 11:58 AM Ian Lepore <[hidden email]> wrote:
>
> > On Wed, 2019-04-17 at 14:56 -0500, Karl Denninger wrote:
> > > On 4/9/2019 19:25, Ian Lepore wrote:
> > > > On Tue, 2019-04-09 at 09:55 -0500, Karl Denninger wrote:
> > > > > On 4/3/2019 11:48, Andrew Gierth wrote:
> > > > > > [...]
> > > >
> > > > I've just posted https://reviews.freebsd.org/D19871 for this.
> > > > Hopefully I'll get it committed in a day or so and merged to
> > > > 12-
> > > > stable
> > > > a few days after that.
> > > >
> > > > -- Ian
> > >
> > > I am running that now on a Pi2 and so far the load problem is
> > > gone
> > > but
> > > the spurious interrupt warnings are not....
> > >
> > >
> >
> > [...]
> > >
> > > On my bench without the I2c inputs connected (which do analog
> > > reads) I
> > > do NOT get the spurious interrupt prints.  With it connected I
> > > do.  The
> > > process that reads them is code that is running in both cases,
> > > but if it
> > > cannot find the I2c devices it logs the error but continues, so
> > > all it
> > > gets to is trying to open the unit, doesn't see it when probed,
> > > and
> > > gives up.
> > >
> > > It appears that I2c is an inherent part of the spurious interrupt
> > > thing
> > > still and while the timer issue appears to be fixed that doesn't
> > > resolve
> > > the other problem.
> > >
> > > Any ideas on how to track down exactly what is generating those
> > > warnings?
> > >
> > >
> >
> > After spending the whole day yesterday trying all the usual driver
> > techniques for eliminating spurious interrupts, I was unable to
> > make
> > them go away completely, but I also convinced myself they're
> > harmless.
> >
> > I was a little surprised that the "read after write" technique
> > didn't
> > work.  That is, after writing to the i2c control register to clear
> > all
> > the interrupt-enable bits, read back that register.  In theory, at
> > least on normal arm chips, that ensures that the prior write has
> > reached the hardware before the read can procede, so it's a way to
> > guarantee that the write has taken effect and the interrupt can no
> > longer be asserted, before returning from the interrupt
> > handler.  But,
> > on the rpi chips even that doesn't work... you can read back the
> > register and verify the interrupt-enable bits are cleared, and
> > still
> > after returning from the handler, it re-interrupts immediately.
> >
> > If you stick in a nice long DELAY() after clearing the control
> > register, the spurious interrupts go away, but that's a horrible
> > fix.
> > It would be especially horrible for i2c devices that do a lot of
> > transfers, you'd end up with the delay time overwhelming the time
> > to do
> > the actual transfers themselves.
> >
> > So, in r346489, I moved the reporting of the spurious transfers
> > under
> > the bootverbose flag, so that normally you just won't see them
> > anymore,
> > but we can still enable the reporting if we suspect some device
> > driver
> > is behaving badly.  I'll mfc that change to 12-stable after a few
> > days.
> >
>
> vmstat -i will also show if you're system has an unusually high interrupt
> rate in general as well, and is preferable to spamming the console with
> printfs :)
>
> Warner

vmstat doesn't report spurious interrupts in any way, though.  I
considered making it do so as one of the possible fixes here, but it
turns out to be complicated... we need to do a bit of reworking of the
INTRNG code as it related to interrupt counts.  For example, on x86 you
get this from vmstat -i:

cpu0:timer                      42006521         80
cpu1:timer                      32510560         62

But on arm, all timer interrupts are counted as belonging to the
generic_timer0 device.  When I tried to figure out how to split that
into per-cpu reporting like x86 does, I discovered what a mess the
intrstats stuff in INTRNG is right now.  So, a project for another day,
I guess.

-- Ian

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arm
To unsubscribe, send any mail to "[hidden email]"
1234