Using kqueue with aio_read/write

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Using kqueue with aio_read/write

Willem Jan Withagen-2
Hi,

Im trying to understand why I cannot get so code to work.
This is the smallest extract I can make to show my problem.

I would expect the kevent() call to return every timeo tick.
Even if I tell it NOT to time-out I get these spurts of errors

Since there is nothing to trigger the AIO-event, I would expect kqueue
to hold indefinitly.

But it does not generate anything other than errors
And instead it repeatedly complains that there is a permission error:
   get_events_kevent: EV_Error(1) kevent(): Operation not permitted

But I'm not getting where that would the case...

Surely a pilot error, but I do overlook it al the time.
So suggestions are welcome.

Thanx,
--WjW

#include <aio.h>
#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <sys/event.h>
#include <unistd.h>

#define BUFFER_SIZE     512
#define MAX_EVENTS 32

#define FILENAME "/tmp/aio_test"
char filename[256];
int fd;
int done = 0;

void get_events_kevent(int fd, int kq)
{
     printf("get_events function fd = %d, kq = %d\n", fd, kq);
     int i = 0, errcnt = 0, err, ret, reterr, rev;
     int search = 1;

     int timeout_ms = 10;
     struct timespec timeo = {
         timeout_ms / 1000,
         (timeout_ms % 1000) * 1000 * 1000
     };
     struct kevent filter[16];
     struct kevent changed[16];

     EV_SET(&filter[0], fd, EVFILT_AIO,
             EV_ADD,
             0, 0, 0 );
     while (!done) {
         printf("+");
         rev = kevent(kq, filter, 1, changed, 16, 0); //&timeo);
         if (rev < 0) {
             perror("kevent error");
         } else if (rev == 0) {
             printf("T");
         } else {
             printf("rev(%d)\n", rev);
             if (changed[0].flags == EV_ERROR) {
                 errno = changed[0].data;
                 printf( "%s: EV_Error(%d) kevent(): %s\n", __func__, errno,
                     strerror(errno));
                 memset(&changed[0], 0, sizeof(struct kevent));
             } else {
                 err = aio_error((struct aiocb*)changed[0].udata);
                 ret = aio_return((struct aiocb*)changed[0].udata);
                 if (ret < 0 )
                     reterr = errno;
                 if (err != 0) {
                     printf( "%s: slot: %d, Error(%d) at aio_error():
%s\n", __func__, i, err, strerror (err));
                     errcnt++;
                     if (errcnt > 50) {
                         exit(3);
                     }
                 }
             }
         }
     }
}

int main()
{
     (void) strcpy(filename, FILENAME);
     unlink(filename);
     fd = open(filename, O_CREAT | O_RDWR, S_IRUSR | S_IWUSR);
     if (fd == -1) {
         printf( "%s: Error at open(): %s\n", __func__, strerror(errno));
         exit(1);
     }

     int kq = kqueue();
     if (fd == -1) {
         printf( "%s: Error at kqueue(): %s\n", __func__, strerror(errno));
         exit(1);
     }

     get_events_kevent( fd, kq);

     close(kq);
     close(fd);
     return 0;
}


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Using kqueue with aio_read/write

Alan Somers-2
On Thu, Dec 27, 2018 at 6:15 PM Willem Jan Withagen <[hidden email]> wrote:

>
> Hi,
>
> Im trying to understand why I cannot get so code to work.
> This is the smallest extract I can make to show my problem.
>
> I would expect the kevent() call to return every timeo tick.
> Even if I tell it NOT to time-out I get these spurts of errors
>
> Since there is nothing to trigger the AIO-event, I would expect kqueue
> to hold indefinitly.
>
> But it does not generate anything other than errors
> And instead it repeatedly complains that there is a permission error:
>    get_events_kevent: EV_Error(1) kevent(): Operation not permitted
>
> But I'm not getting where that would the case...
>
> Surely a pilot error, but I do overlook it al the time.
> So suggestions are welcome.
>
> Thanx,
> --WjW
>
> #include <aio.h>
> #include <errno.h>
> #include <fcntl.h>
> #include <stdio.h>
> #include <stdlib.h>
> #include <string.h>
> #include <sys/stat.h>
> #include <sys/event.h>
> #include <unistd.h>
>
> #define BUFFER_SIZE     512
> #define MAX_EVENTS 32
>
> #define FILENAME "/tmp/aio_test"
> char filename[256];
> int fd;
> int done = 0;
>
> void get_events_kevent(int fd, int kq)
> {
>      printf("get_events function fd = %d, kq = %d\n", fd, kq);
>      int i = 0, errcnt = 0, err, ret, reterr, rev;
>      int search = 1;
>
>      int timeout_ms = 10;
>      struct timespec timeo = {
>          timeout_ms / 1000,
>          (timeout_ms % 1000) * 1000 * 1000
>      };
>      struct kevent filter[16];
>      struct kevent changed[16];
>
>      EV_SET(&filter[0], fd, EVFILT_AIO,
>              EV_ADD,
>              0, 0, 0 );


This is the first problem.  There's no need to explicitly set
EVFILT_AIO on the kqueue.  It gets set by the aio_read(2) or similar
syscall.  And this invocation wouldn't be correct anyway, because for
AIO the ident field refers to the address of the struct aiocb, not the
file descriptor.  If the only events you care about are AIO, then you
can pass NULL as the filter argument to kevent.  I suspect this is the
cause of your problem.  The kernel probably thinks you're trying to
register for an aiocb that's outside of your address space or
something like that.


>      while (!done) {
>          printf("+");
>          rev = kevent(kq, filter, 1, changed, 16, 0); //&timeo);
>          if (rev < 0) {
>              perror("kevent error");
>          } else if (rev == 0) {
>              printf("T");
>          } else {
>              printf("rev(%d)\n", rev);
>              if (changed[0].flags == EV_ERROR) {
>                  errno = changed[0].data;
>                  printf( "%s: EV_Error(%d) kevent(): %s\n", __func__, errno,
>                      strerror(errno));
>                  memset(&changed[0], 0, sizeof(struct kevent));
>              } else {
>                  err = aio_error((struct aiocb*)changed[0].udata);


No need to call aio_error(2) after kevent(2) returns.  You can go
straight to aio_return.  aio_error shouldn't hurt, but it isn't
necessary.


>                  ret = aio_return((struct aiocb*)changed[0].udata);
>                  if (ret < 0 )
>                      reterr = errno;
>                  if (err != 0) {
>                      printf( "%s: slot: %d, Error(%d) at aio_error():
> %s\n", __func__, i, err, strerror (err));
>                      errcnt++;
>                      if (errcnt > 50) {
>                          exit(3);
>                      }
>                  }
>              }
>          }
>      }
> }
>
> int main()
> {
>      (void) strcpy(filename, FILENAME);
>      unlink(filename);
>      fd = open(filename, O_CREAT | O_RDWR, S_IRUSR | S_IWUSR);
>      if (fd == -1) {
>          printf( "%s: Error at open(): %s\n", __func__, strerror(errno));
>          exit(1);
>      }
>
>      int kq = kqueue();
>      if (fd == -1) {
>          printf( "%s: Error at kqueue(): %s\n", __func__, strerror(errno));
>          exit(1);
>      }
>
>      get_events_kevent( fd, kq);
>
>      close(kq);
>      close(fd);
>      return 0;
> }
>
>
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "[hidden email]"
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Using kqueue with aio_read/write

Willem Jan Withagen-2
On 28-12-2018 02:47, Alan Somers wrote:

> On Thu, Dec 27, 2018 at 6:15 PM Willem Jan Withagen <[hidden email]> wrote:
>>
>> Hi,
>>
>> Im trying to understand why I cannot get so code to work.
>> This is the smallest extract I can make to show my problem.
>>
>> I would expect the kevent() call to return every timeo tick.
>> Even if I tell it NOT to time-out I get these spurts of errors
>>
>> Since there is nothing to trigger the AIO-event, I would expect kqueue
>> to hold indefinitly.
>>
>> But it does not generate anything other than errors
>> And instead it repeatedly complains that there is a permission error:
>>     get_events_kevent: EV_Error(1) kevent(): Operation not permitted
>>
>> But I'm not getting where that would the case...
>>
>> Surely a pilot error, but I do overlook it al the time.
>> So suggestions are welcome.
>>
>> Thanx,
>> --WjW
>>
>> #include <aio.h>
>> #include <errno.h>
>> #include <fcntl.h>
>> #include <stdio.h>
>> #include <stdlib.h>
>> #include <string.h>
>> #include <sys/stat.h>
>> #include <sys/event.h>
>> #include <unistd.h>
>>
>> #define BUFFER_SIZE     512
>> #define MAX_EVENTS 32
>>
>> #define FILENAME "/tmp/aio_test"
>> char filename[256];
>> int fd;
>> int done = 0;
>>
>> void get_events_kevent(int fd, int kq)
>> {
>>       printf("get_events function fd = %d, kq = %d\n", fd, kq);
>>       int i = 0, errcnt = 0, err, ret, reterr, rev;
>>       int search = 1;
>>
>>       int timeout_ms = 10;
>>       struct timespec timeo = {
>>           timeout_ms / 1000,
>>           (timeout_ms % 1000) * 1000 * 1000
>>       };
>>       struct kevent filter[16];
>>       struct kevent changed[16];
>>
>>       EV_SET(&filter[0], fd, EVFILT_AIO,
>>               EV_ADD,
>>               0, 0, 0 );
>
>
> This is the first problem.  There's no need to explicitly set
> EVFILT_AIO on the kqueue.  It gets set by the aio_read(2) or similar
> syscall.  And this invocation wouldn't be correct anyway, because for
> AIO the ident field refers to the address of the struct aiocb, not the
> file descriptor.  If the only events you care about are AIO, then you
> can pass NULL as the filter argument to kevent.  I suspect this is the
> cause of your problem.  The kernel probably thinks you're trying to
> register for an aiocb that's outside of your address space or
> something like that.

Hi Alan,

That at least helps against getting EPERM.
And I get a nice stream of timeouts hitting kevent.

It sort of makes sense when you write it like this, but then this is not
clear at all from the man pages (for me that is)
But it seems sort of weird to "not filter" to get AIO events.

>>       while (!done) {
>>           printf("+");
>>           rev = kevent(kq, filter, 1, changed, 16, 0); //&timeo);
>>           if (rev < 0) {
>>               perror("kevent error");
>>           } else if (rev == 0) {
>>               printf("T");
>>           } else {
>>               printf("rev(%d)\n", rev);
>>               if (changed[0].flags == EV_ERROR) {
>>                   errno = changed[0].data;
>>                   printf( "%s: EV_Error(%d) kevent(): %s\n", __func__, errno,
>>                       strerror(errno));
>>                   memset(&changed[0], 0, sizeof(struct kevent));
>>               } else {
>>                   err = aio_error((struct aiocb*)changed[0].udata);
>
>
> No need to call aio_error(2) after kevent(2) returns.  You can go
> straight to aio_return.  aio_error shouldn't hurt, but it isn't
> necessary.

This is a sort of leftover from earlier trials where I was using a "home
made" Poll iterating over the aiocb blocks.

Thanx for the help, now I can go on trying getting this into Ceph.
Perhaps I'll try an clean this up, and post a working example somewhere
online.

--WjW

>
>
>>                   ret = aio_return((struct aiocb*)changed[0].udata);
>>                   if (ret < 0 )
>>                       reterr = errno;
>>                   if (err != 0) {
>>                       printf( "%s: slot: %d, Error(%d) at aio_error():
>> %s\n", __func__, i, err, strerror (err));
>>                       errcnt++;
>>                       if (errcnt > 50) {
>>                           exit(3);
>>                       }
>>                   }
>>               }
>>           }
>>       }
>> }
>>
>> int main()
>> {
>>       (void) strcpy(filename, FILENAME);
>>       unlink(filename);
>>       fd = open(filename, O_CREAT | O_RDWR, S_IRUSR | S_IWUSR);
>>       if (fd == -1) {
>>           printf( "%s: Error at open(): %s\n", __func__, strerror(errno));
>>           exit(1);
>>       }
>>
>>       int kq = kqueue();
>>       if (fd == -1) {
>>           printf( "%s: Error at kqueue(): %s\n", __func__, strerror(errno));
>>           exit(1);
>>       }
>>
>>       get_events_kevent( fd, kq);
>>
>>       close(kq);
>>       close(fd);
>>       return 0;
>> }
>>
>>
>> _______________________________________________
>> [hidden email] mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
>> To unsubscribe, send any mail to "[hidden email]"

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Using kqueue with aio_read/write

Willem Jan Withagen-2
In reply to this post by Alan Somers-2
On 28/12/2018 02:47, Alan Somers wrote:

> On Thu, Dec 27, 2018 at 6:15 PM Willem Jan Withagen <[hidden email]> wrote:
>> Hi,
>>
>> Im trying to understand why I cannot get so code to work.
>> This is the smallest extract I can make to show my problem.
>>
>> I would expect the kevent() call to return every timeo tick.
>> Even if I tell it NOT to time-out I get these spurts of errors
>>
>> Since there is nothing to trigger the AIO-event, I would expect kqueue
>> to hold indefinitly.
>>
>> But it does not generate anything other than errors
>> And instead it repeatedly complains that there is a permission error:
>>     get_events_kevent: EV_Error(1) kevent(): Operation not permitted
>>
>> But I'm not getting where that would the case...
>>
>> Surely a pilot error, but I do overlook it al the time.
>> So suggestions are welcome.
>>
>> Thanx,
>> --WjW
>>
>> #include <aio.h>
>> #include <errno.h>
>> #include <fcntl.h>
>> #include <stdio.h>
>> #include <stdlib.h>
>> #include <string.h>
>> #include <sys/stat.h>
>> #include <sys/event.h>
>> #include <unistd.h>
>>
>> #define BUFFER_SIZE     512
>> #define MAX_EVENTS 32
>>
>> #define FILENAME "/tmp/aio_test"
>> char filename[256];
>> int fd;
>> int done = 0;
>>
>> void get_events_kevent(int fd, int kq)
>> {
>>       printf("get_events function fd = %d, kq = %d\n", fd, kq);
>>       int i = 0, errcnt = 0, err, ret, reterr, rev;
>>       int search = 1;
>>
>>       int timeout_ms = 10;
>>       struct timespec timeo = {
>>           timeout_ms / 1000,
>>           (timeout_ms % 1000) * 1000 * 1000
>>       };
>>       struct kevent filter[16];
>>       struct kevent changed[16];
>>
>>       EV_SET(&filter[0], fd, EVFILT_AIO,
>>               EV_ADD,
>>               0, 0, 0 );
>
> This is the first problem.  There's no need to explicitly set
> EVFILT_AIO on the kqueue.  It gets set by the aio_read(2) or similar
> syscall.  And this invocation wouldn't be correct anyway, because for
> AIO the ident field refers to the address of the struct aiocb, not the
> file descriptor.  If the only events you care about are AIO, then you
> can pass NULL as the filter argument to kevent.  I suspect this is the
> cause of your problem.  The kernel probably thinks you're trying to
> register for an aiocb that's outside of your address space or
> something like that.
>
>
>>       while (!done) {
>>           printf("+");
>>           rev = kevent(kq, filter, 1, changed, 16, 0); //&timeo);
>>           if (rev < 0) {
>>               perror("kevent error");
>>           } else if (rev == 0) {
>>               printf("T");
>>           } else {
>>               printf("rev(%d)\n", rev);
>>               if (changed[0].flags == EV_ERROR) {
>>                   errno = changed[0].data;
>>                   printf( "%s: EV_Error(%d) kevent(): %s\n", __func__, errno,
>>                       strerror(errno));
>>                   memset(&changed[0], 0, sizeof(struct kevent));
>>               } else {
>>                   err = aio_error((struct aiocb*)changed[0].udata);
>
> No need to call aio_error(2) after kevent(2) returns.  You can go
> straight to aio_return.  aio_error shouldn't hurt, but it isn't
> necessary.

According to kevent(2) calling kevent can return errors on the called
aio_calls.
It then returns with EV_ERROR in flags, and errno is stored in the
event.data.

But what would be going on when the event's flag contains EV_ERROR but
event's data is still 0???

the udata field still seems to point to the aio data that was passed
into the aio block when calling aio_read().

Should I ignore this as a non-error?

--WjW



_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Using kqueue with aio_read/write

Alan Somers-2
On Tue, Jan 1, 2019 at 6:56 PM Willem Jan Withagen <[hidden email]> wrote:

>
> On 28/12/2018 02:47, Alan Somers wrote:
> > On Thu, Dec 27, 2018 at 6:15 PM Willem Jan Withagen <[hidden email]> wrote:
> >> Hi,
> >>
> >> Im trying to understand why I cannot get so code to work.
> >> This is the smallest extract I can make to show my problem.
> >>
> >> I would expect the kevent() call to return every timeo tick.
> >> Even if I tell it NOT to time-out I get these spurts of errors
> >>
> >> Since there is nothing to trigger the AIO-event, I would expect kqueue
> >> to hold indefinitly.
> >>
> >> But it does not generate anything other than errors
> >> And instead it repeatedly complains that there is a permission error:
> >>     get_events_kevent: EV_Error(1) kevent(): Operation not permitted
> >>
> >> But I'm not getting where that would the case...
> >>
> >> Surely a pilot error, but I do overlook it al the time.
> >> So suggestions are welcome.
> >>
> >> Thanx,
> >> --WjW
> >>
> >> #include <aio.h>
> >> #include <errno.h>
> >> #include <fcntl.h>
> >> #include <stdio.h>
> >> #include <stdlib.h>
> >> #include <string.h>
> >> #include <sys/stat.h>
> >> #include <sys/event.h>
> >> #include <unistd.h>
> >>
> >> #define BUFFER_SIZE     512
> >> #define MAX_EVENTS 32
> >>
> >> #define FILENAME "/tmp/aio_test"
> >> char filename[256];
> >> int fd;
> >> int done = 0;
> >>
> >> void get_events_kevent(int fd, int kq)
> >> {
> >>       printf("get_events function fd = %d, kq = %d\n", fd, kq);
> >>       int i = 0, errcnt = 0, err, ret, reterr, rev;
> >>       int search = 1;
> >>
> >>       int timeout_ms = 10;
> >>       struct timespec timeo = {
> >>           timeout_ms / 1000,
> >>           (timeout_ms % 1000) * 1000 * 1000
> >>       };
> >>       struct kevent filter[16];
> >>       struct kevent changed[16];
> >>
> >>       EV_SET(&filter[0], fd, EVFILT_AIO,
> >>               EV_ADD,
> >>               0, 0, 0 );
> >
> > This is the first problem.  There's no need to explicitly set
> > EVFILT_AIO on the kqueue.  It gets set by the aio_read(2) or similar
> > syscall.  And this invocation wouldn't be correct anyway, because for
> > AIO the ident field refers to the address of the struct aiocb, not the
> > file descriptor.  If the only events you care about are AIO, then you
> > can pass NULL as the filter argument to kevent.  I suspect this is the
> > cause of your problem.  The kernel probably thinks you're trying to
> > register for an aiocb that's outside of your address space or
> > something like that.
> >
> >
> >>       while (!done) {
> >>           printf("+");
> >>           rev = kevent(kq, filter, 1, changed, 16, 0); //&timeo);
> >>           if (rev < 0) {
> >>               perror("kevent error");
> >>           } else if (rev == 0) {
> >>               printf("T");
> >>           } else {
> >>               printf("rev(%d)\n", rev);
> >>               if (changed[0].flags == EV_ERROR) {
> >>                   errno = changed[0].data;
> >>                   printf( "%s: EV_Error(%d) kevent(): %s\n", __func__, errno,
> >>                       strerror(errno));
> >>                   memset(&changed[0], 0, sizeof(struct kevent));
> >>               } else {
> >>                   err = aio_error((struct aiocb*)changed[0].udata);
> >
> > No need to call aio_error(2) after kevent(2) returns.  You can go
> > straight to aio_return.  aio_error shouldn't hurt, but it isn't
> > necessary.
>
> According to kevent(2) calling kevent can return errors on the called
> aio_calls.
> It then returns with EV_ERROR in flags, and errno is stored in the
> event.data.
>
> But what would be going on when the event's flag contains EV_ERROR but
> event's data is still 0???
>
> the udata field still seems to point to the aio data that was passed
> into the aio block when calling aio_read().
>
> Should I ignore this as a non-error?
>
> --WjW

Are you sure you bzero()ed your aiocb before initializing it?  Any
stack garbage that was present in its
aio_sigevent.sigev_notify_kevent_flags field will be dutifully copied
into the returned kevent.  And in any case, the definitive way to get
the final status of a completed aio operation is with aio_return.
-Alan
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Using kqueue with aio_read/write

Willem Jan Withagen-2
On 02/01/2019 03:23, Alan Somers wrote:

> On Tue, Jan 1, 2019 at 6:56 PM Willem Jan Withagen <[hidden email]> wrote:
>> On 28/12/2018 02:47, Alan Somers wrote:
>>> On Thu, Dec 27, 2018 at 6:15 PM Willem Jan Withagen <[hidden email]> wrote:
>>>> Hi,
>>>>
>>>> Im trying to understand why I cannot get so code to work.
>>>> This is the smallest extract I can make to show my problem.
>>>>
>>>> I would expect the kevent() call to return every timeo tick.
>>>> Even if I tell it NOT to time-out I get these spurts of errors
>>>>
>>>> Since there is nothing to trigger the AIO-event, I would expect kqueue
>>>> to hold indefinitly.
>>>>
>>>> But it does not generate anything other than errors
>>>> And instead it repeatedly complains that there is a permission error:
>>>>      get_events_kevent: EV_Error(1) kevent(): Operation not permitted
>>>>
>>>> But I'm not getting where that would the case...
>>>>
>>>> Surely a pilot error, but I do overlook it al the time.
>>>> So suggestions are welcome.
>>>>
>>>> Thanx,
>>>> --WjW
>>>>
>>>> #include <aio.h>
>>>> #include <errno.h>
>>>> #include <fcntl.h>
>>>> #include <stdio.h>
>>>> #include <stdlib.h>
>>>> #include <string.h>
>>>> #include <sys/stat.h>
>>>> #include <sys/event.h>
>>>> #include <unistd.h>
>>>>
>>>> #define BUFFER_SIZE     512
>>>> #define MAX_EVENTS 32
>>>>
>>>> #define FILENAME "/tmp/aio_test"
>>>> char filename[256];
>>>> int fd;
>>>> int done = 0;
>>>>
>>>> void get_events_kevent(int fd, int kq)
>>>> {
>>>>        printf("get_events function fd = %d, kq = %d\n", fd, kq);
>>>>        int i = 0, errcnt = 0, err, ret, reterr, rev;
>>>>        int search = 1;
>>>>
>>>>        int timeout_ms = 10;
>>>>        struct timespec timeo = {
>>>>            timeout_ms / 1000,
>>>>            (timeout_ms % 1000) * 1000 * 1000
>>>>        };
>>>>        struct kevent filter[16];
>>>>        struct kevent changed[16];
>>>>
>>>>        EV_SET(&filter[0], fd, EVFILT_AIO,
>>>>                EV_ADD,
>>>>                0, 0, 0 );
>>> This is the first problem.  There's no need to explicitly set
>>> EVFILT_AIO on the kqueue.  It gets set by the aio_read(2) or similar
>>> syscall.  And this invocation wouldn't be correct anyway, because for
>>> AIO the ident field refers to the address of the struct aiocb, not the
>>> file descriptor.  If the only events you care about are AIO, then you
>>> can pass NULL as the filter argument to kevent.  I suspect this is the
>>> cause of your problem.  The kernel probably thinks you're trying to
>>> register for an aiocb that's outside of your address space or
>>> something like that.
>>>
>>>
>>>>        while (!done) {
>>>>            printf("+");
>>>>            rev = kevent(kq, filter, 1, changed, 16, 0); //&timeo);
>>>>            if (rev < 0) {
>>>>                perror("kevent error");
>>>>            } else if (rev == 0) {
>>>>                printf("T");
>>>>            } else {
>>>>                printf("rev(%d)\n", rev);
>>>>                if (changed[0].flags == EV_ERROR) {
>>>>                    errno = changed[0].data;
>>>>                    printf( "%s: EV_Error(%d) kevent(): %s\n", __func__, errno,
>>>>                        strerror(errno));
>>>>                    memset(&changed[0], 0, sizeof(struct kevent));
>>>>                } else {
>>>>                    err = aio_error((struct aiocb*)changed[0].udata);
>>> No need to call aio_error(2) after kevent(2) returns.  You can go
>>> straight to aio_return.  aio_error shouldn't hurt, but it isn't
>>> necessary.
>> According to kevent(2) calling kevent can return errors on the called
>> aio_calls.
>> It then returns with EV_ERROR in flags, and errno is stored in the
>> event.data.
>>
>> But what would be going on when the event's flag contains EV_ERROR but
>> event's data is still 0???
>>
>> the udata field still seems to point to the aio data that was passed
>> into the aio block when calling aio_read().
>>
>> Should I ignore this as a non-error?
>>
>> --WjW
> Are you sure you bzero()ed your aiocb before initializing it?  Any
> stack garbage that was present in its
> aio_sigevent.sigev_notify_kevent_flags field will be dutifully copied
> into the returned kevent.  And in any case, the definitive way to get
> the final status of a completed aio operation is with aio_return.
That seems to help in getting things clear...

    -13> 2019-01-02 13:32:31.834 dc15a80  1 bdev:327 paio
get_next_completed processing event i = 0 aio_return(22) (22) Invalid
argument

Disadvantage is that it is not clear yet which of the many arguments
that is?

--WjW

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Using kqueue with aio_read/write

Alan Somers-2
On Wed, Jan 2, 2019 at 5:53 AM Willem Jan Withagen <[hidden email]> wrote:

>
> On 02/01/2019 03:23, Alan Somers wrote:
> > On Tue, Jan 1, 2019 at 6:56 PM Willem Jan Withagen <[hidden email]> wrote:
> >> On 28/12/2018 02:47, Alan Somers wrote:
> >>> On Thu, Dec 27, 2018 at 6:15 PM Willem Jan Withagen <[hidden email]> wrote:
> >>>> Hi,
> >>>>
> >>>> Im trying to understand why I cannot get so code to work.
> >>>> This is the smallest extract I can make to show my problem.
> >>>>
> >>>> I would expect the kevent() call to return every timeo tick.
> >>>> Even if I tell it NOT to time-out I get these spurts of errors
> >>>>
> >>>> Since there is nothing to trigger the AIO-event, I would expect kqueue
> >>>> to hold indefinitly.
> >>>>
> >>>> But it does not generate anything other than errors
> >>>> And instead it repeatedly complains that there is a permission error:
> >>>>      get_events_kevent: EV_Error(1) kevent(): Operation not permitted
> >>>>
> >>>> But I'm not getting where that would the case...
> >>>>
> >>>> Surely a pilot error, but I do overlook it al the time.
> >>>> So suggestions are welcome.
> >>>>
> >>>> Thanx,
> >>>> --WjW
> >>>>
> >>>> #include <aio.h>
> >>>> #include <errno.h>
> >>>> #include <fcntl.h>
> >>>> #include <stdio.h>
> >>>> #include <stdlib.h>
> >>>> #include <string.h>
> >>>> #include <sys/stat.h>
> >>>> #include <sys/event.h>
> >>>> #include <unistd.h>
> >>>>
> >>>> #define BUFFER_SIZE     512
> >>>> #define MAX_EVENTS 32
> >>>>
> >>>> #define FILENAME "/tmp/aio_test"
> >>>> char filename[256];
> >>>> int fd;
> >>>> int done = 0;
> >>>>
> >>>> void get_events_kevent(int fd, int kq)
> >>>> {
> >>>>        printf("get_events function fd = %d, kq = %d\n", fd, kq);
> >>>>        int i = 0, errcnt = 0, err, ret, reterr, rev;
> >>>>        int search = 1;
> >>>>
> >>>>        int timeout_ms = 10;
> >>>>        struct timespec timeo = {
> >>>>            timeout_ms / 1000,
> >>>>            (timeout_ms % 1000) * 1000 * 1000
> >>>>        };
> >>>>        struct kevent filter[16];
> >>>>        struct kevent changed[16];
> >>>>
> >>>>        EV_SET(&filter[0], fd, EVFILT_AIO,
> >>>>                EV_ADD,
> >>>>                0, 0, 0 );
> >>> This is the first problem.  There's no need to explicitly set
> >>> EVFILT_AIO on the kqueue.  It gets set by the aio_read(2) or similar
> >>> syscall.  And this invocation wouldn't be correct anyway, because for
> >>> AIO the ident field refers to the address of the struct aiocb, not the
> >>> file descriptor.  If the only events you care about are AIO, then you
> >>> can pass NULL as the filter argument to kevent.  I suspect this is the
> >>> cause of your problem.  The kernel probably thinks you're trying to
> >>> register for an aiocb that's outside of your address space or
> >>> something like that.
> >>>
> >>>
> >>>>        while (!done) {
> >>>>            printf("+");
> >>>>            rev = kevent(kq, filter, 1, changed, 16, 0); //&timeo);
> >>>>            if (rev < 0) {
> >>>>                perror("kevent error");
> >>>>            } else if (rev == 0) {
> >>>>                printf("T");
> >>>>            } else {
> >>>>                printf("rev(%d)\n", rev);
> >>>>                if (changed[0].flags == EV_ERROR) {
> >>>>                    errno = changed[0].data;
> >>>>                    printf( "%s: EV_Error(%d) kevent(): %s\n", __func__, errno,
> >>>>                        strerror(errno));
> >>>>                    memset(&changed[0], 0, sizeof(struct kevent));
> >>>>                } else {
> >>>>                    err = aio_error((struct aiocb*)changed[0].udata);
> >>> No need to call aio_error(2) after kevent(2) returns.  You can go
> >>> straight to aio_return.  aio_error shouldn't hurt, but it isn't
> >>> necessary.
> >> According to kevent(2) calling kevent can return errors on the called
> >> aio_calls.
> >> It then returns with EV_ERROR in flags, and errno is stored in the
> >> event.data.
> >>
> >> But what would be going on when the event's flag contains EV_ERROR but
> >> event's data is still 0???
> >>
> >> the udata field still seems to point to the aio data that was passed
> >> into the aio block when calling aio_read().
> >>
> >> Should I ignore this as a non-error?
> >>
> >> --WjW
> > Are you sure you bzero()ed your aiocb before initializing it?  Any
> > stack garbage that was present in its
> > aio_sigevent.sigev_notify_kevent_flags field will be dutifully copied
> > into the returned kevent.  And in any case, the definitive way to get
> > the final status of a completed aio operation is with aio_return.
> That seems to help in getting things clear...
>
>     -13> 2019-01-02 13:32:31.834 dc15a80  1 bdev:327 paio
> get_next_completed processing event i = 0 aio_return(22) (22) Invalid
> argument
>
> Disadvantage is that it is not clear yet which of the many arguments
> that is?
>
> --WjW

It could be failing for any reason that read(2) can fail.  Or, the
iocb might not be complete.  aio_error(2) would tell you.  One common
problem is if you allocate the iocb on the stack and accidentally move
it after calling aio_read().  That's bitten me before.
-Alan
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Using kqueue with aio_read/write

Willem Jan Withagen-2
On 02/01/2019 15:59, Alan Somers wrote:

> On Wed, Jan 2, 2019 at 5:53 AM Willem Jan Withagen <[hidden email]> wrote:
>> On 02/01/2019 03:23, Alan Somers wrote:
>>> On Tue, Jan 1, 2019 at 6:56 PM Willem Jan Withagen <[hidden email]> wrote:
>>>> On 28/12/2018 02:47, Alan Somers wrote:
>>>>> On Thu, Dec 27, 2018 at 6:15 PM Willem Jan Withagen <[hidden email]> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Im trying to understand why I cannot get so code to work.
>>>>>> This is the smallest extract I can make to show my problem.
>>>>>>
>>>>>> I would expect the kevent() call to return every timeo tick.
>>>>>> Even if I tell it NOT to time-out I get these spurts of errors
>>>>>>
>>>>>> Since there is nothing to trigger the AIO-event, I would expect kqueue
>>>>>> to hold indefinitly.
>>>>>>
>>>>>> But it does not generate anything other than errors
>>>>>> And instead it repeatedly complains that there is a permission error:
>>>>>>       get_events_kevent: EV_Error(1) kevent(): Operation not permitted
>>>>>>
>>>>>> But I'm not getting where that would the case...
>>>>>>
>>>>>> Surely a pilot error, but I do overlook it al the time.
>>>>>> So suggestions are welcome.
>>>>>>
>>>>>> Thanx,
>>>>>> --WjW
>>>>>>
>>>>>> #include <aio.h>
>>>>>> #include <errno.h>
>>>>>> #include <fcntl.h>
>>>>>> #include <stdio.h>
>>>>>> #include <stdlib.h>
>>>>>> #include <string.h>
>>>>>> #include <sys/stat.h>
>>>>>> #include <sys/event.h>
>>>>>> #include <unistd.h>
>>>>>>
>>>>>> #define BUFFER_SIZE     512
>>>>>> #define MAX_EVENTS 32
>>>>>>
>>>>>> #define FILENAME "/tmp/aio_test"
>>>>>> char filename[256];
>>>>>> int fd;
>>>>>> int done = 0;
>>>>>>
>>>>>> void get_events_kevent(int fd, int kq)
>>>>>> {
>>>>>>         printf("get_events function fd = %d, kq = %d\n", fd, kq);
>>>>>>         int i = 0, errcnt = 0, err, ret, reterr, rev;
>>>>>>         int search = 1;
>>>>>>
>>>>>>         int timeout_ms = 10;
>>>>>>         struct timespec timeo = {
>>>>>>             timeout_ms / 1000,
>>>>>>             (timeout_ms % 1000) * 1000 * 1000
>>>>>>         };
>>>>>>         struct kevent filter[16];
>>>>>>         struct kevent changed[16];
>>>>>>
>>>>>>         EV_SET(&filter[0], fd, EVFILT_AIO,
>>>>>>                 EV_ADD,
>>>>>>                 0, 0, 0 );
>>>>> This is the first problem.  There's no need to explicitly set
>>>>> EVFILT_AIO on the kqueue.  It gets set by the aio_read(2) or similar
>>>>> syscall.  And this invocation wouldn't be correct anyway, because for
>>>>> AIO the ident field refers to the address of the struct aiocb, not the
>>>>> file descriptor.  If the only events you care about are AIO, then you
>>>>> can pass NULL as the filter argument to kevent.  I suspect this is the
>>>>> cause of your problem.  The kernel probably thinks you're trying to
>>>>> register for an aiocb that's outside of your address space or
>>>>> something like that.
>>>>>
>>>>>
>>>>>>         while (!done) {
>>>>>>             printf("+");
>>>>>>             rev = kevent(kq, filter, 1, changed, 16, 0); //&timeo);
>>>>>>             if (rev < 0) {
>>>>>>                 perror("kevent error");
>>>>>>             } else if (rev == 0) {
>>>>>>                 printf("T");
>>>>>>             } else {
>>>>>>                 printf("rev(%d)\n", rev);
>>>>>>                 if (changed[0].flags == EV_ERROR) {
>>>>>>                     errno = changed[0].data;
>>>>>>                     printf( "%s: EV_Error(%d) kevent(): %s\n", __func__, errno,
>>>>>>                         strerror(errno));
>>>>>>                     memset(&changed[0], 0, sizeof(struct kevent));
>>>>>>                 } else {
>>>>>>                     err = aio_error((struct aiocb*)changed[0].udata);
>>>>> No need to call aio_error(2) after kevent(2) returns.  You can go
>>>>> straight to aio_return.  aio_error shouldn't hurt, but it isn't
>>>>> necessary.
>>>> According to kevent(2) calling kevent can return errors on the called
>>>> aio_calls.
>>>> It then returns with EV_ERROR in flags, and errno is stored in the
>>>> event.data.
>>>>
>>>> But what would be going on when the event's flag contains EV_ERROR but
>>>> event's data is still 0???
>>>>
>>>> the udata field still seems to point to the aio data that was passed
>>>> into the aio block when calling aio_read().
>>>>
>>>> Should I ignore this as a non-error?
>>>>
>>>> --WjW
>>> Are you sure you bzero()ed your aiocb before initializing it?  Any
>>> stack garbage that was present in its
>>> aio_sigevent.sigev_notify_kevent_flags field will be dutifully copied
>>> into the returned kevent.  And in any case, the definitive way to get
>>> the final status of a completed aio operation is with aio_return.
>> That seems to help in getting things clear...
>>
>>      -13> 2019-01-02 13:32:31.834 dc15a80  1 bdev:327 paio
>> get_next_completed processing event i = 0 aio_return(22) (22) Invalid
>> argument
>>
>> Disadvantage is that it is not clear yet which of the many arguments
>> that is?
>>
>> --WjW
> It could be failing for any reason that read(2) can fail.  Or, the
> iocb might not be complete.  aio_error(2) would tell you.  One common
> problem is if you allocate the iocb on the stack and accidentally move
> it after calling aio_read().  That's bitten me before.

Well the main part that is biting here, is that it sort of has to be
compatible with what
Sage ea. have cooked up to manage the iov/iocb/aiocb  stuff.
And I need to do it with keeping the Linux code working as well.

And the real mess was to sort out get the correct pointer into the
aio_write() and
find it back in get_next_event() to finish that.
But it looks like I've sorted that part now. Get_next_event() actually
now finds the
correct aiocb. and I can aio_return() without errors.

Now the next trick is going to take it of the list off outstanding aio's
in the ceph-list.
That still asserts somewhere.

But slowly getting there. Just running my head into a wall at some times.

--WjW


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"