userland process rpc.lockd opens untraceable ports...is something wrong here?

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

userland process rpc.lockd opens untraceable ports...is something wrong here?

BBlister

HI,

During a security auditing on one FreeBSD 11.2 server I noticed that
something was listening on a tcp4 and tcp6 port. This could not be traced
back using lsof or sockstat. sockstat returned ? for the process name, and
lsof did not list the port opened. The port was opened because i could
telnet to it.

I opened a thread at freebsd-questions (Cannot identify process of listening
port 600/tcp6). You can find the archive of that thread here:
http://freebsd.1045724.x6.nabble.com/Cannot-identify-process-of-listening-port-600-tcp6-td6314916.html 

After many trials, I found out that these ports were opened by rpc.locked.
Killing that process removed the two listening ports. Restarting the
process, opened two new random ports bellow 1024 that could not be traced
back using all FreeBSD tools that I know to the userland process.

And here is my question: How is this happening? What magic trick did
rpc.lockd process utilizes and hides itself from security auditing tools
like lsof or sockstat? Why rpc.lockd is the only process that hides itself
from locating what ports it has opened? Is there any other tool except
lsof/sockstat that can backtrace the  listening port to the process
rpc.locked?

but the most important question: Can this trickery being exploited by a
malicious process and open listening ports without being traced using
lsof/sockstat?

Yours valuable thoughts are most welcome.

Thanks.



--
Sent from: http://freebsd.1045724.x6.nabble.com/freebsd-hackers-f4034256.html
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: userland process rpc.lockd opens untraceable ports...is something wrong here?

Freddie Cash-8
On Tue, Feb 19, 2019 at 1:15 PM BBlister <[hidden email]> wrote:

> During a security auditing on one FreeBSD 11.2 server I noticed that
> something was listening on a tcp4 and tcp6 port. This could not be traced
> back using lsof or sockstat. sockstat returned ? for the process name, and
> lsof did not list the port opened. The port was opened because i could
> telnet to it.
>
> I opened a thread at freebsd-questions (Cannot identify process of
> listening
> port 600/tcp6). You can find the archive of that thread here:
>
> http://freebsd.1045724.x6.nabble.com/Cannot-identify-process-of-listening-port-600-tcp6-td6314916.html
>
> After many trials, I found out that these ports were opened by rpc.locked.
> Killing that process removed the two listening ports. Restarting the
> process, opened two new random ports bellow 1024 that could not be traced
> back using all FreeBSD tools that I know to the userland process.
>
> And here is my question: How is this happening? What magic trick did
> rpc.lockd process utilizes and hides itself from security auditing tools
> like lsof or sockstat? Why rpc.lockd is the only process that hides itself
> from locating what ports it has opened? Is there any other tool except
> lsof/sockstat that can backtrace the  listening port to the process
> rpc.locked?
>
> but the most important question: Can this trickery being exploited by a
> malicious process and open listening ports without being traced using
> lsof/sockstat?
>

While it doesn't take you from a socket/port to a process, does procstat at
least show you the sockets that rpc.lockd has open?

Something like:  procstat -f <pid-of-rpc.lockd>

Although, one could probably run the following to get from the socket/port
number to the process:  procstat -f -a | grep 600

--
Freddie Cash
[hidden email]
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: userland process rpc.lockd opens untraceable ports...is something wrong here?

mdtancsa
On 2/19/2019 4:24 PM, Freddie Cash wrote:
> While it doesn't take you from a socket/port to a process, does
> procstat at
> least show you the sockets that rpc.lockd has open?
>
> Something like:  procstat -f <pid-of-rpc.lockd>
>
> Although, one could probably run the following to get from the socket/port
> number to the process:  procstat -f -a | grep 600

It doesnt seem to.  sockstat shows

# sockstat | grep "^?"
?        ?          ?     ?  tcp4   *:845                 *:*
?        ?          ?     ?  udp4   *:833                 *:*
?        ?          ?     ?  udp4   *:2049                *:*
?        ?          ?     ?  udp6   *:976                 *:*
?        ?          ?     ?  tcp6   *:882                 *:*
?        ?          ?     ?  udp4   *:*                   *:*
?        ?          ?     ?  udp6   *:938                 *:*
?        ?          ?     ?  udp6   *:2049                *:*

# procstat -f 2449

  PID COMM                FD T V FLAGS    REF  OFFSET PRO NAME       
 2449 rpc.lockd         text v r r-------   -       - -  
/usr/sbin/rpc.lockd
 2449 rpc.lockd          cwd v d r-------   -       - -   /                
 2449 rpc.lockd         root v d r-------   -       - -   /                
 2449 rpc.lockd            0 v c rw------   3       0 -   /dev/null        
 2449 rpc.lockd            1 v c rw------   3       0 -   /dev/null        
 2449 rpc.lockd            2 v c rw------   3       0 -   /dev/null        
 2449 rpc.lockd            3 s - rw------   1       0 UDD /var/run/logpriv

 # sockstat | grep 845
?        ?          ?     ?  tcp4   *:845                 *:*
# kill 2449
# sockstat | grep 845
#



--

-------------------
Mike Tancsa, tel +1 519 651 3400 x203
Sentex Communications, [hidden email]
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada  

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: userland process rpc.lockd opens untraceable ports...is something wrong here?

Steve Kargl
On Tue, Feb 19, 2019 at 04:38:50PM -0500, Mike Tancsa wrote:

> On 2/19/2019 4:24 PM, Freddie Cash wrote:
> > While it doesn't take you from a socket/port to a process, does
> > procstat at
> > least show you the sockets that rpc.lockd has open?
> >
> > Something like:  procstat -f <pid-of-rpc.lockd>
> >
> > Although, one could probably run the following to get from the socket/port
> > number to the process:  procstat -f -a | grep 600
>
> It doesnt seem to.  sockstat shows
>
> # sockstat | grep "^?"
> ?        ?          ?     ?  tcp4   *:845                 *:*
> ?        ?          ?     ?  udp4   *:833                 *:*
> ?        ?          ?     ?  udp4   *:2049                *:*
> ?        ?          ?     ?  udp6   *:976                 *:*
> ?        ?          ?     ?  tcp6   *:882                 *:*
> ?        ?          ?     ?  udp4   *:*                   *:*
> ?        ?          ?     ?  udp6   *:938                 *:*
> ?        ?          ?     ?  udp6   *:2049                *:*

The sockstat(8) manuals states

  If a socket is not associated with any file descriptor,
  the first four columns have no meaning.

--
Steve
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: userland process rpc.lockd opens untraceable ports...is something wrong here?

BBlister
After one suggestion on the questions list,  I used the rpcinfo -p but this
does not print every unknown port. For example:

# netstat -an | grep -E '874|815'
tcp4       0      0 *.815                  *.*                    LISTEN
tcp6       0      0 *.874                  *.*                    LISTEN

sockstat reports ?
# sockstat | grep -E '874|815'
?        ?          ?     ?  tcp4   *:815                 *:*
?        ?          ?     ?  tcp6   *:874                 *:*

rpcinfo -p reports just one port
# rpcinfo -p| grep -E '874|815'
    100021    0   tcp    815  nlockmgr
    100021    1   tcp    815  nlockmgr
    100021    3   tcp    815  nlockmgr
    100021    4   tcp    815  nlockmgr


The 874/tcp6 which belongs to rpc.lockd does not appear on this list.
Is rpcinfo only for IPv4 and if yes,what tool do I use for IPv6 ?





The grand question is of course, is there any tool to actually locate the
processes that open ports and cannot be identified with sockstat?

The second grand question. Why rpc.lockd is a different kind of process that
cannot be located from sockstat? Other RPC processes are found using
sockstat, as the following printing shows:

# rpcinfo -p | grep 2049
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs


sockstat |grep 2049
root     nfsd       41279 5  tcp4   *:2049                *:*
root     nfsd       41279 6  tcp6   *:2049                *:*


nfs is found using rpcinfo and also using sockstat.

What rpc.lockd does and it is not found. After 25 years of sysadmin, I find
it very strange for Freebsd to not being able to trace a listening port to
an executable.



--
Sent from: http://freebsd.1045724.x6.nabble.com/freebsd-hackers-f4034256.html
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: userland process rpc.lockd opens untraceable ports...is something wrong here?

Cy Schubert-4
On February 20, 2019 6:02:17 AM PST, BBlister <[hidden email]> wrote:

>After one suggestion on the questions list,  I used the rpcinfo -p but
>this
>does not print every unknown port. For example:
>
># netstat -an | grep -E '874|815'
>tcp4       0      0 *.815                  *.*                  
>LISTEN
>tcp6       0      0 *.874                  *.*                  
>LISTEN
>
>sockstat reports ?
># sockstat | grep -E '874|815'
>?        ?          ?     ?  tcp4   *:815                 *:*
>?        ?          ?     ?  tcp6   *:874                 *:*
>
>rpcinfo -p reports just one port
># rpcinfo -p| grep -E '874|815'
>    100021    0   tcp    815  nlockmgr
>    100021    1   tcp    815  nlockmgr
>    100021    3   tcp    815  nlockmgr
>    100021    4   tcp    815  nlockmgr
>
>
>The 874/tcp6 which belongs to rpc.lockd does not appear on this list.
>Is rpcinfo only for IPv4 and if yes,what tool do I use for IPv6 ?
>
>
>
>
>
>The grand question is of course, is there any tool to actually locate
>the
>processes that open ports and cannot be identified with sockstat?
>
>The second grand question. Why rpc.lockd is a different kind of process
>that
>cannot be located from sockstat? Other RPC processes are found using
>sockstat, as the following printing shows:
>
># rpcinfo -p | grep 2049
>    100003    2   udp   2049  nfs
>    100003    3   udp   2049  nfs
>    100003    2   tcp   2049  nfs
>    100003    3   tcp   2049  nfs
>
>
>sockstat |grep 2049
>root     nfsd       41279 5  tcp4   *:2049                *:*
>root     nfsd       41279 6  tcp6   *:2049                *:*
>
>
>nfs is found using rpcinfo and also using sockstat.
>
>What rpc.lockd does and it is not found. After 25 years of sysadmin, I
>find
>it very strange for Freebsd to not being able to trace a listening port
>to
>an executable.
>
>
>
>--
>Sent from:
>http://freebsd.1045724.x6.nabble.com/freebsd-hackers-f4034256.html
>_______________________________________________
>[hidden email] mailing list
>https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
>To unsubscribe, send any mail to
>"[hidden email]"

Rpcinfo  displays rpcbind's mapping of RPC program numbers to ports.

Sockstat and lsof provide the output you desire. Sockstat output below, lsof output is too difficult to cut and paste on a phone.

3443  4  udp6   *:652                 *:*
root     rpc.statd  3443  5  tcp6   *:652                 *:*
root     rpc.statd  3443  6  udp4   *:652                 *:*
root     rpc.statd  3443  7  tcp4   *:652                 *:*

Your kernel and userland are not in sync.


--
Pardon the typos and autocorrect, small keyboard in use.
Cheers,
Cy Schubert <[hidden email]>
FreeBSD UNIX: <[hidden email]> Web: http://www.FreeBSD.org

        The need of the many outweighs the greed of the few.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: userland process rpc.lockd opens untraceable ports...is something wrong here?

Ian Lepore-3
In reply to this post by BBlister
On Wed, 2019-02-20 at 07:02 -0700, BBlister wrote:

> After one suggestion on the questions list,  I used the rpcinfo -p but this
> does not print every unknown port. For example:
>
> # netstat -an | grep -E '874|815'
> tcp4       0      0 *.815                  *.*                    LISTEN
> tcp6       0      0 *.874                  *.*                    LISTEN
>
> sockstat reports ?
> # sockstat | grep -E '874|815'
> ?        ?          ?     ?  tcp4   *:815                 *:*
> ?        ?          ?     ?  tcp6   *:874                 *:*
>
> rpcinfo -p reports just one port
> # rpcinfo -p| grep -E '874|815'
>     100021    0   tcp    815  nlockmgr
>     100021    1   tcp    815  nlockmgr
>     100021    3   tcp    815  nlockmgr
>     100021    4   tcp    815  nlockmgr
>
>
> The 874/tcp6 which belongs to rpc.lockd does not appear on this list.
> Is rpcinfo only for IPv4 and if yes,what tool do I use for IPv6 ?
>
>
>
>
>
> The grand question is of course, is there any tool to actually locate the
> processes that open ports and cannot be identified with sockstat?
>
> The second grand question. Why rpc.lockd is a different kind of process that
> cannot be located from sockstat? Other RPC processes are found using
> sockstat, as the following printing shows:
>
> # rpcinfo -p | grep 2049
>     100003    2   udp   2049  nfs
>     100003    3   udp   2049  nfs
>     100003    2   tcp   2049  nfs
>     100003    3   tcp   2049  nfs
>
>
> sockstat |grep 2049
> root     nfsd       41279 5  tcp4   *:2049                *:*
> root     nfsd       41279 6  tcp6   *:2049                *:*
>
>
> nfs is found using rpcinfo and also using sockstat.
>
> What rpc.lockd does and it is not found. After 25 years of sysadmin, I find
> it very strange for Freebsd to not being able to trace a listening port to
> an executable.

The situation here is that the socket is neither opened by nor owned by
any userland process. The rpc.lockd implementation is split into a
kernel piece and a userland piece, and much of the work is done in-
kernel. The in-kernel part of the code contacts the userland daemon
part for help when it needs to.

So the socket is created by the in-kernel part of lockd, and it is not
tied to any file descriptor. Tools which report on userland processes
use file descriptors to associate kernel resources with the processes
that own them. In this case, it is the kernel itself that owns the
socket, so it can't be reported as belonging to any userland process.

If you're interested in poking around in the code involved, see
nlm_server_main() in src/sys/nlm/nlm_prot_impl.c

-- Ian


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: userland process rpc.lockd opens untraceable ports...is something wrong here?

Cy Schubert-4
In reply to this post by Cy Schubert-4
On February 20, 2019 6:56:49 AM PST, Cy Schubert <[hidden email]> wrote:

>On February 20, 2019 6:02:17 AM PST, BBlister <[hidden email]>
>wrote:
>>After one suggestion on the questions list,  I used the rpcinfo -p but
>>this
>>does not print every unknown port. For example:
>>
>># netstat -an | grep -E '874|815'
>>tcp4       0      0 *.815                  *.*                  
>>LISTEN
>>tcp6       0      0 *.874                  *.*                  
>>LISTEN
>>
>>sockstat reports ?
>># sockstat | grep -E '874|815'
>>?        ?          ?     ?  tcp4   *:815                 *:*
>>?        ?          ?     ?  tcp6   *:874                 *:*
>>
>>rpcinfo -p reports just one port
>># rpcinfo -p| grep -E '874|815'
>>    100021    0   tcp    815  nlockmgr
>>    100021    1   tcp    815  nlockmgr
>>    100021    3   tcp    815  nlockmgr
>>    100021    4   tcp    815  nlockmgr
>>
>>
>>The 874/tcp6 which belongs to rpc.lockd does not appear on this list.
>>Is rpcinfo only for IPv4 and if yes,what tool do I use for IPv6 ?
>>
>>
>>
>>
>>
>>The grand question is of course, is there any tool to actually locate
>>the
>>processes that open ports and cannot be identified with sockstat?
>>
>>The second grand question. Why rpc.lockd is a different kind of
>process
>>that
>>cannot be located from sockstat? Other RPC processes are found using
>>sockstat, as the following printing shows:
>>
>># rpcinfo -p | grep 2049
>>    100003    2   udp   2049  nfs
>>    100003    3   udp   2049  nfs
>>    100003    2   tcp   2049  nfs
>>    100003    3   tcp   2049  nfs
>>
>>
>>sockstat |grep 2049
>>root     nfsd       41279 5  tcp4   *:2049                *:*
>>root     nfsd       41279 6  tcp6   *:2049                *:*
>>
>>
>>nfs is found using rpcinfo and also using sockstat.
>>
>>What rpc.lockd does and it is not found. After 25 years of sysadmin, I
>>find
>>it very strange for Freebsd to not being able to trace a listening
>port
>>to
>>an executable.
>>
>>
>>
>>--
>>Sent from:
>>http://freebsd.1045724.x6.nabble.com/freebsd-hackers-f4034256.html
>>_______________________________________________
>>[hidden email] mailing list
>>https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
>>To unsubscribe, send any mail to
>>"[hidden email]"
>
>Rpcinfo  displays rpcbind's mapping of RPC program numbers to ports.
>
>Sockstat and lsof provide the output you desire. Sockstat output below,
>lsof output is too difficult to cut and paste on a phone.
>
>3443  4  udp6   *:652                 *:*
>root     rpc.statd  3443  5  tcp6   *:652                 *:*
>root     rpc.statd  3443  6  udp4   *:652                 *:*
>root     rpc.statd  3443  7  tcp4   *:652                 *:*
>
>Your kernel and userland are not in sync.

My mistake.  This thread is about lockd, not statd.
--
Pardon the typos and autocorrect, small keyboard in use.
Cheers,
Cy Schubert <[hidden email]>
FreeBSD UNIX: <[hidden email]> Web: http://www.FreeBSD.org

        The need of the many outweighs the greed of the few.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: userland process rpc.lockd opens untraceable ports...is something wrong here?

BBlister
In reply to this post by Ian Lepore-3
Nice explanation, but why the rpcinfo -p does not print all the ports that
rpc.lockd is using? Perhaps rpcinfo is not IPV6 compatible, but can only
locate ipv4 ports?

The conclusion is that if something is reported by sockstat with '?' in the
process name, this means that the kernel  has opened it. Is this correct?

Thanks again for clarifying this,

Rgz,

BB



--
Sent from: http://freebsd.1045724.x6.nabble.com/freebsd-hackers-f4034256.html
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: userland process rpc.lockd opens untraceable ports...is something wrong here?

Ben Woods
In reply to this post by Ian Lepore-3
On Wed, 20 Feb 2019 at 11:03 pm, Ian Lepore <[hidden email]> wrote:

> On Wed, 2019-02-20 at 07:02 -0700, BBlister wrote:
> > # sockstat | grep -E '874|815'
> > ?        ?          ?     ?  tcp4   *:815                 *:*
> > ?        ?          ?     ?  tcp6   *:874                 *:*
> >
> > rpcinfo -p reports just one port
> > # rpcinfo -p| grep -E '874|815'
> >     100021    0   tcp    815  nlockmgr
> >     100021    1   tcp    815  nlockmgr
> >     100021    3   tcp    815  nlockmgr
> >     100021    4   tcp    815  nlockmgr
> >
>
> The situation here is that the socket is neither opened by nor owned by
> any userland process. The rpc.lockd implementation is split into a
> kernel piece and a userland piece, and much of the work is done in-
> kernel. The in-kernel part of the code contacts the userland daemon
> part for help when it needs to.
>
> So the socket is created by the in-kernel part of lockd, and it is not
> tied to any file descriptor. Tools which report on userland processes
> use file descriptors to associate kernel resources with the processes
> that own them. In this case, it is the kernel itself that owns the
> socket, so it can't be reported as belonging to any userland process.
>
> If you're interested in poking around in the code involved, see
> nlm_server_main() in src/sys/nlm/nlm_prot_impl.c
>
> -- Ian



A similar issue is discussed in this bug report:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212608

My personal opinion is that this is confusing and worrying for users, and a
better user experience would be if sockstat and lsof were able to detail
the owner of these open ports (either just “kernel” or better yet which
part of the kernel).

I have no idea if this is technically possible or how complicated it is. Is
anyone able to comment on this?

Regards,
Ben
--

--
From: Benjamin Woods
[hidden email]
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"