pkg audit false negatives

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

pkg audit false negatives

Roger Marquis
In the past pkg-audit and even pkg-version have not been reliable tools
where installed ports or packages have been subsequently discontinued or
renamed.  Today, however, I notice that dovecot2 is still showing up in
the output of pkg-version despite the port having been renamed to
dovecot (without the numeric suffix) several days ago.

Does this mean there has been a policy change?  If so does it cover
pkg-audit as well?

Roger
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-security
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: pkg audit false negatives

Remko Lodder
Hi Roger,

> On 11 Aug 2017, at 04:41, Roger Marquis <[hidden email]> wrote:
>
> In the past pkg-audit and even pkg-version have not been reliable tools
> where installed ports or packages have been subsequently discontinued or
> renamed.  Today, however, I notice that dovecot2 is still showing up in
> the output of pkg-version despite the port having been renamed to
> dovecot (without the numeric suffix) several days ago.

Yes, there is a difference between renaming a port, and renaming the vuxml (which is the
database behind pkg audit etc.) entries. The entries are listed as ‘dovecot2-*’ there and
when renaming a port these entries should ideally be renamed too.

It seems that that was not under consideration at the name change moment(s).

I’ll try to look into this (starting by prodding the person(s) who did the rename) and asking them
to rename the entries in vuxml as well.

>
> Does this mean there has been a policy change?  If so does it cover
> pkg-audit as well?

There had been no policy change. The application backend is just matching on what
was recorded at the moment it was added.

Thanks for the notification though, we should add that to the porters-handbook.

Cheers
REmko

>
> Roger
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-security
> To unsubscribe, send any mail to "[hidden email]"


signature.asc (817 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: pkg audit false negatives

Remko Lodder

Hi Roger,

> On 11 Aug 2017, at 17:14, Remko Lodder <[hidden email]> wrote:
>
> Hi Roger,
>
>> On 11 Aug 2017, at 04:41, Roger Marquis <[hidden email]> wrote:
>>
>> In the past pkg-audit and even pkg-version have not been reliable tools
>> where installed ports or packages have been subsequently discontinued or
>> renamed.  Today, however, I notice that dovecot2 is still showing up in
>> the output of pkg-version despite the port having been renamed to
>> dovecot (without the numeric suffix) several days ago.
>
It had been resolved for dovecot (it will now match both variants, since people might still have
the old variant of the port installed) and there is a new paragraph added to the porters handbook
which tells that we need to have a look at the vuxml entries.

Hope this solves your issue,
Remko


signature.asc (817 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: pkg audit false negatives

Roger Marquis
> It had been resolved for dovecot (it will now match both variants, since people might still have
> the old variant of the port installed) and there is a new paragraph added to the porters handbook
> which tells that we need to have a look at the vuxml entries.

Thanks Remko.

> Hope this solves your issue,

It may for renamed ports/pkgs but doesn't appear to for deprecations.
Once ports are dropped they do not show up in pkg-audit despite having
been installed via pkg and/or ports.  That's the false negative that
appears to still be a problem.

Roger
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-security
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: pkg audit false negatives

Remko Lodder

> On 11 Aug 2017, at 23:47, Roger Marquis <[hidden email]> wrote:
>
>> It had been resolved for dovecot (it will now match both variants, since people might still have
>> the old variant of the port installed) and there is a new paragraph added to the porters handbook
>> which tells that we need to have a look at the vuxml entries.
>
> Thanks Remko.

No problemo :)

>
>> Hope this solves your issue,
>
> It may for renamed ports/pkgs but doesn't appear to for deprecations.
> Once ports are dropped they do not show up in pkg-audit despite having
> been installed via pkg and/or ports.  That's the false negative that
> appears to still be a problem.

Ports / pkgs that get renamed are now changed and/or added in VuXML as well.
So the old variant and the new variant of the name’s would both be listed in pkg audit.

pkg audit parses VuXML, it also does a check on what is locally registered in it’s database.

For example if you have a/b installed. And that has a marking in VuXML : <package>b</package>
then it would hit on the package you have. If a/b gets removed for some reason, and it is still in VuXML
and you have it locally registered. Then it would be still be matched (or should).

If an entry is removed from the ports/pkg tree’s and it is also removed from VuXML, then yes, it will
no longer get marked in your local installation. That’s a bit of a chicken and egg basically. Although
I do not recall that it ever happened that ports that are no longer there, are removed from VuXML as
well. (And I follow that since 2004).

Do you have a more concrete example that we can dive into to see what is going on/going wrong?

Cheers
Remko


>
> Roger


signature.asc (817 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: pkg audit false negatives

Roger Marquis
On Fri, 11 Aug 2017, Remko Lodder wrote:

> If an entry is removed from the ports/pkg tree?s and it is also removed
> from VuXML, then yes, it will no longer get marked in your local
> installation. That?s a bit of a chicken and egg basically. Although I do
> not recall that it ever happened that ports that are no longer there, are
> removed from VuXML as well. (And I follow that since 2004).
>
> Do you have a more concrete example that we can dive into to see what is
> going on/going wrong?

Should be able to find missing vulxml entries for most anything that has
been deprecated from the ports tree but most of the ones I've seen are
for web programming languages, particularly php.

For example when php5X was dropped it also disappeared from vulxml, with
no small number of servers still using it.  If those sites depended on
pkg-audit to tell them they had a vulnerability, well, they were out of
luck.  There was no warning, no error, no disclaimer, pkg-audit did and
still does nothing different than it would for a non-vulnerable port or
package.

There may be more vulnerabilities in the wild from non-packaged base as
it is larger but at least people are working on that.  Pkg-audit
tracking of installed but deprecated ports OTOH, seems to have fallen
through the cracks.  Even the FreeBSD Foundation and the ports-security
teams appear to be ignoring this issue.

Roger Marquis
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-security
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: pkg audit false negatives

Remko Lodder

> On 12 Aug 2017, at 02:37, Roger Marquis <[hidden email]> wrote:
>
> On Fri, 11 Aug 2017, Remko Lodder wrote:
>
>> If an entry is removed from the ports/pkg tree?s and it is also removed
>> from VuXML, then yes, it will no longer get marked in your local
>> installation. That?s a bit of a chicken and egg basically. Although I do
>> not recall that it ever happened that ports that are no longer there, are
>> removed from VuXML as well. (And I follow that since 2004).
>> Do you have a more concrete example that we can dive into to see what is
>> going on/going wrong?
>
> Should be able to find missing vulxml entries for most anything that has
> been deprecated from the ports tree but most of the ones I've seen are
> for web programming languages, particularly php.
I do not think that holds:

<vuln vid="b6402385-533b-11e6-a7bd-14dae9d210b8">
17521    <topic>php -- multiple vulnerabilities</topic>
17522    <affects>
17523      <package>
17524        <name>php55</name>
17525        <range><lt>5.5.38</lt></range>
17526      </package>

This is an entry from svnweb, for php55, which was added in 2016(07-26).

So this entry is there. Thus it did not disappear from VuXML at least.

Can you show such a packet from your local installation(s) and present a
``pkg audit -F`` along side it. I would also like to see a detailed pkg info
from the affected pkg.

Thanks a lot in advance,
Remko

>
> For example when php5X was dropped it also disappeared from vulxml, with
> no small number of servers still using it.  If those sites depended on
> pkg-audit to tell them they had a vulnerability, well, they were out of
> luck.  There was no warning, no error, no disclaimer, pkg-audit did and
> still does nothing different than it would for a non-vulnerable port or
> package.
>
> There may be more vulnerabilities in the wild from non-packaged base as
> it is larger but at least people are working on that.  Pkg-audit
> tracking of installed but deprecated ports OTOH, seems to have fallen
> through the cracks.  Even the FreeBSD Foundation and the ports-security
> teams appear to be ignoring this issue.
>
> Roger Marquis


signature.asc (817 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: pkg audit false negatives

Roger Marquis
> I do not think that holds:
>
> <vuln vid="b6402385-533b-11e6-a7bd-14dae9d210b8">
> 17521    <topic>php -- multiple vulnerabilities</topic>
> 17522    <affects>
> 17523      <package>
> 17524        <name>php55</name>
> 17525        <range><lt>5.5.38</lt></range>
> 17526      </package>
>
> This is an entry from svnweb, for php55, which was added in 2016(07-26).
>
> So this entry is there. Thus it did not disappear from VuXML at least.

You are right Remko.  It looks like there was a policy or at least a
practice change about a year ago.  Even have an archived email from
Gerhard Schmidt who first noticed it back in Aug 2016.  My fault for not
doing sufficient fact rechecking,

So we are safe from false negatives after all.  Hurray, I can stop
relying on pkg-version (for this).

That leaves just unpackaged base as FreeBSD's remaining audit weakness.

Roger


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-security
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: pkg audit false negatives

Remko Lodder

> On 14 Aug 2017, at 05:32, Roger Marquis <[hidden email]> wrote:
>
>> I do not think that holds:
>>
>> <vuln vid="b6402385-533b-11e6-a7bd-14dae9d210b8">
>> 17521    <topic>php -- multiple vulnerabilities</topic>
>> 17522    <affects>
>> 17523      <package>
>> 17524        <name>php55</name>
>> 17525        <range><lt>5.5.38</lt></range>
>> 17526      </package>
>>
>> This is an entry from svnweb, for php55, which was added in 2016(07-26).
>>
>> So this entry is there. Thus it did not disappear from VuXML at least.
>
> You are right Remko.  It looks like there was a policy or at least a
> practice change about a year ago.  Even have an archived email from
> Gerhard Schmidt who first noticed it back in Aug 2016.  My fault for not
> doing sufficient fact rechecking,
>
> So we are safe from false negatives after all.  Hurray, I can stop
> relying on pkg-version (for this).
>
> That leaves just unpackaged base as FreeBSD's remaining audit weakness.
Hi, I am happy that I can reduce your worry factor a bit ;-)

Can you share what the audit weakness is? freebsd-update cron checks
whether or not an update is available and then emails you. If you run
-RELEASE, then that means that either an EN or SA had been released..

Cheers
Remko


>
> Roger
>
>
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-security
> To unsubscribe, send any mail to "[hidden email]"


signature.asc (817 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: pkg audit false negatives

Roger Marquis
>> That leaves just unpackaged base as FreeBSD's remaining audit weakness.
>
> Hi, I am happy that I can reduce your worry factor a bit ;-)
>
> Can you share what the audit weakness is? freebsd-update cron checks
> whether or not an update is available and then emails you. If you run
> -RELEASE, then that means that either an EN or SA had been released..

Can you run freebsd-update on a -RELEASE system installed and maintained
with buildworld/buildkernel/installkernel/installworld?

Though it's been more than a year since the last time I tested
freebsd-update, on Virtualbox VMs, it resulted in too many bricked
systems to rely on.  That may have changed but it would still be better
to build a packaged base or have reproduceable builds as lighter-weight
solutions to the base audit issue.

Roger
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-security
To unsubscribe, send any mail to "[hidden email]"
Loading...