[PATCH] pmcannotate tool

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

[PATCH] pmcannotate tool

Attilio Rao-2
pmcannotate is a tool that prints out sources of a tool (in C or
assembly) with inlined profiling informations retrieved by a prior
pmcstat analysis.
If compared with things like callgraph generation, it prints out
profiling on a per-instance basis and this can be useful to find, for
example, badly handled caches, too high latency instructions, etc.

The tool usage is pretty simple:
pmcannotate [-a] [-h] [-k path] [-l level] samples.out binaryobj

where samples.out is a pmcstat raw output and binaryobj is the binary
object that has been profiled and is accessible for (ELF) symbols
retrieving.
The options are better described in manpages but briefly:
- a: performs analysis on the assembly rather than the C source
- h: usage and informations
- k: specify a path for the kernel in order to locate correct objects for it
- l: specify a lower boundary (in total percentage time) after which
functions will be displayed nomore.

A typical usage of pmcannotate can be some way of kernel annotation.
For example, you can follow the steps below:
1) Generate a pmc raw output of system samples:
# pmcstat -S ipm-unhalted-core-cycles -O samples.out
2) Copy the samples in the kernel building dir and cd there
# cp samples.out /usr/src/sys/i386/compile/GENERIC/ ; cd
/usr/src/sys/i386/compile/GENERIC/
3) Run pmcannotate
# pmcannotate -k . samples.out kernel.debug > kernel.ann

In the example above please note that kernel.debug has to be used in
order to produce a C annotated source. This happens because in order
to get the binary sources we rely on the "objdump -S" command which
wants binary compiled with debugging options.
If not debugging options are present assembly analynsis is still
possible, but no C-backed one will be available.
objdump is not the only one tool on which pmcannotare rely. Infact, in
order to have it working, pmcstat needs to be present too because we
need to retrieve, from the pmcstat raw output, informations about the
sampled PCs (in particular the name of the function they live within,
its start and ending addresses). As long as currently pmcstat doesn't
return those informations, a new option has been added to the tool
(-m) which can extract (from a raw pmcstat output) all pc sampled,
name of the functions and symbol bundaries they live within.

Also please note that pmcannotate suffers of 2 limitations.
Firstly, relying on objdump to dump the C source, with heavy
optimization levels and lots of inlines the code gets difficult to
read. Secondly, in particular on x86 but I guess it is not the only
one case, the sample is always attributed to the instruction directly
following the one that was interrupted. So in a C source view some
samples may be attributed to the line below the one you're interested
in.  It's also important to keep in mind that if a line is a jump
target or the start of a function the sample really belongs elsewhere.

The patch can be found here:
http://www.freebsd.org/~attilio/pmcannotate.diff/

where pmcannotate/ dir contains the code and needs to go under
/usr/src/usr.sbin/ and the patch has diffs against pmcstat and
Makefile.

This work has been developed on the behalf of Nokia with important
feedbacks and directions from Jeff Roberson.

Testing and feedbacks (before it hits the tree) are welcome.

Thanks,
Attilio


--
Peace can only be achieved by understanding - A. Einstein
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] pmcannotate tool

Joseph Koshy
> pmcannotate is a tool that prints out sources of a tool (in C or
> assembly) with inlined profiling informations retrieved by a prior
> pmcstat analysis.

[snip]

> This work has been developed on the behalf of Nokia with important
> feedbacks and directions from Jeff Roberson.

Are you  "rookie" on IRC, who used to catch me with tons of questions
whenever I logged on?

> Testing and feedbacks (before it hits the tree) are welcome.

The pmcstat changes seem fine.

Koshy
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] pmcannotate tool

Alexander Leidinger
In reply to this post by Attilio Rao-2
Quoting Attilio Rao <[hidden email]> (from Sun, 23 Nov 2008  
14:02:22 +0100):

> pmcannotate is a tool that prints out sources of a tool (in C or
> assembly) with inlined profiling informations retrieved by a prior
> pmcstat analysis.
> If compared with things like callgraph generation, it prints out
> profiling on a per-instance basis and this can be useful to find, for
> example, badly handled caches, too high latency instructions, etc.

Can this also be used to do some code coverage analysis? What I'm  
interested in is to enable something, run some tests in userland,  
disable this something, and then run a tool which tells me which parts  
of specific functions where run or not.

At first I hoped I can use dtrace for this... I had a dtrace training  
and seen the userland probes in action, where you can trace every ASM  
instruction, but unfortunately you can not do this with kernel probes.  
I tried with fbt and syscall on a Solaris 10 machine. I haven't tested  
with FreeBSD-dtrace yet, but I doubt it is more advanced in this  
regard than the Solaris dtrace. So I'm still searching.

Bye,
Alexander.

--
We should keep the Panama Canal.  After all, we stole it fair and square.
                -- S. I. Hayakawa

http://www.Leidinger.net    Alexander @ Leidinger.net: PGP ID = B0063FE7
http://www.FreeBSD.org       netchild @ FreeBSD.org  : PGP ID = 72077137
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] pmcannotate tool

Ray Kinsella
In reply to this post by Attilio Rao-2
I know I am going to really show my FreeBSD ignorance here, but this is a
patch of FreeBSD 8.0 Current isn't it ?

Thanks

Ray Kinsella

On Sun, Nov 23, 2008 at 1:02 PM, Attilio Rao <[hidden email]> wrote:

> pmcannotate is a tool that prints out sources of a tool (in C or
> assembly) with inlined profiling informations retrieved by a prior
> pmcstat analysis.
> If compared with things like callgraph generation, it prints out
> profiling on a per-instance basis and this can be useful to find, for
> example, badly handled caches, too high latency instructions, etc.
>
> The tool usage is pretty simple:
> pmcannotate [-a] [-h] [-k path] [-l level] samples.out binaryobj
>
> where samples.out is a pmcstat raw output and binaryobj is the binary
> object that has been profiled and is accessible for (ELF) symbols
> retrieving.
> The options are better described in manpages but briefly:
> - a: performs analysis on the assembly rather than the C source
> - h: usage and informations
> - k: specify a path for the kernel in order to locate correct objects for
> it
> - l: specify a lower boundary (in total percentage time) after which
> functions will be displayed nomore.
>
> A typical usage of pmcannotate can be some way of kernel annotation.
> For example, you can follow the steps below:
> 1) Generate a pmc raw output of system samples:
> # pmcstat -S ipm-unhalted-core-cycles -O samples.out
> 2) Copy the samples in the kernel building dir and cd there
> # cp samples.out /usr/src/sys/i386/compile/GENERIC/ ; cd
> /usr/src/sys/i386/compile/GENERIC/
> 3) Run pmcannotate
> # pmcannotate -k . samples.out kernel.debug > kernel.ann
>
> In the example above please note that kernel.debug has to be used in
> order to produce a C annotated source. This happens because in order
> to get the binary sources we rely on the "objdump -S" command which
> wants binary compiled with debugging options.
> If not debugging options are present assembly analynsis is still
> possible, but no C-backed one will be available.
> objdump is not the only one tool on which pmcannotare rely. Infact, in
> order to have it working, pmcstat needs to be present too because we
> need to retrieve, from the pmcstat raw output, informations about the
> sampled PCs (in particular the name of the function they live within,
> its start and ending addresses). As long as currently pmcstat doesn't
> return those informations, a new option has been added to the tool
> (-m) which can extract (from a raw pmcstat output) all pc sampled,
> name of the functions and symbol bundaries they live within.
>
> Also please note that pmcannotate suffers of 2 limitations.
> Firstly, relying on objdump to dump the C source, with heavy
> optimization levels and lots of inlines the code gets difficult to
> read. Secondly, in particular on x86 but I guess it is not the only
> one case, the sample is always attributed to the instruction directly
> following the one that was interrupted. So in a C source view some
> samples may be attributed to the line below the one you're interested
> in.  It's also important to keep in mind that if a line is a jump
> target or the start of a function the sample really belongs elsewhere.
>
> The patch can be found here:
> http://www.freebsd.org/~attilio/pmcannotate.diff/<http://www.freebsd.org/%7Eattilio/pmcannotate.diff/>
>
> where pmcannotate/ dir contains the code and needs to go under
> /usr/src/usr.sbin/ and the patch has diffs against pmcstat and
> Makefile.
>
> This work has been developed on the behalf of Nokia with important
> feedbacks and directions from Jeff Roberson.
>
> Testing and feedbacks (before it hits the tree) are welcome.
>
> Thanks,
> Attilio
>
>
> --
> Peace can only be achieved by understanding - A. Einstein
> _______________________________________________
> [hidden email] mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-performance
> To unsubscribe, send any mail to "
> [hidden email]"
>
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] pmcannotate tool

Attilio Rao-2
In reply to this post by Alexander Leidinger
2008/11/23, Alexander Leidinger <[hidden email]>:

> Quoting Attilio Rao <[hidden email]> (from Sun, 23 Nov 2008 14:02:22
> +0100):
>
>
> > pmcannotate is a tool that prints out sources of a tool (in C or
> > assembly) with inlined profiling informations retrieved by a prior
> > pmcstat analysis.
> > If compared with things like callgraph generation, it prints out
> > profiling on a per-instance basis and this can be useful to find, for
> > example, badly handled caches, too high latency instructions, etc.
> >
>
>  Can this also be used to do some code coverage analysis? What I'm
> interested in is to enable something, run some tests in userland, disable
> this something, and then run a tool which tells me which parts of specific
> functions where run or not.

Yes, this is exactly what it does.
You can see traces for any sampled PC and so get a profiling anslysis
on a per-instance basis.

Thanks,
Attilio


--
Peace can only be achieved by understanding - A. Einstein
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] pmcannotate tool

Attilio Rao-2
In reply to this post by Ray Kinsella
2008/11/23, Ray Kinsella <[hidden email]>:
> I know I am going to really show my FreeBSD ignorance here, but this is a
>  patch of FreeBSD 8.0 Current isn't it ?

Yes, it is for 8.0.
Is it giving you problems?

Thanks,
Attilio


--
Peace can only be achieved by understanding - A. Einstein
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] pmcannotate tool

Jeff Roberson-2
In reply to this post by Attilio Rao-2

On Mon, 24 Nov 2008, Attilio Rao wrote:

> 2008/11/23, Alexander Leidinger <[hidden email]>:
>> Quoting Attilio Rao <[hidden email]> (from Sun, 23 Nov 2008 14:02:22
>> +0100):
>>
>>
>>> pmcannotate is a tool that prints out sources of a tool (in C or
>>> assembly) with inlined profiling informations retrieved by a prior
>>> pmcstat analysis.
>>> If compared with things like callgraph generation, it prints out
>>> profiling on a per-instance basis and this can be useful to find, for
>>> example, badly handled caches, too high latency instructions, etc.
>>>
>>
>>  Can this also be used to do some code coverage analysis? What I'm
>> interested in is to enable something, run some tests in userland, disable
>> this something, and then run a tool which tells me which parts of specific
>> functions where run or not.
>
> Yes, this is exactly what it does.
> You can see traces for any sampled PC and so get a profiling anslysis
> on a per-instance basis.

I would add that it is only sampled so you don't see every instruction
executed.  You can use gcov for that however.  That's precisely what it's
for.

Thanks,
Jeff

>
> Thanks,
> Attilio
>
>
> --
> Peace can only be achieved by understanding - A. Einstein
> _______________________________________________
> [hidden email] mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-arch
> To unsubscribe, send any mail to "[hidden email]"
>
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] pmcannotate tool

Alexander Leidinger
In reply to this post by Attilio Rao-2
Quoting Attilio Rao <[hidden email]> (from Mon, 24 Nov 2008  
00:46:29 +0100):

> 2008/11/23, Alexander Leidinger <[hidden email]>:
>> Quoting Attilio Rao <[hidden email]> (from Sun, 23 Nov 2008 14:02:22
>> +0100):
>>
>>
>> > pmcannotate is a tool that prints out sources of a tool (in C or
>> > assembly) with inlined profiling informations retrieved by a prior
>> > pmcstat analysis.
>> > If compared with things like callgraph generation, it prints out
>> > profiling on a per-instance basis and this can be useful to find, for
>> > example, badly handled caches, too high latency instructions, etc.
>> >
>>
>>  Can this also be used to do some code coverage analysis? What I'm
>> interested in is to enable something, run some tests in userland, disable
>> this something, and then run a tool which tells me which parts of specific
>> functions where run or not.
>
> Yes, this is exactly what it does.
> You can see traces for any sampled PC and so get a profiling anslysis
> on a per-instance basis.

Cool. Would be great if you could provide some example in the man page  
or as a script which shows how to do this for kernel code. This would  
immediately show us how good our regression tests are for their  
specific areas of test.

Bye,
Alexander.

--
In a family recipe you just discovered in an old book,
the most vital measurement will be illegible.

http://www.Leidinger.net    Alexander @ Leidinger.net: PGP ID = B0063FE7
http://www.FreeBSD.org       netchild @ FreeBSD.org  : PGP ID = 72077137
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] pmcannotate tool

Alexander Leidinger
In reply to this post by Jeff Roberson-2
Quoting Jeff Roberson <[hidden email]> (from Sun, 23 Nov 2008  
13:50:35 -1000 (HST)):

>
> On Mon, 24 Nov 2008, Attilio Rao wrote:
>
>> 2008/11/23, Alexander Leidinger <[hidden email]>:
>>> Quoting Attilio Rao <[hidden email]> (from Sun, 23 Nov 2008 14:02:22
>>> +0100):
>>>
>>>
>>>> pmcannotate is a tool that prints out sources of a tool (in C or
>>>> assembly) with inlined profiling informations retrieved by a prior
>>>> pmcstat analysis.
>>>> If compared with things like callgraph generation, it prints out
>>>> profiling on a per-instance basis and this can be useful to find, for
>>>> example, badly handled caches, too high latency instructions, etc.
>>>>
>>>
>>> Can this also be used to do some code coverage analysis? What I'm
>>> interested in is to enable something, run some tests in userland, disable
>>> this something, and then run a tool which tells me which parts of specific
>>> functions where run or not.
>>
>> Yes, this is exactly what it does.
>> You can see traces for any sampled PC and so get a profiling anslysis
>> on a per-instance basis.
>
> I would add that it is only sampled so you don't see every  
> instruction executed.  You can use gcov for that however.  That's  
> precisely what it's for.

How to use gcov for the kernel?

Bye,
Alexander.

--
If only you knew she loved you, you could
face the uncertainty of whether you love her.

http://www.Leidinger.net    Alexander @ Leidinger.net: PGP ID = B0063FE7
http://www.FreeBSD.org       netchild @ FreeBSD.org  : PGP ID = 72077137
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] pmcannotate tool

Ray Kinsella
In reply to this post by Attilio Rao-2
No, not at all.
I am using 6.2 and 7.0 at the moment, I will build another disk with FreeBSD
8.0-CURRENT.

Thanks

Ray Kinsella


On Sun, Nov 23, 2008 at 11:47 PM, Attilio Rao <[hidden email]> wrote:

> 2008/11/23, Ray Kinsella <[hidden email]>:
> > I know I am going to really show my FreeBSD ignorance here, but this is a
> >  patch of FreeBSD 8.0 Current isn't it ?
>
> Yes, it is for 8.0.
> Is it giving you problems?
>
> Thanks,
> Attilio
>
>
> --
> Peace can only be achieved by understanding - A. Einstein
>
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] pmcannotate tool

George Neville-Neil-3
In reply to this post by Attilio Rao-2
At Sun, 23 Nov 2008 14:02:22 +0100,
Attilio Rao wrote:

>
> pmcannotate is a tool that prints out sources of a tool (in C or
> assembly) with inlined profiling informations retrieved by a prior
> pmcstat analysis.
> If compared with things like callgraph generation, it prints out
> profiling on a per-instance basis and this can be useful to find, for
> example, badly handled caches, too high latency instructions, etc.
>
> The tool usage is pretty simple:
> pmcannotate [-a] [-h] [-k path] [-l level] samples.out binaryobj
>
> where samples.out is a pmcstat raw output and binaryobj is the binary
> object that has been profiled and is accessible for (ELF) symbols
> retrieving.
> The options are better described in manpages but briefly:
> - a: performs analysis on the assembly rather than the C source
> - h: usage and informations
> - k: specify a path for the kernel in order to locate correct objects for it
> - l: specify a lower boundary (in total percentage time) after which
> functions will be displayed nomore.
>
> A typical usage of pmcannotate can be some way of kernel annotation.
> For example, you can follow the steps below:
> 1) Generate a pmc raw output of system samples:
> # pmcstat -S ipm-unhalted-core-cycles -O samples.out
> 2) Copy the samples in the kernel building dir and cd there
> # cp samples.out /usr/src/sys/i386/compile/GENERIC/ ; cd
> /usr/src/sys/i386/compile/GENERIC/
> 3) Run pmcannotate
> # pmcannotate -k . samples.out kernel.debug > kernel.ann
>
> In the example above please note that kernel.debug has to be used in
> order to produce a C annotated source. This happens because in order
> to get the binary sources we rely on the "objdump -S" command which
> wants binary compiled with debugging options.
> If not debugging options are present assembly analynsis is still
> possible, but no C-backed one will be available.
> objdump is not the only one tool on which pmcannotare rely. Infact, in
> order to have it working, pmcstat needs to be present too because we
> need to retrieve, from the pmcstat raw output, informations about the
> sampled PCs (in particular the name of the function they live within,
> its start and ending addresses). As long as currently pmcstat doesn't
> return those informations, a new option has been added to the tool
> (-m) which can extract (from a raw pmcstat output) all pc sampled,
> name of the functions and symbol bundaries they live within.
>
> Also please note that pmcannotate suffers of 2 limitations.
> Firstly, relying on objdump to dump the C source, with heavy
> optimization levels and lots of inlines the code gets difficult to
> read. Secondly, in particular on x86 but I guess it is not the only
> one case, the sample is always attributed to the instruction directly
> following the one that was interrupted. So in a C source view some
> samples may be attributed to the line below the one you're interested
> in.  It's also important to keep in mind that if a line is a jump
> target or the start of a function the sample really belongs elsewhere.
>
> The patch can be found here:
> http://www.freebsd.org/~attilio/pmcannotate.diff/
>
> where pmcannotate/ dir contains the code and needs to go under
> /usr/src/usr.sbin/ and the patch has diffs against pmcstat and
> Makefile.
>
> This work has been developed on the behalf of Nokia with important
> feedbacks and directions from Jeff Roberson.
>
> Testing and feedbacks (before it hits the tree) are welcome.
>
Hi,

First of all, this is excellent work. As soon as this and some other
changes in PMC hit 7.x I'll be rolling this out to all the developers
I work with.  I've tested this on amd64 on HEAD, and with the changes
we have talked about privately (%jx vs. %x) it works quite well.

Secondly, I would like to request a feature.  I would like to be able
to get output in a more easily parsable format so I can write some
Emacs code to highlight C code with the output.  I'd like something
along the lines of:

path:function:line:percentage

Keep up the good work!

Later,
George
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] pmcannotate tool

Vadim Goncharov
In reply to this post by Attilio Rao-2
Hi Attilio Rao!

On Sun, 23 Nov 2008 14:02:22 +0100; Attilio Rao wrote about '[PATCH] pmcannotate tool':

> pmcannotate is a tool that prints out sources of a tool (in C or
> assembly) with inlined profiling informations retrieved by a prior
> pmcstat analysis.
> If compared with things like callgraph generation, it prints out
> profiling on a per-instance basis and this can be useful to find, for
> example, badly handled caches, too high latency instructions, etc.
[...]
> objdump is not the only one tool on which pmcannotare rely. Infact, in
> order to have it working, pmcstat needs to be present too because we
> need to retrieve, from the pmcstat raw output, informations about the
> sampled PCs (in particular the name of the function they live within,
> its start and ending addresses). As long as currently pmcstat doesn't
> return those informations, a new option has been added to the tool
> (-m) which can extract (from a raw pmcstat output) all pc sampled,
> name of the functions and symbol bundaries they live within.
[...]

> The patch can be found here:
> http://www.freebsd.org/~attilio/pmcannotate.diff/

> where pmcannotate/ dir contains the code and needs to go under
> /usr/src/usr.sbin/ and the patch has diffs against pmcstat and
> Makefile.

It looks like this the thing I need, please see parallel thread
"hwpmc granularity and 6.4 network performance", I have CPU hog ipfw_chk()
which is 1200+ lines long and want to know which parts are most hogs.

But that part of your patch which is for pmcstat isn't applying to 6.4 :(
Could you please provide another for this ver?

--
WBR, Vadim Goncharov. ICQ#166852181       mailto:[hidden email]
[Moderator of RU.ANTI-ECOLOGY][FreeBSD][http://antigreen.org][LJ:/nuclight]

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"