Oprofile - a Visual interface

So in performance, we profile systems a lot. In our performance testing and analysis, we usually rely on normal Linux command-line interfaces and a tailored automated analysis framework for running workloads, micro-benchmarks, and industry standard benchmarks. The test framework is similar to several derivative branches worked on in the Linux community by people like Martin Bligh[1], Andy Whitcroft, and many others. For example, in the community, AutoTest is one of the latest incarnations of automated test environments[2]. Our current autobench framework is driven primarily these days by a member of our Austin analysis team, Karl Rister.

For our performance work, we wrap and re-wrap things with simplifying scripts and strive for parse'able text output. The key focus for our performance work is to be able to crank the same workload across different OS and software levels, and use/compare varying IBM hardware platforms that Linux supports (x86, Power, Blades, Servers, etc). This helps us more quickly understand where Linux needs to be improved, or if there is something to be improved in the various hardware/firmware combinations and configurations that customers would want to use. Or in some cases when working with new workloads, in figuring out what exactly we're running, and working to understand what we're really measuring.

But what about the more elaborate user-graphical interfaces to performance tools? Not so much. The two user interface assists we do deploy are wrapping things with web interfaces (easy to crank out, easy to adjust), and graphing the generated data - more simply put: "a picture is worth a thousand words".

That said, someone recently asked about the latest Visual Performance Analyzer (VPA) tool which can be used with oprofile on Linux. Last time I had played with it was several years ago, so it was time to give it a new shot. And I found it was pretty nice. It won't replace our automation framework, but it was a nice alternative way to get and view profile information, especially from a remote system looking "into" a system under test. And sometime over the coming weeks, I'd like to see if we can tap into our pool of previously run and saved profiled workloads. Not sure if that's possible, but we'll see.

So here's what I found, with some quick context definitions as needed.

Using a profiler[3] is the simple process of measuring a program or system's behavior while things are running. The simple approach is to sample what each CPU is doing on a regular basis - say every 100,000 timer ticks - and keep track of where in the program (or the system) things are spending the most time. Profilers can do this "automatically", and can be tailored to do some pretty clever tricks using different hardware counters.

Application programmers use profiling to figure out how to optimize their programs, usually improving data structures, loops, and program structure. Each programming language is unique, and has unique performance challenges. In our world though, we're focused on a hierarchy of performance challenges. With new hardware or new operating system code, we are generally focused more on how to optimize Linux for the hardware, looking for the typical "performance hazards" which can be improved in software, firmware, or hardware.

For VPA, I downloaded the x86 rpm version of VPA from IBM's alphaWorks[4] and installed that on a recently installed RHEL 5.1 Linux laptop. and also tried the zip'ed version for Windows. While the files were downloading, I did read on the alphaWorks website:

What is Visual Performance Analyzer?

Visual Performance Analyzer (VPA) is an Eclipse-based performance visualization toolkit. It consists of five major components: Profile Analyzer, Code Analyzer, Pipeline Analyzer, Counter Analyzer, Trace Analyzer, and Control Flow Analyzer.

Interesting. Eclipse-based? So do I need to install the full Eclipse environment? Apparently not. Maynard Johnson (our dependable oprofile Answer-Man) told me that Eclipse now supports a "Rich Client Platform" (RCP) which allows GUI applications to build with Eclipse widgets, without a full Eclipse platform. And sure enough, the two files I downloaded from alphaWorks were both called "vpa-rcp-6.0.0*".

VPA comes with several tools included. The one I was interested in was the Profiler Analyzer. I tried examples from both the Linux client and the Windows client.

These screen shot images are from the Windows client using SnagIt [5]. I really wish we had this tool available on Linux... I recommend going to the Techsmith website and request a SnagIt product port to Linux clients, especially since they're already considering one to the Mac (per their website).

Invoking the Profile Analyzer, I simply defined the connection to the server I was testing. Plenty of options. Lots of flexibility. I simply wanted to see if I could connect to the system, and profile a couple of seconds of the system sitting there idling along. By the way, oprofile was previously installed on the system being tested - but that's easy to do as well.

Worked easily. Connected cleanly. Push-buttons to start and stop profiling. Then I could click on the output, and poke around to see where things were running.

(Quick aside: for those with good eyes, the system being tested was named xbox.ltc.austin.ibm.com - it's just a normal Power 5 based system running SLES 10. The analyst who named a set of three related systems has caused no end of raised eyebrows as we report on work we've done on systems named xbox, ps3, and wii.... clever... but mis-leading at times....)

The system under test wasn't doing anything (just idle), but I was impressed with how easy this was. In the coming weeks, we'll play more. But you should consider downloading VPA as something easy to try, play with, and get results.

Bill B

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
References:

[1] Martin Bligh - http://www.linuxsymposium.org/2006/view_bio.php?id=1697
[2] Autotest - http://test.kernel.org/autotest/
[3] Profilers - http://en.wikipedia.org/wiki/Profiler_%28computer_science%29
[4] Visual Performance Analyzer - http://www.alphaworks.ibm.com/tech/vpa
[5] SnagIt - http://www.techsmith.com/screen-capture.asp