Tuesday, March 18, 2008

Reboot? Real customers don't reboot.

To get clear, clean, and repeat'able performance results, performance teams in the labs generally don't think twice about rebooting the system and starting fresh (after all, who knows what someone has done on the system - particularly those kernel programmers).

In my experience though, to balance this automatic tendency, it's important to note that re-booting a system isn't normal behavior for real-life customers.

For example, when I suggest that a customer re-boot his/her system, there's usually a perceptible pause, sometimes even a quiet chuckle. Turns out most customers are quite happy with their systems running Linux, and simply don't consider the process of re-booting as anything normal. I'm not surprised, but it does mean that the tuning options practically available to customers have to be done dynamically and not a kernel boot option.

This aspect continues to be improved in the Linux with work across the operating system. The ability to control energy consumption, system resource usage, adding/removing CPUs, and adding/removing memory are all examples of cool things being worked on in the Linux community.

For the performance team, we're particularly interested in the ability to control things like SMT on and off (something needed on Power systems), the number of CPUs running, and minimizing kernel memory fragmentation.

Some pieces are there, some are emerging, and some are being invested in. I'll hunt some examples down and post them here in the coming days.

1 comment:

strosaker said...

Hi, Bill:

Great post; good to hear that many Linux folks consider a reboot to be an unusual operation!

The ppc64_cpu script (in powerpc-utils-papr) will let you view if SMT is enabled, and let you enable/disable it on the fly. Run "/usr/sbin/ppc64_cpu --smt" to view if SMT is enabled.

Will CPU hotplug do what you need in terms of adding/removing processors on the fly? You can use the drmgr command (also in powerpc-utils-papr) to add/remove CPUs on systems that aren't managed by an HMC.

Mike

One of many bloggers.

Bill Buros

Bill leads an IBM Linux performance team in Austin Tx (the only place really to live in Texas). The team is focused on IBM's Power offerings (old, new, and future) working with IBM's Linux Technology Center (the LTC). While the focus is primarily on Power systems, the team also analyzes and improves overall Linux performance for IBM's xSeries products (both Intel and AMD) , driving performance improvements which are both common for Linux and occasionally unique to the hardware offerings.

Performance analysis techniques, tools, and approaches are nicely common across Linux. Having worked for years in performance, there are still daily reminders of how much there is to learn in this space, so in many ways this blog is simply another vehicle in the continuing journey to becoming a more experienced "performance professional". One of several journeys in life.

The Usual Notice

The postings on this site are my own and don't necessarily represent IBM's positions, strategies, or opinions, try as I might to influence them.