To get clear, clean, and repeat'able performance results, performance teams in the labs generally don't think twice about rebooting the system and starting fresh (after all, who knows what someone has done on the system - particularly those kernel programmers).
In my experience though, to balance this automatic tendency, it's important to note that re-booting a system isn't normal behavior for real-life customers.
For example, when I suggest that a customer re-boot his/her system, there's usually a perceptible pause, sometimes even a quiet chuckle. Turns out most customers are quite happy with their systems running Linux, and simply don't consider the process of re-booting as anything normal. I'm not surprised, but it does mean that the tuning options practically available to customers have to be done dynamically and not a kernel boot option.
This aspect continues to be improved in the Linux with work across the operating system. The ability to control energy consumption, system resource usage, adding/removing CPUs, and adding/removing memory are all examples of cool things being worked on in the Linux community.
For the performance team, we're particularly interested in the ability to control things like SMT on and off (something needed on Power systems), the number of CPUs running, and minimizing kernel memory fragmentation.
Some pieces are there, some are emerging, and some are being invested in. I'll hunt some examples down and post them here in the coming days.
