Optimizing a Supermicro Server BIOS for High Performance Computing (HPC)

Optimizing a Supermicro server's BIOS for High Performance Computing (HPC) involves adjusting several key settings to maximize the performance of the processor, memory, and network and storage components. Note that exact options may vary slightly depending on the specific motherboard model and processor generation (Intel Xeon, AMD EPYC, etc.).

Here's a general guide to the most important settings to consider:

1. System Performance Mode (System Profile):

• System Profile: Set it to "Max Performance" or "High Performance Mode ." Avoid "Performance Per Watt" or power-saving modes, as they prioritize efficiency over raw performance, which is undesirable for HPC.
• Workload Profile (if available): Some newer BIOSes offer predefined profiles. Look for a profile called "HPL" (High Performance Linpack) or "HPC" if available, as these are specifically designed for HPC workloads.

2. CPU Configuration:

• CPU Power Management / Power Technology: Disable or set to "Custom" and then adjust individual options for maximum performance.
• EIST (Intel SpeedStep) / AMD Cool'n'Quiet: Disable . These features reduce the CPU frequency when not under full load, which can introduce latency or limit sustained performance in HPC.
• Turbo Boost / AMD Precision Boost: Enable . Allows the CPU to increase its frequency above the base frequency when conditions permit, which is crucial for performance.
• C-States (Intel) / Global C-state Control (AMD): Disable these (or set them to "Disabled" or "No C-states"). C-states are power-saving states for the CPU. Disabling them ensures that the CPU remains in its highest performance state.
• P-States: Make sure EIST (P-States) is Enabled and Turbo Mode is also Enabled . The P-States coordination must be set to "HW_ALL" .
• SMT (Simultaneous Multi-threading) / Hyper-threading (Intel): For some HPC workloads (especially those with high memory usage or that don't benefit from thread-level parallelization), disabling SMT/Hyper-threading can improve performance. However, for others (especially those that scale well with more threads), enabling it can be beneficial. Test and compare performance for your specific application.
• Determinism Slider / Determinism Control: Set this to "Performance Determinism" . This prioritizes performance consistency over power efficiency.
• CPU cTDP (Configurable TDP): If your CPU allows it, make sure it is set to the maximum TDP value to allow for the highest sustained performance.
• Core Enabled: Make sure all CPU cores are enabled .
• Hardware Prefetchers (L1/L2 Stream HW Prefetcher): Enabling these prefetchers can generally improve performance by loading data into the cache early.

3. Memory Configuration:

• Memory Frequency: Set it to "Max Performance" or the highest frequency supported by your RAM and CPU modules.
• Memory Patrol Scrub / Memory Write Data CRC: Disable for HPC. These functions are for data integrity and may introduce latency or CPU usage.
• Memory Refresh Rate: Leave it at "1x" or the default unless you have specific requirements.
• Per-Socket Memory Interleaving: If your system has multiple NUMA (Non-Uniform Memory Access) nodes, enable memory interleaving (if available and appropriate for your application) to distribute memory requests across channels and sockets, which can improve bandwidth.
• NUMA Nodes per Socket (NPS): For AMD EPYC CPUs, NPS settings (e.g., 4 NPS) can significantly impact HPC performance. Recommendations vary by workload, but 4 NPS is often suggested for HPC.

4. PCIe and Peripherals Configuration:

• PCI ASPM L1 Link Power Management: Disable . This may save power on PCIe devices, but may introduce latency.
• PCIe Speed PMM Control: Set to "Auto" or the highest PCIe generation supported by your devices (GPUs, network cards, NVMe) and motherboard (e.g., Gen 5).
• Preferred IO Device / Preferred IO Bus: If you're using a high-performance network card (e.g., Mellanox InfiniBand), some Supermicro BIOSes allow you to configure a "Preferred IO Device" to prioritize traffic over that specific PCIe bus, which can reduce latency for inter-node communications. You'll need your card's PCIe bus number (you can get it with lspci in Linux).
• UEFI Boot: Consider using UEFI boot mode instead of Legacy/CSM mode if your operating system and devices support it. This can offer faster boot times and better support for modern hardware.

5. Other Settings:

• Virtualization Technology (Intel VT-x / AMD-V): If your HPC application does not use virtualization, you may consider disabling it to free up resources, although the impact is usually minimal.
• Serial Port Console Redirection: Disable this if you don't need it for remote management.
• USB Support: Disable support for unused USB ports to reduce resource usage.
• Boot Options: Make sure the boot order is set to boot directly from your operating system disc, disabling unnecessary options such as booting from CD/DVD or network if they are not used regularly.

Important considerations:

• BIOS/Firmware Update: Always make sure you have the latest version of the BIOS and firmware (BMC/IPMI, network firmware, etc.) from Supermicro for your motherboard. Updates often include performance improvements and bug fixes.
• Supermicro Documentation: Refer to the manual for your specific Supermicro motherboard . It will contain details on all BIOS options and their functions. Supermicro also provides FAQs and optimization guides for certain CPU models and architectures.
• Performance Testing: After making BIOS changes, it's crucial to test the performance of your HPC applications with relevant benchmarks to verify that the adjustments are having the desired effect. Some changes may be counterproductive for certain workloads.
• Cooling: An HPC system generates a lot of heat. Make sure your cooling solution (air or liquid) is adequate for maximum sustained performance. Thermal throttling (performance reduction due to overheating) will negate any BIOS optimizations.

By carefully adjusting these settings, you can optimize your Supermicro server for maximum performance in high-performance computing workloads.