Ampere PMU Profiler: A Guide to Microarchitecture Profiling
Bhakti HindujaPublished inPerformance & Scaling·DevOps·Hardware·
October 19, 2026
The AI briefing for Developers
Stay up to date with AI tools, model releases, and developer workflows that matter.
Weekly. Free. One click to leave.
SitePoint Premium
Stay Relevant and Grow Your Career in Tech
- Premium Results
- Publish articles on SitePoint
- Daily curated jobs
- Learning Paths
- Discounts to dev tools
7 Day Free Trial. Cancel Anytime.
Executive Summary
The Ampere® PMU Profiler (APP) is a Python-based tool designed to provide deep insight into the microarchitectural behavior of applications running on Ampere CPUs (e.g., Ampere® Altra® and AmpereOne®). Unlike standard profilers that identify where time is spent (for example, which functions consume CPU time), the PMU Profiler explains why time is being spent by measuring low-level hardware events associated with the CPU pipeline and execution behavior.
A key outcome of APP is that it enables performance engineers to move from coarse symptoms to actionable causes. For example, while application-level profiling can show an expensive code path, APP can help identify whether the expense stems from inefficient instruction fetching, data cache misses, or other microarchitectural factors that are difficult or impossible to isolate using application-level tools alone. The document outlines a top-down performance analysis methodology and positions APP as an essential final step for expert-level tuning, particularly on Ampere platforms, where you must understand hardware-level bottlenecks and then apply targeted code optimizations.
APP is intended to complement system-level analysis rather than replace it. System-level profilers are useful for identifying high-level bottlenecks such as recture-level analysis by collecting hardware events. This makes APP especially valuable after system bottlenecks have been eliminated or ruled out, leaving “microarchitecture inefficiency” as the remaining likely cause of slowdowns
The Ampere PMU Profiler uses the Linux perf utility with validated PMUs and metrics on Ampere CPUs. Its purpose is to capture microarchitectural performance indicators through hardware event measurement. In practice, this means APP collects events measured by perf stat that relate to the CPU pipeline and execution mechanisms, allowing engineers to determine not only what is slow, but the underlying microarchitectural reason.
A central distinction between APP and application profiling tools is the level of visibility. Tools that sample stack traces (or count function invocations) typically answer: “Which functions are active during the slow period?” APP answers a more hardware-specific question: “Which microarchitectural mechanisms are consuming cycles, and what stalls or inefficiencies are present?”
The APP workflow assumes that developers can form a hypothesis about where the bottleneck likely originates, such as a particular loop or data access pattern, and then rely on PMU event measurements to confirm or refute those hypotheses at the microarchitectural level.
Why Do We Need APP?
Performance problems are frequently multi-layered. Even after system-level bottlenecks are addressed (for example, ensuring that the CPU is not idling due to I/O, ensuring there is sufficient memory, and verifying re, some workloads still perform poorly because the CPU spends cycles in inefficient pipeline states
APP helps solve this class of problems by measuring hardware-level behavior. For example, APP can identify microarchitectural bottlenecks such as:
- Inefficient instruction fetching
- Data cache misses
- Branch-related pipeline effects
- Other pipeline-level stall sources that manifest as lost cycles
This capability is important because microarchitectural causes often do not map cleanly to application symptoms. Code can appear “hot” in a profiler, but the reason it is slow might be due to how it interacts with cache hierarchies, how it causes translation or fetch inefficiencies, or how the processor recovers from pipeline disruptions. Those details are what PMU-based measurement aims to expose.
APP also links investigation to unlocking the full performance potential of the hardware. By understanding CPU-level bottlenecks, engineers can choose targeted optimizations that application-level tools alone cannot determine with confidence. This ultimately leads to more efficient software and better utilization of Ampere hardware for competitive workloads.
Understanding the APEX Framework
Performance tuning is a process of systematic investigation, moving from a broad, system-wide view down to the specific interactions between code and hardware.
Performance optimization is as much art as it is science. The APEX framework uses tools and methodologies to add structure and rigor to the process and can bridge the gap between creative intuition and empirical fact.
We propose applying the APEX (Adaptive Profiling and Execution) methodology to enable root cause analysis for solving performance problems. Follow the funnel above from top to bottom to effectively use the procedure.
The methodology recommends starting with assessing platform health as a first step to ensure that the platform used for performance analysis is set up well. An unhealthy platform may mislead the performance analysis.
Consider capturing initial performance metrics before tuning any system or application settings. This establishes a clear understanding of the current workload and identifies key scalability knobs. We recommend using Ampere’s PerfKit Benchmarker (APB), which supports many open-ysis and tuning
Next is to assess system performance and any hardware or system bottlenecks while the code runs — this is where the Ampere System Profiler (ASP) is useful to eliminate any system or re shape and ensure that the compute re
One method which may be used is to leverage APB’s automated benchmarking framework to start and stop ASP’s collectors during the run phase of a given APB benchmark. This ensures that the profile is collected while critical code paths are executed and a clear profile report is generated.
Once system and res and points to CPU cycles not being used efficiently, we propose going to the next step in the pyramid and using the Ampere PMU Profiler to root-cause the issue further
Finally, system benchmarking should be done after all bottlenecks are resolved or analyzed to effectively measure the system’s performance for the workload.
Following this systematic APEX methodology ensures that we eliminate possible issues as part of a structured process to efficiently conduct root cause analysis.
Microarchitecture-Level Analysis
At the microarchitecture level, performance is shaped by how the CPU pipeline handles instruction delivery, execution, and memory access. APP leverages PMU measurements to identify pipeline behavior and stall
Memory Hierarchy and Performance Loss
APP emphasizes the performance significance of the memory hierarchy. As data access moves from registers to L1 cache, then L2 cache, then L3 cache, and finally DRAM, access becomes exponentially slower. Because of this, cache misses are a primary cause of performance loss. This provides a conceptual foundation for many APP investigations: if a workload touches large working sets or accesses data in a non-contiguous pattern, it may trigger cache misses that increase effective latency and reduce throughput.
Microarchitectural Bottleneck Identification
APP can be used at the microarchitecture level to understand where stalls might be in the pipeline. The APP role is to collect hardware events related to pipeline stall behavior and to use those events to characterize the workload’s execution profile. This capability matters because pipeline stalls and inefficiencies can dominate runtime even when application-level profiling points to a “hot” function without explaining the root cause.
Key Questions
APP is structured around answering questions that cannot be fully resolved with application-level profiling alone. Based on the described APP workflow and report interpretation strategy, APP can help you answer:
- Where are cycles going at the microarchitectural level? The APP HTML report and TDA sunburst charts are used to broadly characterize whether time is dominated by categories such as instruction retirement behavior, frontend bound behavior, or backend bound behavior.
- Which stall or inefficiency class is consistent with the hot code path? Once you hypothesize a bottleneck mechanism (e.g., cache misses from non-contiguous access), APP measurements can confirm whether the observed behavior aligns with that mechanism.
- What microarchitectural reason explains a hot function’s cost? APP’s purpose is explicitly to explain why time is spent by measuring hardware events. This allows developers to translate hot functions into hardware interactions that can be optimized.
- Is the workload limited by instruction delivery versus execution/memory? By inspecting broad characterization categories (frontend vs. backend bound) in the APP HTML report, engineers can determine which side of the pipeline is more likely to be limiting performance.
Example Usage and Output
app -n 120 -c 1,2 -i 1 --tda -o <folder> -j "taskset -c1,2 <workload>"The above example command attempts to collect:
- PMU profiling samples for 120s
- With a sampling interval of 1s
- Profiles on cores 1 and 2
- TopDown metrics and render TDA sunburst chart
- PMU profiles while running the workload affinitized to cores 1 and 2
Metrics Reported by APP
Core Metrics
| Metric Name | Description |
|---|---|
| IPC | Instructions retired per CPU cycle (user + kernel unless separated). |
| IPC_kernel | Instructions retired per CPU cycle while executing in kernel/EL1. |
| cpu_freq | Average core frequency during the measurement interval (typically in GHz or MHz). |
Cycle Accounting Metrics
| Metric Name | Description |
|---|---|
| frontend_bound | Fraction of cycles where retirement is limited by the front-end (fetch/branch prediction/decode/ICache/ITLB/queueing). |
| backend_bound | Fraction of cycles where retirement is limited by the back-end (execution resources, cache/memory latency/bandwidth, ROB/LSQ pressure). |
Branch Effectiveness Metrics
| Metric Name | Description |
|---|---|
| branch_mispredict% | Percent of retired branch instructions that were mispredicted (causing pipeline recovery/flush). |
| branch_mpki | Branch mispredictions per 1,000 instructions retired. |
DTLB Effectiveness Metrics
| Metric Name | Description |
|---|---|
| dtlb_mpki | Data TLB misses per 1,000 instructions retired (that require translation refill/walk beyond L1 DTLB). |
| dtlb_walk% | Percent of DTLB misses that trigger a page-table walk (vs. being satisfied by a next-level TLB). |
| l1d_tlb_miss% | L1 DTLB miss rate (L1 DTLB misses divided by DTLB accesses). |
| l1d_tlb_mpki | L1 DTLB misses per 1,000 instructions retired. |
| l2_tlb_miss% | L2/second-level DTLB miss rate (misses divided by L2 TLB accesses). |
| l2_tlb_mpki | L2/second-level DTLB misses per 1,000 instructions retired (typically correlates with page walks). |
ITLB Effectiveness Metrics
| Metric Name | Description |
|---|---|
| itlb_mpki | Instruction TLB misses per 1,000 instructions retired. |
| itlb_walk% | Percent of ITLB misses that trigger a page-table walk (vs. hit in a next-level TLB). |
| l1i_tlb_miss% | L1 ITLB miss rate (misses divided by ITLB accesses). |
| l1i_tlb_mpki | L1 ITLB misses per 1,000 instructions retired. |
L1 Cache Effectiveness Metrics
| Metric Name | Description |
|---|---|
| l1i_mpki | L1 instruction cache misses per 1,000 instructions retired. |
| l1d_mpki | L1 data cache misses per 1,000 instructions retired. |
| l1i_miss% | L1 instruction cache miss rate (misses / accesses). |
| l1d_miss% | L1 data cache miss rate (misses / accesses). |
L2 Cache Effectiveness Metrics
| Metric Name | Description |
|---|---|
| l2_mpki | L2 cache misses per 1,000 instructions retired (typically demand misses; exact scope depends on event mapping). |
| l2_miss% | L2 cache miss rate (misses / L2 accesses). |
| l2d_inv_pki | L2 data cache invalidations per 1,000 instructions (coherency-driven invalidations affecting this core/cluster). |
| l2_snoops_pki | L2 snoop transactions per 1,000 instructions (coherency probes observed/handled). |
| l2d_inv_per_snoop | Average invalidations generated per snoop (invalidation intensity per probe). |
Operation Mix Metrics
| Metric Name | Description |
|---|---|
| branch_percentage | Percent of retired instructions that are branch instructions. |
| crypto_percentage | Percent of retired instructions that are crypto/CRC/hash-class instructions (Arm crypto extensions). |
| integer_dp_percentage | Percent of retired instructions that are integer data-processing (ALU) operations. |
| load_percentage | Percent of retired instructions that are loads. |
| store_percentage | Percent of retired instructions that are stores. |
| scalar_fp_percentage | Percent of retired instructions that are scalar floating-point operations. |
| simd_percentage | Percent of retired instructions that are SIMD/NEON (vector) operations. |
Pipeline Stall
| Metric Name | Description |
|---|---|
| stall_frontend_cache_rate | Fraction of cycles stalled due to instruction-side cache/fetch delivery issues (e.g., I-cache misses/linefill effects). |
| stall_frontend_tlb_rate | Fraction of cycles stalled due to ITLB/translation-related front-end stalls. |
| stall_recovery_rate | Fraction of cycles spent recovering from pipeline flushes (commonly branch mispredict recovery and similar redirects). |
| stall_fronetend_bob_rate | Fraction of cycles stalled because the front-end buffer/queue (often “branch/order buffer” or fetch/decode buffering) is full/blocked, limiting delivery to the back-end. |
Pipeline Stall
| Metric Name | Description |
|---|---|
| stall_backend_cache_rate | Fraction of cycles stalled due to cache hierarchy latency on the data side (L1D/L2/SLC effects excluding pure DRAM bandwidth). |
| stall_backend_tlb_rate | Fraction of cycles stalled due to DTLB misses/page-walk latency impacting loads/stores. |
| stall_backend_mem_rate | Fraction of cycles stalled due to main-memory/DRAM latency or bandwidth limits (off-chip). |
| stall_backend_core_rate | Fraction of cycles stalled due to core execution limits (e.g., dependency chains, execution-unit throughput). |
| stall_backend_resource_rate | Fraction of cycles stalled due to internal resource pressure (queues, buffers, credits) not attributed to a specific unit. |
| stall_rob_id_rate | Fraction of cycles where progress is limited by reorder buffer / in-flight instruction capacity (ROB full / cannot dispatch/retire). |
| stall_ixu_sched_rate | Fraction of cycles stalled due to integer execution scheduler/issue queue pressure (IXU scheduling bottleneck). |
| stall_fsu_sched_rate | Fraction of cycles stalled due to FP/SIMD execution scheduler/issue queue pressure (FSU scheduling bottleneck). |
| stall_lob_id_rate | Fraction of cycles stalled due to load buffer/queue (load-order/load buffer) being full or blocked. |
| stall_sob_id_rate | Fraction of cycles stalled due to store buffer/queue (store-order/store buffer) being full or blocked. |
Uncore Metrics
| Metric Name | Description |
|---|---|
| slc_miss% | System-level cache (SLC/LLC) miss rate for requests reaching SLC (misses / accesses). |
| mc_retry_rate% | Percent of memory-controller transactions that are retried (e.g., due to contention/queue full/flow-control), indicating fabric/MC pressure. |
| memrd_bw_GBps | Estimated DRAM read bandwidth consumed (GB/s). |
| memwr_bw_GBps | Estimated DRAM write bandwidth consumed (GB/s). |
| ccix_in_bw_MBps | CCIX (coherent interconnect) inbound bandwidth to the socket/system (MB/s). |
| ccix_out_bw_MBps | CCIX outbound bandwidth from the socket/system (MB/s). |
Refer to a detailed Tuning Guide here.
Conclusion
APP enables PMU hardware event measurement to provide microarchitecture-level performance insight on Ampere CPUs. It is designed to answer the “why” behind performance problems by identifying microarchitectural causes such as inefficient instruction fetching and data cache misses, which are difficult to detect through application-level profiling alone.
The APP workflow is top-down and hypothesis-driven: form a hypothesis from the hot function, measure with APP profiles, then analyze using the APP HTML report with TDA sunburst charts to characterize where cycles are being spent (instruction retirement, frontend bound, backend bound). APP is most valuable when system-level bottlenecks have been characterized or ruled out and microarchitecture-level explanation is required for expert tuning.
Bhakti H. is a Director of Engineering at Ampere, specializing in semiconductors, ML hardware-software co-design, and data center infrastructure.
