Tracing Profilers

Tracing profilers operate by intercepting calls to functions in the application and running some additional profiling code. This extra code will capture information such as the function which is called, when it was called and how long it took to execute. Typically this will be written to a trace file to be post-processed and displayed.

For example, consider an application which contains three functions A, B and C. To capture information about these functions using a tracing profiler, the application must first be prepared by instrumenting code. This can be done in different ways depending on whether the functions are in our application code or in an external library. If the functions are in our application code we can directly instrument the code either by using an instrumenting compiler provided by the profiling tool, such as ScoreP, or by manually adding instrumentation using the profiler API. If the functions are in an external library, then the calls can be instrumented by interception, typically this is done by creating an additional library containing the instrumentation functions, the application is linked against this, and the instrumentation library is linked against the library containing the functions to be profiled. This approach is commonly used to trace calls to parallelism runtimes such as MPI and OpenMP, and parallel profilers will often automate the library interception process at runtime.

Now when the instrumented application is run, the profiler code will then be run every time an instrumented function is called and record the information we need. However, because we have inserted this additional code into the application, we have increased the total amount of work our application does, and so we will have increased the runtime, this additional time is known as the profiling overhead. This is shown pictorially below, where the addition of instrumentation adds overhead (orange) to each function call.

It is important to be aware of the overhead and how large it is, because this overhead time can distort the view of how long different parts of the application take to run. This is a particular issue if there are many calls to very short functions e.g inside loops, because the overheads can often be comparable to or even be larger than the amount of time spent in the function.

Although the overheads can potentially be large, tracing is an extremely powerful technique because the profiling code is executed in the application. This means that almost any information available inside the program can be gathered, e.g for MPI it is very useful to know how much data is communicated and the sender and receiver of the data. The tracing approach is commonly used for the profiling of parallel code by tracing calls to parallel libraries. For example, it is the method used by the Extrae, Scalasca and Intel ITAC profiling tools, amongst others, to trace MPI and OpenMP behaviour.

Sampling Profilers

Sampling profilers do not alter the application code and instead rely on methods provided by the operating system to periodically inspect the application being profiled. Sampling in this manner requires operating system support, but unlike tracing does not require any kind of additional instrumentation and so does not require any specific library or language support to insert tracing code. The profiler instead makes periodic requests to the operating system kernel to return information about the application under investigation, such as the current function being executed and the hardware resources used since the previous query. This is typically a lighter weight approach than tracing and gathers less detailed information in exchange for reduced overhead.

Although sampling does not require modification of the code it will still normally require the application to be compiled with debugging symbol names. This enables the profiler to express the information using human-readable function names.

Use of a sampling profiler will still incur some overhead as the kernel must interrupt the application in order to inspect its state. however, compared to a tracing profiler, sampling overhead is essentially fixed by the choice of sampling rate, rather than being dependent on application behaviour, and can be used on all code types. The downside however, is that sampling can only provide a statistical picture of the application behaviour, rather than a deterministic one. An example of this behaviour is given below: notice there are certain function calls which are not sampled (orange lines), while other function calls are sampled twice. This means that the profiler is not able to construct a complete timeline of function executions. Instead it will count the number of times it finds the code in any given function and provide information on the relative amount of time spent in each of these functions.

Notice that there is still some overhead from the sampling because the application must be interrupted by the OS kernel. However, this will typically be lower than the overhead due to tracing because the method is kernel-based, and the independence of the overhead from the number of function calls.

Hybrid Profilers

When performing analysis of HPC code it is common to combine tracing and sampling methods for a "hybrid" profiling approach. Consider for example a code parallelized with MPI or OpenMP, there are likely to be relatively few calls to the parallelization library, and so it is useful to trace these calls and build a complete picture of the parallel behaviour. At the same time, the user code may be a complex C++ code with many small function calls that cannot be traced without large overhead, so sampling can be used to profile this without major overhead. This is the approach used by, for example, Intel Vtune to simultaneously profile parallel library calls and user code behaviour while minimizing overhead costs.