Profiling
The following functions control a profiler attached to the running server. The functions let a policy or a trigger choose which part of a run the profiler measures, instead of measuring from process start to process exit.
The functions for a profiler are only available when the server was built on a system with the development headers for that profiler installed. The runtime-valgrind and runtime-gperftools feature flags report whether the profiler is attached. The server prints the feature flags at startup when run with -x. A call to a function whose profiler is not attached fails the expansion and logs the reason.
This page uses the terms of each profiler. Callgrind counts costs (instruction fetches, cache misses, branch mispredictions) into one cost centre per function, and writes the cost centres as a profile dump. gperftools records samples of the program counter on a timer, and writes the samples to a profile file.
The usual way to use these functions is from the server.start and server.stop triggers. The server.start trigger fires once the worker threads are running. The server.stop trigger fires after the server receives a request to exit. The -S option sets a trigger from the command line without editing the configuration:
radiusd -f -S 'trigger.server.start=%callgrind.start()' -S 'trigger.server.stop=%callgrind.stop()'Callgrind
Callgrind is the call graph profiler in valgrind. Start the server with instrumentation switched off, so that callgrind does not profile configuration parsing and module loading. Then switch instrumentation on from a trigger:
valgrind --tool=callgrind --instr-atstart=no \
radiusd -f -S 'trigger.server.start=%callgrind.start()'Instrumentation is process-wide, so the thread that runs the function does not matter.
%callgrind.start()
Switch callgrind instrumentation on. When cache simulation is enabled, callgrind flushes the simulated cache, so the first moments after the call show an artificial burst of cache misses.
Use this function when the profile should begin later than process start. The server.start trigger is the usual place to call this function, because configuration parsing and module loading have finished when the trigger fires.
%callgrind.stop()
Switch callgrind instrumentation off. Callgrind keeps the costs already collected, and writes the costs when the process exits or when %callgrind.dump() runs.
Use this function when the profile should end before process exit. A call from the server.stop trigger keeps thread teardown out of the profile. A call from a policy ends the profile after a chosen section of unlang.
%callgrind.dump([<label>])
Write the costs collected so far to a new profile dump, then zero the costs. Callgrind records the optional label in the dump, so the reader can distinguish the dumps when one run produces several.
Use this function when one run should produce several profile dumps, such as one dump per load step, or one dump for the load phase and one dump for shutdown. Instrumentation stays on, and each dump holds only the costs counted since the previous dump. callgrind_annotate accepts several dumps at once.
%callgrind.dump('load complete')%callgrind.zero()
Zero the costs collected so far without writing a dump.
Use this function to discard the costs of a period that the profile should not include. Examples are the cache warmup after %callgrind.start(), and a settling period after a load step changes the request rate. Call this function when the period ends. The next profile dump, or the dump at process exit, then holds only the costs counted after the call.
gperftools
The gperftools CPU profiler samples the program counter on a timer and writes the samples to a file for pprof to read. The profiler runs at close to native speed, so use the profiler to measure real timing. The profiler does not count instructions or simulate caches.
%gperftools.start(<filename>)
Start the profiler, writing samples to filename. The function fails if the profiler is already running, so a second start never discards a running profile. The limit files { … } section restricts the filename in the same way as for the file handling functions.
Use this function to measure where the server spends wall-clock time. The server.start trigger is the usual place to call this function.
radiusd -f -S 'trigger.server.start=%gperftools.start("/var/log/radius/cpu.prof")' \
-S 'trigger.server.stop=%gperftools.stop()'%gperftools.stop()
Write any buffered samples to the profile file, then stop the profiler. The function fails if the profiler is not running.
Use this function to end the profile at a known point, usually from the server.stop trigger. %gperftools.start() starts a stopped profiler again with a new profile file.
%gperftools.flush()
Write any buffered samples to the profile file and keep the profiler running. The function fails if the profiler is not running.
Use this function during a long run, so that pprof can read a partial profile while the server keeps running. Also use this function before a step that might crash the server. The buffered samples then reach the profile file before the crash.