# A Monitoring Agent for Go  ## Metadata - Author: [[Michael Obermüller]] - Full Title: A Monitoring Agent for Go - Category: #articles - Published: 2017-10-03 - Source: reader - Last Highlighted: 2025-03-18 10:30:43.823061+00:00 - URL: https://readwise.io/reader/document_raw_content/282928582 ## Highlights - This includes the following metrics: • Heap (e.g., memory usage, allocations) • Process (e.g., thread count, native invocations) • Garbage Collector (e.g., pause times) • Special metrics about the Go runtime (e.g., Goroutine count) ([View Highlight](https://read.readwise.io/read/01jpmcwfe98t6113f2kj051tbd)) ## New highlights added March 18, 2025 at 12:32 PM - Memory consumption: When the Go runtime creates a new Goroutine it only needs to allocate 2kB of stack space, so Goroutines start with a pretty small stack which grows as needed. Threads on the other hand start out with 1MB stack space on most OS’s (Golang [11], Sundarram [21]). ([View Highlight](https://read.readwise.io/read/01jpmd6wb9t5z5xg4zw66h0tza)) - Creation overhead: Threads have a big setup and teardown overhead, because to create a thread it is needed to request resources from the OS, and hand it back once the work is done (Sundarram [21]). One possible approach to fix this issue is ([View Highlight](https://read.readwise.io/read/01jpmd73hq1te9c0hh0z9hkny0)) - to create a thread pool. Goroutines on the other hand are created by the runtime which is a very cheap operation, because it only needs to allocate the small stack as described before (Sundarram [21]). Once the Goroutine is done the Go runtime does not release its resources, it is saved within a special Gfree list and is going to be reused (Golang [11]). ([View Highlight](https://read.readwise.io/read/01jpmd761vzaetnetejeb35s3b)) - Context switching cost: Threads are preempted asynchronously by the OS sche- duler when a context switch happens. This means it needs to save/restore the complete thread state (all registers, frame pointer. . . ). The Go scheduler has enough information to preempt Goroutines cooperatively at well known points (on function entry). This means that Goroutines may only be scheduled when they invoke Go functions. The scheduler only needs to safe/restore what is known to be alive at function entry points, and in Go that are just three re- gisters (instruction pointer, stack pointer and one general purpose register) (Golang [12], Sundarram [21]). ([View Highlight](https://read.readwise.io/read/01jpmd9bv1zckbr3kjewgek91v)) - each Goroutine is executed by an OS thread. If a Goroutine writes data to the hard disk, or it must wait for an incoming connection, it will be blocked in a system call. Thus all Goroutines in the local run queue will also be blocked. ([View Highlight](https://read.readwise.io/read/01jpmdba2cfr2mry32jsf7bmjv)) - M0 which needs to do the syscall hands over its context (P) to another OS thread (M1) provided by the Go runtime. This ensures that Goroutines within the local run queue are processed and not blocked. M0 keeps the Goroutine (G1) that made the syscall, ([View Highlight](https://read.readwise.io/read/01jpmdbrscvdhzwn1n6k4kwxch)) - and is blocked until the syscall is finished. When the syscall is finished it tries to steal a context (P) from some other thread and continues execution. ([View Highlight](https://read.readwise.io/read/01jpmdbxnv0738abmwqgg52bgx)) - Go compiles Go programs directly into machine code and links the resulting compilation units to monolithic binaries, which contain the user- and the complete Go runtime code ([View Highlight](https://read.readwise.io/read/01jpmdd8gqz7xgdgva283dvcpr)) - Go binaries contain symbol information by default. Symbols are managed within a symbol table. It holds data needed to locate global fields and functions defined within process memory at run-time. ([View Highlight](https://read.readwise.io/read/01jpmddgc7tnh29aetb1r8sscv)) - Go binaries contain debug information by default. It is noteworthy that it is given in DWARF1 format on all platforms supported by Go. Debug information is managed within a debug info table. It contains details about all functions (i.e., its parameters) and data types (i.e., internal structure, fields) used by the user and Go runtime code. ([View Highlight](https://read.readwise.io/read/01jpmddrzggra9m8jb7h5scx34)) - On platform Linux the Go binaries are statically linked, except when using special code packages from the Go runtime which rely on OS services. ([View Highlight](https://read.readwise.io/read/01jpmddy8xkfe70xnkab4arpbv)) - Nethercote [17] or Kempf et al. [14] divide instrumentation into two types; static and dynamic instrumentation. ([View Highlight](https://read.readwise.io/read/01jpmdew6vsp7ttmw171jzjhhn)) - Static instrumentation techniques range from modifying the source code of a program at compile-time (as shown above), to modifying the executable file after compilation. ([View Highlight](https://read.readwise.io/read/01jpmdf4ryy0rjrqgn7xmpey51)) - Dynamic instrumentation techniques on the other hand inject code during run-time, by modifying the process memory. ([View Highlight](https://read.readwise.io/read/01jpmdf7kmz7tvq6nebr86jraj)) - static code instrumentation has also major drawbacks: • The source code of the Go program must be available. • Is restricted to use Go facilities which are available through the Go public API. • Recompilation of the whole program is necessary. • Full instrumentation of Go code is not possible. Only the user code can be modified, but not the Go runtime code. This is due to the fact, that the Go runtime code is precompiled. So when building a Go application (with the default Go toolchain), only the user code gets compiled, and the precompiled Go runtime code is atta- ched to it during linking. Modifying the Go runtime code is only possible when recompiling it from source, which is described at the Golang website2. ([View Highlight](https://read.readwise.io/read/01jpmdhrxhd31zgww780vd6dee)) - Go binaries are generally dynamically linked. ([View Highlight](https://read.readwise.io/read/01jpmdn9h1dyhy081eqzgn5w0x)) - This implies, that it is possible to load code in form of shared libraries at run-time. ([View Highlight](https://read.readwise.io/read/01jpmdne6zh22qk1n0rq0cm9jc)) - it is based on LD_PRELOAD3. LD_PRELOAD is an environment variable which, when set, for- ces a process to load a given list of shared objects before all others. ([View Highlight](https://read.readwise.io/read/01jpmdp4npkahvjhs1gk7xvcyz)) - To remove the burden of manually setting LD_PRELOAD for each process, the list of shared objects can also be added to the file /etc/ld.so.preload. ([View Highlight](https://read.readwise.io/read/01jpmdpmscy2tns21xw8783w2s)) - One important fact is, that the library injection happens after the Go application code is loaded, but before any Go code gets executed. ([View Highlight](https://read.readwise.io/read/01jpmdqj94f6f7txbfc52nzvxw)) - The next task is to divert the execution of a running process to some injected code as described above. ([View Highlight](https://read.readwise.io/read/01jpmdr5hp4vrke1dgtyj7d29b)) - PTrace The first possible way to divert the execution is to work like a debugger which uses the ptrace4 system call (abbr. syscall). ([View Highlight](https://read.readwise.io/read/01jpmdrdg8qhqfptk7p1ykzhf8)) - Ptrace also allows to read/write the tracee’s process me- mory and registers (Bendersky [3]). ([View Highlight](https://read.readwise.io/read/01jpmdrqmr4gxzhtgyhdesh02j)) - The tracing process starts the tracee in suspended mode using ptrace. This allows the tracer to modify the process memory of the tracee before it starts execution. ([View Highlight](https://read.readwise.io/read/01jpmdsh5tjht1yzt0wnjgez3p)) - Whenever this instruction is executed, a syscall is performed (Keniston et al. [15]). ([View Highlight](https://read.readwise.io/read/01jpmdt50dre9cpnbkma4hpm8k)) - Whenever the tracee performs a syscall, the tracing process is automatically infor- med by the OS. Now the tracer is allowed to modify registers of the syscalling thread. This is used to set the instruction pointer (IP) register (which contains the address of the next instruction) to the address of the function cFuncMethodEnter, shown in Figure 3.6 (skape [20]). ([View Highlight](https://read.readwise.io/read/01jpmdtp5cj0b2wsy47nfnhdaw)) - After setting the IP, the tracer allows the tracee continue execution. The function cFuncMethodEnter must also inform the tracer when it is finished, by having an "int3" instruction at the end of its function body (skape [20]). ([View Highlight](https://read.readwise.io/read/01jpmdvf3efhbenrvqawget8zx)) - Whenever the trap instruction at the end of function cFuncMethodEnter is executed, the tracer replaces the value of the IP with the original address, and continues execution of the tracee (skape [20]). ([View Highlight](https://read.readwise.io/read/01jpmdw4c7qx9rmmb3pb1ajz0c)) - The overhead associated with accessing the tracee’s memory and registers is significantly. Keniston et al. [15] state that the performance of syscalls is decreased on the order of 10x to 100x compared to untraced ones. P ([View Highlight](https://read.readwise.io/read/01jpmdwg835329r0fh77sb8s6q)) - Function Trampolines Function trampolines are another way to divert the execution of arbitrary functions de- fined in memory. ([View Highlight](https://read.readwise.io/read/01jpmdww49z6y0zy43m5e7m9g0)) - To use a function trampoline, the first task is to install it on a target function by modifying the process memory. ([View Highlight](https://read.readwise.io/read/01jpmdx764kpws4x7v9gr23g06)) - This is done during run-time by replacing the first few instructions of the function with an unconditional relative jump to the trampoline. ([View Highlight](https://read.readwise.io/read/01jpmdxa6fb328r1rg46teqh0e)) - • The source function calls the target function, and immediately jumps to the function trampoline (1), due to the jump instruction described above. ([View Highlight](https://read.readwise.io/read/01jpmdy50rp8a5st171cbxyzke)) - • The function trampoline calls the instrumentation code (2), executes the replaced instruction, and jumps back to the target function (3, 4), again with a relative jump. ([View Highlight](https://read.readwise.io/read/01jpmdyv1wwtxjmccwc1ztxzj0)) - The target function continues its normal execution and finally returns to its source function (5). ([View Highlight](https://read.readwise.io/read/01jpmdz6t55gzpgt2q1crcapdj)) - Compared to the ptrace approach, function trampolines do not require syscalls at all. ([View Highlight](https://read.readwise.io/read/01jpmdzc8wb6aac1vaby6d9rpd)) - The dynamic instrumentation approach requires a more complex implementation compa- red to the static one ([View Highlight](https://read.readwise.io/read/01jpme005vdep51sbdhzvhhrtk)) - No source code is required, instead symbol information must be available to know where functions are defined within memory. ([View Highlight](https://read.readwise.io/read/01jpme0adecamj26s50qfn03b1)) - The Go application does not need to be prepared in any way, which makes dynamic instrumentation very convenient for the user ([View Highlight](https://read.readwise.io/read/01jpme0gwzjc6j7ymrs8gqsvx5)) - Full instrumentation of both the user and runtime code is possible. ([View Highlight](https://read.readwise.io/read/01jpme0pn3v5xzh4mc3jtcc6wc)) - All user and runtime functions are defined within process memory, and thus can be modified. ([View Highlight](https://read.readwise.io/read/01jpme0swhj1x1vxxc4h21pjqg)) - Besides the more complex implementation, the dynamic approach also has the disadvan- tage that the cost of instrumentation is incurred at run-time ( ([View Highlight](https://read.readwise.io/read/01jpme14pagtbag7q0eadx1njq)) - The static instrumentation approach would have a simpler implementation compared to the dynamic one, but showed some major drawbacks. The source code must be avai- lable, which is not always the case, especially when using third-party Go applications. ([View Highlight](https://read.readwise.io/read/01jpme5zwzbzn58fd4g6eewcqk)) - The Go Agent is purely written in C++ and compiled as a shared library. ([View Highlight](https://read.readwise.io/read/01jpmeb6wa8ws4hzc0wc8kdzzj)) - • Transformer: The Transformer is responsible for the instrumentation of target Go functions which is based on the dynamic instrumentation approach described in Section 3.3. To be able to instrument functions, the Transformer must know where they are defined in memory. Thus, it uses the Symbol Resolver described above. The Transformer is a core component of the Go Agent. Its implementation will be described thoroughly in Section 5.1. ([View Highlight](https://read.readwise.io/read/01jpmecfdw3bjccq7fyxeq26gb)) - • Sensors: Sensors contain instrumentation code. A Sensor hooks one or more met- hods with the help of the Transformer, to collect interesting performance data. By now, the Go Agent implements one Sensor which is placed in the application under diagnosis. This sensor is described in Section 5.3 ([View Highlight](https://read.readwise.io/read/01jpmecmtgbc0y9xngdfsd9mxz)) - the Go Agent is ready to place and initialize sensors. During initialization of the sensor all needed Go functions are instrumented with the help of the Transformer. If the initialization of Go specific agent parts fails (i.e. checks described above) the Go Agent switches to a restricted monitoring mode. ([View Highlight](https://read.readwise.io/read/01jpmeenwc58v0611pykhbzyqf)) - The instrumented Go code (functions) fires method-enter/exit events when it is invoked. The sensors which are registered as listeners will be informed when such events occur. During an event the listener is able to read/write function parameters. ([View Highlight](https://read.readwise.io/read/01jpmefpm1hwz5nqg6727pa914)) - The Trans- former component of the Go Agent (see Chapter 4), which implements the instrumentation algorithm, defines two types of trampolines: ([View Highlight](https://read.readwise.io/read/01jpmega6jqacw9frwa58nxr88)) - Method Enter Trampoline: This trampoline should be executed when an instrumented Go function is called. ([View Highlight](https://read.readwise.io/read/01jpmegegw17exk04gw2fpdhjc)) - Method Exit Trampoline: This trampoline should be executed when an instrumented Go function finishes execution. A Go function finishes execution by calling a return statement, or by panicking. ([View Highlight](https://read.readwise.io/read/01jpmegr6stk96n6b4pmqk4vsh)) - • The symbol name of the instrumented Go function, for debugging purposes. • The method id used as an unique identifier for the Go function. It is generated from the functions symbol name. • Information about method-enter/exit listeners, because multiple listeners may register for the same function. ([View Highlight](https://read.readwise.io/read/01jpmehkq3rbv1k4qzdz00ghgm)) - the Transformer initializes the code of both trampolines. This is accomplis- hed, by copying predefined template assembly code into the memory block described above. ([View Highlight](https://read.readwise.io/read/01jpmehy8f34zpvsnp1ykw3s00)) - The reason for using trampoline code templates is, that at the time of their definition some details were missing. F ([View Highlight](https://read.readwise.io/read/01jpmej7fpz3eym96k5tth2aqf)) - Patterns emerge when disassembling many different Go functions. It is remarkable, that the prologue (1 ) and the epilogue (2) of Go functions are always the same, l ([View Highlight](https://read.readwise.io/read/01jpmepmqdfq9srweq5pa25sp9)) - prologue: The prologue checks if the current Goroutine has enough stack available to execute the function, if not it jumps to the epilogue. ([View Highlight](https://read.readwise.io/read/01jpmepvyb1xs9yb770tzqafsj)) - epilogue: The epilogue calls the function runtime.morestack defined within the Go runtime, which is responsible to allocate more Goroutine stack. Finally, the current Goroutine continues execution by jumping back to the beginning of the Go function. ([View Highlight](https://read.readwise.io/read/01jpmeqebqk3ds54rgnx029z29)) - The Transformer overwrites the first instruction of an instrumented Go function, in order to install a method-enter trampoline. It is replaced by a relative jmp assembly instruction, ([View Highlight](https://read.readwise.io/read/01jpmer40m0fpahhvvmxb846d3)) - Such modifications of the Go code are only allowed before the Go runtime starts executing it which is during the agent initialization phase ([View Highlight](https://read.readwise.io/read/01jpmerctpjqcxe0n4p3rz4hf8)) - The jmp instruction jumps to its corresponding method-enter trampoline whenever it is executed. ([View Highlight](https://read.readwise.io/read/01jpmerky7r8j63md8axf98kfe)) - The Transformer implements assembly code which contains the jmp instruction shown in Figure 5.5, so there is no need to deal with machine code directly. However, this instruction is again just a template, because at the time of its definition the memory addresses of the method-enter trampoline and the Go function were not known. ([View Highlight](https://read.readwise.io/read/01jpmes8k8s4wrw6bbzb35ywb9)) - Calling a C function with the small dynamic Go Stack is not possible. Thus the macro CALL_C_FUNCTION internally uses the system stack described in Section 2.3.1. ([View Highlight](https://read.readwise.io/read/01jpmevd210v8b1vef315gr9kr)) - To be able to distinguish between different method-enter events, information about the currently running Goroutine and the Go function must be available. ([View Highlight](https://read.readwise.io/read/01jpmew9pnyqyf5wye512a2cdc)) - Regarding information about the running Goroutine, Go stores a memory address pointing to such information in the thread local storage (TLS) of an OS thread (M) which is executing a Goroutine (G) ([View Highlight](https://read.readwise.io/read/01jpmewpe24e8x40h2hz2y78hv)) - The function cFuncMethodEnter informs all listeners which are registered for the event. If there is at least one method-exit listener registered, it additionally replaces the return address with the address of the method-exit trampoline. ([View Highlight](https://read.readwise.io/read/01jpmexwxkc4g5dfntgg1gf1zd)) - The original return address and meta data of the function is stored within an internally used data structure called Go Context. ([View Highlight](https://read.readwise.io/read/01jpmey84mnssby7wgc26p9mcq)) - Method-enter/exit events have stack-based behavior. Thus, Go Contexts must be saved within a stack called the Go Context Stack. T ([View Highlight](https://read.readwise.io/read/01jpmeyh34xt3tew5tnecx0m7n)) - Transformer binds a Goroutine Local Storage (GLS) object to each Goroutine which contains the Go Context Stack ([View Highlight](https://read.readwise.io/read/01jpmeyqyxbh23j66fdfb8fn0r)) - After calling function cFuncMethodEnter, the method-enter trampoline executes the replaced assembly instruction ([View Highlight](https://read.readwise.io/read/01jpmeyyrj8jmtbhe0fw2vvgxb)) - Finally, the program continues execution by jumping back to the calling Go function. ([View Highlight](https://read.readwise.io/read/01jpmez877t4r87vav5r4qn6df)) ## New highlights added March 19, 2025 at 11:32 AM - Exit events should be fired, whenever an instrumented Go function finishes execution. ([View Highlight](https://read.readwise.io/read/01jppxjvtzek0ymg146zec825c)) - The function finishes by calling a return statement. ([View Highlight](https://read.readwise.io/read/01jppxk00cdh65v3py2dfwa5es)) - The function finishes (aborts) execution by panicking. Detailed information about panics, and how they are handled, ([View Highlight](https://read.readwise.io/read/01jppxk5df7j9pw8t5pzvxxytf)) - Next, the function body is executed, it finishes by executing a return instruction which allows the function to return to its caller. ([View Highlight](https://read.readwise.io/read/01jppxm2r3nqmyzdymzyzarvvg)) - A ret2 assembly instruction will pop (4 ) the return address from the stack and jump to it. Since the return address was replaced, it jumps (5) to the method-exit trampoline. ([View Highlight](https://read.readwise.io/read/01jppxm81074d2c3q3ye9wr685)) - The assembly code template implementing method-exit trampolines is shown in Fi- gure 5.10. First, it let the stack pointer (SP) point to the exact same location before the return instruction was executed, which is the location of the return address. The next task is to call the C function cFuncMethodExit shown in Figure 5.11 as pseudo code. ([View Highlight](https://read.readwise.io/read/01jppxmgxe43vvhkpake0h0k0k)) - Unlike other programming languages which use exceptions for handling of any (expected / unexpected) errors, in Go it is idiomatic to use error-indicating return values where pos- sible. ([View Highlight](https://read.readwise.io/read/01jppxnkfk6cxp8050ym01nrmy)) - If a function returns an error code that should not occur during normal execution, then Go’s panic3 mechanism should be used which additionally provides stacktrace infor- mation. ([View Highlight](https://read.readwise.io/read/01jppxnx3qkk0gfpatwj6qnevh)) - A panic in Go means that something unexpectedly went wrong during program execution. ([View Highlight](https://read.readwise.io/read/01jppxp687vcgrkrh3c1qtb9kn)) - This problem is solved by instrumenting the special Go function runtime.dopanic_m defined within the Go runtime. ([View Highlight](https://read.readwise.io/read/01jppxpmh3yw8gk46f21tckbbj)) - This function is called as a last resort before the Go runtime prints a stacktrace to stdout and terminates the program due to an unrecovered panic as shown i ([View Highlight](https://read.readwise.io/read/01jppxpzspsh6jbh1jq5z2hns4)) - 3 each method-enter event pushes a Go Context element onto the Go Context Stack which contains a pointer to the function meta data. ([View Highlight](https://read.readwise.io/read/01jppxskc4dg94h0yb9pdq7w15)) - Information about the panicking Goroutine (and access to its Go context stack) is provided by the functions (runtime.dopanic_m) input parameters. ([View Highlight](https://read.readwise.io/read/01jppxsxnqx99q0nk34xqgx71x)) - Go also allows to recover from panics. For this, it provides the Go runtime function recover which regains control of a panicking Goroutine. Recovering is only useful within a deferred function call, because those are guaranteed to be executed when the surroun- ding Go function finishes, either by returning normally or by panicking. ([View Highlight](https://read.readwise.io/read/01jppxtkpc0t5fzb2vjfvh4akq)) - In order to read information from global data structures, the Go Agent needs to locate these Go objects in process memory at run-time. ([View Highlight](https://read.readwise.io/read/01jppxvzs584em215zwh3gzvc1)) - During compilation the Go tool chain generates symbol tables into the executable by default. The symbol tables (used by the Symbol Resolver) will only tell about the memory location of the objects, but not about their internal structure. ([View Highlight](https://read.readwise.io/read/01jppxw6sjxdm809cxtkyq4wsm)) - the Go Type Info Provider is responsible to provide structural type information about global data and function parameters. It consists for the most part of generated C++ code produced by a tool written in Go. This code generating tool which is called GoGen parses debug information of sample Go binaries a ([View Highlight](https://read.readwise.io/read/01jppxx3shq24bvhyyjs9fm9ec)) - a global object with name memstats defined in package runtime. Since type names and field names may change over versions the configuration file allows to define cu- stom names. All possible alias names must be specified within the goAliases JSON array. ([View Highlight](https://read.readwise.io/read/01jppxxp9vbdb4rmrkyw9yffhn)) - a field with name numgc which is defined within the type mstats and is contained in package runtime. ([View Highlight](https://read.readwise.io/read/01jppxxwg928vebnn9fsg82kd3)) - a function parameter with name gp of function dopanic_m defined in package run- time. ([View Highlight](https://read.readwise.io/read/01jppxy1jm5zzxjxahdjga34ts)) - Performance Evaluation ([View Highlight](https://read.readwise.io/read/01jppy7artzxjb016cpz2wdapp)) - The overhead caused by instrumented functions seems to be very high compared to the time it takes to execute an uninstrumented one. ([View Highlight](https://read.readwise.io/read/01jppy9c67b3e30nt7ddh5nafn)) - The reason is, that an empty Go function consists of very few instructions (see Section 5.1.2), so that each additional statement has a high influence. T ([View Highlight](https://read.readwise.io/read/01jppy9p0w7q2j15hhc7n62cha)) - a memory block is allocated for each instrumented Go function. 256 bytes are used for the assembly trampoline code (due to alignment) and additionally 40 bytes for the function meta data. ([View Highlight](https://read.readwise.io/read/01jppya8x1yqjdtb4ktgvvtgpc)) - This results in an overall memory overhead of 296 bytes (192 + 24 = 216 bytes for Linux 386) for an instrumented Go function. ([View Highlight](https://read.readwise.io/read/01jppyaedqt3atw8xgzhc5y830)) - A Go process is considered to be fully initialized when the Go runtime calls the function main. ([View Highlight](https://read.readwise.io/read/01jppybjsrp506zhgzzbm7682f)) - The biggest part of the overhead is caused by connecting to the Dynatrace server, whereas the initialization of the agent ([View Highlight](https://read.readwise.io/read/01jppyby81868cep58nyq1ckbn)) - It shows that the Go application additionally needs 19 MB when the Go Agent is injected. ([View Highlight](https://read.readwise.io/read/01jppyc65zydsrzkxqcfbgb31f)) - shows the average execution time of 1000 http web requests sent to Caddy with (1.317ms, SD = 0.055) and without (1.339ms, SD = 0.067ms) injection of the Go Agent that generates PurePaths™. ([View Highlight](https://read.readwise.io/read/01jppycyhch9pj37v24wkmyr88)) - The exe- cution times are indistinguishable due to the low absolute overhead of instrumented functions (see Section 8.2.1). ([View Highlight](https://read.readwise.io/read/01jppyd4vaf87b7c2vekbd0hgx))