Java clients, log4j2 and logback appenders for Grafana Loki
On top of a Core component with rather simplistic API we intend to build several layers that make it truly useful.
Log4j2 appender seemed like a good first.
Grab the no-dependency version of the appender:
<dependency>
<groupId>pl.tkowalcz.tjahzi</groupId>
<artifactId>log4j2-appender-nodep</artifactId>
<version>0.9.39</version>
</dependency>
If you already use a compatible version of Netty in your project, then to reduce the size of your dependencies, include regular appender distribution:
<dependency>
<groupId>pl.tkowalcz.tjahzi</groupId>
<artifactId>log4j2-appender</artifactId>
<version>0.9.39</version>
</dependency>
packages="pl.tkowalcz.tjahzi.log4j2" attribute to your existing log4j2 configuration file.
<Loki name="Loki">
<host>${sys:loki.host}</host>
<port>${sys:loki.port}</port>
<PatternLayout>
<Pattern>%X{tid} [%t] %d{MM-dd HH:mm:ss.SSS} %5p %c{1} - %m%n%exception{full}</Pattern>
</PatternLayout>
<Label name="server" value="${hostName}"/>
</Loki>
<Root level="INFO">
<AppenderRef ref="Loki"/>
</Root>
Tjahzi by default sends POST requests to /loki/api/v1/push HTTP endpoint. Specifying
e.g. <host>loki.mydomain.com</host><port>3100</port>
will configure the appender to call to URL: http://loki.mydomain.com:3100/loki/api/v1/push.
Tjahzi can send logs to Grafana Cloud. It needs two things to be configured:
443 which switches an HTTP client into HTTPS mode.username and password for HTTP basic authentication that Grafana uses.Password is your “Grafana.com API Key” and can be generated in “Grafana datasource settings”. The host in below example is just for illustrative purposes.
<Loki name="Loki">
<!-- example host -->
<host>logs-prod-us-central1.grafana.net</host>
<port>443</port>
<username>...</username>
<password>...</password>
...
</Loki>
This example sets up a root logger with a Loki appender. Note that pl.tkowalcz.tjahzi.log4j2 is added to packages
attribute of configuration so that the appender can be found.
<?xml version="1.0" encoding="UTF-8"?>
<configuration packages="pl.tkowalcz.tjahzi.log4j2">
<Loggers>
<Root level="INFO">
<AppenderRef ref="Loki"/>
</Root>
</Loggers>
<appenders>
<Loki name="Loki" bufferSizeMegabytes="64">
<host>${sys:loki.host}</host>
<port>${sys:loki.port}</port>
<ThresholdFilter level="ALL"/>
<PatternLayout>
<Pattern>%X{tid} [%t] %d{MM-dd HH:mm:ss.SSS} %5p %c{1} - %m%n%exception{full}</Pattern>
</PatternLayout>
<Header name="X-Scope-OrgID" value="Circus"/>
<Label name="server" value="127.0.0.1"/>
<Metadata name="environment" value="production"/>
<Metadata name="service_version" pattern="${ctx:version}"/>
<LogLevelLabel>log_level</LogLevelLabel>
</Loki>
</appenders>
</configuration>
Connection is configured by providing parameters like host or port explicitly in dedicated tags or by using a URL that has them all embedded. First, we will show how the individual parameters work. At a minimum, Tjahzi needs host and port configuration to connect to Loki, e.g.:
<host>example.com</host>
<port>3100</port>
If port is equal to 443 then SSL will be used. You can also configure SSL manually:
<host>example.com</host>
<port>3100</port>
<useSSL>true</useSSL>
You can also override the default endpoint to which Tjahzi sends data. This can be useful if Loki is behind the reverse proxy and additional path mapping is used:
<host>example.com</host>
<port>3100</port>
<logEndpoint>/monitoring/loki/api/v1/push</logEndpoint>
All these parameters can be configured in one place using a URL:
<url>https://example.com:56654/monitoring/loki/api/v1/push</url>
Note that all previously mentioned tags (host, port, useSSL, logEndpoint) cannot be used when using URL.
URL consists of four parts: protocol, host, port, and path. Some of them may be omitted, and there are defaults that depend on the contents of other parts of the URL. This table has a rundown of all viable configurations:
| Section | Default | Comment |
|---|---|---|
| Protocol | None (must be provided) | Supported protocols are http and https. Https is equivalent to setting useUSSL |
| Host | None (must be provided) | |
| Port | 80 for http, 443 for https | You can use any port and SSL will still be used if protocol is set to https |
| Path | ‘/loki/api/v1/push’ |
Some examples of correct URLs:
<url>http://example.com</url>
<url>https://example.com:56654</url>
<url>http://example.com/monitoring/loki/api/v1/push</url>
<url>https://example.com:3100/monitoring/foo/bar</url>
Contents of the properties
are automatically interpolated by Log4j2
. All environment, system etc. variable references will be replaced by their values during initialization of the
appender. The exception to this rule is context/MDC (${ctx:foo}) value lookup - it is performed for each message at
runtime (allocation free).
NOTE: This process could have been executed for every lookup type at runtime (for each log message). This approach was deemed too expensive. If you need a mechanism to replace a variable (other than context/MDC) after logging system initialization, I would love to hear your use case - please file an issue.
An alternative way of specifying label contents is via a pattern attribute:
<Label name="server" pattern="%C{1.}"/>
This pattern is compatible with Log4j pattern layout. In fact, we reuse log4j internal classes for this implementation. It is generally efficient and allocation-free as per documentation.
Properties file is a simple configuration format, but it is not always clear how to implement more advanced features such as components instantiated more than once. For a basic overview of how to configure log4j using a properties file see official documentation.
Let’s go through the example config used in previous sections and analyze configuration options (Note: Tags are case-insensitive).
Network host address of Loki instance. Either IP address or host name. It will be passed to Netty and end up being
resolved by call to InetSocketAddress::createUnresolved.
Port used for connecting to running Loki. Tjahzi by default uses plain HTTP but if the port is 443 then it will
automatically switch to HTTPS.
Enable secure (HTTPS) communication regardless of the configured port number.
Overrides the default endpoint to which Tjahzi sends data. This can be useful if Loki is behind a reverse proxy and additional path mapping is used.
Configure connection in one place instead of using host, port, etc. See this section.
Username for HTTP basic auth.
Password for HTTP basic auth.
This tag can be used multiple times to specify additional headers that are passed to the Loki instance. One example is to
pass a X-Scope-OrgID header
when running Loki in multi-tenant mode.
Specify additional labels attached to each log line sent via this appender instance. See also a note about label naming.
You can use value attribute to specify static text. You can use ${} variable substitution inside that text and Tjahzi
will resolve variables once at startup. If the variable is a context/MDC lookup, it will be resolved dynamically for each
log line.
This tag also supports pattern attribute where you can use pattern layout expressions that will be resolved at
runtime.
If defined, then a log level label of the configured name will be added to each line sent to Loki. It will contain Log4j log
level e.g. INFO, WARN etc. See also note about label naming.
Specify structured metadata attached to each log line sent via this appender instance. Unlike labels, structured metadata does not affect log stream grouping and is stored alongside the log entry. See the official documentation of Loki.
Note: Structured metadata was added to chunk format V4 and will not work prior to version 2.9 of Loki.
You can use the value attribute to specify static text, or use the pattern attribute with Log4j pattern layout
expressions (including context/MDC lookups like ${ctx:variable_name}) that will be resolved at runtime.
Example configuration:
<Metadata name="fixed" value="stuff"/>
<Metadata name="tx_id_metadata" pattern="${ctx:tx_id}"/>
<Metadata name="thread_id_metadata" pattern="${ctx:thread_id}"/>
This tag supports the same variable substitution and pattern features as the Label tag, but the resulting
key-value pairs are sent as structured metadata rather than labels.
Size of the log buffer. Must be power of two between 1MB and 1GB.
See log buffer sizing for more explanations.
Size of an intermediate thread local buffer that log4j uses to serialise a single log message into. Log lines larger than that will be split into multiple log entries (see wiki for discussion).
Maximum number of retries to perform when delivering a log message to Loki. Log buffer data is delivered in order, one batch after the other, so too many retries will block delivery of subsequent log batches (on the other hand if we need to retry many times, then next batches will probably fail too).
This configures socket connect timeout when connecting to Loki. After an unsuccessful connection attempt, it will continue to retry indefinitely employing exponential backoff (initial backoff = 250ms, maximum backoff = 30s, multiplier = 3).
Sets socket read timeout on Loki connection.
Like
in promtail configuration maximum batch size (in bytes) of logs to accumulate before sending the batch to Loki
.
Like
in promtail configuration maximum amount of time to wait before sending a batch, even if that batch isn't full
.
The agent that reads data from log buffer, compresses it, and sends to Loki via http is called LogShipper. This property
controls how often it wakes up to perform its duties. Other properties control how often the data should be sent to Loki
(batchSize, batchWait) this one just controls how often to wake up and check for these conditions. In versions before
0.9.17 it was left at default 1ms which caused high CPU usage on some setups.
On logging system shutdown (or config reload) Tjahzi will flush its internal buffers so that no logs are lost. This property sets a limit on how long to wait for this to complete before proceeding with shutdown.
If set to true, Tjahzi will run all its threads as daemon threads.
Use this option if you do not want to explicitly close the logging system and still want to make sure Tjahzi internal threads will not prevent JVM from closing down. Note that this can result in unflushed logs not being delivered when the JVM is closed.
If set to true, Tjahzi will log internal errors and connection errors to Log4j2’s internal status logger. This includes agent errors, pipeline errors, dropped log entries, HTTP errors, failed HTTP requests, and connection issues.
It’s best used in conjunction with -Dlog4j2.debug setting,
see log4j2 docs for more information.
This functionality is provided using LokiAppender::setMonitoringModule. It will not work if you set your own
custom monitoring module, which should be fine—you already have set up your own way to monitor errors.