Configuring the Elasticsearch Log Appender (2022.1.0 and later)

Learn how to configure Elasticsearch logging using Log4j.

Note: Log4j 2.x requires Elasticsearch version 7.16.2.

Valid in Version: 2020.2.17 and later, 2022.1.0 and later

Table of Contents

Overview

Elasticsearch can be used as a repository for log data in the Akana product. In versions prior to 2020.2.16, the Akana SaaS platform used an in-house Elasticsearch Appender for Log4j 1.x. With the transition to Log4j 2.x, this legacy appender no longer works. Instead, the Akana product uses an off-the-shelf implementation that is hosted in a public GitHub repository: https://github.com/rfoltyns/log4j2-elasticsearch. The new implementation is more powerful, but also more complex to configure. The Akana platform provides defaults which should be sufficient in most cases; you can also customize the configuration from the templates provided.

The Layouts available in the core Log4j release do not work with the Elasticsearch Appender; only a simple JSON layout is supported. To compensate for this limitation, there is a new Akana Layout for Log4j 2.x, which encapsulates the off-the-shelf JSON Template Layout and makes it compatible with the Elasticsearch Appender.

Elasticsearch Appender features

The Elasticsearch Appender provides many features, including the following:

  • Asynchronous log delivery
  • Batch size and flush interval configuration
  • Failover (redirect failed batch to alternative target)
  • Index rollover (hourly, daily, and so on)
  • Index template configuration
  • Basic Authentication (XPack Security and Shield support)
  • HTTPS support (XPack Security and Shield support)
  • Failover with persistence and retry
  • Log overflow prevention with backoff policies
  • Optimized HTTP client

Elasticsearch Appender general configuration

Because it is feature-rich, the Appender provides many tuning parameters. The product ships with sensible defaults, but there are some important configuration parameters that might change on a per-installation basis.

Parameter Type Description
serverUris (ES_LOG_SERVER_URLS) Comma-separated String A set of Elasticsearch server URLs. Defaults to HTTP://localhost:9200.
connTimeout (ES_LOG_SERVER_CONNECT_TIMEOUT) Integer

The server connection timeout in milliseconds.

Default: 500
readTimeout (ES_LOG_SERVER_READ_TIMEOUT) Integer

The read timeout in milliseconds.

Default: 20000
maxTotalConnections (ES_LOG_SERVER_MAX_CONNECTIONS) Integer

The maximum number of concurrent connections that will be opened to the server.

Default: 8
auth.credentials.username (ES_LOG_SERVER_USERNAME) String The username for connecting to Elasticsearch.
auth.credentials.password (ES_LOG_SERVER_PASSWORD) String The password for connecting to Elasticsearch.

You can modify the default values in either of two ways:

  • At deployment time or runtime, using automation recipes. For information about the new recipe that deploys the necessary bundles and redirects logging to Elasticsearch, see Deploying the Elasticsearch appender below.

    For general information on using automation recipes, see Automation Reference Guide.

  • In the Admin Console: In Configuration Properties, find the com.soa.log configuration categories. These Elasticsearch logging parameters all have this prefix: appender.elastic.batchDelivery.objectFactory.

Index naming and templates

There are two configuration parameters that allow fine-grained control of index names.

Parameter Type Description
indexName (ES_LOG_INDEX_NAME) String The base name for the index. The eventual index name will be a concatenation of the index name and pattern.
pattern (ES_LOG_INDEX_NAME_PATTERN) String

A pattern to apply to the index name. This uses the Java Date formatting pattern strings. The pattern controls when the index names roll over. For example:

  • If the pattern is yyyy-MM-dd, the index name rolls over every day.
  • If the pattern is yyyy-MM-dd-HH, the index rolls over every hour.

You can modify the default values in either of two ways:

  • At deployment time or runtime, using an automation recipe. For information about the recipe, see Deploying the Elasticsearch appender below.

    For general information on using automation recipes, see Automation Reference Guide.

  • In the Admin Console: In Configuration Properties, find the com.soa.log configuration categories. These Elasticsearch logging parameters all have this prefix: appender.elastic.indexNameFormatter.

Index templates are specified using the indexTemplate.sourceString property. The source is an Elasticsearch mapping document. For more information, see https://www.elastic.co/guide/en/elasticsearch/reference/current/explicit-mapping.html.

JSON Template Layout

The Log4j JSON Template Layout (see https://logging.apache.org/log4j/2.x/manual/json-template-layout.html) is a highly customizable layout generator. The product embeds this layout generator in a new bundle that adapts it to the Elasticsearch Appender. Most configuration options remain the same between the Log4j template layout and the Akana template layout, with some simplifications. The new Layout name is AkanaTemplateLayout rather than JsonTemplateLayout.

The table below describes the configuration properties available in the new Layout.

Parameter Type Description
charset Charset Charset used for String encoding (defaults to UTF-8)
stackTraceEnabled boolean Toggles access to the stack traces (defaults to true)
eventTemplate String Inline JSON template for rendering LogEvents (has priority over eventTemplateUri)
eventTemplateUri String URI pointing to the JSON template for rendering LogEvents
eventTemplateAdditionalField EventTemplateAdditionalField[] Additional key-value pairs appended to the root of the event template
stackTraceElementTemplate String Inline JSON template for rendering StackTraceElements (has priority over stackTraceElementTemplateUri)
stackTraceElementTemplateUri String URI pointing to the JSON template for rendering StackTraceElements
maxStringLength int Truncate string values longer than the specified limit (defaults to 16384)

Default JSON Template

The template layout uses JSON documents to describe the format of log entries (events) and stack trace entries. Examples of different formats are available from the Log4j repository: see https://github.com/apache/logging-log4j2/tree/release-2.x/log4j-layout-template-json/src/main/resources. By default, the product uses a template that generates log entries that are compatible with ECS (Elasticsearch Common Schema). The default template for log entries is reproduced below.

{
  "@timestamp":{
    "$resolver":"timestamp",
    "pattern":{
      "format":"yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",
      "timeZone":"UTC"
    }
  },
  "ecs.version":"1.2.0",
  "log.level":{
    "$resolver":"level",
    "field":"name"
  },
  "message":{
    "$resolver":"message",
    "stringified":true
  },
  "process.thread.name":{
    "$resolver":"thread",
    "field":"name"
  },
  "log.logger":{
    "$resolver":"logger",
    "field":"name"
  },
  "error.type":{
    "$resolver":"exception",
    "field":"className"
  },
  "error.message":{
    "$resolver":"exception",
    "field":"message"
  },
  "error.stack_trace":{
    "$resolver":"exception",
    "field":"stackTrace",
    "stackTrace":{
      "stringified":true
    }
  }
}

Deploying the Elasticsearch appender

The Elasticsearch appender can be installed and configured using deployment automation recipes or using the Admin Console. To use automation, there is a single recipe, elasticsearch-logger.json, that deploys the necessary bundles and redirects logging to Elasticsearch. The elasticsearch-logger.json recipe can accept the properties listed below.

Parameter Description Default Value
ES_LOG_SERVER_URLS Comma-separated list of Elasticsearch server URLs http://localhost:9200
ES_LOG_APPENDERS

The root appenders to use. By default, deployment of the Elasticsearch Appender directs all logging to Elasticsearch.

If file logging is also required, you can use the default logger as well as Elasticsearch. To do this, use the following value to specify that both the rolling file appender and the Elasticsearch appender should be used:

elastic, rolling.

Note: Writing to multiple loggers introduces additional overhead.

 elastic
ES_LOG_BATCH_INTERVAL The interval, in milliseconds, when batches will be written unless the batch size is reached first. 3000
ES_LOG_BATCH_SIZE The batch size that will force a write, unless the interval is reached first. 2048
ES_LOG_EVENT_TEMPLATE The layout template for log events. $INSTALL_DIR/recipes/elasticsearch-ecs-layout-template.json
ES_LOG_INDEX_NAME The base index name. akana
ES_LOG_INDEX_NAME_PATTERN The index name pattern. yyyy-MM-dd
ES_LOG_INDEX_TEMPLATE The Elasticsearch mapping document for the index. $INSTALL_DIR/recipes/elasticsearch-index-template.json
ES_LOG_SERVER_CONNECT_TIMEOUT The server connect timeout in milliseconds. 500
ES_LOG_SERVER_MAPPING_TYPE The document type in Elasticsearch. _doc
ES_LOG_SERVER_MAX_CONNECTIONS The maximum number of connections to establish to the server. 8
ES_LOG_SERVER_USERNAME The server username.
ES_LOG_SERVER_PASSWORD The server password.
ES_LOG_SERVER_READ_TIMEOUT The client read timeout in milliseconds. 20000
ES_LOG_STACK_TEMPLATE The layout template for stack elements. $INSTALL_DIR/recipes/elasticsearch-ecs-stack-template.json