Elasticsearch Log Plug-In (2020.2.18 and later)

Learn how to install and configure Elasticsearch logging using Log4j.

Note: Log4j 2.x requires Elasticsearch version 7.16.2.

Valid in Version: 2020.2.17 and later, 2022.1.0 and later

Table of Contents

• Overview
• Elasticsearch Appender features
• Installing the Akana Elasticsearch Log Plugin
• Elasticsearch Log Plugin general configuration
• Index naming and templates
• Akana Template Layout
• Default JSON Template
• Uninstalling the Akana Elasticsearch Log Plugin

Overview

Elasticsearch can be used as a repository for log data in the Akana product. In versions prior to 2020.2.16, the Akana SaaS platform used an in-house Elasticsearch Appender for Log4j 1.x. With the transition to Log4j 2.x, this legacy appender no longer works. Instead, the Akana product uses an off-the-shelf implementation that is hosted in a public GitHub repository: https://github.com/rfoltyns/log4j2-elasticsearch. The new implementation is more powerful, but also more complex to configure. The Akana platform provides defaults which should be sufficient in most cases; you can also customize the configuration from the templates provided.

The Layouts available in the core Log4j release do not work with the Elasticsearch Appender; only a simple JSON layout is supported. To compensate for this limitation, there is a new Akana Layout for Log4j 2.x, which encapsulates the off-the-shelf JSON Template Layout and makes it compatible with the Elasticsearch Appender.

Elasticsearch Appender features

The Elasticsearch Appender provides many features, including the following:

• Asynchronous log delivery
• Batch size and flush interval configuration
• Failover (redirect failed batch to alternative target)
• Index rollover (hourly, daily, and so on)
• Index template configuration
• Basic Authentication (XPack Security and Shield support)
• HTTPS support (XPack Security and Shield support)
• Failover with persistence and retry
• Log overflow prevention with backoff policies
• Optimized HTTP client

Installing the Akana Elasticsearch Log Plugin

The Akana Elasticsearch Log Plugin can be installed using the Admin Console. To install:

1. On the Available Features tab, select Plug-in from the dropdown list, and select the Akana Elasticsearch Log Appenders Plugin checkbox.

2. Click Install Feature. The required bundles are installed for the Akana Elasticsearch Log Appenders Plugin. If prompted, restart the container after installing the plugin.

   

The Akana Elasticsearch Log Plugin lets you configure and write the container log files and HTTP access log files to the Elasticsearch server.

To use automation, there is a single deployment automation recipe, elasticsearch-logger.json, that deploys the necessary bundles and redirects logging to Elasticsearch server. The elasticsearch-logger.json recipe can accept the properties listed below.

Parameter	Description	Default Value
ES_LOG_SERVER_URLS	Comma-separated list of Elasticsearch server URLs	http://localhost:9200
ES_LOG_BATCH_INTERVAL	The interval, in milliseconds, when batches will be written unless the batch size is reached first.	3000
ES_LOG_BATCH_SIZE	The batch size that will force a write, unless the interval is reached first.	2048
ES_LOG_EVENT_TEMPLATE	The layout template for log events.	$INSTALL_DIR/recipes/elasticsearch-ecs-layout-template.json
ES_LOG_INDEX_NAME	The base index name.	akana
ES_LOG_INDEX_NAME_PATTERN	The index name pattern.	yyyy-MM-dd
ES_LOG_INDEX_TEMPLATE	The Elasticsearch mapping document for the index.	$INSTALL_DIR/recipes/elasticsearch-index-template.json
ES_LOG_SERVER_CONNECT_TIMEOUT	The server connect timeout in milliseconds.	500
ES_LOG_SERVER_MAPPING_TYPE	The document type in Elasticsearch.	_doc
ES_LOG_SERVER_MAX_CONNECTIONS	The maximum number of connections to establish to the server.	8
ES_LOG_SERVER_USERNAME	The server username.
ES_LOG_SERVER_PASSWORD	The server password.
ES_LOG_SERVER_READ_TIMEOUT	The client read timeout in milliseconds.	20000
ES_LOG_STACK_TEMPLATE	The layout template for stack elements.	$INSTALL_DIR/recipes/elasticsearch-ecs-stack-template.json

Note: The rootLogger.level parameter can be set to all, fatal, error, warn, info, debug, trace, or off. The default value is set to error. However, if you set the rootLogger.level to all, the log files can be huge on a system and may fill up its mass storage over time.

Elasticsearch Log Plugin general configuration

Because it is feature-rich, the Elasticsearch Log Plug-in provides many tuning parameters. The product ships with sensible defaults, but there are some important configuration parameters that might change on a per-installation basis.

Parameter	Type	Description
serverUris (ES_LOG_SERVER_URLS)	Comma-separated String	A set of Elasticsearch server URLs. Defaults to HTTP://localhost:9200.
connTimeout (ES_LOG_SERVER_CONNECT_TIMEOUT)	Integer	The server connection timeout in milliseconds. Default: 500
readTimeout (ES_LOG_SERVER_READ_TIMEOUT)	Integer	The read timeout in milliseconds. Default: 20000
maxTotalConnections (ES_LOG_SERVER_MAX_CONNECTIONS)	Integer	The maximum number of concurrent connections that will be opened to the server. Default: 8
auth.credentials.username (ES_LOG_SERVER_USERNAME)	String	The username for connecting to Elasticsearch.
auth.credentials.password (ES_LOG_SERVER_PASSWORD)	String	The password for connecting to Elasticsearch.

You can modify the default values in either of two ways:

• At deployment time or runtime, using automation recipes. For information about the new recipe that deploys the necessary bundles and redirects logging to Elasticsearch, see Installing the Akana Elasticsearch Log Plugin.

  For general information on using automation recipes, see Automation Reference Guide.

• In the Admin Console: In Configuration Properties, find the com.soa.log configuration categories. These Elasticsearch logging parameters all have this prefix: appender.elastic.batchDelivery.objectFactory.

Index naming and templates

There are two configuration parameters that allow fine-grained control of index names.

Parameter	Type	Description
indexName (ES_LOG_INDEX_NAME)	String	The base name for the index. The eventual index name will be a concatenation of the index name and pattern.
pattern (ES_LOG_INDEX_NAME_PATTERN)	String	A pattern to apply to the index name. This uses the Java Date formatting pattern strings. The pattern controls when the index names roll over. For example: If the pattern is yyyy-MM-dd, the index name rolls over every day. If the pattern is yyyy-MM-dd-HH, the index rolls over every hour.

You can modify the default values in either of two ways:

• At deployment time or runtime, using an automation recipe. For information about the recipe, see Installing the Akana Elasticsearch Log Plugin.

  For general information on using automation recipes, see Automation Reference Guide.

• In the Admin Console: In Configuration Properties, find the com.soa.log configuration categories. These Elasticsearch logging parameters all have this prefix: appender.elastic.indexNameFormatter.

Index templates are specified using the indexTemplate.sourceString property. The source is an Elasticsearch mapping document. For more information, see https://www.elastic.co/guide/en/elasticsearch/reference/current/explicit-mapping.html.

Akana Template Layout

The Log4j JSON Template Layout (see https://logging.apache.org/log4j/2.x/manual/json-template-layout.html) is a highly customizable layout generator. The product embeds this layout generator in a new bundle that adapts it to the Elasticsearch Appender. Most configuration options remain the same between the Log4j template layout and the Akana template layout, with some simplifications. The new Layout name is AkanaTemplateLayout rather than JsonTemplateLayout.

The table below describes the configuration properties available in the new Layout.

Parameter	Type	Description
charset	Charset	Charset used for String encoding (defaults to UTF-8)
stackTraceEnabled	boolean	Toggles access to the stack traces (defaults to true)
eventTemplate	String	Inline JSON template for rendering LogEvents (has priority over eventTemplateUri)
eventTemplateUri	String	URI pointing to the JSON template for rendering LogEvents
eventTemplateAdditionalField	EventTemplateAdditionalField[]	Additional key-value pairs appended to the root of the event template
stackTraceElementTemplate	String	Inline JSON template for rendering StackTraceElements (has priority over stackTraceElementTemplateUri)
stackTraceElementTemplateUri	String	URI pointing to the JSON template for rendering StackTraceElements
maxStringLength	int	Truncate string values longer than the specified limit (defaults to 16384)

Default JSON Template

The template layout uses JSON documents to describe the format of log entries (events) and stack trace entries. Examples of different formats are available from the Log4j repository: see https://github.com/apache/logging-log4j2/tree/release-2.x/log4j-layout-template-json/src/main/resources. By default, the product uses a template that generates log entries that are compatible with ECS (Elasticsearch Common Schema). The default template for log entries is reproduced below.

{
  "@timestamp":{
    "$resolver":"timestamp",
    "pattern":{
      "format":"yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",
      "timeZone":"UTC"
    }
  },
  "ecs.version":"1.2.0",
  "log.level":{
    "$resolver":"level",
    "field":"name"
  },
  "message":{
    "$resolver":"message",
    "stringified":true
  },
  "process.thread.name":{
    "$resolver":"thread",
    "field":"name"
  },
  "log.logger":{
    "$resolver":"logger",
    "field":"name"
  },
  "error.type":{
    "$resolver":"exception",
    "field":"className"
  },
  "error.message":{
    "$resolver":"exception",
    "field":"message"
  },
  "error.stack_trace":{
    "$resolver":"exception",
    "field":"stackTrace",
    "stackTrace":{
      "stringified":true
   }
  },
  "enclosedExceptions": {
    "$resolver": "mdc",
    "key":"enclosedExceptions"
  },
  "request": {
    "$resolver": "mdc",
    "key":"request"
  },
  "tenant": {
    "$resolver": "mdc",
    "key":"tenant"
  },
  "eventId": {
    "$resolver": "mdc",
    "key":"eventId"
  },
  "alertCode": {
    "$resolver": "mdc",
    "key":"alertCode"
  }
}

Related Topics