Troubleshooting Elasticsearch

Some troubleshooting information to help resolve issues with Elasticsearch.

Table of Contents

Elasticsearch client can't connect to the nodes

If the Elasticsearch client can't connect to the nodes, and/or the nodes can't connect to each other, check that you have the correct Elasticsearch ports available on every container that the Elasticsearch feature is installed on.

The ports are specific to the REST Client mode in use. The default is RESTClient mode: 9200.

However, it depends on what's specified in the Elasticsearch YAML file. Make sure the ports specified in the YAML file are available.

For more information, see What changes do I need to make to the Elasticsearch YAML file?

Elasticsearch search results hang or are not returned

In certain circumstances you might encounter an issue that Elasticsearch search results are not returned, and the search query seems to hang. If restarting the CM container resolves the issue, you could try reducing the default timeout connection settings to help free up connections.

There are also additional Elasticsearch security settings that you can modify, if needed, to help resolve Elasticsearch connection issues by modifying the timeout values.

If you have Elasticsearch connections that use a centralized server behind a firewall, it might be possible that the firewall is closing the TCP connection from the CM DMZ containers to the internal Elasticsearch cluster. In this scenario, reducing the timeout value results in CM freeing up those database connections more quickly.

In most cases the default settings should be fine.

Information on the Elasticsearch timeout settings is given below.

To modify the Elasticsearch timeout values

  1. Log in to the Akana Administrator Console for the Policy Manager container.
  2. Go to Configuration > Configuration Categories and find the com.akana.es.client.security category.
  3. Modify the settings as needed:
    • elastic.client.connectTimeout. Default: 1000.
    • elastic.client.socketTimeout. Default: 30000.

New fields not showing up in search index

If new fields are added to Elasticsearch in a new version of the product, manual steps are needed so that the new fields are included in the search index. If these actions are not completed, the new fields are not indexed.

  • Versions prior to 2020.2.0: Follow the instructions in the Akana Release Notes. New fields were added in the following versions:
    • 2019.1.14
    • 2019.1.12
    • 2019.1.8
  • 2020.2.0 and later: run the cm-es-index-upgrade.json automation recipe. For more information, see Updating the Elasticsearch index.

Connection errors with Elasticsearch Java API Client on AWS NLB: Configuring TCP keepalive values

Valid in Version: 2020.2.18 and later

If you're using Elasticsearch on Amazon Web Services (AWS) with the AWS NLB load balancer, you'll need to update the keepalive settings on any servers that Akana containers using Elasticsearch are running on; essentially, any container that has Policy Manager or Community Manager installed. By default, the AWS NLB closes connections after 350 seconds. In practical terms, if a Community Manager user does a search, gets the results, then does something else for more than 350 seconds and then searches again, there will be an error because the connection was closed by the load balancer.

To help ensure a smooth user experience, you can configure keepalive probe settings on the Elasticsearch server.

The most important setting is net.ipv4.tcp_keepalive_time. The default is 7200 seconds; you could change it to a lower number such as 300 seconds.

For additional information and instructions, see Using Elasticsearch with AWS NLB: Enabling TCP keepalive values in the Elasticsearch installation and configuration documentation.

Elasticsearch throwing maxClauseCount limit error for large data manipulation APIs, APPs, or Groups

Elasticsearch will now dynamically set the maximum number of allowed clauses in a query based on the size of the search thread pool and the size of the heap allocated to the JVM.

However, if you are getting a "maxClauseCount" limit error in Elasticsearch while querying, you can increase the default configuration value for the maximum number of Boolean clauses allowed within a search string by using the indices.query.bool.max_clause_count property. For more information, see Search Settings.

Elasticsearch indices can't sync causing APIs not to return any results

You may notice that the Elasticsearch indices are not synchronized when upgrading from an earlier version to 2022.1.3, causing the APIs to not return any results.

This error occurs when the indexes are not properly synchronized. To resolve this error, follow these steps to resynchronize the indices.

  1. Delete the default_apiversion by using the following command.
    curl -k -sS "https://${IP}:9200/default_apiversion" -X DELETE 
{"acknowledged":true}
  2. Delete the default_api by using the following command. 
    curl -k -sS "https://${IP}:9200/default_api" -X DELETE 
{"acknowledged":true}
  3. Delete the indices from the INDEX_STATUS table to sync indices by running the following command.
    delete from INDEX_STATUS where OBJECTTYPE in ('api', 'apiversion');