Skip to main content

ElasticSearch max file descriptors too low error

ElasticSearch 5.x requires a minimum of Max file descriptors 65536 and Max virtual memory areas 262144.
It throws an error on start-up if these are set to very low value.
ERROR: bootstrap checks failed
max file descriptors [16384] for elasticsearch process is too low, increase to at least [65536]
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

Check current values using:
$ cat /proc/sys/fs/file-max 
16384
$ cat /proc/sys/vm/max_map_count
65530
$ ulimit -Hn
16384
$ ulimit -Sn
4096

To fix this, following files need to change/add below settings:
  • Recommended: Add a new file 99-elastic.conf under /etc/security/limits.d with following settings:
    elasticsearch          - nofile  800000
elasticsearch          - nproc   16384
defaultusername        - nofile  800000
defaultusername        - nproc   16384
  • Alternatively, edit /etc/sysctl.conf with following settings:
    fs.file-max = 800000
vm.max_map_count=300000
Labels:
Location: London, UK

Comments

sudo said…
99-elastic.conf is that a correct file name?
17 April, 2019 13:11
Post a Comment

Popular posts from this blog

wget and curl behind corporate proxy throws certificate is not trusted or certificate doesn't have a known issuer

If you try to run wget or curl in Ununtu/Debian behind corporate proxy, you might receive errors like: ERROR: The certificate of 'apertium.projectjj.com' is not trusted. ERROR: The certificate of 'apertium.projectjj.com' doesn't have a known issuer. wget https://apertium.projectjj.com/apt/apertium-packaging.public.gpg ERROR: cannot verify apertium.projectjj.com's certificate, issued by 'emailAddress=proxyteam@corporate.proxy.com,CN=diassl.corporate.proxy.com,OU=Division UK,O=Group name,L=Company,ST=GB,C=UK': Unable to locally verify the issuer's authority. To connect to apertium.projectjj.com insecurely, use `--no-check-certificate'. To solution is to install your company's CA certificate in Ubuntu. In Windows, open the first part of URL in your web browser. e.g. open https://apertium.projectjj.com in web browser. If you inspect the certifcate, you will see the same CN (diassl.corporate.proxy.com), as reported by the error above ...
Post a Comment
Read more

Kafka performance tuning

Performance Tuning of Kafka is critical when your cluster grow in size. Below are few points to consider to improve Kafka performance: Consumer group ID : Never use same exact consumer group ID for dozens of machines consuming from different topics. All of those commits will end up on the same exact partition of __consumer_offsets , hence the same broker, and this might in turn cause performance problems. Choose the consumer group ID to group_id+topic_name . Skewed : A broker is skewed if its number of partitions is greater that the average of partitions per broker on the given topic. Example: 2 brokers share 4 partitions, if one of them has 3 partitions, it is skewed (3 > 2). Try to make sure that none of the brokers is skewed. Spread : Brokers spread is the percentage of brokers in the cluster that has partitions for the given topic. Example: 3 brokers share a topic that has 2 partitions, so 66% of the brokers have partitions for this topic. Try to achieve 100% broker spread...
1 comment
Read more

ElasticSearch Curator

Curator is a tool from Elastic to help manage your ElasticSearch cluster. For certain logs/data, we use one ElasticSearch index per year/month/day and might keep a rolling 7 day window of history. This means that every day we need to create, backup, and delete some indices. Curator helps make this process automated and repeatable. Installation Curator is written in Python , so will need pip to install it: pip install elasticsearch-curator curator --config ./curator_cluster_config.yml curator_actions.yml --dry-run Configuration Create a file curator_cluster_config.yml with following contents: --- # Remember, leave a key empty if there is no value. None will be a string, not a Python "NoneType" client: hosts: - "es_coordinating_01.singhaiuklimited.com" port: 9200 url_prefix: use_ssl: True # The certificate file is the CA certificate used to sign all ES node certificates. # Use same CA certificate to generate and sign the certificate running ...
Post a Comment
Read more