How to Install Elasticsearch on Debian 11 Bullseye

Elasticsearch is a highly scalable open-source full-text search and analytics engine. The software supports RESTful operations that allow you to store, search, and analyze big volumes of data quickly and in near real-time. Elasticsearch is well-liked and popular amongst sysadmins and developers as it is a mighty search engine based on the Lucene library. It is generally used as the underlying engine/technology that powers applications with complex search features and requirements.

In the following tutorial, you will learn how to install Elastic Search on Debian 11 Bullseye.

Prerequisites

• Recommended OS: Debian 11 Bullseye
• User account: A user account with sudo or root access.
• Required Packages: Curl and Java

Installing Curl

Curl is needed for some parts of this guide. To install this package, type the following command:

sudo apt install curl -y

Installing Java

To successfully install and, more importantly, use Elasticsearch, you need to install Java. The process is quite easy.

Type the following command to install the OpenJDK package:

sudo apt install default-jdk

Example output with packages to be installed:

To proceed with the installation, type (Y) then press the (ENTER) key.

Next, verify the Java version installed and the build with the following command:

java -version

Example output below:

Installing Elasticsearch

Elasticsearch is not available in the standard Debian 11 repositories, so you will need to install it from the Elasticsearch APT repository.

Before adding the repository, import the GPG key with the following command:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Example output with (Ok) confirming it was successful:

Now that you have added the GPG key, the Elasticsearch repositories will be now trusted. Proceed to install the official repository with the following terminal command:

sudo sh -c 'echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" > /etc/apt/sources.list.d/elastic-7.x.list'

At the time of the tutorial being created, Elasticsearch 7.13.4 is the latest version but will change in time, and the repository you added will work for any 7.x.x versions in the future.

Next, update your repository list and install Elasticsearch as follows:

sudo apt update && sudo apt install elasticsearch

By default, the Elasticsearch service is disabled on boot and not active. To start the service and enable it on system boot, type the following (systemctl) command:

sudo systemctl enable elasticsearch.service --now

Next, to verify that Elasticsearch is running correctly, you will use the curl command to send an HTTP request to port 9200 on localhost as follows:

curl -X GET "localhost:9200/"

Example output to verify Elasticsearch working correctly:

To view the system message that Elasticsearch logs on your system, type the following command:

sudo journalctl -u elasticsearch

Example output log:

Configuring Elasticsearch

Elasticsearch data is stored in the default directory location (/var/lib/elasticsearch). To view or edit the configuration files, you can find them in the directory location (/etc/elasticsearch), and java start-up options can be configured in the (/etc/default/elasticsearch) configuration file.

The default settings are mostly fine for single operating servers as Elasticsearch runs on localhost only. However, if you are going to set up a cluster, you will need to modify the configuration file to allow remote connections.

Set-up Remote Access (Optional)

By default, Elasticsearch listens only to localhost. To change this, open up the configuration file as follows:

sudo nano /etc/elasticsearch/elasticsearch.yml

Next, scroll down and find the Network section and uncomment (#) the following line and replace it with the Internal Private IP address or External IP address as follows:

In the example, we uncommented (#) the (network.host) and changed it to an Internal Private IP address.

For security purposes, it is ideal for specifying addresses; however, if you have multiple Internal or External IP addresses hitting the server change the network interface to listen to all with entering (0.0.0.0) as follows:

Next, save the configuration file (CTRL+O) then exit (CLTR+X).

You will need to restart the Elasticsearch service with the following command for changes to take effect:

sudo systemctl restart elasticsearch

Configure UFW Firewall to allow Remote Connections

If you allow remote connections, you will need to allow your firewall to allow those IP addresses to connect. This can be done with the following command:

sudo ufw allow from <IP Address> to any port 9200

More information on Debian and UFW firewall can be found to configure the UFW firewall on Debian.

How to use Elasticsearch

To use Elasticsearch using the curl command is a straightforward process. Below are some of the most commonly used:

Delete index

Below the index is named samples.

curl -X DELETE 'http://localhost:9200/samples'

List all indexs

curl -X GET 'http://localhost:9200/_cat/indices?v'

List all docs in index

curl -X GET 'http://localhost:9200/sample/_search'

Query using URL parameters

Here we use Lucene query format to write q=school:Harvard.

curl -X GET http://localhost:9200/samples/_search?q=school:Harvard

Query with JSON aka Elasticsearch Query DSL

You can query using parameters on the URL. But you can also use JSON, as shown in the next example. JSON would be easier to read and debug when you have a complex query than one giant string of URL parameters.

curl -XGET --header 'Content-Type: application/json' http://localhost:9200/samples/_search -d '{
      "query" : {
        "match" : { "school": "Harvard" }
    }
}'

List index mapping

All Elasticsearch fields are indexes. So this lists all fields and their types in an index.

curl -X GET http://localhost:9200/samples

Add Data

curl -XPUT --header 'Content-Type: application/json' http://localhost:9200/samples/_doc/1 -d '{
   "school" : "Harvard"			
}'

Update Doc

Here is how to add fields to an existing document. First, we create a new one. Then we update it.

curl -XPUT --header 'Content-Type: application/json' http://localhost:9200/samples/_doc/2 -d '
{
    "school": "Clemson"
}'

curl -XPOST --header 'Content-Type: application/json' http://localhost:9200/samples/_doc/2/_update -d '{
"doc" : {
               "students": 50000}
}'

Backup index

curl -XPOST --header 'Content-Type: application/json' http://localhost:9200/_reindex -d '{
  "source": {
    "index": "samples"
  },
  "dest": {
    "index": "samples_backup"
  }
}'

Bulk load data in JSON format

export pwd="elastic:"

curl --user $pwd  -H 'Content-Type: application/x-ndjson' -XPOST 'https://58571402f5464923883e7be42a037917.eu-central-1.aws.cloud.es.io:9243/0/_bulk?pretty' --data-binary @<file>

Show cluster health

curl --user $pwd  -H 'Content-Type: application/json' -XGET https://58571402f5464923883e7be42a037917.eu-central-1.aws.cloud.es.io:9243/_cluster/health?pretty

Aggregation and Bucket Aggregation

For an Nginx web server, this produces web hit counts by user city:

curl -XGET --user $pwd --header 'Content-Type: application/json'  https://58571402f5464923883e7be42a037917.eu-central-1.aws.cloud.es.io:9243/logstash/_search?pretty -d '{
        "aggs": {
             "cityName": {
                    "terms": {
                     "field": "geoip.city_name.keyword",
                                "size": 50

        }
   }
  }
}
'

This expands that to product response code count by the city in an Nginx web server log

curl -XGET --user $pwd --header 'Content-Type: application/json'  https://58571402f5464923883e7be42a037917.eu-central-1.aws.cloud.es.io:9243/logstash/_search?pretty -d '{
        "aggs": {
          "city": {
                "terms": {
                        "field": "geoip.city_name.keyword"
                },
        "aggs": {
          "responses": {
                "terms": {
                     "field": "response"
                 }
           }
         }
      },
      "responses": {
                "terms": {
                     "field": "response"
                 }
        }
   }
}'

Using ElasticSearch with Basic Authentication

If you have turned on security with ElasticSearch, then you need to supply the user and password like shown below to every curl command:

curl -X GET 'http://localhost:9200/_cat/indices?v' -u elastic:(password)

Pretty Print

Add ?pretty=true to any search to pretty-print the JSON. Like this:

 curl -X GET 'http://localhost:9200/(index)/_search'?pretty=true

To query and return only certain fields

To return only certain fields, put them into the _source array:

GET filebeat-7.6.2-2020.05.05-000001/_search
 {
    "_source": ["suricata.eve.timestamp","source.geo.region_name","event.created"],
    "query":      {
        "match" : { "source.geo.country_iso_code": "GR" }
    }
}

To Query by Date

When the field is of type date, you can use date math, like this:

GET filebeat-7.6.2-2020.05.05-000001/_search
 {
    "query": {
        "range" : {
            "event.created": {
                "gte" : "now-7d/d"
            }
        }
}
}

Uninstalling Elasticsearch

If you no longer require Elasticsearch, you can remove the software with the following command:

sudo apt remove elasticsearch

Remove the apt repository as follows:

sudo rm /etc/apt/sources.list.d/elastic-7.x.list

Then update your repository list to reflect the changes:

sudo apt update

Comments and Conclusion

You have learned how to install Elasticsearch on Debian 11 Bullseye from Elasticsearch’s official repository and configure the basics in the tutorial. Overall, ElasticSearch has many popular features, some were mentioned at the start of the tutorial, but others include enabling users to search various fields by using a single query. ElasticSearch provides a great level of sharding, which means horizontal scalability, which enhances the performance even with an increase in load.

For further reading, visit the official documentation page.