README.md 10.5 KB
Newer Older
Anthony Lapenna's avatar
Anthony Lapenna committed
1
# Docker ELK stack
tony's avatar
tony committed
2

ivanrome's avatar
ivanrome committed
3
[![Join the chat at https://gitter.im/deviantony/docker-elk](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/deviantony/docker-elk?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
Mario Kozjak's avatar
Mario Kozjak committed
4
[![Elastic Stack version](https://img.shields.io/badge/ELK-6.2.3-blue.svg?style=flat)](https://github.com/deviantony/docker-elk/issues/266)
ivanrome's avatar
ivanrome committed
5
[![Build Status](https://api.travis-ci.org/deviantony/docker-elk.svg?branch=master)](https://travis-ci.org/deviantony/docker-elk)
The Gitter Badger's avatar
The Gitter Badger committed
6

Daniel Macedo's avatar
Daniel Macedo committed
7
Run the latest version of the ELK (Elasticsearch, Logstash, Kibana) stack with Docker and Docker Compose.
Anthony Lapenna's avatar
Anthony Lapenna committed
8

Daniel Macedo's avatar
Daniel Macedo committed
9
10
It will give you the ability to analyze any data set by using the searching/aggregation capabilities of Elasticsearch
and the visualization power of Kibana.
Anthony Lapenna's avatar
Anthony Lapenna committed
11

Daniel Macedo's avatar
Daniel Macedo committed
12
Based on the official Docker images:
Anthony Lapenna's avatar
Anthony Lapenna committed
13

Antoine Cotten's avatar
Antoine Cotten committed
14
15
16
* [elasticsearch](https://github.com/elastic/elasticsearch-docker)
* [logstash](https://github.com/elastic/logstash-docker)
* [kibana](https://github.com/elastic/kibana-docker)
Anthony Lapenna's avatar
Anthony Lapenna committed
17

18
19
**Note**: Other branches in this project are available:

20
21
22
* ELK 6 with X-Pack support: https://github.com/deviantony/docker-elk/tree/x-pack
* ELK 6 in Vagrant: https://github.com/deviantony/docker-elk/tree/vagrant
* ELK 6 with Search Guard: https://github.com/deviantony/docker-elk/tree/searchguard
23

Daniel Macedo's avatar
Daniel Macedo committed
24
25
26
27
28
## Contents

1. [Requirements](#requirements)
   * [Host setup](#host-setup)
   * [SELinux](#selinux)
29
   * [DockerForWindows](#dockerforwindows)
Daniel Macedo's avatar
Daniel Macedo committed
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
2. [Getting started](#getting-started)
   * [Bringing up the stack](#bringing-up-the-stack)
   * [Initial setup](#initial-setup)
3. [Configuration](#configuration)
   * [How can I tune the Kibana configuration?](#how-can-i-tune-the-kibana-configuration)
   * [How can I tune the Logstash configuration?](#how-can-i-tune-the-logstash-configuration)
   * [How can I tune the Elasticsearch configuration?](#how-can-i-tune-the-elasticsearch-configuration)
   * [How can I scale out the Elasticsearch cluster?](#how-can-i-scale-up-the-elasticsearch-cluster)
4. [Storage](#storage)
   * [How can I persist Elasticsearch data?](#how-can-i-persist-elasticsearch-data)
5. [Extensibility](#extensibility)
   * [How can I add plugins?](#how-can-i-add-plugins)
   * [How can I enable the provided extensions?](#how-can-i-enable-the-provided-extensions)
6. [JVM tuning](#jvm-tuning)
   * [How can I specify the amount of memory used by a service?](#how-can-i-specify-the-amount-of-memory-used-by-a-service)
   * [How can I enable a remote JMX connection to a service?](#how-can-i-enable-a-remote-jmx-connection-to-a-service)

## Requirements

### Host setup

1. Install [Docker](https://www.docker.com/community-edition#/download) version **1.10.0+**
2. Install [Docker Compose](https://docs.docker.com/compose/install/) version **1.6.0+**
Anthony Lapenna's avatar
Anthony Lapenna committed
53
3. Clone this repository
54

Daniel Macedo's avatar
Daniel Macedo committed
55
### SELinux
tony's avatar
tony committed
56

Daniel Macedo's avatar
Daniel Macedo committed
57
58
59
On distributions which have SELinux enabled out-of-the-box you will need to either re-context the files or set SELinux
into Permissive mode in order for docker-elk to start properly. For example on Redhat and CentOS, the following will
apply the proper context:
tony's avatar
tony committed
60

61
```console
62
$ chcon -R system_u:object_r:admin_home_t:s0 docker-elk/
Anthony Lapenna's avatar
Anthony Lapenna committed
63
64
```

65
66
67
68
### DockerForWindows

If you're using Docker for Windows, ensure the 'Shared Drives' feature is enabled for the C: drive (Docker for Windows > Settings > Shared Drives). [MSDN article detailing Shared Drives config](https://blogs.msdn.microsoft.com/stevelasker/2016/06/14/configuring-docker-for-windows-volumes/).

Daniel Macedo's avatar
Daniel Macedo committed
69
70
71
## Usage

### Bringing up the stack
tony's avatar
tony committed
72

73
74
**Note**: In case you switched branch or updated a base image - you may need to run `docker-compose build` first

Daniel Macedo's avatar
Daniel Macedo committed
75
Start the ELK stack using `docker-compose`:
tony's avatar
tony committed
76

77
```console
tony's avatar
tony committed
78
79
80
81
82
$ docker-compose up
```

You can also choose to run it in background (detached mode):

83
```console
tony's avatar
tony committed
84
85
86
$ docker-compose up -d
```

87
Give Kibana a few seconds to initialize, then access the Kibana web UI by hitting
Daniel Macedo's avatar
Daniel Macedo committed
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
[http://localhost:5601](http://localhost:5601) with a web browser.

By default, the stack exposes the following ports:
* 5000: Logstash TCP input.
* 9200: Elasticsearch HTTP
* 9300: Elasticsearch TCP transport
* 5601: Kibana

**WARNING**: If you're using `boot2docker`, you must access it via the `boot2docker` IP address instead of `localhost`.

**WARNING**: If you're using *Docker Toolbox*, you must access it via the `docker-machine` IP address instead of
`localhost`.

Now that the stack is running, you will want to inject some log entries. The shipped Logstash configuration allows you
to send content via TCP:
tony's avatar
tony committed
103

104
```console
tony's avatar
tony committed
105
106
107
$ nc localhost 5000 < /path/to/logfile.log
```

Daniel Macedo's avatar
Daniel Macedo committed
108
## Initial setup
109

Daniel Macedo's avatar
Daniel Macedo committed
110
### Default Kibana index pattern creation
111

Daniel Macedo's avatar
Daniel Macedo committed
112
When Kibana launches for the first time, it is not configured with any index pattern.
113

Daniel Macedo's avatar
Daniel Macedo committed
114
115
116
117
118
119
120
121
#### Via the Kibana web UI

**NOTE**: You need to inject data into Logstash before being able to configure a Logstash index pattern via the Kibana web
UI. Then all you have to do is hit the *Create* button.

Refer to [Connect Kibana with
Elasticsearch](https://www.elastic.co/guide/en/kibana/current/connect-to-elasticsearch.html) for detailed instructions
about the index pattern configuration.
tony's avatar
tony committed
122

Daniel Macedo's avatar
Daniel Macedo committed
123
#### On the command line
Jih-Chi Lee's avatar
Jih-Chi Lee committed
124

125
Create an index pattern via the Kibana API:
tony's avatar
tony committed
126

127
```console
128
$ curl -XPOST -D- 'http://localhost:5601/api/saved_objects/index-pattern' \
Daniel Macedo's avatar
Daniel Macedo committed
129
    -H 'Content-Type: application/json' \
Mario Kozjak's avatar
Mario Kozjak committed
130
    -H 'kbn-version: 6.2.3' \
131
    -d '{"attributes":{"title":"logstash-*","timeFieldName":"@timestamp"}}'
Daniel Macedo's avatar
Daniel Macedo committed
132
133
```

134
The created pattern will automatically be marked as the default index pattern as soon as the Kibana UI is opened for the first time.
Daniel Macedo's avatar
Daniel Macedo committed
135
136

## Configuration
tony's avatar
tony committed
137

Daniel Macedo's avatar
Daniel Macedo committed
138
139
**NOTE**: Configuration is not dynamically reloaded, you will need to restart the stack after any change in the
configuration of a component.
tony's avatar
tony committed
140

Daniel Macedo's avatar
Daniel Macedo committed
141
### How can I tune the Kibana configuration?
tony's avatar
tony committed
142

tony's avatar
tony committed
143
The Kibana default configuration is stored in `kibana/config/kibana.yml`.
tony's avatar
tony committed
144

Antoine Cotten's avatar
Antoine Cotten committed
145
It is also possible to map the entire `config` directory instead of a single file.
tony's avatar
tony committed
146

Daniel Macedo's avatar
Daniel Macedo committed
147
### How can I tune the Logstash configuration?
tony's avatar
tony committed
148

Antoine Cotten's avatar
Antoine Cotten committed
149
The Logstash configuration is stored in `logstash/config/logstash.yml`.
Antoine Cotten's avatar
Antoine Cotten committed
150

Daniel Macedo's avatar
Daniel Macedo committed
151
152
153
154
It is also possible to map the entire `config` directory instead of a single file, however you must be aware that
Logstash will be expecting a
[`log4j2.properties`](https://github.com/elastic/logstash-docker/tree/master/build/logstash/config) file for its own
logging.
155

Daniel Macedo's avatar
Daniel Macedo committed
156
### How can I tune the Elasticsearch configuration?
tony's avatar
tony committed
157

158
The Elasticsearch configuration is stored in `elasticsearch/config/elasticsearch.yml`.
tony's avatar
tony committed
159

Antoine Cotten's avatar
Antoine Cotten committed
160
You can also specify the options you want to override directly via environment variables:
161
162
163

```yml
elasticsearch:
Daniel Macedo's avatar
Daniel Macedo committed
164

165
  environment:
Antoine Cotten's avatar
Antoine Cotten committed
166
167
    network.host: "_non_loopback_"
    cluster.name: "my-cluster"
168
169
```

Daniel Macedo's avatar
Daniel Macedo committed
170
### How can I scale out the Elasticsearch cluster?
171

Daniel Macedo's avatar
Daniel Macedo committed
172
173
Follow the instructions from the Wiki: [Scaling out
Elasticsearch](https://github.com/deviantony/docker-elk/wiki/Elasticsearch-cluster)
174

Daniel Macedo's avatar
Daniel Macedo committed
175
## Storage
tony's avatar
tony committed
176

Daniel Macedo's avatar
Daniel Macedo committed
177
### How can I persist Elasticsearch data?
tony's avatar
tony committed
178

179
180
The data stored in Elasticsearch will be persisted after container reboot but not after container removal.

Daniel Macedo's avatar
Daniel Macedo committed
181
182
In order to persist Elasticsearch data even after removing the Elasticsearch container, you'll have to mount a volume on
your Docker host. Update the `elasticsearch` service declaration to:
tony's avatar
tony committed
183
184
185

```yml
elasticsearch:
Daniel Macedo's avatar
Daniel Macedo committed
186

tony's avatar
tony committed
187
188
189
  volumes:
    - /path/to/storage:/usr/share/elasticsearch/data
```
tony's avatar
tony committed
190

191
192
This will store Elasticsearch data inside `/path/to/storage`.

193
194
195
196
197
198
199
200
201
**NOTE:** beware of these OS-specific considerations:
* **Linux:** the [unprivileged `elasticsearch` user][esuser] is used within the Elasticsearch image, therefore the
  mounted data directory must be owned by the uid `1000`.
* **macOS:** the default Docker for Mac configuration allows mounting files from `/Users/`, `/Volumes/`, `/private/`,
  and `/tmp` exclusively. Follow the instructions from the [documentation][macmounts] to add more locations.

[esuser]: https://github.com/elastic/elasticsearch-docker/blob/016bcc9db1dd97ecd0ff60c1290e7fa9142f8ddd/templates/Dockerfile.j2#L22
[macmounts]: https://docs.docker.com/docker-for-mac/osxfs/

Daniel Macedo's avatar
Daniel Macedo committed
202
## Extensibility
203

Daniel Macedo's avatar
Daniel Macedo committed
204
### How can I add plugins?
205
206
207
208
209

To add plugins to any ELK component you have to:

1. Add a `RUN` statement to the corresponding `Dockerfile` (eg. `RUN logstash-plugin install logstash-filter-json`)
2. Add the associated plugin code configuration to the service configuration (eg. Logstash input/output)
210
3. Rebuild the images using the `docker-compose build` command
211

Daniel Macedo's avatar
Daniel Macedo committed
212
213
214
215
216
217
218
219
220
### How can I enable the provided extensions?

A few extensions are available inside the [`extensions`](extensions) directory. These extensions provide features which
are not part of the standard Elastic stack, but can be used to enrich it with extra integrations.

The documentation for these extensions is provided inside each individual subdirectory, on a per-extension basis. Some
of them require manual changes to the default ELK configuration.

## JVM tuning
221

Daniel Macedo's avatar
Daniel Macedo committed
222
### How can I specify the amount of memory used by a service?
223

Daniel Macedo's avatar
Daniel Macedo committed
224
225
226
By default, both Elasticsearch and Logstash start with [1/4 of the total host
memory](https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/parallel.html#default_heap_size) allocated to
the JVM Heap Size.
227

Daniel Macedo's avatar
Daniel Macedo committed
228
229
The startup scripts for Elasticsearch and Logstash can append extra JVM options from the value of an environment
variable, allowing the user to adjust the amount of memory that can be used by each component:
230
231
232
233
234
235

| Service       | Environment variable |
|---------------|----------------------|
| Elasticsearch | ES_JAVA_OPTS         |
| Logstash      | LS_JAVA_OPTS         |

Daniel Macedo's avatar
Daniel Macedo committed
236
237
238
To accomodate environments where memory is scarce (Docker for Mac has only 2 GB available by default), the Heap Size
allocation is capped by default to 256MB per service in the `docker-compose.yml` file. If you want to override the
default JVM configuration, edit the matching environment variable(s) in the `docker-compose.yml` file.
239
240
241
242
243

For example, to increase the maximum JVM Heap Size for Logstash:

```yml
logstash:
Daniel Macedo's avatar
Daniel Macedo committed
244

245
246
247
248
  environment:
    LS_JAVA_OPTS: "-Xmx1g -Xms1g"
```

Daniel Macedo's avatar
Daniel Macedo committed
249
### How can I enable a remote JMX connection to a service?
250

Daniel Macedo's avatar
Daniel Macedo committed
251
252
As for the Java Heap memory (see above), you can specify JVM options to enable JMX and map the JMX port on the docker
host.
253

Daniel Macedo's avatar
Daniel Macedo committed
254
255
256
Update the `{ES,LS}_JAVA_OPTS` environment variable with the following content (I've mapped the JMX service on the port
18080, you can change that). Do not forget to update the `-Djava.rmi.server.hostname` option with the IP address of your
Docker host (replace **DOCKER_HOST_IP**):
257
258
259

```yml
logstash:
Daniel Macedo's avatar
Daniel Macedo committed
260

261
262
263
  environment:
    LS_JAVA_OPTS: "-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=18080 -Dcom.sun.management.jmxremote.rmi.port=18080 -Djava.rmi.server.hostname=DOCKER_HOST_IP -Dcom.sun.management.jmxremote.local.only=false"
```