Pkgsrc changes:
* Add patches to work around runtime.support_avx and runtime.support_avx2
being undefined when building with go 1.11.
* Add patch to cludge around build issue by commenting out BindToInterface
invocations, which is only implemented for Linux and Darwin.
Upstream changes:
6.6.2:
======
Auditbeat
* System module: Fix and unify bucket closing logic. 10897
Filebeat
* Fix a bug when converting NetFlow fields to snake_case. 10950
Metricbeat
* Fix issue in kubernetes module preventing usage percentages to
be properly calculated. 10946
Packetbeat
* Avoid reporting unknown MongoDB opcodes more than once. 10878
Winlogbeat
* Prevent Winlogbeat from dropping events with invalid XML. 11006
6.6.1:
======
Breaking changes
Affecting all Beats
* Fix stopping of modules started by kubernetes autodiscover. 10476
Auditbeat
* Enable System module config on Windows. 10237
Filebeat
* Fix bad bytes count in docker input when filtering by stream. 10211
* Add convert_timezone option to Logstash module to convert dates
to UTC. 9756 9797
* Add convert_timezone option to Elasticsearch module to convert
dates to UTC. 9756 9761
* Make elasticsearch/audit fileset be more lenient in parsing node
name. 10035 10135
Journalbeat
* Fix fields.yml indentation of audit group which had the effect
of creating an incomplete Elasticsearch index template. 10556
Metricbeat
* Fix issue with elasticsearch/node_stats metricset (x-pack) not
indexing source_node field. 10639
Packetbeat
* Fixed a crash when using af_packet capture 10477
Functionbeat
* Ensure that functionbeat is logging at info level not debug. 10262
Added
Filebeat
* Teach elasticsearch/audit fileset to parse out some more fields.
10134 10137
Journalbeat
* Migrate registry from previously incorrect path. 10486
6.6.0:
======
Breaking changes
Affecting all Beats
* Dissect syntax change, use * instead of ? when working with field
reference. 8054
Filebeat
* Allow beats to blacklist certain part of the configuration while
using Central Management. 9099
Metricbeat
* Allow beats to blacklist certain part of the configuration while
using Central Management. 9099
Functionbeat
* The CLI will now log CloudFormation Stack events. 8912
* Correctly normalize Cloudformation resource name. 10087
Bugfixes
Affecting all Beats
* Fix autodiscover configurations stopping when metadata is missing. 8851
* Refresh host metadata in add_host_metadata. 9359
* When collecting swap metrics for beats telemetry or system
metricbeat module handle cases of free swap being bigger than
total swap by assuming no swap is being used. 6271 9383
* Ignore non index fields in default_field for Elasticsearch. 9549
* Update Golang to 1.10.6. 9563
* Update Kibana index pattern attributes for objects that are disabled. 9644
* Enforce validation for the Central Management access token. 9621
* Fix registry handle leak on Windows
(https://github.com/elastic/go-sysinfo/pull/33). 9920
* Gracefully handle TLS options when enrolling a Beat. 9129
* Allow to unenroll a Beat from the UI. 9452
* The backing off now implements jitter to better distribute the load. 10172
* Fix config appender registration. 9873
* Fix TLS certificate DoS vulnerability. 10304
Filebeat
* Fix improperly set config for CRI Flag in Docker Input 8899
* Just enabling the elasticsearch fileset and starting Filebeat
no longer causes an error. 8891
* Support IPv6 addresses with zone id in IIS ingest pipeline. 9836
error log: 9869 access log: 10030
* Support haproxy log lines without captured headers. 9463 9958
Heartbeat
* Heartbeat now always downloads the entire body of HTTP endpoints,
even if no checks against the body content are declared. This
fixes an issue where timing metrics would be incorrect in
scenarios where the body wasn't used since the connection would
be closed soon after the headers were sent, but before the entire
body was. 8894
Metricbeat
* Add missing namespace field in http server metricset 7890
* Fix issue with not collecting Elasticsearch cross-cluster
replication stats correctly. 9179
* The node.name field in the elasticsearch/node metricset now
correctly reports the Elasticsarch node name. Previously this
field was incorrectly reporting the node ID instead. 9209
* Fix panics in vsphere module when certain values where not
returned by the API. 9784
* Fix pod UID metadata enrichment in Kubernetes module. 10081
Packetbeat
* Fix issue with process monitor associating traffic to the wrong
process. 9151 9443
* Fix DHCPv4 dashboard that wouldn't load in Kibana. 9850
Added
Affecting all Beats
* Unify dashboard exporter tools. 9097
* Dissect will now flag event on parsing error. 8751
* Added the redirect_stderr option that allows panics to be logged
to log files. 8430
* Add cache.ttl to add_host_metadata. 9359
* Add support for index lifecycle management (beta). 7963
* Always include Pod UID as part of Pod metadata. {pull]9517[9517]
* Release Jolokia autodiscover as GA. 9706
Auditbeat
* Add system module. 9546
Filebeat
* Added detect_null_bytes selector to detect null bytes from a io.reader. 9210
* Added syslog_host variable to HAProxy module to allow syslog
listener to bind to configured host. 9366
* Allow to force CRI format parsing for better performance 8424
* Add event.dataset to module events. 9457
* Add field log.source.address and log.file.path to replace source. 9435
* Add support for multi-core thread_id in postgresql module 9156 9482
* Added netflow input type that supports NetFlow v1, v5, v6, v7,
v8, v9 and IPFIX. 9399
Journalbeat
* Add the ability to check against JSON HTTP bodies with conditions. 8667
* Add cursor_seek_fallback option. 9234
Metricbeat
* Collect custom cluster display_name in elasticsearch/cluster_stats
metricset. 8445
* Test etcd module with etcd 3.3. 9068
* All elasticsearch metricsets now have module-level cluster.id
and cluster.name fields. 8770 8771 9164 9165 9166 9168
* All elasticsearch node-level metricsets now have node.id and
node.name fields. 9168 9209
* Add settings to disable docker and cgroup cpu metrics per core.
9187 9194 9589
* The elasticsearch/node metricset now reports the Elasticsearch
cluster UUID. 8771
* Support GET requests in Jolokia module. 8566 9226
* Add freebsd support for the uptime metricset. 9413
* Add host.os.name field to add_host_metadata processor. 8948 9405
* Add field event.dataset which is {module}.{metricset). 9393
Deprecated
Filebeat
* Deprecate field source. Will be replaced by log.source.address
and log.file.path in 7.0. 9435
Metricbeat
* Deprecate field metricset.rtt. Replaced by event.duration which
is in nano instead of micro seconds. 9393
Packetbeat
* Support new TLS version negotiation introduced in TLS 1.3. 8647.
6.5.4:
======
Bugfixes
Affecting all Beats
* Update Golang to 1.10.6. This fixes an issue in remote certificate
validation CVE-2018-16875. 9563
Filebeat
* Fix saved objects in filebeat haproxy dashboard. 9417
* Fixed a memory leak when harvesters are closed. 7820
Added
Filebeat
* Added support on Traefik for Common Log Format and Combined Log
Format mixed which is the default Traefik format 8015 6111 8768.
6.5.3:
======
Bugfixes
Affecting all Beats
* Log events at the debug level when dropped by encoding problems. 9251
Filebeat
* Correctly parse December or Dec in the Syslog input. 9349
* Don't generate incomplete configurations when logs collection
is disabled by hints. 9305
* Stop runners disabled by hints after previously being started. 9305
* Fix installation of haproxy dashboard. 9307 9313
6.5.2:
======
Bugfixes
Affecting all Beats
* Propagate Sync error when running SafeFileRotate. 9069
Metricbeat
* Fix panic on docker healthcheck collection on dockers without
healthchecks. 9171
* Fix issue preventing diskio metrics collection for idle disks.
9124 9125
6.5.1:
======
Bugfixes
Affecting all Beats
* Fix windows binaries not having an enroll command. 9096 8836
Journalbeat
* Fix journalbeat sometimes hanging if output is unavailable. 9106
Metricbeat
* Fix race condition when enriching events with kubernetes metadata. 9055 9067
Added
Journalbeat
* Add minimal kibana dashboard. 9106
6.5.0:
======
Bugfixes
Affecting all Beats
* Fixed add_host_metadata not initializing correctly on Windows.
7715
* Fixed missing file unlock in spool file on Windows, so file can
be reopened and locked. 7859
* Fix spool file opening/creation failing due to file locking on
Windows. 7859
* Fix size of maximum mmaped read area in spool file on Windows.
7859
* Fix potential data loss on OS X in spool file by using fcntl
with F_FULLFSYNC. 7859
* Improve fsync on linux, by assuming the kernel resets error
flags of failed writes. 7859
* Remove unix-like permission checks on Windows, so files can be
opened. 7849
* Replace index patterns in TSVB visualizations. 7929
* Deregister pipeline loader callback when inputsRunner is stopped.
7893[7893]
* Add backoff support to x-pack monitoring outputs. 7966
* Removed execute permissions systemd unit file. 7873
* Fix a race condition with the add_host_metadata and the event
serialization. 8223 8653
* Enforce that data used by k8s or docker doesn't use any reference. 8240
* Switch to different UUID lib due to to non-random generated UUIDs. 8485
* Fix race condition when publishing monitoring data. 8646
* Fix bug in loading dashboards from zip file. 8051
* Fix in-cluster kubernetes configuration on IPv6. 8754
* The export config subcommand should not display real value for
field reference. 8769
* The setup command will not fail if no dashboard is available to
import. 8977
* Fix central management configurations reload when a configuration
is removed in Kibana. 9010
Auditbeat
* Fixed a crash in the file_integrity module under Linux. 7753
* Fixed the RPM by designating the config file as configuration
data in the RPM spec. 8075
* Fixed a concurrent map write panic in the auditd module. 8158
* Fixed a data race in the file_integrity module. 8009
* Fixed a deadlock in the file_integrity module. 8027
Filebeat
* Fix date format in Mongodb Ingest pipeline. 7974
* Fixed a docker input error due to the offset update bug in
partial log join.8177
* Update CRI format to support partial/full tags. 8265
* Fix some errors happening when stopping syslog input. 8347
* Fix RFC3339 timezone and nanoseconds parsing with the syslog input. 8346
* Mark the TCP and UDP input as GA. 8125
* Support multiline logs in logstash/log fileset of Filebeat. 8562
Heartbeat
* Fixed bug where HTTP responses with larger bodies would
incorrectly report connection errors. 8660
Metricbeat
* Fix golang.heap.gc.cpu_fraction type from long to float in Golang
module. 7789
* Fixed the RPM by designating the modules.d config files as
configuration data in the RPM spec. 8075
* Fixed the location of the modules.d dir in Deb and RPM packages. 8104
* Add docker diskio stats on Windows. 6815 8126
* Fix incorrect type conversion of average response time in Haproxy
dashboards 8404
* Added io disk read and write times to system module 8473 8508
* Avoid mapping issues in kubernetes module. 8487
* Recover metrics for old apache versions removed by mistake on #6450. 7871
* Fix dropwizard module parsing of metric names. 8365 8385
* Fix issue that would prevent kafka module to find a proper broker
when port is not set 8613
* Fix range colors in multiple visualizations. 8633 8634
* Fix incorrect header parsing on http metricbeat module 8564 8585
* Fixed a panic when the kvm module cannot establish a connection
to libvirtd. 7792.
* Add missing namespace field in http server metricset 7890
Packetbeat
* Fixed a seccomp related error where the fcntl64 syscall was
not permitted on 32-bit Linux and the sniffer failed to start.
7839
* Added missing cmdline and client_cmdline fields to index template. 8258
Added
Affecting all Beats
* Added time-based log rotation. 8349
* Add backoff on error support to redis output. 7781
* Allow for cloud-id to specify a custom port. This makes cloud-id
work in ECE contexts. 7887
* Add support to grow or shrink an existing spool file between
restarts. 7859
* Make kubernetes autodiscover ignore events with empty container
IDs 7971
* Implement CheckConfig in RunnerFactory to make autodiscover
check configs 7961
* Add DNS processor with support for performing reverse lookups
on IP addresses. 7770
* Support for Kafka 2.0.0 in kafka output 8399
* Add setting setup.kibana.space.id to support Kibana Spaces 7942
* Better tracking of number of open file descriptors. 7986
* Report number of open file handles on Windows. 8329
* Added the add_process_metadata processor to enrich events with
process information. 6789
* Add Beats Central Management 8559
* Report configured queue type. 8091
* Enable host and cloud metadata processors by default. 8596
Filebeat
* Add tag "truncated" to "log.flags" if incoming line is longer
than configured limit. 7991
* Add haproxy module. 8014
* Add tag "multiline" to "log.flags" if event consists of multiple
lines. 7997
* Release docker input as GA. 8328
* Keep unparsed user agent information in user_agent.original. 7832
* Added default and TCP parsing formats to HAproxy module 8311 8637
* Add Suricata IDS/IDP/NSM module. 8153 8693
* Support for Kafka 2.0.0 8853
Heartbeat
* Heartbeat is marked as GA.
* Add automatic config file reloading. 8023
* Added autodiscovery support 8415
* Added support for extra TLS/x509 metadata. 7944
* Added stats and state metrics for number of monitors and endpoints
started. 8621
Journalbeat
* Add journalbeat. 8703
Metricbeat
* Add replstatus metricset to MongoDB module 7604
* Add experimental socket summary metricset to system module 6782
* Move common kafka fields (broker, topic and partition.id) to
the module level to facilitate events correlation 7767
* Add fields for memory fragmentation, memory allocator stats,
copy on write, master-slave status, and active defragmentation
to info metricset of Redis module. 7695
* Increase ignore_above for system.process.cmdline to 2048. 8100
* Add support to renamed fields planned for redis 5.0. 8167
* Allow TCP helper to support delimiters and graphite module to
accept multiple metrics in a single payload. 8278
* Added died PID state to process_system metricset on system module 8275
* Add metrics metricset to MongoDB module. 7611
* Added ccr metricset to Elasticsearch module. 8335
* Support for Kafka 2.0.0 8399
* Added support for query params in configuration 8286 8292
* Add container image for docker metricsets. 8214 8438
* Precalculate composed id fields for kafka dashboards. 8504
* Add support for full status page output for php-fpm module as
a separate metricset called process. 8394
* Add Kafka dashboard. 8457
* Release Kafka module as GA. 8854
Packetbeat
* Added DHCP protocol support. 7647
Functionbeat
* Initial version of Functionbeat. 8678
Deprecated
Heartbeat
* watch.poll_file is now deprecated and superceded by automatic
config file reloading.
Metricbeat
* Redis info replication.master_offset has been deprecated in
favor of replication.master.offset.7695
* Redis info clients fields longest_output_list and biggest_input_buf
have been renamed to max_output_buffer and max_input_buffer
based on the names they will have in Redis 5.0, both fields will
coexist during a time with the same value 8167.
* Move common kafka fields (broker, topic and partition.id) to
the module level 7767.
6.4.3:
======
Bugfixes
Affecting all Beats
* Fix a race condition with the add_host_metadata and the event
serialization. 8223 8653
* Fix race condition when publishing monitoring data. 8646
* Fix bug in loading dashboards from zip file. 8051
* The export config subcommand should not display real value for
field reference. 8769
Filebeat
* Fix typo in Filebeat IIS Kibana visualization. 8604
Metricbeat
* Recover metrics for old Apache versions removed by mistake on
#6450. 7871
* Avoid mapping issues in Kubernetes module. 8487
* Fixed a panic when the KVM module cannot establish a connection
to libvirtd. 7792
6.4.2:
======
Bugfixes
Filebeat
* Fix some errors happening when stopping syslog input. 8347
* Fix RFC3339 timezone and nanoseconds parsing with the syslog input. 8346
Metricbeat
* Fix incorrect type conversion of average response time in Haproxy
dashboards 8404
* Fix dropwizard module parsing of metric names. 8365 8385
6.4.1:
======
Bugfixes
Affecting all Beats
* Add backoff support to x-pack monitoring outputs. 7966
* Removed execute permissions systemd unit file. 7873
* Fix a race condition with the add_host_metadata and the event
serialization. 8223
* Enforce that data used by k8s or docker doesn't use any reference.
8240
* Implement CheckConfig in RunnerFactory to make autodiscover
check configs 7961
* Make kubernetes autodiscover ignore events with empty container
IDs 7971
Auditbeat
* Fixed a concurrent map write panic in the auditd module. 8158
* Fixed the RPM by designating the config file as configuration
data in the RPM spec. 8075
Filebeat
* Fixed a docker input error due to the offset update bug in
partial log join.8177
* Update CRI format to support partial/full tags. 8265
Metricbeat
* Fixed the location of the modules.d dir in Deb and RPM packages.
8104
* Fixed the RPM by designating the modules.d config files as
configuration data in the RPM spec. 8075
* Fix golang.heap.gc.cpu_fraction type from long to float in Golang
module. 7789
Packetbeat
* Added missing cmdline and client_cmdline fields to index template. 8258
6.4.0:
======
Known issue
Due to a packaging mistake, the modules.d configuration directory
is installed in the wrong path in the Metricbeat DEB and RPM
packages. This issue results in an empty list when you run metricbeat
modules list and failures when you try to enable or disable modules.
To work around this issue, run the following command:
sudo cp -r /usr/share/metricbeat/modules.d /etc/metricbeat/
This issue affects all new installations on DEB and RPM. Upgrades
will run, but use old configurations defined in the modules.d
directory from the previous installation.
The issue will be fixed in the 6.4.1 release.
Breaking changes
Affecting all Beats
* Set default kafka version to 1.0.0 in kafka output. Older versions
are still supported by configuring the version setting. Minimally
supported version is 0.11 (older versions might work, but are
untested). 7025
Heartbeat
* Rename http.response.status to http.response.status_code to
align with ECS. 7274
* Remove type field as not needed. 7307
Metricbeat
* Fixed typo in values for state_container status.phase, from
terminate to terminated. 6916
* RabbitMQ management plugin path is now configured at the module
level instead of having to do it in each of the metricsets. New
management_path_prefix option should be used now 7074
* RabbitMQ node metricset only collects metrics of the instance
it connects to, node.collect: cluster can be used to collect
all nodes as before. 6556 6971
* Change http/server metricset to put events by default under
http.server and prefix config options with server.. 7100
* Disable dedotting in docker module configuration. This will
change the out-of-the-box behaviour, but not the one of already
configured instances. 7485
* Fix typo in etcd/self metricset fields from *.bandwithrate to
*.bandwidthrate. 7456
* Changed the definition of the system.cpu.total.pct and
system.cpu.total.norm.cou fields to exclude the IOWait time.
7691
Bugfixes
Affecting all Beats
* Error out on invalid Autodiscover template conditions settings. 7200
* Allow to override the ignore_above option when defining new
field with the type keyword. 7238
* Fix a panic on the Dissect processor when we have data remaining
after the last delimiter. 7449
* When we fail to build a Kubernetes' indexer or matcher we produce
a warning but we don't add them to the execution. 7466
* Fix default value for logging.files.keepfiles. It was being set
to 0 and now it's set to the documented value of 7. 7494
* Retain compatibility with older Docker server versions. 7542
* Fix errors unpacking configs modified via CLI by ignoring -E
key=value pairs with missing value. 7599
Auditbeat
* Allow auditbeat setup to run without requiring elevated privileges
for the audit client. 7111
* Fix goroutine leak that occurred when the auditd module was stopped. 7163
Filebeat
* Fix a data race between stopping and starting of the harvesters. 6879
* Fix an issue when parsing ISO8601 dates with timezone definition 7367
* Fix Grok pattern of MongoDB module. 7568
* Fix registry duplicates and log resending on upgrade. 7634
Metricbeat
* Fix Windows service metricset when using a 32-bit binary on a
64-bit OS. 7294
* Do not report Metricbeat container host as hostname in Kubernetes
deployment. 7199
* Ensure metadata updates don't replace existing pod metrics. 7573
* Fix kubernetes pct fields reporting. 7677
* Add support for new kube_node_status_condition in Kubernetes
state_node. 7699
Added
Affecting all Beats
* Add dissect processor. 6925
* Add IP-addresses and MAC-addresses to add_host_metadata. 6878
* Added a seccomp (secure computing) filter on Linux that whitelists
the necessary system calls used by each Beat. 5213
* Ship fields.yml as part of the binary 4834
* Added options to dev-tools/cmd/dashboards/export_dashboard.go:
-indexPattern to include index-pattern in output, -quiet to be
quiet. 7101
* Add Indexer indexing by pod uid. Enable pod uid metadata gathering
in add_kubernetes_metadata. Extended Matcher log_path matching
to support volume mounts 7072
* Add default_fields to Elasticsearch template when connecting to
Elasticsearch >= 7.0. 7015
* Add support for loading a template.json file directly instead
of using fields.yml. 7039
* Add support for keyword multifields in field.yml. 7131
* Add experimental Jolokia Discovery autodiscover provider. 7141
* Add owner object info to Kubernetes metadata. 7231
* Add Beat export dashboard command. 7239
* Add support for docker autodiscover to monitor containers on
host network 6708
* Add ability to define input configuration as stringified JSON
for autodiscover. 7372
* Add processor definition support for hints builder 7386
* Add support to disable html escaping in outputs. 7445
* Refactor error handing in schema.Apply(). 7335
* Add additional types to Kubernetes metadata 7457
* Add module state reporting for Beats Monitoring. 7075
* Release the rename processor as GA. 7656
* Add support for Openstack Nova in add_cloud_metadata processor.
7663
Auditbeat
* Added XXH64 hash option for file integrity checks. 7311
* Added the show auditd-rules and show auditd-status commands to
show kernel rules and status. 7114
* Add Kubernetes specs for auditbeat file integrity monitoring 7642
Filebeat
* Add Kibana module with log fileset. 7052
* Support MySQL 5.7.19 by mysql/slowlog 6969
* Correctly join partial log lines when using docker input. 6967
* Add support for TLS with client authentication to the TCP input 7056
* Converted part of pipeline from treafik/access metricSet to
dissect to improve efficiency. 7209
* Add GC fileset to the Elasticsearch module. 7305
* Add Audit log fileset to the Elasticsearch module. 7365
* Add Slow log fileset to the Elasticsearch module. 7473
* Add deprecation fileset to the Elasticsearch module. 7474
* Add convert_timezone option to Kafka module to convert dates to
UTC. 7546 7578
* Add patterns for kafka 1.1 logs. 7608
* Move debug messages in tcp input source 7712
Metricbeat
* Add experimental Elasticsearch index metricset. 6881
* Add dashboards and visualizations for haproxy metrics. 6934
* Add Jolokia agent in proxy mode. 6475
* Add message rates to the RabbitMQ queue metricset 6442 6606
* Add exchanges metricset to the RabbitMQ module 6442 6607
* Add Elasticsearch index_summary metricset. 6918
* Add shard metricset to Elasticsearch module. 7006
* Add apiserver metricset to Kubernetes module. 7059
* Add maxmemory to redis info metricset. 7127
* Set guest as default user in RabbitMQ module. 7107
* Add postgresql statement metricset. 7048 7060
* Update state_container metricset to support latest kube-state-metrics
version. 7216
* Add TLS support to MongoDB module. 7401
* Added Traefik module with health metricset. 7413
* Add Elasticsearch ml_job metricsets. 7196
* Add support for bearer token files to HTTP helper. 7527
* Add Elasticsearch index recovery metricset. 7225
* Add locks, global_locks, oplatencies and process fields to status
metricset of MongoDB module. 7613
* Run Kafka integration tests on version 1.1.0 7616
* Release raid and socket metricset from system module as GA. 7658
* Release elasticsearch module and all its metricsets as beta. 7662
* Release munin and traefik module as beta. 7660
* Add envoyproxy module. 7569
* Release prometheus collector metricset as GA. 7660
* Add Elasticsearch cluster_stats metricset. 7638
* Added basepath setting for HTTP-based metricsets 7700
Packetbeat
* The process monitor now reports the command-line for all processes,
under Linux and Windows. 7135
* Updated the TLS protocol parser with new cipher suites added to
TLS 1.3. 7455
* Flows are enriched with process information using the process
monitor. 7507
* Added UDP support to process monitor. 7571
Deprecated
Metricbeat
* Kubernetes state_container cpu.limit.nanocores and cpu.request.nanocores
have been deprecated in favor of cpu.*.cores. 6916
6.3.2:
======
Bugfixes
Affecting all Beats
* Fix registry duplicates and log resending on upgrade. 7634
* Fix default value for logging.files.keepfiles. It was being set
to 0 and now it's set to the documented value of 7. 7494
* Retain compatibility with older Docker server versions. 7542
Metricbeat
* Fix missing hosts config option in Ceph module. 7596
* Ensure metadata updates don't replace existing pod metrics. 7573
Added
Metricbeat
* Add support for bearer token files to HTTP helper. 7527
Packetbeat
* Updated the TLS protocol parser with new cipher suites added
to TLS 1.3. 7455
6.3.1:
======
Bugfixes
Affecting all Beats
* Allow index-pattern only setup when setup.dashboards.only_index=true.
7285
* Preserve the event when source matching fails in add_docker_metadata.
7133
* Negotiate Docker API version from our client instead of using
a hardcoded one. 7165
* Fix duplicating dynamic_fields in template when overwriting the
template. 7352
Auditbeat
* Fixed parsing of AppArmor audit messages. 6978
Filebeat
* Comply with PostgreSQL database name format 7198
* Optimize PostgreSQL ingest pipeline to use anchored regexp and
merge multiple regexp into a single expression. 7269
* Keep different registry entry per container stream to avoid
wrong offsets. 7281
* Fix offset field pointing at end of a line. 6514
* Commit registry writes to stable storage to avoid corrupt registry
files. 6792
Metricbeat
* Fix field mapping for the system process CPU ticks fields. 7230
* Ensure canonical naming for JMX beans is disabled in Jolokia module. 7047
* Fix Jolokia attribute mapping when using wildcards and MBean
names with multiple properties. 7321
Packetbeat
* Fix an out of bounds access in HTTP parser caused by malformed
request. 6997
* Fix missing type for http.response.body field. 7169
Added
Auditbeat
* Added caching of UID and GID values to auditd module. 6978
* Updated syscall tables for Linux 4.16. 6978
* Added better error messages for when the auditd module fails
due to the Linux kernel not supporting auditing (CONFIG_AUDIT=n).
7012
Metricbeat
* Collect accumulated docker network metrics and mark old ones
as deprecated. 7253
6.3.0:
======
Breaking changes
Affecting all Beats
* De dot keys of labels and annotations in kubernetes meta processors
to prevent collisions. 6203
* Rename beat.cpu.*.time metrics to beat.cpu.*.time.ms. 6449
* Add host.name field to all events, to avoid mapping conflicts.
This could be breaking Logstash configs if you rely on the host
field being a string. 7051
Filebeat
* Add validation for Stdin, when Filebeat is configured with Stdin
and any other inputs, Filebeat will now refuse to start. 6463
* Mark system.syslog.message and system.auth.message as text
instead of keyword. 6589
Metricbeat
* De dot keys in kubernetes/event metricset to prevent collisions.
6203
* Add config option for windows/perfmon metricset to ignore non
existent counters. 6432
* Refactor docker CPU calculations to be more consistent with
docker stats. 6608
* Update logstash.node_stats metricset to write data under
logstash.node.stats.*. 6714
Bugfixes
Affecting all Beats
* Fix panic when Events containing a float32 value are normalized. 6129
* Fix setup.dashboards.always_kibana when using Kibana 5.6. 6090
* Fix for Kafka logger. 6430
* Remove double slashes in Windows service script. 6491
* Ensure Kubernetes labels/annotations don't break mapping 6490
* Ensure that the dashboard zip files can't contain files outside
of the kibana directory. 6921
* Fix map overwrite panics by cloning shared structs before doing
the update. 6947
* Fix delays on autodiscovery events handling caused by blocking
runner stops. 7170
* Do not emit Kubernetes autodiscover events for Pods without IP
address. 7235
* Fix self metrics when containerized 6641
Auditbeat
* Add hex decoding for the name field in audit path records. 6687
* Fixed a deadlock in the file_integrity module under Windows. 6864
Filebeat
* Fix panic when log prospector configuration fails to load. 6800
* Fix memory leak in log prospector when files cannot be read. 6797
* Add raw JSON to message field when JSON parsing fails. 6516
Commit registry writes to stable storage to avoid corrupt registry
files. 6877
* Fix a parsing issue in the syslog input for RFC3339 timestamp
and time with nanoseconds. 7046
* Fix an issue with an overflowing wait group when using the TCP
input. 7202
* Fix an issue when parsing ISO8601 dates with timezone definition 7367
Heartbeat
* Fix race due to updates of shared a map, that was not supposed
to be shared between multiple go-routines. 6616
Metricbeat
* Fix the default configuration for Logstash to include the default
port. 6279
* Fix dealing with new process status codes in Linux kernel 4.14+. 6306
* Add filtering option by exact device names in system.diskio.
diskio.include_devices. 6085
* Add connections metricset to RabbitMQ module 6548
* Fix panic in http dependent modules when invalid config was used. 6205
* Fix system.filesystem.used.pct value to match what df reports. 5494
* Fix namespace disambiguation in Kubernetes state_* metricsets. 6281
* Fix Windows perfmon metricset so that it sends metrics when an
error occurs. 6542
* Exclude bind mounts in fsstat and filesystem metricsets. 6819
* Don't stop Metricbeat if aerospike server is down. 6874
* Disk reads and write count metrics in RabbitMQ queue metricset
made optional. 6876
* Add mapping for docker metrics per cpu. 6843
Winlogbeat
* Fixed a crash under Windows 2003 and XP when an event had less
insert strings than required by its format string. 6247
Added
Affecting all Beats
* Update Golang 1.9.4 6326
* Add the ability to log to the Windows Event Log. 5913
* The node name can be discovered automatically by machine-id
matching when beat deployed outside Kubernetes cluster. 6146
* Panics will be written to the logger before exiting. 6199
* Add builder support for autodiscover and annotations builder 6408
* Add plugin support for autodiscover builders, providers 6457
* Preserve runtime from container statuses in Kubernetes autodiscover 6456
* Experimental feature setup.template.append_fields added. 6024
* Add appender support to autodiscover 6469
* Add add_host_metadata processor 5968
* Retry configuration to load dashboards if Kibana is not reachable
when the beat starts. 6560
* Add has_fields conditional to filter events based on the existence
of all the given fields. 6285 6653
* Add support for spooling to disk to the beats event publishing pipeline. 6581
* Added logging of system info at Beat startup. 5946
* Do not log errors if X-Pack Monitoring is enabled but Elastisearch
X-Pack is not. 6627
* Add rename processor. 6292
* Allow override of dynamic template match_mapping_type for fields
with object_type. 6691
Filebeat
* Add IIS module to parse access log and error log. 6127
* Renaming of the prospector type to the input type and all
prospectors are now moved to the input folder, to maintain
backward compatibility type aliasing was used to map the old
type to the new one. This change also affect YAML configuration.
6078
* Addition of the TCP input 6700
* Add option to convert the timestamps to UTC in the system module. 5647
* Add Logstash module support for main log and the slow log,
support the plain text or structured JSON format 5481
* Add stream filtering when using docker prospector. 6057
* Add support for CRI logs format. 5630
* Add json.ignore_decoding_error config to not log json decoding
erors. 6547
* Make registry file permission configurable. 6455
* Add MongoDB module. 6238
* Add Ingest pipeline loading to setup. 6814
* Add support of log_format combined to NGINX access logs. 6858
* Release config reloading feature as GA.
* Add support human friendly size for the UDP input. 6886
* Add Syslog input to ingest RFC3164 Events via TCP and UDP 6842
* Remove the undefined username option from the Redis input and
clarify the documentation. 6662
Heartbeat
* Made the URL field of Heartbeat aggregateable. 6263
* Use match.Matcher for checking Heartbeat response bodies with
regular expressions. 6539
Metricbeat
* Support apache status pages for versions older than 2.4.16. 6450
* Add support for huge pages on Linux. 6436
* Support to optionally de dot keys in http/json metricset to
prevent collisions. 5970
* Add graphite protocol metricbeat module. 4734
* Add http server metricset to support push metrics via http. 4770
* Make config object public for graphite and http server 4820
* Add system uptime metricset. 4848
* Add experimental queue metricset to RabbitMQ module. 4788
* Add additional php-fpm pool status kpis for Metricbeat module 5287
* Add etcd module. 4970
* Add ip address of docker containers to event. 5379
* Add ceph osd tree information to metricbeat 5498
* Add ceph osd_df to metricbeat 5606
* Add basic Logstash module. 5540
* Add dashboard for Windows service metricset. 5603
* Add pct calculated fields for Pod and container CPU and memory usages. 6158
* Add statefulset support to Kubernetes module. 6236
* Refactor prometheus endpoint parsing to look similar to upstream
prometheus 6332
* Making the http/json metricset GA. 6471
* Add support for array in http/json metricset. 6480
* Making the jolokia/jmx module GA. 6143
* Making the MongoDB module GA. 6554
* Allow to disable labels dedot in Docker module, in favor of a
safe way to keep dots. 6490
* Add experimental module to collect metrics from munin nodes. 6517
* Add support for wildcards and explicit metrics grouping in
jolokia/jmx. 6462
* Set collector as default metricset in Prometheus module. 6636 6747
* Set mntr as default metricset in Zookeeper module. 6674
* Set default metricsets in vSphere module. 6676
* Set status as default metricset in Apache module. 6673
* Set namespace as default metricset in Aerospike module. 6669
* Set service as default metricset in Windows module. 6675
* Set all metricsets as default metricsets in uwsgi module. 6688
* Allow autodiscover to monitor unexposed ports 6727
* Mark kubernetes.event metricset as beta. 6715
* Set all metricsets as default metricsets in couchbase module. 6683
* Mark uwsgi module and metricset as beta. 6717
* Mark Golang module and metricsets as beta. 6711
* Mark system.raid metricset as beta. 6710
* Mark http.server metricset as beta. 6712
* Mark metricbeat logstash module and metricsets as beta. 6713
* Set all metricsets as default metricsets in Ceph module. 6676
* Set container, cpu, diskio, healthcheck, info, memory and network
in docker module as default. 6718
* Set cpu, load, memory, network, process and process_summary as
default metricsets in system module. 6689
* Set collector as default metricset in Dropwizard module. 6669
* Set info and keyspace as default metricsets in redis module. 6742
* Set connection as default metricset in rabbitmq module. 6743
* Set all metricsets as default metricsets in Elasticsearch module. 6755
* Set all metricsets as default metricsets in Etcd module. 6756
* Set server metricsets as default in Graphite module. 6757
* Set all metricsets as default metricsets in HAProxy module. 6758
* Set all metricsets as default metricsets in Kafka module. 6759
* Set all metricsets as default metricsets in postgresql module. 6761
* Set status metricsets as default in Kibana module. 6762
* Set all metricsets as default metricsets in Logstash module. 6763
* Set container, node, pod, system, volume as default in Kubernetes
module. https://github.com/elastic/beats/pull/ 6764[6764]
* Set stats as default in memcached module. 6765
* Set all metricsets as default metricsets in Mongodb module. 6766
* Set pool as default metricset for php_fpm module. 6768
* Set status as default metricset for mysql module.
https://github.com/elastic/beats/pull/ 6769[6769]
* Set stubstatus as default metricset for nginx module. 6770
* Added support for haproxy 1.7 and 1.8. 6793
* Add accumulated I/O stats to diskio in the line of docker stats. 6701
* Ignore virtual filesystem types by default in system module. 6819
* Release config reloading feature as GA. 6891
* Kubernetes deployment: Add ServiceAccount config to system metricbeat. 6824
* Kubernetes deployment: Add DNS Policy to system metricbeat. 6656
Packetbeat
* Add support for condition on bool type 5659 5954
* Fix high memory usage on HTTP body if body is not published. 6680
* Allow to capture the HTTP request or response bodies independently. 6784
* HTTP publishes an Error event for unmatched requests or responses. 6794
Winlogbeat
* Use bookmarks to persist the last published event. 6150
Users who have previously installed this package by building as root should
very carefully check their Go installations. Running "pkg_admin check" will
almost certainly fail.
Auditbeat
- Add hex decoding for the name field in audit path records.
Filebeat
- Fix panic when log prospector configuration fails to load.
Packetbeat
- HTTP parses successfully on empty status phrase.
==== Bugfixes
Affecting all Beats
- Add logging when monitoring cannot connect to Elasticsearch.
- Fix infinite loop when event unmarshal fails in Kubernetes
pod watcher.
Filebeat
- Fix a conversion issue for time related fields in the Logstash
module for the slowlog fileset.
==== Breaking changes
Affecting all Beats
- The log format may differ due to logging library changes.
- The default value for pipelining is reduced to 2 to avoid high
memory in the Logstash beats input.
Auditbeat
- Split the audit.kernel and audit.file metricsets into their own
modules
named auditd and file_integrity, respectively. This change requires
existing users to update their config.
- Renamed file_integrity module fields.
- Renamed auditd module fields.
Metricbeat
- Rename `golang.heap.system.optained` field to
`golang.heap.system.obtained`.
- De dot keys in jolokia/jmx metricset to prevent collisions.
==== Bugfixes
Auditbeat
- Fixed an issue where the proctitle value was being truncated.
- Fixed an issue where values were incorrectly interpretted as hex
data.
- Fixed parsing of the `key` value when multiple keys are present.
- Fix possible resource leak if file_integrity module is used with
config
reloading on Windows or Linux.
Filebeat
- Fix variable name for `convert_timezone` in the system module.
Metricbeat
- Fix error `datastore '*' not found` in Vsphere module.
- Fix error `NotAuthenticated` in Vsphere module.
- Fix mongodb session consistency mode to allow command execution on
secondary nodes.
- Fix kubernetes `state_pod` `status.phase` so that the active phase
is returned instead of `unknown`.
- Fix error collecting network_names in Vsphere module.
- Fix process cgroup memory metrics for memsw, kmem, and kmem_tcp.
- Fix kafka OffsetFetch request missing topic and partition
parameters.
Packetbeat
- Fix mysql SQL parser to trim `\r` from Windows Server
`SELECT\r\n\t1`.
==== Added
Affecting all Beats
- Adding a local keystore to allow user to obfuscate password
- Add autodiscover for kubernetes.
- Add Beats metrics reporting to Xpack.
- Update the command line library cobra and add support for zsh
completion
- Update to Golang 1.9.2
- Moved `ip_port` indexer for `add_kubernetes_metadata` to all beats.
- `ip_port` indexer now index both IP and IP:port pairs.
- Add the ability to write structured logs.
- Use structured logging for the metrics that are periodically logged
via the
`logging.metrics` feature.
- Improve Elasticsearch output metrics to count number of dropped and
duplicate (if event ID is given) events.
- Add the abilility for the add_docker_metadata process to enrich
based on process ID.
- The `add_docker_metadata` and `add_kubernetes_metadata` processors
are now GA, instead of Beta.
- Update go-ucfg library to support top level key reference and cyclic
key reference for the
keystore
Auditbeat
- Auditbeat is marked as GA, no longer Beta.
- Add support for BLAKE2b hash algorithms to the file integrity
module.
- Add support for recursive file watches.
Filebeat
- Add Osquery module.
- Add stream filtering when using `docker` prospector.
Metricbeat
- Add ceph osd_df to metricbeat
- Add field network_names of hosts and virtual machines.
- Add experimental system/raid metricset.
- Add a dashboard for the Nginx module.
- Add experimental mongodb/collstats metricset.
- Update the MySQL dashboard to use the Time Series Visual Builder.
- Add experimental uwsgi module.
- Docker and Kubernetes modules are now GA, instead of Beta.
- Support haproxy stats gathering using http (additionaly to tcp
socket).
- Support to optionally 'de dot' keys in http/json metricset to
prevent collisions.
Packetbeat
- Configure good defaults for `add_kubernetes_metadata`.
6.1.2
Auditbeat
- Add an error check to the file integrity scanner to prevent a panic
when there is an error reading file info via lstat.
Filebeat
- Switch to docker prospector in sample manifests for Kubernetes
deployment
=== Beats version 6.1.0
==== Breaking changes
Auditbeat
- Changed `audit.file.path` to be a multi-field so that path is
searchable.
Metricbeat
- Rename `heap_init` field to `heap.init` in the Elasticsearch module.
- Rename `http.response.status_code` field to `http.response.code` in
the HTTP module.
==== Bugfixes
Affecting all Beats
- Remove ID() from Runner interface
- Correctly send configured `Host` header to the remote server.
- Change add_kubernetes_metadata to attempt detection of namespace.
- Avoid double slash when join url and path
- Fix console color output for Windows.
- Fix logstash output debug message.
- Fix isolation of modules when merging local and global field
settings.
Filebeat
- Add support for adding string tags
- Fix race condition when limiting the number of harvesters running in
parallel
- Fix relative paths in the prospector definitions.
- Fix `recursive_globe.enabled` option.
Metricbeat
- Change field type of http header from nested to object
- Fix the fetching of process information when some data is missing
under MacOS X.
- Change `MySQL active connections` visualization title to `MySQL
total connections`.
- Fix `ProcState` on Linux and FreeBSD when process names contain
parentheses.
- Fix incorrect `Mem.Used` calculation under linux.
Packetbeat
- Fix http status phrase parsing not allow spaces.
- Fix http parse to allow to parse get request with space in the URI.
Winlogbeat
- Fix the registry file. It was not correctly storing event log names,
and upon restart it would begin reading at the start of each event log.
==== Added
Affecting all Beats
- Support dashboard loading without Elasticseach
- Changed the hashbang used in the beat helper script from `/bin/bash`
to `/usr/bin/env bash`.
- Changed beat helper script to use `exec` when running the beat.
- Fix reloader error message to only print on actual error
- Add support for enabling TLS renegotiation.
- Add Azure VM support for add_cloud_metadata processor
- Add `output.file.permission` config option.
- Refactor add_kubernetes_metadata to support autodiscovery
- Improve custom flag handling and CLI flags usage message.
- Add number_of_routing_shards config set to 30
- Set log level for kafka output.
- Move TCP UDP start up into `server.Start()`
- Update to Golang 1.9.2
Auditbeat
- Add support for SHA3 hash algorithms to the file integrity module.
- Add dashboards for Linux audit framework events (overview,
executions, sockets).
Filebeat
- Add PostgreSQL module with slowlog support.
- Add Kafka log module.
- Add support for `/var/log/containers/` log path in
`add_kubernetes_metadata` processor.
- Remove error log from runnerfactory as error is returned by API.
- Add experimental Docker `json-file` prospector .
- Add experimental Docker autodiscover functionality.
- Add option to convert the timestamps to UTC in the system module.
- Add Logstash module support for main log and the slow log, support
the plain text or structured JSON format
Metricbeat
- Add graphite protocol metricbeat module.
- Add http server metricset to support push metrics via http.
- Make config object public for graphite and http server
- Add system uptime metricset.
- Add experimental `queue` metricset to RabbitMQ module.
- Add additional php-fpm pool status kpis for Metricbeat module
- Add etcd module.
- Add ip address of docker containers to event.
- Add ceph osd tree information to Metricbeat
- Add basic Logstash module.
- Add dashboard for Windows service metricset.
- Add experimental Docker autodiscover functionality.
- Add Windows service metricset in the windows module.
- Update gosigar to v0.6.0.
Packetbeat
- Add support for decoding the TLS envelopes.
=== Beats version 6.0.1
==== Bugfixes
Affecting all Beats
- Fix documentation links in README.md files.
- Fix `add_docker_metadata` dropping some containers.
Heartbeat
- Fix the "HTTP up status" visualization.
Metricbeat
- Fix map overwrite in docker diskio module.
- Fix connection leak in mongodb module.
- Fix the include top N processes feature for cases where there are
fewer processes than N.
Affecting all Beats
- Fix documentation links in README.md files.
- Fix add_docker_metadata dropping some containers.
Heartbeat
- Fix the "HTTP up status" visualization.
Metricbeat
- Fix map overwrite in docker diskio module.
- Fix connection leak in mongodb module.
- Fix the include top N processes feature for cases where there
are fewer processes than N.
=== Beats version 6.0.0
==== Breaking changes
Affecting all Beats
- The log directory (`path.log`) for Windows services is now set to
`C:\ProgramData\[beatname]\logs`.
- The _all field is disabled in Elasticsearch 6.0.
- Fail if removed setting output.X.flush_interval is explicitly
configured.
- Rename the `/usr/bin/beatname.sh` script (e.g. `metricbeat.sh`) to
`/usr/bin/beatname`.
- Beat does not start if elasticsearch index pattern was modified but
not the template name and pattern.
- Fail if removed setting output.X.flush_interval is explicitly
configured.
- Rename `kubernetes` processor to `add_kubernetes_metadata`.
- Rename `.full.yml` config files to `*.reference.yml`.
- The `scripts/import_dashboards` is removed from packages. Use the
`setup` command instead.
- Change format of the saved kibana dashboards to have a single JSON
file for each dashboard
- Rename `configtest` command to `test config`.
- Remove setting `queue_size` and `bulk_queue_size`.
- Remove setting `dashboard.snapshot` and `dashboard.snapshot_url`.
- Beats can no longer be launched from Windows Explorer (GUI), command
line is required.
Filebeat
- Rename `input_type` field to `prospector.type`
- The `@metadata.type` field, added by the Logstash output, is now
hardcoded to `doc` and will be removed in future versions.
Metricbeat
- Change all `system.cpu..pct` metrics to be scaled by the number of
CPU cores.
- Remove filters setting from metricbeat modules.
- Added `type` field to filesystem metrics.
Heartbeat
- Renamed the heartbeat RPM/DEB name to `heartbeat-elastic`.
Packetbeat
- Remove not-working `runoptions.uid` and `runoptions.gid` options in
Packetbeat.
- Remove the already unsupported `pf_ring` sniffer option.
Auditbeat
- Changed file metricset config to make `file.paths` a list instead of
a dictionary.
==== Bugfixes
Affecting all Beats
- Fix data race accessing watched containers.
- Do not require template if index change and template disabled
- Fix missing ACK in redis output.
- Fix the `/usr/bin/beatname` script to accept `-d ""` as a parameter.
- Combine `fields.yml` properties when they are defined in different
sources.
- Keep Docker & Kubernetes pod metadata after container dies while
they are needed by processors.
- Fix `fields.yml` lookup when using `export template` with a custom
`path.config` param.
- Remove runner creation from every reload check
- Fix add_kubernetes_metadata matcher registry lookup.
- Register kubernetes `field_format` matcher and remove logger in
`Encode` API
- Fix go plugins not loaded when beat starts
- Add support for `initContainers` in `add_kubernetes_metadata`
processor.
- Eliminate deprecated _default_ mapping in 6.x
- Fix pod name indexer to use both namespace, pod name to frame index
key
- Don't stop with error loading the ES template if the ES output is
not enabled.
- Fix race condition in internal logging rotator.
- Normalize all times to UTC to ensure proper index naming.
- Fix issue with loading dashboards to ES 6.0 when .kibana index did
not already exist.
- Fix importing the dashboards when the limit for max open files is
too low.
- Fix configuration documentation for kubernetes processor
- Fix misspelling in `add_locale` configuration option for
abbreviation.
Filebeat
- Fix machine learning jobs setup for dynamic modules.
- Fix default paths for redis 4.0.1 logs on macOS
- Fix Filebeat not starting if command line and modules configs are
used together.
- Fix double `@timestamp` field when JSON decoding was used.
- Fix issue where the `fileset.module` could have the wrong value.
- Fix race condition on harvester stopping with reloading enabled.
- Fix recursive glob config parsing and resolution across restarts.
- Allow string characters in user agent patch version (NGINX and
Apache)
- Fix grok pattern in filebeat module system/auth without hostname.
Winlogbeat
- Removed validation of top-level config keys.
Metricbeat
- Use `beat.name` instead of `beat.hostname` in the Host Overview
dashboard.
- Fix the loading of 5.x dashboards.
- Fix a memory allocation issue where more memory was allocated than
needed in the windows-perfmon metricset.
- Don't start metricbeat if external modules config is wrong and
reload is disabled
- The MongoDB module now connects on each fetch, to avoid stopping the
whole Metricbeat instance if MongoDB is not up when starting.
- Fix kubernetes events module to be able to index time fields
properly.
- Fixed `cmd_set` and `cmd_get` being mixed in the Memcache module.
- Added missing mongodb configuration file to the `modules.d` folder.
- Fix wrong MySQL CRUD queries timelion visualization
- Add new metrics to CPU metricsset
- Fix issue affecting Windows services timing out at startup.
- Fix incorrect docker.diskio.total metric calculation.
- Vsphere module: used memory field corrected.
- Set correct format for percent fields in memory module.
- Fix a debug statement that said a module wrapper had stopped when it
hadn't.
- Use MemAvailable value from /proc/meminfo on Linux 3.14.
- Fix panic when events were dropped by filters.
- Add filtering to system filesystem metricset to remove relative
mountpoints like those from Linux network namespaces.
- Remove unnecessary print statement in schema apis.
- Fix type of field `haproxy.stat.check.health.last`.
Heartbeat
- Fix monitor.name being empty by default.
- Fix wrong event timestamps.
Packetbeat
- Fix missing length check in the PostgreSQL module.
- Fix panic in ACK handler if event is dropped on blocked queue
- Update flow timestamp on each packet being received.
- Enabled /proc/net/tcp6 scanning and fixed ip v6 parsing.
- Enable memcache filtering only if a port is specified in the config
file.
Auditbeat
- Fix `file.max_file_size` config option for the audit file metricset.
==== Added
Affecting all Beats
- Enable flush timeout by default.
- Add @metadata.version to events send to Logstash.
- Add setting to enable/disable the slow start in logstash output.
- Update init scripts to use the `test config` subcommand instead of
the deprecated `-configtest` flag.
- Get by default the credentials for connecting to Kibana from the
Elasticsearch output configuration.
- Added `cloud.id` and `cloud.auth` settings, for simplifying using
Beats with the Elastic Cloud.
- Add lz4 compression support to kafka output.
- Add newer kafka versions to kafka output.
- Configure the index name when loading the dashboards and the index
pattern.
- New cli subcommands interface.
- Allow source path matching in `add_docker_metadata` processor.
- Add support for analyzers and multifields in fields.yml.
- Add support for JSON logging.
- Add `test output` command, to test Elasticsearch and Logstash output
settings.
- Introduce configurable event queue settings: queue.mem.events,
queue.mem.flush.min_events and queue.mem.flush.timeout.
- Enable pipelining in Logstash output by default.
- Added 'result' field to Elasticsearch QueryResult struct for
compatibility with 6.x Index and Delete API responses.
- The sample dashboards are now included in the Beats packages.
- Add `pattern` option to be used in the fields.yml to specify the
pattern for a number field.
- Upgraded to Golang 1.8.3.
- Added the possibility to set Elasticsearch mapping template settings
from the Beat configuration file.
- Add a variable to the SysV init scripts to make it easier to change
the user.
- Add the option to write the generated Elasticsearch mapping template
into a file.
- Add `instance_name` in GCE add_cloud_metadata processor.
- Add `add_docker_metadata` processor.
- Add `logging.files` `permissions` option.
Filebeat
- Add Kubernetes manifests to deploy Filebeat.
- Changed the number of shards in the default configuration to 3.
- Don't start filebeat if external modules/prospectors config is wrong
and reload is disabled
- Add `filebeat.registry_flush` setting, to delay the registry
updates.
- Add experimental Redis module.
- Nginx module: use the first not-private IP address as the remote_ip.
- Load Ingest Node pipelines when the Elasticsearch connection is
established, instead of only once at startup.
- Add support for loading Xpack Machine Learning configurations from
the modules, and added sample configurations for the Nginx module.
- Add udp prospector type.
- Enabled Cgo which means libc is dynamically compiled.
- Add Beta module config reloading mechanism
- Remove spooler and publisher components and settings.
- Added ability to sort harvested files.
- Add experimental Redis slow log prospector type.
Winlogbeat
- Changed the number of shards in the default configuration to 3.
- Add the ability to use LevelRaw if Level isn't populated in the
event XML.
Metricbeat
- Add Kubernetes manifests to deploy Metricbeat.
- Auto-select a hostname (based on the host on which the Beat is
running) in the Host Overview dashboard.
- Add `filesystem.ignore_types` to system module for ignoring
filesystem types.
- Add support to exclude labels from kubernetes pod metadata.
- Add random startup delay to each metricset to avoid the thundering
herd problem.
- Add the ability to configure audit rules to the kernel module.
- Add the ability to configure kernel's audit failure mode.
- Add experimental Aerospike module.
- Vsphere module: collect custom fields from virtual machines.
- Add `test modules` command, to test modules expected output.
- Add `processors` setting to metricbeat modules.
- Support `npipe` protocol (Windows) in Docker module.
- Add macOS implementation of the system diskio metricset.
- Add process_summary metricset that records high level metrics about
processes.
- Add `kube-state-metrics` based metrics to `kubernetes` module
- Add debug logging to Jolokia JMX metricset.
- Add events metricset for kubernetes metricbeat module
- Change Metricbeat default configuration file to be better optimized
for most users.
- Add experimental RabbitMQ module.
- Add Kibana dashboard for the Kubernetes modules.
Heartbeat
- Changed the number of shards in the default configuration to 1.
- Enabled Cgo which means libc is dynamically compiled.
Packetbeat
- Changed the number of shards in the default configuration to 3.
Auditbeat
- Changed the number of shards in the default configuration to 3.
- Add support for receiving audit events using a multicast socket.
- Added `file.hash_types` config option for controlling the hash
types.
- Added the ability to specify byte unit suffixes to
`file.max_file_size`.
- Add file integrity metricset to the audit module.
==== Deprecated
Affecting all Beats
- The `@metadata.type` field, added by the Logstash output, is
deprecated, hardcoded to `doc` and will be removed in future
versions.
Filebeat
- The `filebeat.config_dir` option is deprecated. Use
`filebeat.config.prospector` options instead.
- Deprecate `input_type` prospector config. Use `type` config option
instead.
=== Beats version 5.6.4
- Fix race condition in internal logging rotator.
- Add support for enabling TLS renegotiation.
- Add setting to enable/disable the slow start in logstash output.
- Packetbeat: Fix missing length check in the PostgreSQL module.
=== Beats version 5.6.0
==== Breaking changes
Affecting all Beats
- The _all.norms setting in the Elasticsearch template is no longer
disabled.
==== Bugfixes
Filebeat
- Fix issue where the `fileset.module` could have the wrong value.
Packetbeat
- Update flow timestamp on each packet being received.
Metricbeat
- Fix a debug statement that said a module wrapper had stopped when it
hadn't.
- Use MemAvailable value from /proc/meminfo on Linux 3.14.
- Fix panic when events were dropped by filters.
==== Added
Affecting all Beats
- Add option to the import_dashboards script to load the dashboards via
Kibana API.
Filebeat
- Add support for loading Xpack Machine Learning configurations from the
modules, and added sample configurations for the Nginx module.
- Add ability to parse nginx logs exposing the X-Forwarded-For header
instead of the remote address.
Metricbeat
- Add `filesystem.ignore_types` to system module for ignoring filesystem
types.
==== Deprecated
Affecting all Beats
- Loading more than one output is deprecated and will be removed in 6.0.
No changes in this release.
=== Beats version 5.5.1
==== Bugfixes
Affecting all Beats
- Normalize all times to UTC to ensure proper index naming.
=== Beats version 5.5.0
==== Breaking changes
Affecting all Beats
- Usage of field `_type` is now ignored and hardcoded to `doc`.
Metricbeat
- Change all `system.cpu.*.pct` metrics to be scaled by the number of
CPU cores.
==== Bugfixes
Affecting all Beats
- Fix console output.
Filebeat
- Allow string characters in user agent patch version (NGINX and Apache)
Metricbeat
- Fix type of field `haproxy.stat.check.health.last`.
Packetbeat
- Fix `packetbeat.interface` options that contain underscores (e.g.
`with_vlans` or `bpf_filter`).
- Enabled /proc/net/tcp6 scanning and fixed ip v6 parsing.
==== Deprecated
Filebeat
- Deprecate `document_type` prospector config option as _type is removed
in elasticsearch 6.0. Use fields instead.
Winlogbeat
- Deprecated metrics endpoint. It is superseded by a libbeat feature
that can serve metrics on an HTTP endpoint.
Affecting all Beats
- Removed empty sections from the template files, causing indexing
errors for array objects.
Metricbeat
- Fix issue affecting Windows services timing out at startup.
- Add filtering to system filesystem metricset to remove relative
mountpoints like those from Linux network namespaces.
Packetbeat
- Clean configured geoip.paths before attempting to open the database.
Affecting all Beats
- Fix importing the dashboards when the limit for max open files is
too low.
- Fix console output.
- Binaries upgraded to Go 1.7.6 which contains security fixes.
Filebeat
- Fix issue that new prospector was not reloaded on conflict.
- Fix grok pattern in filebeat module system/auth without hostname.
- Fix the Mysql slowlog parsing of IP addresses.
Winlogbeat
- Add the ability to use LevelRaw if Level isn't populated in the
event XML.
==== Bugfixes
Affecting all Beats
- Improve error message when downloading the dashboards fails.
- Fix potential Elasticsearch output URL parsing error if protocol
scheme is missing.
- Downgrade Elasticsearch per batch item failure log to debug level.
- Make `@timestamp` accessible from format strings.
Filebeat
- Allow log lines without a program name in the Syslog fileset.
- Don't stop Filebeat when modules are used with the Logstash output.
Metricbeat
- Fixing panic on the Prometheus collector when label has a comma.
- Make system process metricset honor the `cpu_ticks` config option.
Winlogbeat
- Fix null terminators include in raw XML string when include_xml is
enabled.
==== Added
Affecting all Beats
- Update index mappings to support future Elasticsearch 6.X.
Filebeat
- Add auditd module for reading audit logs on Linux.
- Add fileset for the Linux authorization logs.
Heartbeat
- Add default ports in HTTP monitor.
Metricbeat
- Add beta Jolokia module.
- Add dashboard for the MySQL module.
- Module configuration reloading is now beta instead of experimental.
- Marked http fields from the HAProxy module optional to improve
compatibility with 1.5.
- Add support for custom HTTP headers and TLS for the Metricbeat
modules.
Packetbeat
- Add DNS dashboard for an overview the DNS traffic.
- Add DNS Tunneling dashboard to highlight domains with large numbers
of subdomains or high data volume.
Affecting all Beats
- Fix panic when testing regex-AST to match against date patterns.
Filebeat
- Fix modules default file permissions.
- Allow - in Apache access log byte count.
Metricbeat
- Avoid errors when some Apache status fields are missing.
==== Breaking changes
Affecting all Beats
- Configuration files must be owned by the user running the Beat or by
root, and they must not be writable by others.
- Change Beat generator. Use
`$GOPATH/src/github.com/elastic/beats/script/generate.py` to
generate a beat.
Filebeat
- Always use absolute path for event and registry. This can lead to
issues when relative paths were used before.
Metricbeat
- Linux cgroup metrics are now enabled by default for the system
process metricset. The configuration option for the feature was
renamed from `cgroups` to `process.cgroups.enabled`.
- Change field names `couchbase.node.couch.*.actual_disk_size.*` to
`couchbase.node.couch.*.disk_size.*`
==== Bugfixes
Affecting all Beats
- Add `_id`, `_type`, `_index` and `_score` fields in the generated
index pattern.
Filebeat
- Fix empty registry file on machine crash.
Metricbeat
- Add error handling to system process metricset for when Linux
cgroups are missing from the kernel.
- Add labels to the Docker healthcheck metricset output.
Winlogbeat
- Fix handling of empty strings in event_data.
==== Added
Affecting all Beats
- Files created by Beats (logs, registry, file output) will have 0600
permissions.
- RPM/deb packages will now install the config file with 0600
permissions.
- Add the option to pass custom HTTP headers to the Elasticsearch
output.
- Unify `regexp` and `contains` conditionals, for both to support
array of strings and convert numbers to strings if required.
- Add the option to load the sample dashboards during the Beat startup
phase.
- Disabled date detection in Elasticsearch index templates. Date
fields must be explicitly defined in index templates.
- Using environment variables in the configuration file is now GA,
instead of experimental.
Filebeat
- Add Filebeat modules for system, apache2, mysql, and nginx.
- Add the `pipeline` config option at the prospector level, for
configuring the Ingest Node pipeline ID.
- Update regular expressions used for matching file names or lines
(multiline, include/exclude functionality) to new matchers improving
performance of simple string matches.
- The `symlinks` and `harverster_limit` settings are now GA, instead
of experimental.
- close_timeout is also applied when the output is blocking.
- Improve handling of different path variants on Windows.
Metricbeat
- Add experimental dbstats metricset to MongoDB module.
- Use persistent, direct connections to the configured nodes for
MongoDB module.
- Add dynamic configuration reloading for modules.
- Add docker health metricset
- Add docker image metricset
- System module uses new matchers for white-listing processes.
- Add Beta CEPH module with health metricset.
- Add Beta php_fpm module with pool metricset.
- The Docker, Kafka, and Prometheus modules are now Beta, instead of
experimental.
- The HAProxy module is now GA, instead of experimental.
- Add the ability to collect the environment variables from system
processes.
==== Deprecated
Affecting all Beats
- Usage of field `_type` is deprecated. It should not be used in
queries or dashboards.
Filebeat
- The experimental `publish_async` option is now deprecated and is
planned to be removed in 6.0.
- Metricbeat: Fix go routine leak in docker module.
- Packetbeat: Fix error in the NFS sample dashboard.
- Winlogbeat: Fix error in the Winlogbeat sample dashboard.
==== Bugfixes
Affecting all Beats
- Fix overwriting explicit empty config sections.
Filebeat
- Fix alignment issue were Filebeat compiled with Go 1.7.4 was crashing
on 32 bits system.
Metricbeat
- Fix service times-out at startup.
- Kafka module case sensitive host name matching.
- Fix interface conversion panic in couchbase module
Packetbeat
- Fix issue where some Cassandra visualizations were showing data from
all protocols.
==== Added
Affecting all Beats
- Add support for passing list and dictionary settings via -E flag.
- Support for parsing list and dictionary setting from environment
variables.
- Added new flags to import_dashboards (-cacert, -cert, -key,
-insecure).
- The limit for the number of fields is increased via the mapping
template.
- Updated to Go 1.7.4.
- Added a NOTICE file containing the notices and licenses of the
dependencies.
Heartbeat
- First release, containing monitors for ICMP, TCP, and HTTP.
Filebeat
- Add enabled config option to prospectors.
- Add target option for decoded_json_field.
Metricbeat
- Kafka module broker matching enhancements.
- Add a couchbase module with metricsets for node, cluster and bucket.
- Export number of cores for CPU module.
- Experimental Prometheus module.
- Add system socket module that reports all TCP sockets.
- Kafka consumer groups metricset.
Winlogbeat
- Reduced amount of memory allocated while reading event log records.
Filebeat
- Fix registry migration issue from old states were files were only
harvested after second restart.
Packetbeat
- Fix error on importing dashboards due to colons in the Cassandra
dashboard.
- Fix error on importing dashboards due to the wrong type for the
geo_point fields.
Winlogbeat
- Fix for "The array bounds are invalid" error when reading large
events.
The Beats are lightweight processes, written in Go, that you install
on your servers to capture all sorts of operational data like logs,
operating system metrics or network packet data, and to send it to
Elasticsearch, either directly or via Logstash, so it can be
visualized with Kibana.
