Commit graph

33 commits

Author SHA1 Message Date
bsiegert
ac232c7df5 Revbump all Go packages after go111 update. 2018-11-04 18:37:54 +00:00
jperkin
53e7b2c198 beats: Don't use "go build -i", it overwrites system files!
Users who have previously installed this package by building as root should
very carefully check their Go installations.  Running "pkg_admin check" will
almost certainly fail.
2018-10-29 13:27:25 +00:00
jperkin
7a79fd2dd8 beats: Create a go tool pointing to the appropriate version.
This should be moved to the go infrastructure if it turns out there are more
packages that call "go" directly in their build.
2018-10-19 16:49:58 +00:00
bsiegert
d22fe57854 Revbump all Go packages after lang/go update. 2018-06-12 17:50:20 +00:00
fhajny
b46cdc9421 sysutils/beats: Update to 6.2.4. Enable auditbeat (outside SunOS).
Auditbeat
- Add hex decoding for the name field in audit path records.

Filebeat
- Fix panic when log prospector configuration fails to load.

Packetbeat
- HTTP parses successfully on empty status phrase.
2018-04-18 12:18:49 +00:00
fhajny
925f02dc7e sysutils/beats: Update to 6.2.3.
- Fix conditions checking on autodiscover Docker labels.
- Avoid panic errors when processing nil Pod events in
  add_kubernetes_metadata.
- Fix infinite failure on Kubernetes watch.
2018-04-06 16:06:23 +00:00
bsiegert
429ee23e50 Revbump all Go packages after 1.10.1 update.
ok wiz@ for committing during freeze
2018-03-30 11:56:19 +00:00
bsiegert
9ab6e4ca7b Revbump all Go packages after Go 1.10 update. 2018-03-04 15:52:12 +00:00
fhajny
f8d87da84c sysutils/beats: Update to 6.2.2.
==== Bugfixes

Affecting all Beats

- Add logging when monitoring cannot connect to Elasticsearch.
- Fix infinite loop when event unmarshal fails in Kubernetes
  pod watcher.

Filebeat

- Fix a conversion issue for time related fields in the Logstash
  module for the slowlog fileset.
2018-02-27 12:54:02 +00:00
fhajny
5ca00ff8d5 sysutils/beats: Update to 6.2.0.
==== Breaking changes

Affecting all Beats

- The log format may differ due to logging library changes.
- The default value for pipelining is reduced to 2 to avoid high
  memory in the Logstash beats input.

Auditbeat

- Split the audit.kernel and audit.file metricsets into their own
  modules
  named auditd and file_integrity, respectively. This change requires
  existing users to update their config.
- Renamed file_integrity module fields.
- Renamed auditd module fields.

Metricbeat

- Rename `golang.heap.system.optained` field to
  `golang.heap.system.obtained`.
- De dot keys in jolokia/jmx metricset to prevent collisions.

==== Bugfixes

Auditbeat

- Fixed an issue where the proctitle value was being truncated.
- Fixed an issue where values were incorrectly interpretted as hex
  data.
- Fixed parsing of the `key` value when multiple keys are present.
- Fix possible resource leak if file_integrity module is used with
  config
  reloading on Windows or Linux.

Filebeat

- Fix variable name for `convert_timezone` in the system module.

Metricbeat

- Fix error `datastore '*' not found` in Vsphere module.
- Fix error `NotAuthenticated` in Vsphere module.
- Fix mongodb session consistency mode to allow command execution on
  secondary nodes.
- Fix kubernetes `state_pod` `status.phase` so that the active phase
  is returned instead of `unknown`.
- Fix error collecting network_names in Vsphere module.
- Fix process cgroup memory metrics for memsw, kmem, and kmem_tcp.
- Fix kafka OffsetFetch request missing topic and partition
  parameters.

Packetbeat

- Fix mysql SQL parser to trim `\r` from Windows Server
  `SELECT\r\n\t1`.


==== Added

Affecting all Beats

- Adding a local keystore to allow user to obfuscate password
- Add autodiscover for kubernetes.
- Add Beats metrics reporting to Xpack.
- Update the command line library cobra and add support for zsh
  completion
- Update to Golang 1.9.2
- Moved `ip_port` indexer for `add_kubernetes_metadata` to all beats.
- `ip_port` indexer now index both IP and IP:port pairs.
- Add the ability to write structured logs.
- Use structured logging for the metrics that are periodically logged
  via the
  `logging.metrics` feature.
- Improve Elasticsearch output metrics to count number of dropped and
  duplicate (if event ID is given) events.
- Add the abilility for the add_docker_metadata process to enrich
  based on process ID.
- The `add_docker_metadata` and `add_kubernetes_metadata` processors
  are now GA, instead of Beta.
- Update go-ucfg library to support top level key reference and cyclic
  key reference for the
  keystore

Auditbeat

- Auditbeat is marked as GA, no longer Beta.
- Add support for BLAKE2b hash algorithms to the file integrity
  module.
- Add support for recursive file watches.

Filebeat

- Add Osquery module.
- Add stream filtering when using `docker` prospector.

Metricbeat

- Add ceph osd_df to metricbeat
- Add field network_names of hosts and virtual machines.
- Add experimental system/raid metricset.
- Add a dashboard for the Nginx module.
- Add experimental mongodb/collstats metricset.
- Update the MySQL dashboard to use the Time Series Visual Builder.
- Add experimental uwsgi module.
- Docker and Kubernetes modules are now GA, instead of Beta.
- Support haproxy stats gathering using http (additionaly to tcp
  socket).
- Support to optionally 'de dot' keys in http/json metricset to
  prevent collisions.

Packetbeat

- Configure good defaults for `add_kubernetes_metadata`.
2018-02-07 16:31:34 +00:00
fhajny
3b3a9e8415 Update sysutils/beats to 6.1.2.
6.1.2

Auditbeat
- Add an error check to the file integrity scanner to prevent a panic
  when there is an error reading file info via lstat.

Filebeat
- Switch to docker prospector in sample manifests for Kubernetes
  deployment
2018-01-22 19:54:56 +00:00
fhajny
513c9fa3b2 Update sysutils/beats to 6.1.0.
=== Beats version 6.1.0

==== Breaking changes

Auditbeat

- Changed `audit.file.path` to be a multi-field so that path is
  searchable.

Metricbeat

- Rename `heap_init` field to `heap.init` in the Elasticsearch module.
- Rename `http.response.status_code` field to `http.response.code` in
  the HTTP module.

==== Bugfixes

Affecting all Beats

- Remove ID() from Runner interface
- Correctly send configured `Host` header to the remote server.
- Change add_kubernetes_metadata to attempt detection of namespace.
- Avoid double slash when join url and path
- Fix console color output for Windows.
- Fix logstash output debug message.
- Fix isolation of modules when merging local and global field
  settings.

Filebeat

- Add support for adding string tags
- Fix race condition when limiting the number of harvesters running in
  parallel
- Fix relative paths in the prospector definitions.
- Fix `recursive_globe.enabled` option.

Metricbeat

- Change field type of http header from nested to object
- Fix the fetching of process information when some data is missing
  under MacOS X.
- Change `MySQL active connections` visualization title to `MySQL
  total connections`.
- Fix `ProcState` on Linux and FreeBSD when process names contain
  parentheses.
- Fix incorrect `Mem.Used` calculation under linux.

Packetbeat

- Fix http status phrase parsing not allow spaces.
- Fix http parse to allow to parse get request with space in the URI.

Winlogbeat

- Fix the registry file. It was not correctly storing event log names,
  and upon restart it would begin reading at the start of each event log.

==== Added

Affecting all Beats

- Support dashboard loading without Elasticseach
- Changed the hashbang used in the beat helper script from `/bin/bash`
  to `/usr/bin/env bash`.
- Changed beat helper script to use `exec` when running the beat.
- Fix reloader error message to only print on actual error
- Add support for enabling TLS renegotiation.
- Add Azure VM support for add_cloud_metadata processor
- Add `output.file.permission` config option.
- Refactor add_kubernetes_metadata to support autodiscovery
- Improve custom flag handling and CLI flags usage message.
- Add number_of_routing_shards config set to 30
- Set log level for kafka output.
- Move TCP UDP start up into `server.Start()`
- Update to Golang 1.9.2

Auditbeat

- Add support for SHA3 hash algorithms to the file integrity module.
- Add dashboards for Linux audit framework events (overview,
  executions, sockets).

Filebeat

- Add PostgreSQL module with slowlog support.
- Add Kafka log module.
- Add support for `/var/log/containers/` log path in
  `add_kubernetes_metadata` processor.
- Remove error log from runnerfactory as error is returned by API.
- Add experimental Docker `json-file` prospector .
- Add experimental Docker autodiscover functionality.
- Add option to convert the timestamps to UTC in the system module.
- Add Logstash module support for main log and the slow log, support
  the plain text or structured JSON format

Metricbeat

- Add graphite protocol metricbeat module.
- Add http server metricset to support push metrics via http.
- Make config object public for graphite and http server
- Add system uptime metricset.
- Add experimental `queue` metricset to RabbitMQ module.
- Add additional php-fpm pool status kpis for Metricbeat module
- Add etcd module.
- Add ip address of docker containers to event.
- Add ceph osd tree information to Metricbeat
- Add basic Logstash module.
- Add dashboard for Windows service metricset.
- Add experimental Docker autodiscover functionality.
- Add Windows service metricset in the windows module.
- Update gosigar to v0.6.0.

Packetbeat

- Add support for decoding the TLS envelopes.


=== Beats version 6.0.1

==== Bugfixes

Affecting all Beats

- Fix documentation links in README.md files.
- Fix `add_docker_metadata` dropping some containers.

Heartbeat

- Fix the "HTTP up status" visualization.

Metricbeat

- Fix map overwrite in docker diskio module.
- Fix connection leak in mongodb module.
- Fix the include top N processes feature for cases where there are
  fewer processes than N.
2017-12-18 14:37:02 +00:00
fhajny
fb3befce9a Update sysutils/beats to 6.0.1.
Affecting all Beats
- Fix documentation links in README.md files.
- Fix add_docker_metadata dropping some containers.

Heartbeat
- Fix the "HTTP up status" visualization.

Metricbeat
- Fix map overwrite in docker diskio module.
- Fix connection leak in mongodb module.
- Fix the include top N processes feature for cases where there
  are fewer processes than N.
2017-12-13 12:44:54 +00:00
fhajny
777650d1b3 Update sysutils/beats to 6.0.0.
=== Beats version 6.0.0

==== Breaking changes

Affecting all Beats

- The log directory (`path.log`) for Windows services is now set to
  `C:\ProgramData\[beatname]\logs`.
- The _all field is disabled in Elasticsearch 6.0.
- Fail if removed setting output.X.flush_interval is explicitly
  configured.
- Rename the `/usr/bin/beatname.sh` script (e.g. `metricbeat.sh`) to
  `/usr/bin/beatname`.
- Beat does not start if elasticsearch index pattern was modified but
  not the template name and pattern.
- Fail if removed setting output.X.flush_interval is explicitly
  configured.
- Rename `kubernetes` processor to `add_kubernetes_metadata`.
- Rename `.full.yml` config files to `*.reference.yml`.
- The `scripts/import_dashboards` is removed from packages. Use the
  `setup` command instead.
- Change format of the saved kibana dashboards to have a single JSON
  file for each dashboard
- Rename `configtest` command to `test config`.
- Remove setting `queue_size` and `bulk_queue_size`.
- Remove setting `dashboard.snapshot` and `dashboard.snapshot_url`.
- Beats can no longer be launched from Windows Explorer (GUI), command
  line is required.

Filebeat

- Rename `input_type` field to `prospector.type`
- The `@metadata.type` field, added by the Logstash output, is now
  hardcoded to `doc` and will be removed in future versions.

Metricbeat

- Change all `system.cpu..pct` metrics to be scaled by the number of
  CPU cores.
- Remove filters setting from metricbeat modules.
- Added `type` field to filesystem metrics.

Heartbeat

- Renamed the heartbeat RPM/DEB name to `heartbeat-elastic`.

Packetbeat

- Remove not-working `runoptions.uid` and `runoptions.gid` options in
  Packetbeat.
- Remove the already unsupported `pf_ring` sniffer option.

Auditbeat

- Changed file metricset config to make `file.paths` a list instead of
  a dictionary.

==== Bugfixes

Affecting all Beats

- Fix data race accessing watched containers.
- Do not require template if index change and template disabled
- Fix missing ACK in redis output.
- Fix the `/usr/bin/beatname` script to accept `-d ""` as a parameter.
- Combine `fields.yml` properties when they are defined in different
  sources.
- Keep Docker & Kubernetes pod metadata after container dies while
  they are needed by processors.
- Fix `fields.yml` lookup when using `export template` with a custom
  `path.config` param.
- Remove runner creation from every reload check
- Fix add_kubernetes_metadata matcher registry lookup.
- Register kubernetes `field_format` matcher and remove logger in
  `Encode` API
- Fix go plugins not loaded when beat starts
- Add support for `initContainers` in `add_kubernetes_metadata`
  processor.
- Eliminate deprecated _default_ mapping in 6.x
- Fix pod name indexer to use both namespace, pod name to frame index
  key
- Don't stop with error loading the ES template if the ES output is
  not enabled.
- Fix race condition in internal logging rotator.
- Normalize all times to UTC to ensure proper index naming.
- Fix issue with loading dashboards to ES 6.0 when .kibana index did
  not already exist.
- Fix importing the dashboards when the limit for max open files is
  too low.
- Fix configuration documentation for kubernetes processor
- Fix misspelling in `add_locale` configuration option for
  abbreviation.

Filebeat

- Fix machine learning jobs setup for dynamic modules.
- Fix default paths for redis 4.0.1 logs on macOS
- Fix Filebeat not starting if command line and modules configs are
  used together.
- Fix double `@timestamp` field when JSON decoding was used.
- Fix issue where the `fileset.module` could have the wrong value.
- Fix race condition on harvester stopping with reloading enabled.
- Fix recursive glob config parsing and resolution across restarts.
- Allow string characters in user agent patch version (NGINX and
  Apache)
- Fix grok pattern in filebeat module system/auth without hostname.

Winlogbeat

- Removed validation of top-level config keys.

Metricbeat

- Use `beat.name` instead of `beat.hostname` in the Host Overview
  dashboard.
- Fix the loading of 5.x dashboards.
- Fix a memory allocation issue where more memory was allocated than
  needed in the windows-perfmon metricset.
- Don't start metricbeat if external modules config is wrong and
  reload is disabled
- The MongoDB module now connects on each fetch, to avoid stopping the
  whole Metricbeat instance if MongoDB is not up when starting.
- Fix kubernetes events module to be able to index time fields
  properly.
- Fixed `cmd_set` and `cmd_get` being mixed in the Memcache module.
- Added missing mongodb configuration file to the `modules.d` folder.
- Fix wrong MySQL CRUD queries timelion visualization
- Add new metrics to CPU metricsset
- Fix issue affecting Windows services timing out at startup.
- Fix incorrect docker.diskio.total metric calculation.
- Vsphere module: used memory field corrected.
- Set correct format for percent fields in memory module.
- Fix a debug statement that said a module wrapper had stopped when it
  hadn't.
- Use MemAvailable value from /proc/meminfo on Linux 3.14.
- Fix panic when events were dropped by filters.
- Add filtering to system filesystem metricset to remove relative
  mountpoints like those from Linux network namespaces.
- Remove unnecessary print statement in schema apis.
- Fix type of field `haproxy.stat.check.health.last`.

Heartbeat

- Fix monitor.name being empty by default.
- Fix wrong event timestamps.

Packetbeat

- Fix missing length check in the PostgreSQL module.
- Fix panic in ACK handler if event is dropped on blocked queue
- Update flow timestamp on each packet being received.
- Enabled /proc/net/tcp6 scanning and fixed ip v6 parsing.
- Enable memcache filtering only if a port is specified in the config
  file.

Auditbeat

- Fix `file.max_file_size` config option for the audit file metricset.

==== Added

Affecting all Beats

- Enable flush timeout by default.
- Add @metadata.version to events send to Logstash.
- Add setting to enable/disable the slow start in logstash output.
- Update init scripts to use the `test config` subcommand instead of
  the deprecated `-configtest` flag.
- Get by default the credentials for connecting to Kibana from the
  Elasticsearch output configuration.
- Added `cloud.id` and `cloud.auth` settings, for simplifying using
  Beats with the Elastic Cloud.
- Add lz4 compression support to kafka output.
- Add newer kafka versions to kafka output.
- Configure the index name when loading the dashboards and the index
  pattern.
- New cli subcommands interface.
- Allow source path matching in `add_docker_metadata` processor.
- Add support for analyzers and multifields in fields.yml.
- Add support for JSON logging.
- Add `test output` command, to test Elasticsearch and Logstash output
  settings.
- Introduce configurable event queue settings: queue.mem.events,
  queue.mem.flush.min_events and queue.mem.flush.timeout.
- Enable pipelining in Logstash output by default.
- Added 'result' field to Elasticsearch QueryResult struct for
  compatibility with 6.x Index and Delete API responses.
- The sample dashboards are now included in the Beats packages.
- Add `pattern` option to be used in the fields.yml to specify the
  pattern for a number field.
- Upgraded to Golang 1.8.3.
- Added the possibility to set Elasticsearch mapping template settings
  from the Beat configuration file.
- Add a variable to the SysV init scripts to make it easier to change
  the user.
- Add the option to write the generated Elasticsearch mapping template
  into a file.
- Add `instance_name` in GCE add_cloud_metadata processor.
- Add `add_docker_metadata` processor.
- Add `logging.files` `permissions` option.

Filebeat

- Add Kubernetes manifests to deploy Filebeat.
- Changed the number of shards in the default configuration to 3.
- Don't start filebeat if external modules/prospectors config is wrong
  and reload is disabled
- Add `filebeat.registry_flush` setting, to delay the registry
  updates.
- Add experimental Redis module.
- Nginx module: use the first not-private IP address as the remote_ip.
- Load Ingest Node pipelines when the Elasticsearch connection is
  established, instead of only once at startup.
- Add support for loading Xpack Machine Learning configurations from
  the modules, and added sample configurations for the Nginx module.
- Add udp prospector type.
- Enabled Cgo which means libc is dynamically compiled.
- Add Beta module config reloading mechanism
- Remove spooler and publisher components and settings.
- Added ability to sort harvested files.
- Add experimental Redis slow log prospector type.

Winlogbeat

- Changed the number of shards in the default configuration to 3.
- Add the ability to use LevelRaw if Level isn't populated in the
  event XML.

Metricbeat

- Add Kubernetes manifests to deploy Metricbeat.
- Auto-select a hostname (based on the host on which the Beat is
  running) in the Host Overview dashboard.
- Add `filesystem.ignore_types` to system module for ignoring
  filesystem types.
- Add support to exclude labels from kubernetes pod metadata.
- Add random startup delay to each metricset to avoid the thundering
  herd problem.
- Add the ability to configure audit rules to the kernel module.
- Add the ability to configure kernel's audit failure mode.
- Add experimental Aerospike module.
- Vsphere module: collect custom fields from virtual machines.
- Add `test modules` command, to test modules expected output.
- Add `processors` setting to metricbeat modules.
- Support `npipe` protocol (Windows) in Docker module.
- Add macOS implementation of the system diskio metricset.
- Add process_summary metricset that records high level metrics about
  processes.
- Add `kube-state-metrics` based metrics to `kubernetes` module
- Add debug logging to Jolokia JMX metricset.
- Add events metricset for kubernetes metricbeat module
- Change Metricbeat default configuration file to be better optimized
  for most users.
- Add experimental RabbitMQ module.
- Add Kibana dashboard for the Kubernetes modules.

Heartbeat

- Changed the number of shards in the default configuration to 1.
- Enabled Cgo which means libc is dynamically compiled.

Packetbeat

- Changed the number of shards in the default configuration to 3.

Auditbeat

- Changed the number of shards in the default configuration to 3.
- Add support for receiving audit events using a multicast socket.
- Added `file.hash_types` config option for controlling the hash
  types.
- Added the ability to specify byte unit suffixes to
  `file.max_file_size`.
- Add file integrity metricset to the audit module.

==== Deprecated

Affecting all Beats

- The `@metadata.type` field, added by the Logstash output, is
  deprecated, hardcoded to `doc` and will be removed in future
versions.


Filebeat

- The `filebeat.config_dir` option is deprecated. Use
  `filebeat.config.prospector` options instead.
- Deprecate `input_type` prospector config. Use `type` config option
  instead.




=== Beats version 5.6.4

- Fix race condition in internal logging rotator.
- Add support for enabling TLS renegotiation.
- Add setting to enable/disable the slow start in logstash output.
- Packetbeat: Fix missing length check in the PostgreSQL module.
2017-11-20 15:55:16 +00:00
fhajny
3eb05a106d sysutils/beats: Provide a better hint to where libpcap is. May fix
the build in some situations.
2017-11-02 15:30:50 +00:00
fhajny
b354414ea7 Update sysutils/beats to 5.6.3.
No changes - sync version with databases/elasticsearch.
2017-10-17 11:31:36 +00:00
fhajny
1c1e7fdb75 Update sysutils/beats to 5.6.2.
=== Beats version 5.6.0

==== Breaking changes

Affecting all Beats
- The _all.norms setting in the Elasticsearch template is no longer
  disabled.

==== Bugfixes

Filebeat
- Fix issue where the `fileset.module` could have the wrong value.

Packetbeat
- Update flow timestamp on each packet being received.

Metricbeat
- Fix a debug statement that said a module wrapper had stopped when it
  hadn't.
- Use MemAvailable value from /proc/meminfo on Linux 3.14.
- Fix panic when events were dropped by filters.

==== Added

Affecting all Beats
- Add option to the import_dashboards script to load the dashboards via
  Kibana API.

Filebeat
- Add support for loading Xpack Machine Learning configurations from the
  modules, and added sample configurations for the Nginx module.
- Add ability to parse nginx logs exposing the X-Forwarded-For header
  instead of the remote address.

Metricbeat
- Add `filesystem.ignore_types` to system module for ignoring filesystem
  types.

==== Deprecated

Affecting all Beats
- Loading more than one output is deprecated and will be removed in 6.0.
2017-09-27 12:40:44 +00:00
tnn
970f01378d beats: regen distinfo, bump rev
Upstream moved the release tag after a documentation fix.
2017-09-22 12:25:44 +00:00
fhajny
3053fe06bc === Beats version 5.5.2
No changes in this release.


=== Beats version 5.5.1

==== Bugfixes

Affecting all Beats
- Normalize all times to UTC to ensure proper index naming.


=== Beats version 5.5.0

==== Breaking changes

Affecting all Beats
- Usage of field `_type` is now ignored and hardcoded to `doc`.

Metricbeat
- Change all `system.cpu.*.pct` metrics to be scaled by the number of
  CPU cores.

==== Bugfixes

Affecting all Beats
- Fix console output.

Filebeat
- Allow string characters in user agent patch version (NGINX and Apache)

Metricbeat
- Fix type of field `haproxy.stat.check.health.last`.

Packetbeat
- Fix `packetbeat.interface` options that contain underscores (e.g.
  `with_vlans` or `bpf_filter`).
- Enabled /proc/net/tcp6 scanning and fixed ip v6 parsing.

==== Deprecated

Filebeat
- Deprecate `document_type` prospector config option as _type is removed
  in elasticsearch 6.0. Use fields instead.

Winlogbeat
- Deprecated metrics endpoint. It is superseded by a libbeat feature
  that can serve metrics on an HTTP endpoint.
2017-08-18 09:08:54 +00:00
wiz
7284dffc22 More RELRO check disabling for go packages. 2017-07-23 07:20:42 +00:00
fhajny
2c389e3e5c Update sysutils/beats to 5.4.3.
Doc changes only.
2017-07-04 14:22:48 +00:00
fhajny
92c7374a7a Update sysutils/beats to 5.4.2
Affecting all Beats
- Removed empty sections from the template files, causing indexing
  errors for array objects.

Metricbeat
- Fix issue affecting Windows services timing out at startup.
- Add filtering to system filesystem metricset to remove relative
  mountpoints like those from Linux network namespaces.

Packetbeat
- Clean configured geoip.paths before attempting to open the database.
2017-06-21 13:38:41 +00:00
fhajny
b7202fd01c Update sysutils/beats to 5.4.1.
Affecting all Beats
- Fix importing the dashboards when the limit for max open files is
  too low.
- Fix console output.
- Binaries upgraded to Go 1.7.6 which contains security fixes.

Filebeat
- Fix issue that new prospector was not reloaded on conflict.
- Fix grok pattern in filebeat module system/auth without hostname.
- Fix the Mysql slowlog parsing of IP addresses.

Winlogbeat
- Add the ability to use LevelRaw if Level isn't populated in the
  event XML.
2017-06-14 13:56:09 +00:00
fhajny
97350ec422 Update sysutils/beats to 5.4.0.
==== Bugfixes

Affecting all Beats

- Improve error message when downloading the dashboards fails.
- Fix potential Elasticsearch output URL parsing error if protocol
  scheme is missing.
- Downgrade Elasticsearch per batch item failure log to debug level.
- Make `@timestamp` accessible from format strings.

Filebeat

- Allow log lines without a program name in the Syslog fileset.
- Don't stop Filebeat when modules are used with the Logstash output.

Metricbeat

- Fixing panic on the Prometheus collector when label has a comma.
- Make system process metricset honor the `cpu_ticks` config option.

Winlogbeat

- Fix null terminators include in raw XML string when include_xml is
  enabled.

==== Added

Affecting all Beats

- Update index mappings to support future Elasticsearch 6.X.

Filebeat

- Add auditd module for reading audit logs on Linux.
- Add fileset for the Linux authorization logs.

Heartbeat

- Add default ports in HTTP monitor.

Metricbeat

- Add beta Jolokia module.
- Add dashboard for the MySQL module.
- Module configuration reloading is now beta instead of experimental.
- Marked http fields from the HAProxy module optional to improve
  compatibility with 1.5.
- Add support for custom HTTP headers and TLS for the Metricbeat
  modules.

Packetbeat

- Add DNS dashboard for an overview the DNS traffic.
- Add DNS Tunneling dashboard to highlight domains with large numbers
  of subdomains or high data volume.
2017-05-11 08:48:27 +00:00
fhajny
23533fb9e4 Update sysutils/beats to 5.3.2.
Filebeat
- Properly shut down crawler in case one prospector is misconfigured.
- Fix panic in JSON decoding code if the input line is "null".
2017-04-28 08:49:32 +00:00
fhajny
175da1e21d Update sysutils/beats to 5.3.1.
Affecting all Beats
- Fix panic when testing regex-AST to match against date patterns.

Filebeat
- Fix modules default file permissions.
- Allow - in Apache access log byte count.

Metricbeat
- Avoid errors when some Apache status fields are missing.
2017-04-21 11:58:38 +00:00
bsiegert
65e3f16e8f Revbump all Go packages after the Go 1.8.1 update. 2017-04-13 15:12:03 +00:00
fhajny
af0e01cead Update sysutils/beats to 5.3.0.
==== Breaking changes

Affecting all Beats
- Configuration files must be owned by the user running the Beat or by
  root, and they must not be writable by others.
- Change Beat generator. Use
  `$GOPATH/src/github.com/elastic/beats/script/generate.py` to
  generate a beat.

Filebeat
- Always use absolute path for event and registry. This can lead to
  issues when relative paths were used before.

Metricbeat
- Linux cgroup metrics are now enabled by default for the system
  process metricset. The configuration option for the feature was
  renamed from `cgroups` to `process.cgroups.enabled`.
- Change field names `couchbase.node.couch.*.actual_disk_size.*` to
  `couchbase.node.couch.*.disk_size.*`

==== Bugfixes

Affecting all Beats
- Add `_id`, `_type`, `_index` and `_score` fields in the generated
  index pattern.

Filebeat
- Fix empty registry file on machine crash.

Metricbeat
- Add error handling to system process metricset for when Linux
  cgroups are missing from the kernel.
- Add labels to the Docker healthcheck metricset output.

Winlogbeat
- Fix handling of empty strings in event_data.

==== Added

Affecting all Beats
- Files created by Beats (logs, registry, file output) will have 0600
  permissions.
- RPM/deb packages will now install the config file with 0600
  permissions.
- Add the option to pass custom HTTP headers to the Elasticsearch
  output.
- Unify `regexp` and `contains` conditionals, for both to support
  array of strings and convert numbers to strings if required.
- Add the option to load the sample dashboards during the Beat startup
  phase.
- Disabled date detection in Elasticsearch index templates. Date
  fields must be explicitly defined in index templates.
- Using environment variables in the configuration file is now GA,
  instead of experimental.

Filebeat
- Add Filebeat modules for system, apache2, mysql, and nginx.
- Add the `pipeline` config option at the prospector level, for
  configuring the Ingest Node pipeline ID.
- Update regular expressions used for matching file names or lines
  (multiline, include/exclude functionality) to new matchers improving
  performance of simple string matches.
- The `symlinks` and `harverster_limit` settings are now GA, instead
  of experimental.
- close_timeout is also applied when the output is blocking.
- Improve handling of different path variants on Windows.


Metricbeat
- Add experimental dbstats metricset to MongoDB module.
- Use persistent, direct connections to the configured nodes for
  MongoDB module.
- Add dynamic configuration reloading for modules.
- Add docker health metricset
- Add docker image metricset
- System module uses new matchers for white-listing processes.
- Add Beta CEPH module with health metricset.
- Add Beta php_fpm module with pool metricset.
- The Docker, Kafka, and Prometheus modules are now Beta, instead of
  experimental.
- The HAProxy module is now GA, instead of experimental.
- Add the ability to collect the environment variables from system
  processes.

==== Deprecated

Affecting all Beats
- Usage of field `_type` is deprecated. It should not be used in
  queries or dashboards.

Filebeat
- The experimental `publish_async` option is now deprecated and is
  planned to be removed in 6.0.
2017-03-30 15:37:12 +00:00
fhajny
fefc06d52f Update sysutils/beats to 5.2.2
Metricbeat
- Fix bug docker module hanging when docker container killed.
- Set timeout to period instead of 1s by default as documented.
2017-03-01 14:21:27 +00:00
fhajny
b8dd87d46f Update sysutils/beats to 5.2.1.
- Metricbeat: Fix go routine leak in docker module.
- Packetbeat: Fix error in the NFS sample dashboard.
- Winlogbeat: Fix error in the Winlogbeat sample dashboard.
2017-02-17 13:45:42 +00:00
fhajny
e2982a0031 Update sysutils/beats to 5.2.0.
==== Bugfixes

Affecting all Beats

- Fix overwriting explicit empty config sections.

Filebeat

- Fix alignment issue were Filebeat compiled with Go 1.7.4 was crashing
  on 32 bits system.

Metricbeat

- Fix service times-out at startup.
- Kafka module case sensitive host name matching.
- Fix interface conversion panic in couchbase module

Packetbeat

- Fix issue where some Cassandra visualizations were showing data from
  all protocols.

==== Added

Affecting all Beats

- Add support for passing list and dictionary settings via -E flag.
- Support for parsing list and dictionary setting from environment
  variables.
- Added new flags to import_dashboards (-cacert, -cert, -key,
  -insecure).
- The limit for the number of fields is increased via the mapping
  template.
- Updated to Go 1.7.4.
- Added a NOTICE file containing the notices and licenses of the
  dependencies.

Heartbeat

- First release, containing monitors for ICMP, TCP, and HTTP.

Filebeat

- Add enabled config option to prospectors.
- Add target option for decoded_json_field.

Metricbeat

- Kafka module broker matching enhancements.
- Add a couchbase module with metricsets for node, cluster and bucket.
- Export number of cores for CPU module.
- Experimental Prometheus module.
- Add system socket module that reports all TCP sockets.
- Kafka consumer groups metricset.

Winlogbeat

- Reduced amount of memory allocated while reading event log records.
2017-02-14 10:43:20 +00:00
fhajny
b2dc944bff Update sysutils/beats to 5.1.2.
Filebeat
- Fix registry migration issue from old states were files were only
  harvested after second restart.

Packetbeat
- Fix error on importing dashboards due to colons in the Cassandra
  dashboard.
- Fix error on importing dashboards due to the wrong type for the
  geo_point fields.

Winlogbeat
- Fix for "The array bounds are invalid" error when reading large
  events.
2017-01-17 17:39:48 +00:00
fhajny
5b8bd78a1b Import beats 5.1.1 as sysutils/beats (based on wip/beats).
The Beats are lightweight processes, written in Go, that you install
on your servers to capture all sorts of operational data like logs,
operating system metrics or network packet data, and to send it to
Elasticsearch, either directly or via Logstash, so it can be
visualized with Kibana.
2017-01-05 16:43:09 +00:00