See upgrade notes:
http://kafka.apache.org/100/documentation.html#upgrade
New Features
- KIP-72 Allow putting a bound on memory consumed by Incoming requests
- Expose states of active tasks to public API
- Add UnderMinIsrPartitionCount and per-partition UnderMinIsr metrics
- KIP-182: Reduce Streams DSL overloads and allow easier use of custom
storage engines
- Add ChangeReplicaDirRequest and DescribeReplicaDirRequest (KIP-113)
- Add cumulative count attribute for all Kafka rate metrics
- Add new metrics to support health checks
- Handle SASL authentication failures as non-retriable exceptions in
clients
Improvement
- Remove deprecated and slower scala JSON parser
- Add Kafka Connect metrics
- Allow setting of default topic configs via StreamsConfig
- Improve DumpLogSegments offsets-decoder output format
- Improve documentation of configuration
listeners=PLAINTEXT://0.0.0.0:9092
- StateRestoreCallback begin and end indication
- Update the usage description of clean shutdown file.
- Add a '--group' option to the console consumer
- Support Java 9
- Details of retention.bytes property at Topic level are not clear on
how they impact partition size
- Handle disk failure for JBOD (KIP-112)
- Improve diagnostics for SASL authentication failures
- Remove usage of java.xml.bind.* classes hidden by default in JDK9
- Make server start script work against Java 9
- Only log invalid user configs and overwrite with correct one
- Changes to punctuate semantics (KIP-138)
- Use per-task converters in Connect
- Consumer perf tool should count rebalance time separately
- Add ability to batch restore and receive restoration stats.
- KIP-162: Enable topic deletion by default
- Fix taskClass() method name in Connector and flush() signature in
SinkTask
- Add checks in integration tests to verify that threads have been
shutdown
- Add a new Kafka Streams example IoT oriented
- KIP-168: Add GlobalTopicCount metric per cluster
- Change Source offset commit message to info to match Sink behaviour
- Replace -XX:+DisableExplicitGC with -XX:+ExplicitGCInvokesConcurrent
in kafka-run-class
- Streams should not suspend tasks twice
- Replace the deprecated 'ConsumerOffsetChecker' in documentation
- Consider removing date formatting from Segments class
- Formatting verifiable producer/consumer output in a similar fashion
- Streams reset tool should have same console output with or without
dry-run
- Improve Java doc for LeaderEpochFileCache.endOffsetFor()
- Return topic authorization failed if no topic describe access
- Explain that `client.id` is just used as a prefix within Streams
- Using a logPrefix inside the StreamPartitionAssignor
- Add --custom-ducktape flag to ducker-ak
- Console Consumer overrides auto.offset.reset property when provided on
the command line without warning about it.
- Connect REST API should include the connector type when describing a
connector
- Test DeleteRecordsRequest in AuthorizerIntegrationTest
- KafkaConsumer.subscribe() overload that takes just Pattern without
ConsumerRebalanceListener
- All ducktape services should store their files in subdirectories of
/mnt
- ShellTest: add tests for attempting to run nonexistent program, error
return
- Elevate log messages for denials to INFO in SimpleAclAuthorizer class
- Refactor Streams to use LogContext
- Refactor Producer to use LogContext
- Refactor AdminClient to use LogContext
- Refactor NetworkClient to use LogContext
- Upgrade ducktape version to 0.7.1, and use new kill_java_processes
- Implement KafkaPrincipalBuilder interface with support for SASL
(KIP-189)
- Remove unneeded synchronized keyword in StreamThread
- Consistent logging of topic partitions
- Add AdminClient.createPartitions()
- Support unmapping of mapped/direct buffers in Java 9
- Handle SSL authentication failures as non-retriable exceptions in
clients
- Avoid call fetchPrevious in FlushListeners
- Handle authentication failures from transactional producer and
KafkaAdminClient
- User StoreListener not available for global stores
- Mention the need to close store iterators
- Enable custom authentication plugins to return error messages to
clients
- There is something wrong in Kafka Streams document
Bugs
- NullPointerException in client on MetadataRequest
- The kafka-consumer-perf-test.sh script help information print useless
parameters.
- Remove ConsumerOffsetChecker, deprecated in 0.9, in 0.11
- Make KStreamTestDriver extending from ExternalResource
- Cleanup Kafka Streams builder API
- completedReceives can contain closed channels
- KIP-163: Offset fetch and commit requests use the same permissions
- KafkaProducer.flush hangs when NetworkClient.handleCompletedReceives
throws exception
- Records from internal repartitioning topics should always use
RecordMetadataTimestampExtractor
- Augment KStream.print() to allow users pass in extra parameters in the
printed string
- Calling KafkaProducer.close() from multiple threads may cause spurious
error
- Kafka batch files does not support path with spaces
- creating a window store with one segment throws division by zero error
- Replace pattern matching with foreach where the case None is unused
- Flaky ResetIntegrationTest
- Kafka Streams keeps restoring state after shutdown is initiated during
startup
- streams task gets stuck after re-balance due to LockException
- StreamsResetter doesn't allow custom Consumer properties
- Producer buffer pool allocates memory inside a lock.
- ProcessorContext.appConfigs() should return parsed/validated values
- JmxReporter Fetch metrics for kafka.server should not be created when
client quotas are not enabled
- Using connect-console-sink/source.properties raises an exception
related to "file" property not found
- Clients get inconsistent connection states when SASL/SSL connection is
marked CONECTED and DISCONNECTED at the same time
- LogCleaner stopped due to
org.apache.kafka.common.errors.CorruptRecordException
- StreamsKafkaClient should not use StreamsConfig.POLL_MS_CONFIG
- Created state changelog topics not logged correctly
- ListOffsetResponse isn't logged in the right way with trace level
enabled
- Refactor kafkatest docker support
- Rolling upgrade test for streams
- bin/kafka-consumer-groups.sh failing to query offsets
- kafka-run-class.sh doesn't check for empty $CLASSPATH
- Documentation for altering topics
- KafkaConsumer: High memory allocation rate when idle
- KafkaConsumer offsetsForTimes should include partitions in result even
if no offset could be found
- Transformations - tranformations for value broken on tombstone events
- The LastStableOffsetLag metric should be removed when partition is
deleted
- SchemaBuilder does not validate input.
- KafkaConsumer.commitSync throws IllegalStateException: Attempt to
retrieve exception from future which hasn't failed
- Instable test
QueryableStateIntegrationTest.shouldAllowToQueryAfterThreadDied
- With transformations that mutate the topic-partition committing
offsets should to refer to the original topic-partition
- Support Power platform by updating rocksdb version
- SchemaBuilder.type(Schema.Type) should not allow null.
- Processor got uncaught exception: NullPointerException
- Illegal state in SocketServer; attempt to send with another send in
progress
- ConsoleConsumer : --new-consumer option as deprecated
- Group loading regression causing stale metadata/offsets cache
- Streams should not abort transaction when closing zombie task
- Improve error handling in SocketServer to reduce illegal states later
- System test failure due to timeout starting Jmx tool
- KafkaApis.handleWriteTxnMarkerRequest can return
UNSUPPORTED_FOR_MESSAGE_FORMAT error on partition emigration
- Make --new-consumer option as deprecated in all tools
- SerializationException in doSend() masks class cast exception
- ducktape kafka service: do not assume Service contains num_nodes
- Consumer poll loop over the same record after a CorruptRecordException
- Using _DUCKTAPE_OPTIONS has no effect on executing tests
- Transient test failure:
ResetConsumerGroupOffsetTest.testResetOffsetsToZonedDateTime
- Support bulk attributes request on KafkaMbean where some attributes do
not exist
- LogDirFailureTest system test fails
- queryable state window store range scan only returns results from one
store
- Add logging to distinguish between internally and externally initiated
shutdown of Kafka
- Sort processor node based on name suffix in
TopologyDescription.toString()
- Unit test shouldTogglePrepareForBulkLoadDuringRestoreCalls fails
- Auto topic creation causes failure with older clusters
- Bulk Restore Should Handle Deletes
- [streams] 'null' values in state stores
- In Jenkins, kafka.api.SaslSslAdminClientIntegrationTest failed with
org.apache.kafka.common.errors.TimeoutException
- Consumer should ensure consistent ordering between sync and async
offset commits
- System tests get exception RocksDBException: db has more levels than
options.num_levels
- Client-ids are not handled consistently by clients and broker
- KafkaAdminClient thread should be daemon
- Partition.makeLeader() should convert HW to OffsetMetadata before
becoming the leader
- Delete topic and re-create topic immediate will delete the new topic's
timeindex
- ShellTest.testRunProgramWithErrorReturn fails on macOS
- Synchronization issue on flush
- Reassigning a topic's partitions can adversely impact other topics
- Move merge() from StreamsBuilder to KStream
- Kafka server should halt if IBP < 1.0.0 and there is log directory
failure
- org.apache.kafka.streams.state.internals.Segments#segments method
returns incorrect results when segments were added out of order
- WorkerSinkTask failure exception should include cause
- StoreChangeLogReader needs to restore partitions that were added post
initialization
- SocketServer.processNewResponses should not skip a response if
exception is thrown
- StoreChangelogReader should be resilient to broker-side metadata not
available
- ChangeLoggingWindowBytesStore needs to retain duplicates when writing
to the log
- KafkaStreams state transitions not correct
- Speedup broker startup after unclean shutdown by reducing unnecessary
snapshot files deletion
- Reset thread interrupt state in case of InterruptedException
- ReassignPartitionsCommand fails if default throttle/timeout used
- Mx4jLoader.maybeLoad should only be executed if kafka_mx4jenable is
set to true
- Remove ZK dependency from Streams reset tool
- ReplicaFetcherThread should not die due to replica in offline log
directory
- Expiring batches with idempotence enabled could cause data loss.
- Kafka Connect applications should log info message when starting up
- Transient failure in SslSelectorTest.testMuteOnOOM
- Controller should read the latest IsrChangeNotification znodes when
handling IsrChangeNotification event
- CompositeReadOnlyWindowStore range fetch doesn't return all values
when fetching with different start and end times
- Fix minor typos and errors in the Kafka Streams turotial
- Connect classloader isolation may be broken for JDBC drivers
- Failure in Connect system test: ConnectRestApiTest
- StreamBuilder#table and StreamsBuilder#globalTable should use serdes
from Materialized
- Producer IllegalStateException due to second deallocate after aborting
a batch
- NPE in NetworkClient
- Producer uses unsupported ProduceRequest version against older brokers
- Ineffective check of negative value in
CompositeReadOnlyKeyValueStore#approximateNumEntries()
- Remove all broker metrics during shutdown
- Deadlock due to locking of DelayedProduce and group
- RequestChannel.sendReponse records incorrect size for NetworkSend with
TRACE logging
- kip-120 changed internal streams topic naming
- FailOnInvalidTimestamp does not log error
- Streams State Restoration never completes when logging is disabled
- disableLogging() causes partitions to not be consumed
- Rename AlterReplicaDir to AlterReplicaDirs
- Replication Fetcher thread for a partition with no data fails to start
- Reject JoinGroup request from first member with empty protocol
type/protocol list
- Kafka Connect: Unsanitized workerID causes exception during startup
- Transient failure:
MemoryRecordsBuilderTest.convertToV1WithMixedV0AndV2Data
- NoSuchElementException in markErrorMeter during TransactionsBounceTest
- NPE in RecordAccumulator
- Use the idempotent producer in the reassign_partitions_test
- KafkaFuture timeout fails to fire if a narrow race condition is hit
- System test failure: LogDirFailureTest
- Integer overflow in log cleaner cleanable ratio computation
- Kafka Request Handler deadlocks and brings down the cluster.
- NoSuchMethodError when creating ProducerRecord in upgrade system tests
- Running tools on Windows fail due to typo in JVM config
- ducker-ak: add ipaddress and enum34 dependencies to docker image
- Scanning plugin.path needs to support relative symlinks
- replica dir not deleted after topic deletion
- Streams quick start crashes Java on Windows
- Reconnecting to broker does not exponentially backoff
- Major performance issue due to excessive logging during leader
election
- Silent Data Loss in Kafka011 Transactional Producer
- Transaction markers are sometimes discarded if txns complete
concurrently
- High memory usage on controller during partition reassignment
Task
- Update Jackson dependencies (from 2.8.5 to 2.9.x)
- Enable PowerMock tests when running on Java 9
- Upgrade RocksDB dependency to legally acceptable version
Test
- Support passing ZK chroot in system tests
- Add unit tests for ClusterConnectionStates
Sub-task
- Replace ControlledShutdown{Request,Response} with
org.apache.kafka.common.requests equivalent
- Remove temporary mapping to deserialize functions in RequestChannel
- Task migration during rebalance callback process could lead the
obsoleted task's IllegalStateException
- Improve test coverage of StreamsKafkaClient
- Improve test coverage of RocksDBStore
- Improve test coverage of CompositeReadOnlySessionStore
- Improve test coverage of CompositeReadOnlyKeyValueStore
- Improve test coverage InMemoryKeyValueLoggedStore
- Improve test coverage of CachingKeyValueStore
- Improve test coverage UsePreviousTimeOnInvalidTimestamp
- Optimize RocksDb config for fast recovery/bulk load
- KeyValueStore.put may throw exception unrelated to the current put
attempt
- Replace zkClient.getChildren method with zkUtils.getChildren
- Options for handling corrupt data during deserialization
- Improve exception handling for state stores
- Distinguish abortable failures in transactional producer
- Unexpected state transition Dead to PendingShutdown
- Replace zkClient.subscribe*Changes method with an equivalent zkUtils
method
- Replace zkClient.exists method with zkUtils.pathExists
- Replace zkClient.delete* method with an equivalent zkUtils method
- Idempotent producer should not require
max.in.flight.requests.per.connection=1
- introduce async ZookeeperClient
- Check exception handling in all streams tests
- Autogenerate Producer sender metrics
- Provide a simple way for custom storage engines to use streams wrapped
stores (KIP-182)
- Add new API methods to KTable
- Add new API methods to KGroupedStream
- Add new API methods to KGroupedTable
- Add Topology and deprecate TopologyBuilder
- Add StreamsBuilder and deprecate KStreamBuilder
- Refactor KeyValueStore hierarchy so that MeteredKeyValueStore is the
outermost store
- Refactor WindowStore hierarchy so that Metered Store is the outermost
store
- Add the archetype project along with "write applications" web docs.
- Refactor SessionStore hierarchy
- Add the Trogdor fault injection daemon
- Add ducktape integration for the Trogdor Fault injection daemon
- Tighten up situations where OutOfOrderSequence may be returned
- Fix transient unit test failure in trogdor coordinator shutdown
- Add Printed class and KStream#print(Printed)
- Add Produced class and new to and through overloads to KStream
- Add Serialized class and KStream groupBy and groupByKey overloads
- Add Joined class and relevant KStream join overloads
- Update Docs
- Add Consumed class and overloads to StreamBuilder
- Upgrade Guide doc changes for KIP-130
- Add groupBy(KeyValueMapper, Serialized) to KTable
- Add filter, filterNot, mapValues and Materialized to KTable
- Add WindowedKStream interface and implemenation
- Add Materialized overloads to StreamBuilder
- Transactions system test should check for message order
- Create Connect metrics for connectors
- Create Connect metrics common to source and sink tasks
- Create Connect metrics for source tasks
- Create Connect metrics for sink tasks
- Create Connect metrics for workers
- Create Connect metrics for worker rebalances
- Add `hasOffset` and `hasTimestamp` methods to RecordMetadata to
indicate whether metadata is available
- Add Materialized overloads to WindowedKStream
- Add SessionWindowedKStream
- Deprecate KTable#to and KTable#through
- Improve ProcessorStateManager exception handling
- User Callback Exceptions need to be handled properly
- Add generated documentation for Connect metrics
ef550bfea3