-- producer
A Kafka client that publishes records to the Kafka cluster.
The producer is <i>thread safe</i> and sharing a single producer instance across threads will generally be faster than
having multiple instances.
Here is a simple example of using the producer to send records with strings containing sequential numbers as the key/value
pairs.
{@code
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("acks", "all");
props.put("retries", 0);
props.put("linger.ms", 1);
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
Producer<String, String> producer = new KafkaProducer<>(props);
for (int i = 0; i < 100; i++)
producer.send(new ProducerRecord<String, String>("my-topic", Integer.toString(i), Integer.toString(i)));
producer.close();
}
The producer consists of a pool of buffer space that holds records that haven't yet been transmitted to the server
as well as a background I/O thread that is responsible for turning these records into requests and transmitting them
to the cluster. Failure to close the producer after use will leak these resources.
The {@link #send(ProducerRecord) send()} method is asynchronous. When called it adds the record to a buffer of pending record sends
and immediately returns. This allows the producer to batch together individual records for efficiency.
The <code>acks</code> config controls the criteria under which requests are considered complete. The "all" setting
we have specified will result in blocking on the full commit of the record, the slowest but most durable setting.
If the request fails, the producer can automatically retry, though since we have specified <code>retries</code>
as 0 it won't. Enabling retries also opens up the possibility of duplicates (see the documentation on
<a href="http://kafka.apache.org/documentation.html#semantics">message delivery semantics</a> for details).
The producer maintains buffers of unsent records for each partition. These buffers are of a size specified by
the <code>batch.size</code> config. Making this larger can result in more batching, but requires more memory (since we will
generally have one of these buffers for each active partition).
By default a buffer is available to send immediately even if there is additional unused space in the buffer. However if you
want to reduce the number of requests you can set <code>linger.ms</code> to something greater than 0. This will
instruct the producer to wait up to that number of milliseconds before sending a request in hope that more records will
arrive to fill up the same batch. This is analogous to Nagle's algorithm in TCP. For example, in the code snippet above,
likely all 100 records would be sent in a single request since we set our linger time to 1 millisecond. However this setting
would add 1 millisecond of latency to our request waiting for more records to arrive if we didn't fill up the buffer. Note that
records that arrive close together in time will generally batch together even with <code>linger.ms=0</code> so under heavy load
batching will occur regardless of the linger configuration; however setting this to something larger than 0 can lead to fewer, more
efficient requests when not under maximal load at the cost of a small amount of latency.
The <code>buffer.memory</code> controls the total amount of memory available to the producer for buffering. If records
are sent faster than they can be transmitted to the server then this buffer space will be exhausted. When the buffer space is
exhausted additional send calls will block. The threshold for time to block is determined by <code>max.block.ms</code> after which it throws
a TimeoutException.
The <code>key.serializer</code> and <code>value.serializer</code> instruct how to turn the key and value objects the user provides with
their <code>ProducerRecord</code> into bytes. You can use the included {@link org.apache.kafka.common.serialization.ByteArraySerializer} or
{@link org.apache.kafka.common.serialization.StringSerializer} for simple string or byte types.
From Kafka 0.11, the KafkaProducer supports two additional modes: the idempotent producer and the transactional producer.
The idempotent producer strengthens Kafka's delivery semantics from at least once to exactly once delivery. In particular
producer retries will no longer introduce duplicates. The transactional producer allows an application to send messages
to multiple partitions (and topics!) atomically.
To enable idempotence, the <code>enable.idempotence</code> configuration must be set to true. If set, the
<code>retries</code> config will default to <code>Integer.MAX_VALUE</code> and the <code>acks</code> config will
default to <code>all</code>. There are no API changes for the idempotent producer, so existing applications will
not need to be modified to take advantage of this feature.
To take advantage of the idempotent producer, it is imperative to avoid application level re-sends since these cannot
be de-duplicated. As such, if an application enables idempotence, it is recommended to leave the <code>retries</code>
config unset, as it will be defaulted to <code>Integer.MAX_VALUE</code>. Additionally, if a {@link #send(ProducerRecord)}
returns an error even with infinite retries (for instance if the message expires in the buffer before being sent),
then it is recommended to shut down the producer and check the contents of the last produced message to ensure that
it is not duplicated. Finally, the producer can only guarantee idempotence for messages sent within a single session.
To use the transactional producer and the attendant APIs, you must set the <code>transactional.id</code>
configuration property. If the <code>transactional.id</code> is set, idempotence is automatically enabled along with
the producer configs which idempotence depends on. Further, topics which are included in transactions should be configured
for durability. In particular, the <code>replication.factor</code> should be at least <code>3</code>, and the
<code>min.insync.replicas</code> for these topics should be set to 2. Finally, in order for transactional guarantees
to be realized from end-to-end, the consumers must be configured to read only committed messages as well.
The purpose of the <code>transactional.id</code> is to enable transaction recovery across multiple sessions of a
single producer instance. It would typically be derived from the shard identifier in a partitioned, stateful, application.
As such, it should be unique to each producer instance running within a partitioned application.
All the new transactional APIs are blocking and will throw exceptions on failure. The example
below illustrates how the new APIs are meant to be used. It is similar to the example above, except that all
100 messages are part of a single transaction.
{@code
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("transactional.id", "my-transactional-id");
Producer<String, String> producer = new KafkaProducer<>(props, new StringSerializer(), new StringSerializer());
producer.initTransactions();
try {
producer.beginTransaction();
for (int i = 0; i < 100; i++)
producer.send(new ProducerRecord<>("my-topic", Integer.toString(i), Integer.toString(i)));
producer.commitTransaction();
} catch (ProducerFencedException | OutOfOrderSequenceException | AuthorizationException e) {
// We can't recover from these exceptions, so our only option is to close the producer and exit.
producer.close();
} catch (KafkaException e) {
// For all other exceptions, just abort the transaction and try again.
producer.abortTransaction();
}
producer.close();
}
As is hinted at in the example, there can be only one open transaction per producer. All messages sent between the
{@link #beginTransaction()} and {@link #commitTransaction()} calls will be part of a single transaction. When the
<code>transactional.id</code> is specified, all messages sent by the producer must be part of a transaction.
The transactional producer uses exceptions to communicate error states. In particular, it is not required
to specify callbacks for <code>producer.send()</code> or to call <code>.get()</code> on the returned Future: a
<code>KafkaException</code> would be thrown if any of the
<code>producer.send()</code> or transactional calls hit an irrecoverable error during a transaction. See the {@link #send(ProducerRecord)}
documentation for more details about detecting errors from a transactional send.
By calling
<code>producer.abortTransaction()</code> upon receiving a <code>KafkaException</code> we can ensure that any
successful writes are marked as aborted, hence keeping the transactional guarantees.
This client can communicate with brokers that are version 0.10.0 or newer. Older or newer brokers may not support
certain client features. For instance, the transactional APIs need broker versions 0.11.0 or later. You will receive an
<code>UnsupportedVersionException</code> when invoking an API that is not available in the running broker version.