卡夫卡消费者启动延迟汇合点网
问题说明
启动融合的dotnet使用者时，在调用订阅和随后的轮询之后，似乎需要很长时间才能从服务器接收分配分区"事件，因此会收到消息(大约10-15秒)
When starting up a confluent-dotnet consumer , after the call to subscribe and subsequent polling, it seems to take a very long time to receive the "Partition assigned" event from the server, and therefore messages (about 10-15sec).
起初我以为会有自动创建主题的开销，但是无论是否已经存在消费者的主题/消费者组，时间都是相同的.
At first I thought there was a auto topic creation overhead, but the time is the same whether the topic/consumer group of the consumer already exist or not.
我从这个配置启动我的使用者，其余的代码与汇合的高级使用者示例中的代码相同:
I start my consumer with this config, the rest of the code is the same as in the confluent advanced consumer example :
var kafkaConfig = new Dictionary<string, object>
{
{"group.id", config.ConsumerGroup},
{"statistics.interval.ms", 60000},
{"fetch.wait.max.ms", 10},
{"bootstrap.servers", config.BrokerList},
{"enable.auto.commit", config.AutoCommit},
{"socket.blocking.max.ms",1},
{"fetch.error.backoff.ms",1 },
{"socket.nagle.disable",true },
{"auto.commit.interval.ms", 5000},
{
"default.topic.config", new Dictionary<string, object>()
{
{"auto.offset.reset", "smallest"}
}
}
};
kafka群集由具有默认设置的远程数据中心中的3台中低端规格机器组成. 是否可以调整代理或客户端设置以减少启动时间?
The kafka cluster consists of 3 low-mid spec machines in a remote datacenter with default settings. Is there a broker or client setting that can be tweaked to lower this startup time?
使用分配"(而不是订阅")为我自己分配分区会导致大约2秒的启动时间
assigning partitions myself with Assign instead of Subscribe results in startup time of around 2sec instead
正确答案
Kafka消费者按设计成组工作-您看到的延迟是组协调员(驻留在群集上，而不是客户端上)等待任何现有/上一个会话超时，并允许在同一组中的任何其他使用者使用前启动，然后再将分区分配给具有活动连接的所有使用者.
Kafka Consumers work in groups by design - the delay you see is the group co-ordinator (which resides on the cluster, not the client side) waiting for any existing/previous session(s) to timeout and to allow any additional consumers in the same group to start before allocating partitions to all the consumers with an active connection.
实际上，如果您足够快地重新启动测试使用者，您会看到延迟跃升到将近30秒，因为session.timeout.ms的默认值为30000，并且群集仍未注意到"以前的使用者直到此超时开始之前都消失了.另外，如果在两次重新启动之间更改group.id，则会看到延迟急剧下降，因为群集不会等待属于另一个组的现有使用者.
In fact, if you re-start your test consumer quickly enough, you'll see that delay jump to almost 30 seconds because session.timeout.ms has a default value of 30000 and the cluster still hasn't "noticed" that the previous consumer has gone until this timeout kicks in. Also if you change group.id between restarts you'll see the delay drop drastically as the cluster won't wait on existing consumers that are part of a different group.
最后，尝试干净地退出您的消费者，然后再次启动(调用Unsubscribe()并确保消费者被处置).
Finally, try cleanly exiting your consumer before firing up again (call Unsubscribe() and make sure the Consumer is disposed).
似乎可以将session.timeout.ms降低到6000，以减少任何现有使用者组连接的超时，但不能降低.
It appears that session.timeout.ms can be lowered to 6000 to reduce the timeout of any existing consumer group connection, but not lower.
即使一切开始干净"，您似乎仍然会延迟多达7秒钟(我猜是标准连接设置，还要等待同一组中的所有其他使用方开始).如果您使用Assign()而不是Subscribe()，那么您将选择自己为用户分配分区，并且自动组平衡不适用.
Even with everything starting "clean" it appears you'll still get a delay of up to 7 seconds (I'm guessing standard connection setup plus waiting for any other consumers in the same group to start). If you use Assign() instead of Subscribe() then you are choosing to assign the partitions to your consumer(s) yourself and automatic group balancing doesn't apply.
这篇好文章是转载于：学新通技术网
