如何修复Kafka connect集群启动错误?

Confluent

1 前言

一个问题,一篇文章,一出故事。
笔者从新初始化Kafka集群后出现Kafka connect服务器无法启动的情况,于是手动启动服务发现如下报错。

2 最佳实践

2.1 手动启动服务排错

sudo -u cp-kafka-connect /usr/bin/connect-distributed /etc/kafka/connect-distributed.properties

错误信息如下,

[2022-06-27 10:13:19,582] ERROR [Worker clientId=connect-1, groupId=connect-cluster] Uncaught exception in herder work thread, exiting:  (org.apache.kafka.connect.runtime.distributed.DistributedHerder:334)
org.apache.kafka.common.config.ConfigException: Topic 'connect-offsets' supplied via the 'status.storage.topic' property is required to have 'cleanup.policy=compact' to guarantee consistency and durability of connector and task statuses, but found the topic currently has 'cleanup.policy=delete'. Continuing would likely result in eventually losing connector and task statuses and problems restarting this Connect cluster in the future. Change the 'status.storage.topic' property in the Connect worker configurations to use a topic with 'cleanup.policy=compact'.

[2022-06-27 10:14:19,583] ERROR [Worker clientId=connect-1, groupId=connect-cluster] Uncaught exception in herder work thread, exiting:  (org.apache.kafka.connect.runtime.distributed.DistributedHerder:334)
org.apache.kafka.common.config.ConfigException: Topic 'connect-status' supplied via the 'status.storage.topic' property is required to have 'cleanup.policy=compact' to guarantee consistency and durability of connector and task statuses, but found the topic currently has 'cleanup.policy=delete'. Continuing would likely result in eventually losing connector and task statuses and problems restarting this Connect cluster in the future. Change the 'status.storage.topic' property in the Connect worker configurations to use a topic with 'cleanup.policy=compact'.

[2022-06-27 10:16:28,480] ERROR [Worker clientId=connect-1, groupId=connect-cluster] Uncaught exception in herder work thread, exiting:  (org.apache.kafka.connect.runtime.distributed.DistributedHerder:334)
org.apache.kafka.common.config.ConfigException: Topic 'connect-configs' supplied via the 'config.storage.topic' property is required to have 'cleanup.policy=compact' to guarantee consistency and durability of connector configurations, but found the topic currently has 'cleanup.policy=delete'. Continuing would likely result in eventually losing connector configurations and problems restarting this Connect cluster in the future. Change the 'config.storage.topic' property in the Connect worker configurations to use a topic with 'cleanup.policy=compact'.

2.2 查询报错的Topis

kafka-topics --bootstrap-server cfkafka02:9092,cfkafka03:9092,cfkafka04:9092 --describe --topic connect-offsets
kafka-topics --bootstrap-server cfkafka02:9092,cfkafka03:9092,cfkafka04:9092 --describe --topic connect-status
kafka-topics --bootstrap-server cfkafka02:9092,cfkafka03:9092,cfkafka04:9092 --describe --topic connect-configs

2.3 删除报错的Topis

kafka-topics --bootstrap-server cfkafka02:9092,cfkafka03:9092,cfkafka04:9092 --delete --topic connect-offsets
kafka-topics --bootstrap-server cfkafka02:9092,cfkafka03:9092,cfkafka04:9092 --delete --topic connect-status
kafka-topics --bootstrap-server cfkafka02:9092,cfkafka03:9092,cfkafka04:9092 --delete --topic connect-configs

注:至此,故障解决!

参阅文档
===================
https://github.com/confluentinc/kafka-connect-jdbc/issues/189

没有评论

发表评论

Confluent
如何更新Kafka topic的Replication Factor?

1 前言 一个问题,一篇文章,一出故事。 笔者部署完Confluent Kafka集群后发现Kafk …

Confluent
如何部署Confluent ksqlDB集群?

1 基础知识 1.1 KSQL 1.1.1 KSQL的概念 – KSQL与SQL数据库完 …

Confluent
如何修复Kafka主题connect-configs报错?

1 前言 一个问题,一篇文章,一出故事。 笔者安装完成Kafka connect后发现Kafka疯狂 …