开发者社区> 问答> 正文

canal设置ZooKeeperMetaManager

1.canal在zookeeper在切换HA时,会存在重复消费问题

2.canal不能很好的修改去position

基于以上两点,想要切换至ZooKeeperMetaManager,故目前通过修改XXX-instance.xml来进行,但会报错:org.I0Itec.zkclient.exception.ZkNoNodeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /otter/canal/destinations/example/1001/mark,即使手动添上节点后,元数据依旧没有只存在zokeeper中,还是先存在内存然后定时刷到zk中的

@agapple 请问,解决上述两个问题有更好的方式吗?如何可以更优雅的指定position的位置来进行消费

完整的错误: exception=org.I0Itec.zkclient.exception.ZkNoNodeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /otter/canal/destinations/example/1001/mark at org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47) at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685) at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:413) at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:409) at com.alibaba.otter.canal.meta.ZooKeeperMetaManager.clearAllBatchs(ZooKeeperMetaManager.java:231) at com.alibaba.otter.canal.server.embedded.CanalServerWithEmbedded.rollback(CanalServerWithEmbedded.java:406) at com.alibaba.otter.canal.server.netty.handler.SessionHandler.messageReceived(SessionHandler.java:182) at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:100) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783) at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:154) at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:100) at org.jboss.netty.handler.timeout.IdleStateAwareChannelHandler.handleUpstream(IdleStateAwareChannelHandler.java:48) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783) at org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:275) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.unfoldAndFireMessageReceived(ReplayingDecoder.java:525) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:506) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:443) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349) at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /otter/canal/destinations/example/1001/mark at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1496) at com.alibaba.otter.canal.common.zookeeper.ZooKeeperx.getChildren(ZooKeeperx.java:107) at org.I0Itec.zkclient.ZkClient$2.call(ZkClient.java:416) at org.I0Itec.zkclient.ZkClient$2.call(ZkClient.java:413) at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675) ... 34 more

如果元数据只存在zk中,在重复的这个问题上会稍微好一些,但程序整体的效率会减低,这是取舍的问题,这个理解。

另外想请问一下, canal 1.0.24 想要切换至ZooKeeperMetaManager,是否可以通过修改XXX-instance.xml来进行,虽然这样会在一定的程度上降低程序的效率(尝试过但失败了)?

另外是否可以有办法优雅的指定位置来进行消费,不通过重启,比如只是修改zk中position,即可实现?

原提问者GitHub用户shengjk

展开
收起
绿子直子 2023-05-09 11:58:25 85 0
1 条回答
写回答
取消 提交回答
  • 理论设计上就没法保证不重复 目前暂时没有这样的API可以直接指定位点启动

    原回答者GitHub用户agapple

    2023-05-10 10:25:41
    赞同 展开评论 打赏
问答分类:
问答地址:
问答排行榜
最热
最新

相关电子书

更多
低代码开发师(初级)实战教程 立即下载
冬季实战营第三期:MySQL数据库进阶实战 立即下载
阿里巴巴DevOps 最佳实践手册 立即下载