2019-04-11 16:49:30.532 [pool-2-thread-1] ERROR c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - java.lang.RuntimeException: java.lang.RuntimeException: TransportSerializationException[Failed to deserialize response from handler [org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler]]; nested: IllegalArgumentException[Unknown NamedWriteable [org.elasticsearch.cluster.metadata.MetaData$Custom][ml]]; java.lang.RuntimeException: java.lang.RuntimeException: java.lang.RuntimeException: TransportSerializationException[Failed to deserialize response from handler [org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler]]; nested: IllegalArgumentException[Unknown NamedWriteable [org.elasticsearch.cluster.metadata.MetaData$Custom][ml]]; at com.alibaba.otter.canal.client.adapter.es.service.ESSyncService.sync(ESSyncService.java:113) at com.alibaba.otter.canal.client.adapter.es.service.ESSyncService.sync(ESSyncService.java:61) at com.alibaba.otter.canal.client.adapter.es.ESAdapter.sync(ESAdapter.java:176) at com.alibaba.otter.canal.client.adapter.es.ESAdapter.sync(ESAdapter.java:155) at com.alibaba.otter.canal.adapter.launcher.loader.AbstractCanalAdapterWorker.batchSync(AbstractCanalAdapterWorker.java:198) at com.alibaba.otter.canal.adapter.launcher.loader.AbstractCanalAdapterWorker.lambda$null$1(AbstractCanalAdapterWorker.java:62) at java.util.ArrayList.forEach(ArrayList.java:1249) at com.alibaba.otter.canal.adapter.launcher.loader.AbstractCanalAdapterWorker.lambda$null$2(AbstractCanalAdapterWorker.java:58) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: java.lang.RuntimeException: TransportSerializationException[Failed to deserialize response from handler [org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler]]; nested: IllegalArgumentException[Unknown NamedWriteable [org.elasticsearch.cluster.metadata.MetaData$Custom][ml]]; at com.alibaba.otter.canal.client.adapter.support.Util.sqlRS(Util.java:47) at com.alibaba.otter.canal.client.adapter.es.service.ESSyncService.mainTableInsert(ESSyncService.java:468) at com.alibaba.otter.canal.client.adapter.es.service.ESSyncService.insert(ESSyncService.java:140) at com.alibaba.otter.canal.client.adapter.es.service.ESSyncService.sync(ESSyncService.java:96) ... 11 common frames omitted Caused by: java.lang.RuntimeException: TransportSerializationException[Failed to deserialize response from handler [org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler]]; nested: IllegalArgumentException[Unknown NamedWriteable [org.elasticsearch.cluster.metadata.MetaData$Custom][ml]]; at com.alibaba.otter.canal.client.adapter.es.service.ESSyncService.lambda$mainTableInsert$3(ESSyncService.java:485) at com.alibaba.otter.canal.client.adapter.support.Util.sqlRS(Util.java:43) ... 14 common frames omitted Caused by: org.elasticsearch.transport.TransportSerializationException: Failed to deserialize response from handler [org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler] at org.elasticsearch.transport.TcpTransport.handleResponse(TcpTransport.java:1441) at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1400) at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:64) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ... 1 common frames omitted Caused by: java.lang.IllegalArgumentException: Unknown NamedWriteable [org.elasticsearch.cluster.metadata.MetaData$Custom][ml] at org.elasticsearch.common.io.stream.NamedWriteableRegistry.getReader(NamedWriteableRegistry.java:112) at org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteable(NamedWriteableAwareStreamInput.java:45) at org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteable(NamedWriteableAwareStreamInput.java:39) at org.elasticsearch.cluster.metadata.MetaData.readFrom(MetaData.java:775) at org.elasticsearch.cluster.ClusterState.readFrom(ClusterState.java:668) at org.elasticsearch.action.admin.cluster.state.ClusterStateResponse.readFrom(ClusterStateResponse.java:81) at org.elasticsearch.transport.TransportResponseHandler.read(TransportResponseHandler.java:47) at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1085) at org.elasticsearch.transport.TcpTransport.handleResponse(TcpTransport.java:1437) ... 27 common frames omitted 2019-04-11 16:49:30.534 [Thread-4] ERROR c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - Outer adapter sync failed! Error sync but ACK! 2019-04-11 16:49:30.534 [elasticsearch[client][transport_client_boss][T#3]] WARN org.elasticsearch.transport.netty4.Netty4Transport - exception caught on transport layer [NettyTcpChannel{localAddress=/192.168.100.207:59176, remoteAddress=/192.168.100.204:9300}], closing connection java.lang.IllegalStateException: Message not fully read (response) for requestId [13], handler [org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler/org.elasticsearch.action.ActionListenerResponseHandler@58cac2ff], error [false]; resetting at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1407) at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:64) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at java.lang.Thread.run(Thread.java:748)
elasticsearch server版本号是6.2.4 Canal server版本号1.1.3 Canal-adapter版本号1.1.3
mysql数据变化时,canal-adapter得到了数据, 但是向elasticsearch同步数据时,抛异常了
原提问者GitHub用户zhuxiaokang0223
应该是 es 配置里面,设置的 _id : id 造成的吧,你的sql语句里面最后select出来的实际上只有 _id字段
原回答者GitHub用户simonlei
根据您提供的信息,Canal-adapter-1.1.3 在将 MySQL 数据同步到 Elasticsearch 时,出现了 TransportSerializationException 异常。这个异常通常是由 Elasticsearch 版本不兼容引起的。具体而言,Elasticsearch 6.x 版本中引入了 NamedWriteable 接口,用于序列化和反序列化 Elasticsearch 中的对象。如果使用的 Elasticsearch 版本低于 6.x,就会出现 TransportSerializationException 异常。
解决这个问题的方法是,将 Elasticsearch 升级到 6.x 或以上版本。同时,您需要确保 Canal-adapter-1.1.3 和 Elasticsearch 版本兼容。如果您使用的是 Canal-adapter-1.1.3,建议将 Elasticsearch 升级到 6.x 版本,并使用 Elasticsearch 6.x 版本的 Java API 进行开发。如果您使用的是其他版本的 Canal-adapter,建议查看对应版本的文档,以确定与 Elasticsearch 版本的兼容性。
另外,如果您已经将 Elasticsearch 升级到 6.x 或以上版本,但仍然出现 TransportSerializationException 异常,可能是由于 Elasticsearch 集群中存在不同版本的节点,导致序列化和反序列化出现问题。这种情况下,您需要检查 Elasticsearch 集群中的节点版本是否一致,并确保所有节点都升级到了 6.x 或以上版本。
版权声明:本文内容由阿里云实名注册用户自发贡献,版权归原作者所有,阿里云开发者社区不拥有其著作权,亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容,填写侵权投诉表单进行举报,一经查实,本社区将立刻删除涉嫌侵权内容。