"Flink CDC中这个问题怎么解决?Failed to deserialize data of EventHeaderV4这个问题。
2023-07-15 05:40:09,119 ERROR io.debezium.pipeline.ErrorHandler [] - Producer failure
io.debezium.DebeziumException: Failed to deserialize data of EventHeaderV4{timestamp=1689370731000, eventType=ROWS_QUERY, serverId=683728822, headerLength=19, dataLength=311, nextPosition=206875527, flags=128}
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.wrap(MySqlStreamingChangeEventSource.java:1489) ~[blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$ReaderThreadLifecycleListener.onCommunicationFailure(MySqlStreamingChangeEventSource.java:1545) [blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1079) [blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:631) [blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:932) [blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
Caused by: com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1689370731000, eventType=ROWS_QUERY, serverId=683728822, headerLength=19, dataLength=311, nextPosition=206875527, flags=128}
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:341) ~[blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:244) ~[blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$1.nextEvent(MySqlStreamingChangeEventSource.java:260) ~[blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1051) ~[blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]
... 3 more
Caused by: java.io.EOFException: Failed to read remaining 72 of 306 bytes from position 3501350. Block length: 72. Initial block length: 307.
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.fill(ByteArrayInputStream.java:115) ~[blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.read(ByteArrayInputStream.java:105) ~[blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.readString(ByteArrayInputStream.java:78) ~[blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]
at com.github.shyiko.mysql.binlog.event.deserialization.RowsQueryEventDataDeserializer.deserialize(RowsQueryEventDataDeserializer.java:31) ~[blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]
at com.github.shyiko.mysql.binlog.event.deserialization.RowsQueryEventDataDeserializer.deserialize(RowsQueryEventDataDeserializer.java:25) ~[blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:335) ~[blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:244) ~[blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$1.nextEvent(MySqlStreamingChangeEventSource.java:260) ~[blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1051) ~[blob_p-daa0aaf6b57c704eb04b1630334fb2252930099a-5e7585dcfc95383a2ad6ee2b9c7f083f:?]"
在 Flink CDC 中,"Failed to deserialize data" 错误通常表示在将数据反序列化为对象时出现了问题。可能原因包括:
数据格式不正确:数据源发送的数据格式可能不符合期望的格式,或者与 Flink 应用程序中定义的数据类型不匹配,导致反序列化失败。
数据类型不匹配:Flink 应用程序中定义的数据类型与实际数据类型不匹配,导致反序列化失败。
序列化/反序列化库版本不匹配:Flink 应用程序使用的序列化/反序列化库版本与数据源发送的数据使用的版本不匹配,导致反序列化失败。
把binlog清了再启动任务看下。sql可以的话,把你的ds代码打个包到集群运行,直print连接器,看看有么有报错,不上生产。 sql要是没问题,大包包ds的代码上集群运行看看有没有报错。此回答整理至钉群“Flink CDC 社区”
这个错误提示表明 Flink CDC 在反序列化 EventHeaderV4 数据时失败了。可能是因为数据格式不正确或者缺少必要的字段。可以尝试检查MySQL binlog 格式是否正确,并确保所有必要的字段都存在。另外,也可以查看 Flink CDC 的文档,了解如何正确配置 MySQL CDC。
版权声明:本文内容由阿里云实名注册用户自发贡献,版权归原作者所有,阿里云开发者社区不拥有其著作权,亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容,填写侵权投诉表单进行举报,一经查实,本社区将立刻删除涉嫌侵权内容。
实时计算Flink版是阿里云提供的全托管Serverless Flink云服务,基于 Apache Flink 构建的企业级、高性能实时大数据处理系统。提供全托管版 Flink 集群和引擎,提高作业开发运维效率。