开发者社区> 问答> 正文

基于k8s安装的flink无法访问hdfs

flink基于k8s安装,hadoop原生安装集群,将任务放到flink上执行的时候,报错不能使用checkpoint,详细报错如下: Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 'hdfs'. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded. For a full list of supported file systems, please see https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/. at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:530) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:407) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.runtime.state.filesystem.FsCheckpointStorageAccess. (FsCheckpointStorageAccess.java:64) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.runtime.state.filesystem.FsStateBackend.createCheckpointStorage(FsStateBackend.java:527) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.streaming.runtime.tasks.StreamTask. (StreamTask.java:337) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.streaming.runtime.tasks.StreamTask. (StreamTask.java:304) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.streaming.runtime.tasks.SourceStreamTask. (SourceStreamTask.java:76) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.streaming.runtime.tasks.SourceStreamTask. (SourceStreamTask.java:72) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_292] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_292] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_292] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_292] at org.apache.flink.runtime.taskmanager.Task.loadAndInstantiateInvokable(Task.java:1515) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:727) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.runtime.taskmanager.Task.run(Task.java:563) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_292] Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Hadoop is not in the classpath/dependencies. at org.apache.flink.core.fs.UnsupportedSchemeFactory.create(UnsupportedSchemeFactory.java:55) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:526) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:407) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.runtime.state.filesystem.FsCheckpointStorageAccess. (FsCheckpointStorageAccess.java:64) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.runtime.state.filesystem.FsStateBackend.createCheckpointStorage(FsStateBackend.java:527) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.streaming.runtime.tasks.StreamTask. (StreamTask.java:337) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.streaming.runtime.tasks.StreamTask. (StreamTask.java:304) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.streaming.runtime.tasks.SourceStreamTask. (SourceStreamTask.java:76) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.streaming.runtime.tasks.SourceStreamTask. (SourceStreamTask.java:72) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_292] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_292] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_292] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_292] at org.apache.flink.runtime.taskmanager.Task.loadAndInstantiateInvokable(Task.java:1515) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:727) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at org.apache.flink.runtime.taskmanager.Task.run(Task.java:563) ~[flink-dist_2.12-1.13.0.jar:1.13.0] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_292]

请求大佬看一下,主要是这两个集群,一个是k8s安装的,一个不是。

展开
收起
游客v4spwg6razmti 2021-09-27 17:19:27 1016 0
0 条回答
写回答
取消 提交回答
问答排行榜
最热
最新

相关电子书

更多
Flink CDC Meetup PPT - 龚中强 立即下载
Flink CDC Meetup PPT - 王赫 立即下载
Flink CDC Meetup PPT - 覃立辉 立即下载

相关镜像