Flink BlobServerConnection NoSuchFileException

我们在使用flink-stream时遇到了如下问题，有谁可以帮助一下吗？怎么避免该问题呢？

flink版本：flink-1.8.1 standalone cluster

node4是我们集群中的一个slave节点，我们的任务运行了差不多1个月才出现的这个问题。这个异常导致任务全部处于失败然后恢复失败的循环中，目前不知道什么原因造成的，查看了 /tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/文件夹是空的。

谢谢。

2019-09-19 16:54:34,328 ERROR org.apache.flink.runtime.blob.BlobServerConnection - GET operation failed for BLOB ba2fb0924f20a093cff50b27aa8ea6c1/p-105986c82d33c1ed56863caabc5f9b6517bc16c3-2122fe23e729d1ea1d0a4f13f16d1e28 from /node04. java.nio.file.NoSuchFileException: /tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/temp-03506073 at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409) at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) at java.nio.file.Files.move(Files.java:1395) at org.apache.flink.runtime.blob.BlobUtils.moveTempFileToStore(BlobUtils.java:429) at org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:497) at org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231) at org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117) 2019-09-19 16:54:34,330 ERROR org.apache.flink.runtime.blob.BlobServerConnection - GET operation failed for BLOB ba2fb0924f20a093cff50b27aa8ea6c1/p-105986c82d33c1ed56863caabc5f9b6517bc16c3-2122fe23e729d1ea1d0a4f13f16d1e28 from /node04 . java.nio.file.NoSuchFileException: /tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/temp-03506074 at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409) at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) at java.nio.file.Files.move(Files.java:1395) at org.apache.flink.runtime.blob.BlobUtils.moveTempFileToStore(BlobUtils.java:429) at org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:497) at org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231) at org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117) 2019-09-19 16:54:34,331 ERROR org.apache.flink.runtime.blob.BlobServerConnection - GET operation failed for BLOB ba2fb0924f20a093cff50b27aa8ea6c1/p-105986c82d33c1ed56863caabc5f9b6517bc16c3-2122fe23e729d1ea1d0a4f13f16d1e28 from /node04 . java.nio.file.NoSuchFileException: /tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/temp-03506075 at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409) at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) at java.nio.file.Files.move(Files.java:1395) at org.apache.flink.runtime.blob.BlobUtils.moveTempFileToStore(BlobUtils.java:429) at org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:497) at org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231) at org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117) 2019-09-19 16:54:34,332 ERROR org.apache.flink.runtime.blob.BlobServerConnection - GET operation failed for BLOB ba2fb0924f20a093cff50b27aa8ea6c1/p-105986c82d33c1ed56863caabc5f9b6517bc16c3-2122fe23e729d1ea1d0a4f13f16d1e28 from /node04 . java.nio.file.NoSuchFileException: /tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/temp-03506076 at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409) at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) at java.nio.file.Files.move(Files.java:1395) at org.apache.flink.runtime.blob.BlobUtils.moveTempFileToStore(BlobUtils.java:429) at org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:497) at org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231) at org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117)*来自志愿者整理的flink邮件归档