Spark - ReturnStatementInClosureException: Return statements aren‘t allowed in Spark closures

简介: Spark 使用 RDD 调用 Filter 函数时,dirver 端卡住,报错 ReturnStatementInClosureException: Return statements aren't allowed in Spark closures,即闭包内无法使用 return 函数。

 一.引言

Spark 使用 RDD 调用 Filter 函数时,dirver 端卡住,报错 ReturnStatementInClosureException: Return statements aren't allowed in Spark closures,即闭包内无法使用 return 函数:

image.gif编辑

二.使用场景

使用 rdd.filter 方法过滤 id 时使用了 return 方法,导致上述报错:

rdd.filter(arr => {
       val id = arr(0)
       val l = id.length()
       if (l <= 8) return false
       if (id.startsWith("1")) {
           true
       } else {
           false
       }
     })

image.gif

修改方案:

闭包函数内不使用 return 关键字即可

rdd.filter(arr => {
        val id = arr(0)
        val l = id.length()
        if (l <= 8) {
            false
        } else if (id.startsWith("1")) {
            true
        } else {
            false
        }
    })

image.gif


目录
相关文章
|
6月前
|
分布式计算 安全 Java
Spark 编译出现 InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
Spark 编译出现 InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
166 0
|
6月前
|
SQL 分布式计算 HIVE
[已解决]Job failed with org.apache.spark.SparkException: Job aborted due to stage failure: Task 3 in st
[已解决]Job failed with org.apache.spark.SparkException: Job aborted due to stage failure: Task 3 in st
304 0
|
6月前
|
资源调度 Shell 开发工具
The token '&&' is not a valid statement separator in this version
The token '&&' is not a valid statement separator in this version
106 0
|
SQL 数据库
this is incompatible with sql_mode=only_full_group_by
this is incompatible with sql_mode=only_full_group_by
86 0
|
SQL 分布式计算 Spark
|
分布式计算 Java Spark
Optimizing Spark job parameters
Optimizing Spark job parameters
257 0
|
关系型数据库 PostgreSQL
PostgreSQL - ERROR: could not determine data type of parameter $1
PostgreSQL - ERROR: could not determine data type of parameter $1
1969 0
my task doorway - where is it parsed
Created by Wang, Jerry on Apr 03, 2015
my task doorway - where is it parsed
成功解决TypeError: Value passed to parameter 'paddings' has DataType float32 not in list of allowed valu
成功解决TypeError: Value passed to parameter 'paddings' has DataType float32 not in list of allowed valu