代码用途:简单测试spark-sql的读hdfs文件,然后用sql 简便提取临时数据
报错猜测:个人刚开始怀疑是pom文件依赖的版本有问题,现在也不确定啥问题,具体描述如下:
1:集群环境spark-shell下执行,很正常,可以很欢快的使用,可以查询people结果。。
val sqlContext = new org.apache.spark.sql.SQLContext(sc) case class Person(name: String) var people = sc.textFile("/workspace/xx/tmp/a").map(_.split(" ")).map(p => Person(p(0))) val peopleSchema = sqlContext.createSchemaRDD(people) peopleSchema.registerTempTable("people") var df=sqlContext.sql("select * from people")2:在本地打包时发现报错,不能编译。
import org.apache.spark.sql.SQLContext import org.apache.spark.{SparkContext, SparkConf} import org.apache.spark.SparkContext._ import org.apache.spark.sql import org.apache.spark.sql._ import org.apache.hadoop.fs.Path import org.apache.spark.sql.SchemaRDD object sparkSql { def main(args: Array[String]) { val conf = new SparkConf().setAppName("sql_data") var sc = new SparkContext(conf) val sqlContext = new org.apache.spark.sql.SQLContext(sc) case class Person(name: String) var people = sc.textFile("/workspace/xx/tmp/a").map(_.split(" ")).map(p => Person(p(0))) val peopleSchema = sqlContext.createSchemaRDD(people) peopleSchema.registerTempTable("people") var df=sqlContext.sql("select * from people") df.map(t => t(0)).collect().foreach(println) } }报错信息:
val peopleSchema = sqlContext.createSchemaRDD(people)
3:pom
<dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.10</artifactId> <version>1.1.0</version> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-sql_2.10</artifactId> <version>1.1.0</version> </dependency>
在本地代码中有修改了,依旧报同样的错误!
importsqlContext.createSchemaRDDcaseclassPerson(name:String)把case放外边就好了。。。啦啦啦啦啦!
Justmoveyourcaseclassoutofthemethoddefinition
http://stackoverflow.com/questions/29143756/scala-spark-app-with-no-typetag-available-error-in-def-main-style-app
thanks版权声明:本文内容由阿里云实名注册用户自发贡献,版权归原作者所有,阿里云开发者社区不拥有其著作权,亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容,填写侵权投诉表单进行举报,一经查实,本社区将立刻删除涉嫌侵权内容。