开发者社区> 问答> 正文

datax hdfswriter报内存溢出

datax hdfswriter报内存溢出,详细日志如下: at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:40) at com.alibaba.datax.core.job.scheduler.processinner.ProcessInnerScheduler.dealFailedStat(ProcessInnerScheduler.java:39) at com.alibaba.datax.core.job.scheduler.AbstractScheduler.schedule(AbstractScheduler.java:99) at com.alibaba.datax.core.job.JobContainer.schedule(JobContainer.java:535) at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:119) at com.alibaba.datax.core.Engine.start(Engine.java:92) at com.alibaba.datax.core.Engine.entry(Engine.java:171) at com.alibaba.datax.core.Engine.main(Engine.java:204) Caused by: java.lang.OutOfMemoryError: Java heap space at java.nio.HeapByteBuffer. (HeapByteBuffer.java:57) at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) at org.apache.hadoop.hive.ql.io.orc.OutStream.getNewOutputBuffer(OutStream.java:117) at org.apache.hadoop.hive.ql.io.orc.OutStream.spill(OutStream.java:168) at org.apache.hadoop.hive.ql.io.orc.OutStream.flush(OutStream.java:239) at org.apache.hadoop.hive.ql.io.orc.RunLengthByteWriter.flush(RunLengthByteWriter.java:58) at org.apache.hadoop.hive.ql.io.orc.BitFieldWriter.flush(BitFieldWriter.java:44) at org.apache.hadoop.hive.ql.io.orc.WriterImpl$TreeWriter.writeStripe(WriterImpl.java:694) at org.apache.hadoop.hive.ql.io.orc.WriterImpl$StringTreeWriter.writeStripe(WriterImpl.java:1138) at org.apache.hadoop.hive.ql.io.orc.WriterImpl$StructTreeWriter.writeStripe(WriterImpl.java:1609) at org.apache.hadoop.hive.ql.io.orc.WriterImpl.flushStripe(WriterImpl.java:1989) at org.apache.hadoop.hive.ql.io.orc.WriterImpl.close(WriterImpl.java:2281) at org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat$OrcRecordWriter.close(OrcOutputFormat.java:106) at org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat$OrcRecordWriter.close(OrcOutputFormat.java:91) at com.alibaba.datax.plugin.writer.hdfswriter.HdfsHelper.orcFileStartWrite(HdfsHelper.java:389) at com.alibaba.datax.plugin.writer.hdfswriter.HdfsWriter$Task.startWrite(HdfsWriter.java:364) at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:56) at java.lang.Thread.run(Thread.java:745)

但实际导出的数据很小 StandAloneJobContainerCommunicator - Total 4 records, 996 bytes | Speed 996B/s, 4 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage

求指点是哪里的内存溢出,并且为啥溢出

展开
收起
lgshsjfo77rxi 2022-04-03 19:11:22 2829 0
1 条回答
写回答
取消 提交回答
  • 客户端进程默认配置的内存过小的话,如果操作的数据量大,可能内存溢出导致操作失败。

    2022-05-19 08:08:22
    赞同 展开评论 打赏
问答排行榜
最热
最新

相关电子书

更多
Flink CDC:新一代数据集成框架 立即下载
DataWorks数据集成实时同步最佳实践(含内测邀请)-2020飞天大数据平台实战应用第一季 立即下载
《DataWorks 数据集成实时同步》 立即下载