datax hdfswriter报内存溢出,详细日志如下: at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:40) at com.alibaba.datax.core.job.scheduler.processinner.ProcessInnerScheduler.dealFailedStat(ProcessInnerScheduler.java:39) at com.alibaba.datax.core.job.scheduler.AbstractScheduler.schedule(AbstractScheduler.java:99) at com.alibaba.datax.core.job.JobContainer.schedule(JobContainer.java:535) at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:119) at com.alibaba.datax.core.Engine.start(Engine.java:92) at com.alibaba.datax.core.Engine.entry(Engine.java:171) at com.alibaba.datax.core.Engine.main(Engine.java:204) Caused by: java.lang.OutOfMemoryError: Java heap space at java.nio.HeapByteBuffer. (HeapByteBuffer.java:57) at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) at org.apache.hadoop.hive.ql.io.orc.OutStream.getNewOutputBuffer(OutStream.java:117) at org.apache.hadoop.hive.ql.io.orc.OutStream.spill(OutStream.java:168) at org.apache.hadoop.hive.ql.io.orc.OutStream.flush(OutStream.java:239) at org.apache.hadoop.hive.ql.io.orc.RunLengthByteWriter.flush(RunLengthByteWriter.java:58) at org.apache.hadoop.hive.ql.io.orc.BitFieldWriter.flush(BitFieldWriter.java:44) at org.apache.hadoop.hive.ql.io.orc.WriterImpl$TreeWriter.writeStripe(WriterImpl.java:694) at org.apache.hadoop.hive.ql.io.orc.WriterImpl$StringTreeWriter.writeStripe(WriterImpl.java:1138) at org.apache.hadoop.hive.ql.io.orc.WriterImpl$StructTreeWriter.writeStripe(WriterImpl.java:1609) at org.apache.hadoop.hive.ql.io.orc.WriterImpl.flushStripe(WriterImpl.java:1989) at org.apache.hadoop.hive.ql.io.orc.WriterImpl.close(WriterImpl.java:2281) at org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat$OrcRecordWriter.close(OrcOutputFormat.java:106) at org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat$OrcRecordWriter.close(OrcOutputFormat.java:91) at com.alibaba.datax.plugin.writer.hdfswriter.HdfsHelper.orcFileStartWrite(HdfsHelper.java:389) at com.alibaba.datax.plugin.writer.hdfswriter.HdfsWriter$Task.startWrite(HdfsWriter.java:364) at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:56) at java.lang.Thread.run(Thread.java:745)
但实际导出的数据很小 StandAloneJobContainerCommunicator - Total 4 records, 996 bytes | Speed 996B/s, 4 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage
求指点是哪里的内存溢出,并且为啥溢出
版权声明:本文内容由阿里云实名注册用户自发贡献,版权归原作者所有,阿里云开发者社区不拥有其著作权,亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容,填写侵权投诉表单进行举报,一经查实,本社区将立刻删除涉嫌侵权内容。