开发者社区> 问答> 正文

我这里有个HFile,从1.1.6升级到2.3.4 原地升级 ,StoreScanner一直不结束

1.1.6原地升级2.3.4 升级后compact不结束,经排查是因为Scan其中一个rowkey一直不结束进入死循环导致,但是我不知道怎么解决?

在一个HFile中一直循环

===========各cell 在hfile中 的trailer、blockBuffer、curBlock、nextIndexedKey 信息==========

cell=350ef10764bc4af960b6b468479aff1e_10032279726411/d:directory/1637112759915/Put/vlen=0/seqid=1555467268

reader.trailer=fileinfoOffset=251902094, loadOnOpenDataOffset=251901646, dataIndexCount=5, metaIndexCount=0, totalUncomressedBytes=616627871, entryCount=3831303, compressionCodec=LZ4, uncompressedDataIndexSize=623813, numDataIndexLevels=2, firstDataBlockOffset=0, lastDataBlockOffset=251756307, comparatorClassName=org.apache.hadoop.hbase.CellComparatorImpl, encryptionKey=NONE, majorVersion=3, minorVersion=3

blockBuffer=SingleByteBuff[pos=64486, lim=75102, cap= 75102]

curBlock=[blockType=DATA, fileOffset=153871189, headerSize=33, onDiskSizeWithoutHeader=31376, uncompressedSizeWithoutHeader=75102, prevBlockOffset=153846578, isUseHBaseChecksum=true, checksumType=CRC32C, bytesPerChecksum=16384, onDiskDataSizeWithHeader=31401, getOnDiskSizeWithHeader=31409, totalChecksumBytes=8, isUnpacked=true, buf=[SingleByteBuff[pos=0, lim=75143, cap= 75143]], dataBeginsWith=\x00\x00\x00E\x00\x00\x00\x00\x00/350eef5defca3e7c5bc6e3, fileContext=[usesHBaseChecksum=true, checksumType=CRC32C, bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true, includesTags=false, compressAlgo=LZ4, compressTags=false, cryptoContext=[cipher=NONE keyHash=NONE], name=93cc194d62fc419792991613332b990d, cellComparator=org.apache.hadoop.hbase.CellComparatorImpl@7cbeac65], nextBlockOnDiskSize=26690]

nextIndexedKey=4//LATEST_TIMESTAMP/Maximum/vlen=0/mvcc=0

cell=350ef10764bc4af960b6b468479aff1e_10032279726411/d:product_ext_nick/1637057318457/Put/vlen=27/seqid=1554882684

reader.trailer=fileinfoOffset=251902094, loadOnOpenDataOffset=251901646, dataIndexCount=5, metaIndexCount=0, totalUncomressedBytes=616627871, entryCount=3831303, compressionCodec=LZ4, uncompressedDataIndexSize=623813, numDataIndexLevels=2, firstDataBlockOffset=0, lastDataBlockOffset=251756307, comparatorClassName=org.apache.hadoop.hbase.CellComparatorImpl, encryptionKey=NONE, majorVersion=3, minorVersion=3

blockBuffer=SingleByteBuff[pos=64568, lim=75102, cap= 75102]

curBlock=[blockType=DATA, fileOffset=153871189, headerSize=33, onDiskSizeWithoutHeader=31376, uncompressedSizeWithoutHeader=75102, prevBlockOffset=153846578, isUseHBaseChecksum=true, checksumType=CRC32C, bytesPerChecksum=16384, onDiskDataSizeWithHeader=31401, getOnDiskSizeWithHeader=31409, totalChecksumBytes=8, isUnpacked=true, buf=[SingleByteBuff[pos=0, lim=75143, cap= 75143]], dataBeginsWith=\x00\x00\x00E\x00\x00\x00\x00\x00/350eef5defca3e7c5bc6e3, fileContext=[usesHBaseChecksum=true, checksumType=CRC32C, bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true, includesTags=false, compressAlgo=LZ4, compressTags=false, cryptoContext=[cipher=NONE keyHash=NONE], name=93cc194d62fc419792991613332b990d, cellComparator=org.apache.hadoop.hbase.CellComparatorImpl@7cbeac65], nextBlockOnDiskSize=26690]

nextIndexedKey=4//LATEST_TIMESTAMP/Maximum/vlen=0/mvcc=0

cell=350ef10764bc4af960b6b468479aff1e_10032279726411/d:store_info_ex_pc/1637125253154/Put/vlen=10329/seqid=1555632639

reader.trailer=fileinfoOffset=251902094, loadOnOpenDataOffset=251901646, dataIndexCount=5, metaIndexCount=0, totalUncomressedBytes=616627871, entryCount=3831303, compressionCodec=LZ4, uncompressedDataIndexSize=623813, numDataIndexLevels=2, firstDataBlockOffset=0, lastDataBlockOffset=251756307, comparatorClassName=org.apache.hadoop.hbase.CellComparatorImpl, encryptionKey=NONE, majorVersion=3, minorVersion=3

blockBuffer=SingleByteBuff[pos=64684, lim=75102, cap= 75102]

curBlock=[blockType=DATA, fileOffset=153871189, headerSize=33, onDiskSizeWithoutHeader=31376, uncompressedSizeWithoutHeader=75102, prevBlockOffset=153846578, isUseHBaseChecksum=true, checksumType=CRC32C, bytesPerChecksum=16384, onDiskDataSizeWithHeader=31401, getOnDiskSizeWithHeader=31409, totalChecksumBytes=8, isUnpacked=true, buf=[SingleByteBuff[pos=0, lim=75143, cap= 75143]], dataBeginsWith=\x00\x00\x00E\x00\x00\x00\x00\x00/350eef5defca3e7c5bc6e3, fileContext=[usesHBaseChecksum=true, checksumType=CRC32C, bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true, includesTags=false, compressAlgo=LZ4, compressTags=false, cryptoContext=[cipher=NONE keyHash=NONE], name=93cc194d62fc419792991613332b990d, cellComparator=org.apache.hadoop.hbase.CellComparatorImpl@7cbeac65], nextBlockOnDiskSize=26690]

nextIndexedKey=4//LATEST_TIMESTAMP/Maximum/vlen=0/mvcc=0

cell=350ef10764bc4af960b6b468479aff1e_10032279726411/d:vender_name/1637112759915/Put/vlen=0/seqid=1555467268

reader.trailer=fileinfoOffset=251902094, loadOnOpenDataOffset=251901646, dataIndexCount=5, metaIndexCount=0, totalUncomressedBytes=616627871, entryCount=3831303, compressionCodec=LZ4, uncompressedDataIndexSize=623813, numDataIndexLevels=2, firstDataBlockOffset=0, lastDataBlockOffset=251756307, comparatorClassName=org.apache.hadoop.hbase.CellComparatorImpl, encryptionKey=NONE, majorVersion=3, minorVersion=3

blockBuffer=SingleByteBuff[pos=0, lim=65594, cap= 65594]

curBlock=[blockType=DATA, fileOffset=153902598, headerSize=33, onDiskSizeWithoutHeader=26657, uncompressedSizeWithoutHeader=65594, prevBlockOffset=153871189, isUseHBaseChecksum=true, checksumType=CRC32C, bytesPerChecksum=16384, onDiskDataSizeWithHeader=26682, getOnDiskSizeWithHeader=26690, totalChecksumBytes=8, isUnpacked=true, buf=[SingleByteBuff[pos=0, lim=65635, cap= 65635]], dataBeginsWith=\x00\x00\x00G\x00\x00\x00\x00\x00/350ef10764bc4af960b6b4, fileContext=[usesHBaseChecksum=true, checksumType=CRC32C, bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true, includesTags=false, compressAlgo=LZ4, compressTags=false, cryptoContext=[cipher=NONE keyHash=NONE], name=93cc194d62fc419792991613332b990d, cellComparator=org.apache.hadoop.hbase.CellComparatorImpl@7cbeac65], nextBlockOnDiskSize=27336]

nextIndexedKey=null

-------------------------分割线-----------------------

当指针执行至列为vender_name 的数据时: generalizedSeek()-》NonLazyKeyValueScanner.doRealSeek()-》scanner.reseek(kv)-》reseekAtOrAfter(hfs, key)=》s.reseekTo(k)=》seekTo(key, false)-》BlockWithScanInfo blockWithScanInfo = indexReader.loadDataBlockWithScanInfo() 触发非正常流程:

==== loadDataBlockWithScanInfo before==== seekKey为:key=350ef10764bc4af960b6b468479aff1e_10032279726411,qualifie=vender_name,time=-9223372036854775808

HFileScanner为:StoreFileScanner[HFileScanner for reader reader=hdfs://snake/hbase/data/realtime_search/WD_MERGED_DATA/2fc7bd24c7d6d8d500f04b2480660de9/d/93cc194d62fc419792991613332b990d, compression=lz4, cacheConf=cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, firstKey=Optional[34f5c0cb717779d9ede9a8b80e46d4d4_10039682866610/d:product_ext_nick/1637109574351/Put/seqid=0], lastKey=Optional[351eb49327ecd6099a11efe0d1347724_10020558744609/d:commentcount/1637134716442/Put/seqid=0], avgKeyLen=70, avgValueLen=76, entries=3831303, length=251907468, cur=350ef10764bc4af960b6b468479aff1e_10032279726411/d:vender_name/1637112759915/Put/vlen=0/seqid=1555467268]

reader.trailer 为:fileinfoOffset=251902094, loadOnOpenDataOffset=251901646, dataIndexCount=5, metaIndexCount=0, totalUncomressedBytes=616627871, entryCount=3831303, compressionCodec=LZ4, uncompressedDataIndexSize=623813, numDataIndexLevels=2, firstDataBlockOffset=0, lastDataBlockOffset=251756307, comparatorClassName=org.apache.hadoop.hbase.CellComparatorImpl, encryptionKey=NONE, majorVersion=3, minorVersion=3

blockBuffer为:SingleByteBuff[pos=0, lim=65594, cap= 65594]

curBlock为:[blockType=DATA, fileOffset=153902598, headerSize=33, onDiskSizeWithoutHeader=26657, uncompressedSizeWithoutHeader=65594, prevBlockOffset=153871189, isUseHBaseChecksum=true, checksumType=CRC32C, bytesPerChecksum=16384, onDiskDataSizeWithHeader=26682, getOnDiskSizeWithHeader=26690, totalChecksumBytes=8, isUnpacked=true, buf=[SingleByteBuff[pos=0, lim=65635, cap= 65635]], dataBeginsWith=\x00\x00\x00G\x00\x00\x00\x00\x00/350ef10764bc4af960b6b4, fileContext=[usesHBaseChecksum=true, checksumType=CRC32C, bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true, includesTags=false, compressAlgo=LZ4, compressTags=false, cryptoContext=[cipher=NONE keyHash=NONE], name=93cc194d62fc419792991613332b990d, cellComparator=org.apache.hadoop.hbase.CellComparatorImpl@7cbeac65], nextBlockOnDiskSize=27336]

====loadDataBlockWithScanInfo after==== 返回预期之外的block, 返回d:store_info_ex_pc所在的block 返回信息如下: block=[blockType=DATA, fileOffset=153871189, headerSize=33, onDiskSizeWithoutHeader=31376, uncompressedSizeWithoutHeader=75102, prevBlockOffset=153846578, isUseHBaseChecksum=true, checksumType=CRC32C, bytesPerChecksum=16384, onDiskDataSizeWithHeader=31401, getOnDiskSizeWithHeader=31409, totalChecksumBytes=8, isUnpacked=true, buf=[SingleByteBuff[pos=0, lim=75143, cap= 75143]], dataBeginsWith=\x00\x00\x00E\x00\x00\x00\x00\x00/350eef5defca3e7c5bc6e3, fileContext=[usesHBaseChecksum=true, checksumType=CRC32C, bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true, includesTags=false, compressAlgo=LZ4, compressTags=false, cryptoContext=[cipher=NONE keyHash=NONE], name=93cc194d62fc419792991613332b990d, cellComparator=org.apache.hadoop.hbase.CellComparatorImpl@7cbeac65], nextBlockOnDiskSize=26690] nextIndexedKey=4//LATEST_TIMESTAMP/Maximum/vlen=0/mvcc=0

所以当indexReader.loadDataBlockWithScanInfo()=》seekTo(key, false)=》s.reseekTo(k)=》reseekAtOrAfter(hfs, key)=》scanner.reseek(kv) 在scanner.reseek(kv) 执行 setCurrentCell(hfs.getCell()); 取到的cell 为350ef10764bc4af960b6b468479aff1e_10032279726411/d:vender_name/1637112759915/Put/vlen=0/seqid=1555467268 导致往后的逻辑一直此逻辑

展开
收起
1662344814864411 2021-12-29 22:07:01 820 0
0 条回答
写回答
取消 提交回答
问答排行榜
最热
最新

相关电子书

更多
低代码开发师(初级)实战教程 立即下载
冬季实战营第三期:MySQL数据库进阶实战 立即下载
阿里巴巴DevOps 最佳实践手册 立即下载