如何修复HBase Regions的孔洞?

Cloudera-Hadoop

1 基础知识

1.1 Region的错误原因

– 由于HBase具有自动分片功能(Automatic sharding)
– 区域(regions)随着数据的增长会自动拆分或重新分布(Region是HBase集群分布数据的最小单位)
– 区域拆分的大小依据来源于变量“hbase.hRegion.max.filesize”,变量默认值10GB
– 区域拆分过程中,由于各种原因可能导致分片过程没有按预期完成,此时需要人工干预

1.2 错误的检查与处理命令

sudo -u hbase hbase hbck -h

可见如下输出,

Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
20/05/05 13:28:16 INFO Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS
Usage: fsck [opts] {only tables}
 where [opts] are:
   -help Display help options (this)
   -details Display full report of all regions.
   -timelag   Process only regions that  have not experienced any metadata updates in the last   seconds.
   -sleepBeforeRerun  Sleep this many seconds before checking if the fix worked if run with -fix
   -summary Print only summary of the tables and status.
   -metaonly Only check the state of the hbase:meta table.
   -sidelineDir  HDFS path to backup existing meta.
   -boundaries Verify that regions boundaries are the same between META and store files.
   -exclusive Abort if another hbck is exclusive or fixing.
   -disableBalancer Disable the load balancer.

  Metadata Repair options: (expert features, use with caution!)
   -fix              Try to fix region assignments.  This is for backwards compatiblity
   -fixAssignments   Try to fix region assignments.  Replaces the old -fix
   -fixMeta          Try to fix meta problems.  This assumes HDFS region info is good.
   -noHdfsChecking   Don't load/check region info from HDFS. Assumes hbase:meta region info is good. Won't check/fix any HDFS issue, e.g. hole, orphan, or overlap
   -fixHdfsHoles     Try to fix region holes in hdfs.
   -fixHdfsOrphans   Try to fix region dirs with no .regioninfo file in hdfs
   -fixTableOrphans  Try to fix table dirs with no .tableinfo file in hdfs (online mode only)
   -fixHdfsOverlaps  Try to fix region overlaps in hdfs.
   -fixVersionFile   Try to fix missing hbase.version file in hdfs.
   -maxMerge      When fixing region overlaps, allow at most  regions to merge. (n=5 by default)
   -sidelineBigOverlaps  When fixing region overlaps, allow to sideline big overlaps
   -maxOverlapsToSideline   When fixing region overlaps, allow at most  regions to sideline per group. (n=2 by default)
   -fixSplitParents  Try to force offline split parents to be online.
   -ignorePreCheckPermission  ignore filesystem permission pre-check
   -fixReferenceFiles  Try to offline lingering reference store files
   -fixEmptyMetaCells  Try to fix hbase:meta entries not referencing any region (empty REGIONINFO_QUALIFIER rows)

  Datafile Repair options: (expert features, use with caution!)
   -checkCorruptHFiles     Check all Hfiles by opening them to make sure they are valid
   -sidelineCorruptHFiles  Quarantine corrupted HFiles.  implies -checkCorruptHFiles

  Metadata Repair shortcuts
   -repair           Shortcut for -fixAssignments -fixMeta -fixHdfsHoles -fixHdfsOrphans -fixHdfsOverlaps -fixVersionFile -sidelineBigOverlaps -fixReferenceFiles -fixTableLocks -fixOrphanedTableZnodes
   -repairHoles      Shortcut for -fixAssignments -fixMeta -fixHdfsHoles

  Table lock options
   -fixTableLocks    Deletes table locks held for a long time (hbase.table.lock.expire.ms, 10min by default)

  Table Znode options
   -fixOrphanedTableZnodes    Set table state in ZNode to disabled if table does not exists

 Replication options
   -fixReplication   Deletes replication queues for removed peers

2 最佳实践

2.1 检查表

sudo -u hbase hbase hbck > /tmp/hbase-hbck.txt

执行完成后,使用如下命令读取提示信息,

less /tmp/hbase-hbck.txt

可见如下错误提示

ERROR: There is a hole in the region chain between 8B9CH20R23 and 8BBFA24C43.  You need to create a new .regioninfo and region dir in hdfs to plug the hole.
ERROR: Found inconsistency in table table1

如果以上觉得信息太多,且知道是特定的表故障,可指定表检查,

sudo -u hbase hbase hbck table1

2.2 修复表

sudo -u hbase hbase hbck -repairHoles table1

参阅文档
======================

hbase hbck命令的使用
———————-
https://community.cloudera.com/t5/Community-Articles/HBase-HBCK-In-Depth/ta-p/245214
http://archive.cloudera.com/cdh5/cdh/5/hbase-0.98.6-cdh5.3.2/book/hbck.in.depth.html

如何修复Region的孔洞问题
———————–
https://www.jianshu.com/p/d5697506741e
https://blog.csdn.net/qq_31598113/article/details/78926596
http://hbasefly.com/2016/09/08/hbase-rit/

没有评论

发表评论

Bash
如何自动备份HDFS Name Node?

1 前言 之前的章节手动完成Hadoop HDFS的NameNode节点备份,本章重点是实现名称节点 …

Cloudera-Hadoop
如何备份恢复HDFS元数据-逻辑级备份?

1 基础知识 1.1 备份命令的简介 – 备份命令用于防止所有名称节点都不可用的情况下可 …

Cloudera-Hadoop
如何备份恢复HDFS元数据-文件级备份?

1 基础知识 1.1 备份的目的 – 当两个名称节点(NameNode)角色都失败是,H …