hadoop查看文件大小(hdfs hive hbase)-CSDN博客

本文链接：https://blog.csdn.net/iteye_3893/article/details/82642657

本文介绍了如何使用Hadoop命令行工具如hadoop fs -count和hdfs dfs -du来检查HDFS、Hive和HBase中文件及目录的大小。通过这些命令，可以获取到文件夹数、文件数以及具体的存储容量，并且能以人类可读的单位（如G或K）显示大小。此外，还提到了查看特定Hive库和分区表大小的方法。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

可以查看 hdfs hive hbase文件存储大小

hadoop fs -count

或者

hdfs dfs -count /51JOB

[root@chinadaas01 ~]# hadoop fs -count /hbase1/zzjg_detail

文件夹数文件数这个目录下的文件bytes

查看文件夹下各个目录的总大小：

hdfs dfs -du -h /user/hive/warehouse/ste_qyjs.db/

能看到这个库下所有表的总大小并以G 或者K 显示

直接看第三个命令：

[bigdata@task1-sandbox tfb]$ hadoop fs -du -s -h /user/hive/warehouse/dev_treasury.db
1.1 T 3.3 T /user/hive/warehouse/dev_treasury.db

查看分区表大小：“

hadoop fs -du /user/hive/warehouse/dev_treasury.db/ods_common_users|awk '{ SUM += $1 } END { print SUM/(1024*1024*1024)}'

awk处理完所有输入行之后才处理END模式，上面命令是讲所有分区表和非分区表的大小统计进去以GB显示

<!--5f39ae17-8c62-4a45-bc43-b32064c9388a:W3siYmxvY2tJZCI6IjEyMjEtMTUyOTM5MzI3MDg1NSIsImJsb2NrVHlwZSI6InBhcmFncmFwaCIsInN0eWxlcyI6eyJhbGlnbiI6ImxlZnQiLCJpbmRlbnQiOjAsInRleHQtaW5kZW50IjowLCJsaW5lLWhlaWdodCI6MS43NX0sInR5cGUiOiJwYXJhZ3JhcGgiLCJyaWNoVGV4dCI6eyJkYXRhIjpbeyJjaGFyIjoiICJ9LHsiY2hhciI6ImgifSx7ImNoYXIiOiJhIn0seyJjaGFyIjoiZCJ9LHsiY2hhciI6Im8ifSx7ImNoYXIiOiJvIn0seyJjaGFyIjoicCJ9LHsiY2hhciI6IiAifSx7ImNoYXIiOiJmIn0seyJjaGFyIjoicyJ9LHsiY2hhciI6IiAifSx7ImNoYXIiOiItIn0seyJjaGFyIjoiZCJ9LHsiY2hhciI6InUifSx7ImNoYXIiOiIgIn0seyJjaGFyIjoiLyJ9LHsiY2hhciI6InUifSx7ImNoYXIiOiJzIn0seyJjaGFyIjoiZSJ9LHsiY2hhciI6InIifSx7ImNoYXIiOiIvIn0seyJjaGFyIjoiaCJ9LHsiY2hhciI6ImkifSx7ImNoYXIiOiJ2In0seyJjaGFyIjoiZSJ9LHsiY2hhciI6Ii8ifSx7ImNoYXIiOiJ3In0seyJjaGFyIjoiYSJ9L