Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-16269

[Fix] Improve NNThroughputBenchmark#blockReport operation

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      When using NNThroughputBenchmark to verify the blockReport, you will get some exception information.
      Commands used:
      ./bin/hadoop org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -fs xxxx -op blockReport -datanodes 3 -reports 1

      The exception information:
      21/10/12 14:35:18 INFO namenode.NNThroughputBenchmark: Starting benchmark: blockReport
      21/10/12 14:35:19 INFO namenode.NNThroughputBenchmark: Creating 10 files with 10 blocks each.
      21/10/12 14:35:19 ERROR namenode.NNThroughputBenchmark: java.lang.ArrayIndexOutOfBoundsException: 50009
      at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.addBlocks(NNThroughputBenchmark.java:1161)
      at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.generateInputs(NNThroughputBenchmark.java:1143)
      at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$OperationStatsBase.benchmark(NNThroughputBenchmark.java:257)
      at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.run(NNThroughputBenchmark.java:1528)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
      at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.runBenchmark(NNThroughputBenchmark.java:1430)
      at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.main(NNThroughputBenchmark.java:1550)

      Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 50009
      at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.addBlocks(NNThroughputBenchmark.java:1161)
      at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.generateInputs(NNThroughputBenchmark.java:1143)
      at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$OperationStatsBase.benchmark(NNThroughputBenchmark.java:257)
      at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.run(NNThroughputBenchmark.java:1528)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
      at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.runBenchmark(NNThroughputBenchmark.java:1430)
      at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.main(NNThroughputBenchmark.java:1550)

      Checked some code and found that the problem appeared here.
      private ExtendedBlock addBlocks(String fileName, String clientName)
      throws IOException {
      for(DatanodeInfo dnInfo: loc.getLocations())

      { int dnIdx = dnInfo.getXferPort()-1; datanodes[dnIdx].addBlock(loc.getBlock().getLocalBlock()); }

      }
      It can be seen from this that what dnInfo.getXferPort() gets is a port information and should not be used as an index of an array.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            jianghuazhu JiangHua Zhu
            jianghuazhu JiangHua Zhu
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 5h 20m
                5h 20m

                Slack

                  Issue deployment