Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13144

Enhancing IPC client throughput via multiple connections per user

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.4.0, 3.3.5
    • 3.4.0, 3.3.5
    • ipc
    • Reviewed

    Description

      The generic IPC client (org.apache.hadoop.ipc.Client) utilizes a single connection thread for each ConnectionId. The ConnectionId is unique to the connection's remote address, ticket and protocol. Each ConnectionId is 1:1 mapped to a connection thread by the client via a map cache.

      The result is to serialize all IPC read/write activity through a single thread for a each user/ticket + address. If a single user makes repeated calls (1k-100k/sec) to the same destination, the IPC client becomes a bottleneck.

      Attachments

        1. HADOOP-13144-performance.patch
          20 kB
          Íñigo Goiri
        2. HADOOP-13144.003.patch
          6 kB
          Íñigo Goiri
        3. HADOOP-13144.002.patch
          6 kB
          Íñigo Goiri
        4. HADOOP-13144.001.patch
          6 kB
          Íñigo Goiri
        5. HADOOP-13144.000.patch
          4 kB
          Íñigo Goiri
        6. HADOOP-13144_overload_enhancement.patch
          43 kB
          Aihua Xu

        Issue Links

        is depended upon by

        Sub-task - The sub-task of the issue HDFS-13274 RBF: Extend RouterRpcClient to use multiple sockets

        • Major - Major loss of function.
        • Resolved

        New Feature - A new feature of the product, which has yet to be developed. HDFS-10467 Router-based HDFS federation

        • Major - Major loss of function.
        • Resolved
        relates to

        Improvement - An improvement or enhancement to an existing feature or task. HADOOP-18406 Adds alignment context to call path for creating RPC proxy with multiple connections per user.

        • Major - Major loss of function.
        • Resolved
        links to

        Web Link GitHub Pull Request #4542

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            elgoiri Íñigo Goiri
            jakace Jason Kace
            Votes:
            0 Vote for this issue
            Watchers:
            33 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 1.5h
                1.5h

                Slack

                  Issue deployment