feat(bigtable/emulator): allow listening on Unix Domain Sockets #9665
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This can be tested as follows: No changes in the client sdk itself needed, only in the emulator. |
4bc0038
to
a65bc68
Compare
|
conventionalcommits.org seems stuck on an old commit message, not sure how to get it unstuck. |
flokli
added a commit
to flokli/bigtable_rs
that referenced
this pull request
Mar 29, 2024
The golang SDK already supports connecting to an emulator listening at a unix domain socket. Getting the emulator itself listening on a unix domain socket can be accomplished by some proxy/socat, some LD_PRELOAD magic (ip2unix) or using a patch as proposed in googleapis/google-cloud-go#9665 ``` $ ./emulator --address $PWD/emulator.sock & Cloud Bigtable emulator running on /home/flokli/dev/google-cloud-go/bigtable/cmd/emulator/emulator.sock $ export BIGTABLE_EMULATOR_HOST=unix://$PWD/emulator.sock $ cbt -instance instance-1 -project project-1 createtable table-1 2024/03/29 11:02:04 -creds flag unset, will use gcloud credential $ cbt -instance instance-1 -project project-1 ls 2024/03/29 11:02:15 -creds flag unset, will use gcloud credential table-1 ``` This adds support to connect to a unix domain socket to bigtable-rs too. We stop constructing a single-item vec of endpoints, and instead pass a channel to `create_client` in the emulator case. The channel is created from the endpoint. In the non-uds case, we use a simple connect_lazy. In the uds case, we use a custom connector, similar to the example in https://github.com/hyperium/tonic/pull/184/files#diff-f27114adeedf7b42e8656c8a86205685a54bae7a7929b895ab62516bdf9ff252. This allows bigtable-rs to talk to an emulator listening on a unix domain socket.
This was referenced Mar 29, 2024
flokli
added a commit
to flokli/bigtable_rs
that referenced
this pull request
Mar 29, 2024
The golang SDK already supports connecting to an emulator listening at a unix domain socket. Getting the emulator itself listening on a unix domain socket can be accomplished by some proxy/socat, some LD_PRELOAD magic (ip2unix) or using a patch as proposed in googleapis/google-cloud-go#9665 ``` $ ./emulator --address $PWD/emulator.sock & Cloud Bigtable emulator running on /home/flokli/dev/google-cloud-go/bigtable/cmd/emulator/emulator.sock $ export BIGTABLE_EMULATOR_HOST=unix://$PWD/emulator.sock $ cbt -instance instance-1 -project project-1 createtable table-1 2024/03/29 11:02:04 -creds flag unset, will use gcloud credential $ cbt -instance instance-1 -project project-1 ls 2024/03/29 11:02:15 -creds flag unset, will use gcloud credential table-1 ``` This adds support to connect to a unix domain socket to bigtable-rs too. We stop constructing a single-item vec of endpoints, and instead pass a channel to `create_client` in the emulator case. The channel is created from the endpoint. In the non-uds case, we use a simple connect_lazy. In the uds case, we use a custom connector, similar to the example in https://github.com/hyperium/tonic/pull/184/files#diff-f27114adeedf7b42e8656c8a86205685a54bae7a7929b895ab62516bdf9ff252. This allows bigtable-rs to talk to an emulator listening on a unix domain socket.
flokli
added a commit
to flokli/bigtable_rs
that referenced
this pull request
Mar 29, 2024
The golang SDK already supports connecting to an emulator listening at a unix domain socket. Getting the emulator itself listening on a unix domain socket can be accomplished by some proxy/socat, some LD_PRELOAD magic (ip2unix) or using a patch as proposed in googleapis/google-cloud-go#9665 ``` $ ./emulator --address $PWD/emulator.sock & Cloud Bigtable emulator running on /home/flokli/dev/google-cloud-go/bigtable/cmd/emulator/emulator.sock $ export BIGTABLE_EMULATOR_HOST=unix://$PWD/emulator.sock $ cbt -instance instance-1 -project project-1 createtable table-1 2024/03/29 11:02:04 -creds flag unset, will use gcloud credential $ cbt -instance instance-1 -project project-1 ls 2024/03/29 11:02:15 -creds flag unset, will use gcloud credential table-1 ``` This adds support to connect to a unix domain socket to bigtable-rs too. We stop constructing a single-item vec of endpoints, and instead pass a channel to `create_client` in the emulator case. The channel is created from the endpoint. In the non-uds case, we use a simple connect_lazy. In the uds case, we use a custom connector, similar to the example in https://github.com/hyperium/tonic/pull/184/files#diff-f27114adeedf7b42e8656c8a86205685a54bae7a7929b895ab62516bdf9ff252. This allows bigtable-rs to talk to an emulator listening on a unix domain socket.
flokli
added a commit
to flokli/bigtable_rs
that referenced
this pull request
Mar 29, 2024
The golang SDK already supports connecting to an emulator listening at a unix domain socket. Getting the emulator itself listening on a unix domain socket can be accomplished by some proxy/socat, some LD_PRELOAD magic (ip2unix) or using a patch as proposed in googleapis/google-cloud-go#9665 ``` $ ./emulator --address $PWD/emulator.sock & Cloud Bigtable emulator running on /home/flokli/dev/google-cloud-go/bigtable/cmd/emulator/emulator.sock $ export BIGTABLE_EMULATOR_HOST=unix://$PWD/emulator.sock $ cbt -instance instance-1 -project project-1 createtable table-1 2024/03/29 11:02:04 -creds flag unset, will use gcloud credential $ cbt -instance instance-1 -project project-1 ls 2024/03/29 11:02:15 -creds flag unset, will use gcloud credential table-1 ``` This adds support to connect to a unix domain socket to bigtable-rs too. We stop constructing a single-item vec of endpoints, and instead pass a channel to `create_client` in the emulator case. The channel is created from the endpoint. In the non-uds case, we use a simple connect_lazy. In the uds case, we use a custom connector, similar to the example in https://github.com/hyperium/tonic/pull/184/files#diff-f27114adeedf7b42e8656c8a86205685a54bae7a7929b895ab62516bdf9ff252. This allows bigtable-rs to talk to an emulator listening on a unix domain socket.
flokli
added a commit
to flokli/bigtable_rs
that referenced
this pull request
Mar 29, 2024
The golang SDK already supports connecting to an emulator listening at a unix domain socket. Getting the emulator itself listening on a unix domain socket can be accomplished by some proxy/socat, some LD_PRELOAD magic (ip2unix) or using a patch as proposed in googleapis/google-cloud-go#9665 ``` $ ./emulator --address $PWD/emulator.sock & Cloud Bigtable emulator running on /home/flokli/dev/google-cloud-go/bigtable/cmd/emulator/emulator.sock $ export BIGTABLE_EMULATOR_HOST=unix://$PWD/emulator.sock $ cbt -instance instance-1 -project project-1 createtable table-1 2024/03/29 11:02:04 -creds flag unset, will use gcloud credential $ cbt -instance instance-1 -project project-1 ls 2024/03/29 11:02:15 -creds flag unset, will use gcloud credential table-1 ``` This adds support to connect to a unix domain socket to bigtable-rs too. We stop constructing a single-item vec of endpoints, and instead pass a channel to `create_client` in the emulator case. The channel is created from the endpoint. In the non-uds case, we use a simple connect_lazy. In the uds case, we use a custom connector, similar to the example in https://github.com/hyperium/tonic/pull/184/files#diff-f27114adeedf7b42e8656c8a86205685a54bae7a7929b895ab62516bdf9ff252. This allows bigtable-rs to talk to an emulator listening on a unix domain socket.
flokli
added a commit
to flokli/bigtable_rs
that referenced
this pull request
Apr 2, 2024
Getting the emulator itself listening on a unix domain socket can be accomplished by some proxy/socat, some LD_PRELOAD magic (ip2unix) or using a patch as proposed in googleapis/google-cloud-go#9665 ``` $ ./emulator --address $PWD/emulator.sock & Cloud Bigtable emulator running on /home/flokli/dev/google-cloud-go/bigtable/cmd/emulator/emulator.sock $ export BIGTABLE_EMULATOR_HOST=unix://$PWD/emulator.sock $ cbt -instance instance-1 -project project-1 createtable table-1 2024/03/29 11:02:04 -creds flag unset, will use gcloud credential $ cbt -instance instance-1 -project project-1 ls 2024/03/29 11:02:15 -creds flag unset, will use gcloud credential table-1 ``` As can be seen through the `cbt` invocations, the golang SDK already supports connecting to an emulator listening at a unix domain socket. This commit adds support to connect to a unix domain socket to bigtable-rs too. `create_client` now accepts a channel, rather than a list of endpoints. The constructors call `Channel::balance_list(endpoints.into_iter())` themselves, in case of a normal bigtable connection.` In case of a emulator connection, we produce a channel directly connecting to the emulator, either via HTTP, or via Unix Socket, instead of using a single endpoint balancer. In the uds case, we use a custom connector, similar to the example in https://github.com/hyperium/tonic/pull/184/files#diff-f27114adeedf7b42e8656c8a86205685a54bae7a7929b895ab62516bdf9ff252. We now wait for the connection, so both `new_with_emulator` and `new_with_auth_manager` now become async (`new` already was). Additionally, `new_with_emulator` is made public, and a docstring is added. It allows connecting to the emulator explicitly, in cases where setting BIGTABLE_EMULATOR_HOST is inconvenient, for example when connecting to a (new, separate) emulator during unit tests.
flokli
added a commit
to flokli/bigtable_rs
that referenced
this pull request
Apr 2, 2024
Getting the emulator itself listening on a unix domain socket can be accomplished by some proxy/socat, some LD_PRELOAD magic (ip2unix) or using a patch as proposed in googleapis/google-cloud-go#9665 ``` $ ./emulator --address $PWD/emulator.sock & Cloud Bigtable emulator running on /home/flokli/dev/google-cloud-go/bigtable/cmd/emulator/emulator.sock $ export BIGTABLE_EMULATOR_HOST=unix://$PWD/emulator.sock $ cbt -instance instance-1 -project project-1 createtable table-1 2024/03/29 11:02:04 -creds flag unset, will use gcloud credential $ cbt -instance instance-1 -project project-1 ls 2024/03/29 11:02:15 -creds flag unset, will use gcloud credential table-1 ``` As can be seen through the `cbt` invocations, the golang SDK already supports connecting to an emulator listening at a unix domain socket. This commit adds support to connect to a unix domain socket to bigtable-rs too. `create_client` now accepts a channel, rather than a list of endpoints. The constructors call `Channel::balance_list(endpoints.into_iter())` themselves, in case of a normal bigtable connection.` In case of a emulator connection, we produce a channel directly connecting to the emulator, either via HTTP, or via Unix Socket, instead of using a single endpoint balancer. In the uds case, we use a custom connector, similar to the example in https://github.com/hyperium/tonic/pull/184/files#diff-f27114adeedf7b42e8656c8a86205685a54bae7a7929b895ab62516bdf9ff252. We now wait for the connection, so both `new_with_emulator` and `new_with_auth_manager` now become async (`new` already was). Additionally, `new_with_emulator` is made public, and a docstring is added. It allows connecting to the emulator explicitly, in cases where setting BIGTABLE_EMULATOR_HOST is inconvenient, for example when connecting to a (new, separate) emulator during unit tests.
flokli
added a commit
to flokli/bigtable_rs
that referenced
this pull request
Apr 2, 2024
Getting the emulator itself listening on a unix domain socket can be accomplished by some proxy/socat, some LD_PRELOAD magic (ip2unix) or using a patch as proposed in googleapis/google-cloud-go#9665 ``` $ ./emulator --address $PWD/emulator.sock & Cloud Bigtable emulator running on /home/flokli/dev/google-cloud-go/bigtable/cmd/emulator/emulator.sock $ export BIGTABLE_EMULATOR_HOST=unix://$PWD/emulator.sock $ cbt -instance instance-1 -project project-1 createtable table-1 2024/03/29 11:02:04 -creds flag unset, will use gcloud credential $ cbt -instance instance-1 -project project-1 ls 2024/03/29 11:02:15 -creds flag unset, will use gcloud credential table-1 ``` As can be seen through the `cbt` invocations, the golang SDK already supports connecting to an emulator listening at a unix domain socket. This commit adds support to connect to a unix domain socket to bigtable-rs too. `create_client` now accepts a channel, rather than a list of endpoints. The constructors call `Channel::balance_list(endpoints.into_iter())` themselves, in case of a normal bigtable connection.` In case of a emulator connection, we produce a channel directly connecting to the emulator, either via HTTP, or via Unix Socket, instead of using a single endpoint balancer. In the uds case, we use a custom connector, similar to the example in https://github.com/hyperium/tonic/pull/184/files#diff-f27114adeedf7b42e8656c8a86205685a54bae7a7929b895ab62516bdf9ff252. We now wait for the connection, so both `new_with_emulator` and `new_with_auth_manager` now become async (`new` already was). Additionally, `new_with_emulator` is made public, and a docstring is added. It allows connecting to the emulator explicitly, in cases where setting BIGTABLE_EMULATOR_HOST is inconvenient, for example when connecting to a (new, separate) emulator during unit tests.
flokli
added a commit
to flokli/bigtable_rs
that referenced
this pull request
Apr 6, 2024
Getting the emulator itself listening on a unix domain socket can be accomplished by some proxy/socat, some LD_PRELOAD magic (ip2unix) or using a patch as proposed in googleapis/google-cloud-go#9665 ``` $ ./emulator --address $PWD/emulator.sock & Cloud Bigtable emulator running on /home/flokli/dev/google-cloud-go/bigtable/cmd/emulator/emulator.sock $ export BIGTABLE_EMULATOR_HOST=unix://$PWD/emulator.sock $ cbt -instance instance-1 -project project-1 createtable table-1 2024/03/29 11:02:04 -creds flag unset, will use gcloud credential $ cbt -instance instance-1 -project project-1 ls 2024/03/29 11:02:15 -creds flag unset, will use gcloud credential table-1 ``` As can be seen through the `cbt` invocations, the golang SDK already supports connecting to an emulator listening at a unix domain socket. This commit adds support to connect to a unix domain socket to bigtable-rs too. `create_client` now accepts a channel, rather than a list of endpoints. The constructors call `Channel::balance_list(endpoints.into_iter())` themselves, in case of a normal bigtable connection.` In case of a emulator connection, we produce a channel directly connecting to the emulator, either via HTTP, or via Unix Socket, instead of using a single endpoint balancer. We use the lazy counterpart to keep behaviour (and function signatures) consistent with the previous `Channel::balance_list`. In the uds case, we use a custom connector, similar to the example in https://github.com/hyperium/tonic/pull/184/files#diff-f27114adeedf7b42e8656c8a86205685a54bae7a7929b895ab62516bdf9ff252. Additionally, `new_with_emulator` is made public, and a docstring is added. It allows connecting to the emulator explicitly, in cases where setting BIGTABLE_EMULATOR_HOST is inconvenient, for example when connecting to a (new, separate) emulator during unit tests.
cbtemulator listening on unix domain sockets is much easier than trying to allocate free TCP ports, especially if many cbtemulators are run at the same time in integration tests. This adds an additional flag, address, which has priority if it's set, rather than host:port. `NewServer` already takes a `laddr string`, so we simply check for it to contain slashes, and if so, listen on unix, rather than TCP.
Call srv.Close when receiving an interrupt, and delete the unix domain socket in that function.
|
I got confused over the |
tvlbot
pushed a commit
to tvlfyi/tvix
that referenced
this pull request
Apr 13, 2024
I updated googleapis/google-cloud-go#9665 in the meantime, and GH decided to GC the patches. Vendor the patch in for now (manually stripping the `bigtable/` prefix in the path). Hopefully the PR itself gets merged soon. Change-Id: I5b7ba78ccaf5c792c1445818b23b52d6f17155a2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11402 Reviewed-by: raitobezarius <[email protected]> Tested-by: BuildkiteCI Autosubmit: flokli <[email protected]>
liufuyang
pushed a commit
to liufuyang/bigtable_rs
that referenced
this pull request
Apr 14, 2024
* feat: support connecting to emulator over unix domain socket Getting the emulator itself listening on a unix domain socket can be accomplished by some proxy/socat, some LD_PRELOAD magic (ip2unix) or using a patch as proposed in googleapis/google-cloud-go#9665 ``` $ ./emulator --address $PWD/emulator.sock & Cloud Bigtable emulator running on /home/flokli/dev/google-cloud-go/bigtable/cmd/emulator/emulator.sock $ export BIGTABLE_EMULATOR_HOST=unix://$PWD/emulator.sock $ cbt -instance instance-1 -project project-1 createtable table-1 2024/03/29 11:02:04 -creds flag unset, will use gcloud credential $ cbt -instance instance-1 -project project-1 ls 2024/03/29 11:02:15 -creds flag unset, will use gcloud credential table-1 ``` As can be seen through the `cbt` invocations, the golang SDK already supports connecting to an emulator listening at a unix domain socket. This commit adds support to connect to a unix domain socket to bigtable-rs too. `create_client` now accepts a channel, rather than a list of endpoints. The constructors call `Channel::balance_list(endpoints.into_iter())` themselves, in case of a normal bigtable connection.` In case of a emulator connection, we produce a channel directly connecting to the emulator, either via HTTP, or via Unix Socket, instead of using a single endpoint balancer. We use the lazy counterpart to keep behaviour (and function signatures) consistent with the previous `Channel::balance_list`. In the uds case, we use a custom connector, similar to the example in https://github.com/hyperium/tonic/pull/184/files#diff-f27114adeedf7b42e8656c8a86205685a54bae7a7929b895ab62516bdf9ff252. Additionally, `new_with_emulator` is made public, and a docstring is added. It allows connecting to the emulator explicitly, in cases where setting BIGTABLE_EMULATOR_HOST is inconvenient, for example when connecting to a (new, separate) emulator during unit tests. * refactor: only split off unix:// once Prevent the endpoint setup code duplication by moving it into a little helper function.
bhshkh
approved these changes
Jul 22, 2024
| var err error | ||
|
|
||
| // If the address contains slashes, listen on a unix domain socket instead. | ||
| if strings.Contains(laddr, "/") { |
There was a problem hiding this comment.
Use strings.HasPrefix("laddr", "unix://") instead
There was a problem hiding this comment.
Please ignore this comment. I looked more into this and understand that the current approach is correct
| @@ -139,6 +148,11 @@ func (s *Server) Close() { | |||
|
|
|||
| s.srv.Stop() | |||
| s.l.Close() | |||
|
|
|||
| // clean up unix socket | |||
| if strings.Contains(s.Addr, "/") { | |||
eliben
pushed a commit
to eliben/google-cloud-go
that referenced
this pull request
Jul 23, 2024
…leapis#9665) * feat(bigtable/emulator): allow listening on Unix Domain Sockets cbtemulator listening on unix domain sockets is much easier than trying to allocate free TCP ports, especially if many cbtemulators are run at the same time in integration tests. This adds an additional flag, address, which has priority if it's set, rather than host:port. `NewServer` already takes a `laddr string`, so we simply check for it to contain slashes, and if so, listen on unix, rather than TCP. * feat(bigtable): clean up unix socket on close Call srv.Close when receiving an interrupt, and delete the unix domain socket in that function. --------- Co-authored-by: Baha Aiman <[email protected]>
13 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
cbtemulator listening on unix domain sockets is much easier than trying to allocate free TCP ports, especially if many cbtemulators are run at the same time in integration tests.
This adds an additional flag, address, which has priority if it's set, rather than host:port.
NewServeralready takes aladdr string, so we simply check for it to contain slashes, and if so, listen on unix, rather than TCP.