More flexible Python Callable type.

[robertwb]

This allows def functions, fully qualified imports, and expressions with preambles.

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment (R: @username).
• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

[robertwb]

R: @ihji

[codecov]

Codecov Report

Merging #17767 (c12f209) into master (f24cedf) will decrease coverage by 0.09%.
The diff coverage is 97.50%.

@@            Coverage Diff             @@
##           master   #17767      +/-   ##
==========================================
- Coverage   74.07%   73.97%   -0.10%     
==========================================
  Files         697      695       -2     
  Lines       91927    91934       +7     
==========================================
- Hits        68092    68010      -82     
- Misses      22590    22678      +88     
- Partials     1245     1246       +1     

Flag	Coverage Δ
python	83.69% <97.50%> (-0.04%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
sdks/python/apache_beam/utils/python_callable.py	95.83% <97.50%> (+40.27%)	⬆️
...ache_beam/runners/dataflow/dataflow_job_service.py	50.00% <0.00%> (-12.17%)	⬇️
sdks/go/pkg/beam/core/graph/fn.go	76.82% <0.00%> (-8.51%)	⬇️
...o/pkg/beam/io/rtrackers/offsetrange/offsetrange.go	75.70% <0.00%> (-2.81%)	⬇️
sdks/go/pkg/beam/runners/dataflow/dataflow.go	56.09% <0.00%> (-2.53%)	⬇️
...eam/runners/portability/fn_api_runner/fn_runner.py	87.51% <0.00%> (-2.50%)	⬇️
sdks/python/apache_beam/utils/interactive_utils.py	95.12% <0.00%> (-2.44%)	⬇️
...python/apache_beam/runners/worker/worker_status.py	77.53% <0.00%> (-2.18%)	⬇️
.../python/apache_beam/transforms/periodicsequence.py	96.72% <0.00%> (-1.64%)	⬇️
...nners/portability/fn_api_runner/worker_handlers.py	77.89% <0.00%> (-1.45%)	⬇️
... and 22 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f24cedf...c12f209. Read the comment docs.

[ihji]

I didn't realize you wrote the similar PR I drafted last night: https://github.com/apache/beam/pull/17763/files
I'm okay with the suggested implementation in this PR but prefer the simpler solution that explicitly separates the statement source for exec and the expression source for eval. Using statements as an expression by our own semantics seems more confusing to me.

@chamikaramj Cham, Do you have any preference / idea?

[robertwb]

Oh, didn't realize you were looking at this as well.

The primary advantage of this form is for anything that is difficult to express as a lambda, e.g. having to write

PythonCallableSource.of(
    "foo")
.withContext("def foo(...): ...")

feels a bit roundabout. I was also leaning towards making things like PythonCallableSource.of("math.sqrt") or PythonCallableSource.of("scipy.special.ellipj") work with minimal overhead, and reduce the amount of overhead required for each calling SDK (go, typescript, ...) by keeping the representation a simple string.

[ihji]

@robertwb Okay. Then can you please add a detailed pydoc about the modified semantics we implemented for a string representation of Python codes? We might need users to know what statements and expressions are allowed and how to use them with examples.

[chamikaramj]

+1 for simple approach (single string) with a detailed pydoc. From an API perspective I like separating out the exec and eval parts but I agree with Roberts concern that this might feel unnatural for a Python user who just want to use a single Python script that needs some statements to build context.

[robertwb]

Added documentation.

[ihji]

imports

[ihji]

imports

[ihji]

LGTM except one typo. Thanks!