In Log Explorer, run the following query for the last 30 days got 51 entries.resource.type="cloud_dataproc_cluster"resource.labels.region="us-central1"resource.labels.project_id="my-dataproc"protoPayload.methodName="google.cloud.dataproc.v1.ClusterCo...

@jangemmarPer doc: "https://cloud.google.com/dataproc/docs/reference/rest/v1/projects.regions.jobs#State"0: STATE_UNSPECIFIED - The job state is unknown.1: PENDING - The job is pending; it has been submitted, but is not yet running.2: SETUP_DONE - Jo...

I created an autoscaling policy and attached to a cluster with 2 primary worker nodes (2 core with 8gb). I ran a PySpark code with number of executors set to 50, but it did not trigger autoscaling.Went through the documentations, most is on the theor...

I did go thru the console to create a dataproc batch, but can't find a way in UI to enable custom metrics on Spark.