notebooks/official/vizier/get_started_vertex_vizier.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "copyright"
      },
      "outputs": [],
      "source": [
        "# Copyright 2022 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "title:generic,gcp"
      },
      "source": [
        "# Get started with Vertex AI Vizier\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/vizier/get_started_vertex_vizier.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fofficial%2Fvizier%2Fget_started_vertex_vizier.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://cloud.google.com/ml-engine/images/colab-enterprise-logo-32px.png\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
        "    </a>\n",
        "  </td>    \n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/vizier/get_started_vertex_vizier.ipynb\">\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/vizier/get_started_vertex_vizier.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "overview:mlops"
      },
      "source": [
        "## Overview\n",
        "\n",
        "\n",
        "This tutorial demonstrates how to use Vertex AI Vizier for hyperparameter tuning.\n",
        "\n",
        "Learn more about [Vertex AI Vizier](https://cloud.google.com/vertex-ai/docs/vizier/overview)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "objective:mlops,stage2,get_started_vertex_vizier"
      },
      "source": [
        "### Objective\n",
        "\n",
        "In this tutorial, you learn how to use Vertex AI Vizier when training with Vertex AI.\n",
        "\n",
        "This tutorial uses the following Google Cloud ML services:\n",
        "\n",
        "- Vertex AI Training\n",
        "- Vertex AI hyperparameter tuning\n",
        "- Vertex AI Vizier\n",
        "\n",
        "The steps performed include:\n",
        "\n",
        "- Hyperparameter tuning with Random algorithm.\n",
        "- Hyperparameter tuning with Vizier (Bayesian) algorithm.\n",
        "- Suggesting trials and updating results for Vizier study"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "recommendation:mlops,stage2,vertex,vizier"
      },
      "source": [
        "### Recommendations\n",
        "\n",
        "When doing E2E MLOps on Google Cloud, the following are best practices for when to use Vertex AI Vizier for hyperparameter tuning:\n",
        "\n",
        "**Grid Search**\n",
        "\n",
        "You have a small number of combinations of discrete values. For example, you have the following two hyperparameters and discrete values:\n",
        "\n",
        "- batch size: \\[ 16, 32, 64\\]\n",
        "- lr: \\[ 0.001, 0.01. 0.1\\]\n",
        "\n",
        "The total number of combinations is 9 (3 x 3).\n",
        "\n",
        "**Random Search**\n",
        "\n",
        "You have a small number of hyperparameters, where at least one is a continuous value. For example, you have the following two hyperparameters and ranges:\n",
        "\n",
        "- batch size: \\[ 16, 32, 64\\]\n",
        "- lr: 0.001 .. 0.1\n",
        "\n",
        "\n",
        "**Vizier Search**\n",
        "\n",
        "You have either a:\n",
        "\n",
        "- large number of hyperparameters and discrete values\n",
        "- vast continuous search space\n",
        "- multiple of objectives"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dataset:custom,boston,lrg"
      },
      "source": [
        "### Dataset\n",
        "\n",
        "The dataset used for this tutorial is the [Boston Housing Prices dataset](https://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html). The version of the dataset used in this tutorial is the one that's available from TensorFlow SDK. The trained model predicts the median price of a house in units of 1K USD."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c480fc50ec3c"
      },
      "source": [
        "### Costs \n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* Vertex AI\n",
        "* Cloud Storage\n",
        "\n",
        "\n",
        "Learn about [Vertex AI\n",
        "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
        "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
        "Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "61RBz8LLbxCR"
      },
      "source": [
        "## Get started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "No17Cw5hgx12"
      },
      "source": [
        "### Install Vertex AI SDK for Python and other required packages\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1fd00fa70a2a"
      },
      "outputs": [],
      "source": [
        "! pip3 install --upgrade --quiet google-cloud-aiplatform \\\n",
        "                                 numpy==1.23.0 \\\n",
        "                                 google-vizier"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R5Xep4W9lq-Z"
      },
      "source": [
        "### Restart runtime (Colab only)\n",
        "\n",
        "To use the newly installed packages, you must restart the runtime on Google Colab."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XRvKdaPDTznN"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "\n",
        "    import IPython\n",
        "\n",
        "    app = IPython.Application.instance()\n",
        "    app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SbmM4z7FOBpM"
      },
      "source": [
        "<div class=\"alert alert-block alert-warning\">\n",
        "<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
        "</div>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dmWOrTJ3gx13"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "Authenticate your environment on Google Colab.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NyKGtVQjgx13"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DF4l8DTdWgPY"
      },
      "source": [
        "### Set Google Cloud project information\n",
        "\n",
        "To get started using Vertex AI, you must have an existing Google Cloud project. Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "set_project_id"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
        "LOCATION = \"us-central1\"  # @param {type:\"string\"}\n",
        "\n",
        "# Set the project id\n",
        "! gcloud config set project {PROJECT_ID}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bucket:mbsdk"
      },
      "source": [
        "### Create a Cloud Storage bucket\n",
        "\n",
        "Create a storage bucket to store intermediate artifacts such as datasets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bucket"
      },
      "outputs": [],
      "source": [
        "BUCKET_URI = f\"gs://your-bucket-name-{PROJECT_ID}-unique\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_bucket"
      },
      "source": [
        "**If your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "create_bucket"
      },
      "outputs": [],
      "source": [
        "! gsutil mb -l {LOCATION} -p {PROJECT_ID} {BUCKET_URI}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "validate_bucket"
      },
      "source": [
        "Finally, validate access to your Cloud Storage bucket by examining its contents:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "validate_bucket"
      },
      "outputs": [],
      "source": [
        "! gsutil ls -al {BUCKET_URI}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "setup_vars"
      },
      "source": [
        "### Set up variables\n",
        "\n",
        "Next, set up some variables used throughout the tutorial.\n",
        "### Import libraries and define constants"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "import_aip:mbsdk"
      },
      "outputs": [],
      "source": [
        "from google.cloud import aiplatform\n",
        "from google.cloud.aiplatform.vizier import Study, pyvizier"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "init_aip:mbsdk"
      },
      "source": [
        "### Initialize Vertex AI SDK for Python\n",
        "\n",
        "Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "init_aip:mbsdk"
      },
      "outputs": [],
      "source": [
        "aiplatform.init(project=PROJECT_ID, staging_bucket=BUCKET_URI)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "accelerators:training,mbsdk"
      },
      "source": [
        "#### Set hardware accelerators\n",
        "\n",
        "You can set hardware accelerators for training.\n",
        "\n",
        "Set the variable `TRAIN_GPU/TRAIN_NGPU` to use a container image supporting a GPU and the number of GPUs allocated to the virtual machine (VM) instance. For example, to use a GPU container image with 4 Nvidia Telsa T4 GPUs allocated to each VM, you'd specify:\n",
        "\n",
        "    (aiplatform.gapic.AcceleratorType.NVIDIA_TESLA_T4, 4)\n",
        "\n",
        "Otherwise specify `(None, None)` to use a container image to run on a CPU.\n",
        "\n",
        "Learn more [here](https://cloud.google.com/vertex-ai/docs/general/locations#accelerators) hardware accelerator support for your region"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "accelerators:training,mbsdk"
      },
      "outputs": [],
      "source": [
        "TRAIN_GPU, TRAIN_NGPU = (aiplatform.gapic.AcceleratorType.NVIDIA_TESLA_T4, 1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "container:training"
      },
      "source": [
        "#### Set pre-built containers\n",
        "\n",
        "Set the pre-built Docker container image for training.\n",
        "\n",
        "- Set the variable `TF` to the TensorFlow version of the container image. For example, `2-1` would be version 2.1, and `1-15` would be version 1.15. The following list shows some of the pre-built images available:\n",
        "\n",
        "\n",
        "For the latest list, see [Pre-built containers for training](https://cloud.google.com/ai-platform-unified/docs/training/pre-built-containers)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "container:training"
      },
      "outputs": [],
      "source": [
        "TF = \"2.5\".replace(\".\", \"-\")\n",
        "\n",
        "if TF[0] == \"2\":\n",
        "    if TRAIN_GPU:\n",
        "        TRAIN_VERSION = \"tf-gpu.{}\".format(TF)\n",
        "    else:\n",
        "        TRAIN_VERSION = \"tf-cpu.{}\".format(TF)\n",
        "else:\n",
        "    if TRAIN_GPU:\n",
        "        TRAIN_VERSION = \"tf-gpu.{}\".format(TF)\n",
        "    else:\n",
        "        TRAIN_VERSION = \"tf-cpu.{}\".format(TF)\n",
        "\n",
        "TRAIN_IMAGE = \"{}-docker.pkg.dev/vertex-ai/training/{}:latest\".format(\n",
        "    LOCATION.split(\"-\")[0], TRAIN_VERSION\n",
        ")\n",
        "\n",
        "print(\"Training:\", TRAIN_IMAGE, TRAIN_GPU, TRAIN_NGPU)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "machine:training"
      },
      "source": [
        "#### Set machine type\n",
        "\n",
        "Next, set the machine type to use for training.\n",
        "\n",
        "- Set the variable `TRAIN_COMPUTE` to configure the compute resources for the VMs used for training.\n",
        " - `machine type`\n",
        "     - `n1-standard`: 3.75GB of memory per vCPU.\n",
        "     - `n1-highmem`: 6.5GB of memory per vCPU\n",
        "     - `n1-highcpu`: 0.9 GB of memory per vCPU\n",
        " - `vCPUs`: number of \\[2, 4, 8, 16, 32, 64, 96 \\]\n",
        "\n",
        "**Note**: The following isn't supported for training:\n",
        "\n",
        " - `standard`: 2 vCPUs\n",
        " - `highcpu`: 2, 4 and 8 vCPUs\n",
        "\n",
        "**Note**: You can also use n2 and e2 machine types for training and deployment, but they don't support GPUs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "machine:training"
      },
      "outputs": [],
      "source": [
        "TRAIN_COMPUTE = \"n1-standard-4\"\n",
        "print(\"Train machine type\", TRAIN_COMPUTE)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vizier_intro"
      },
      "source": [
        "## Standalone Vertex AI Vizer service\n",
        "\n",
        "The Vizier service can be used as a standalone service for selecting the next set of parameters for a trial.\n",
        "\n",
        "**Note:** The service doesn't execute trials. You create your own trial and execution.\n",
        "\n",
        "Learn more about how to [Create Vertex AI Vizier studies](https://cloud.google.com/vertex-ai/docs/vizier/using-vizier)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hpt_intro"
      },
      "source": [
        "## Vertex AI Hyperparameter Tuning service\n",
        "\n",
        "The following example demonstrates how to setup, execute and evaluate trials using the Vertex AI hyperparameter tuning service with `random` search algorithm.\n",
        "\n",
        "Learn more about [Overview of hyperparameter tuning](https://cloud.google.com/vertex-ai/docs/training/hyperparameter-tuning-overview)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "examine_training_package"
      },
      "source": [
        "### Examine the hyperparameter tuning package\n",
        "\n",
        "#### Package layout\n",
        "\n",
        "Before you start the hyperparameter tuning, take a look at how a Python package is assembled for a custom hyperparameter tuning job. When unarchived, the package contains the following directory/file layout.\n",
        "\n",
        "- PKG-INFO\n",
        "- README.md\n",
        "- setup.cfg\n",
        "- setup.py\n",
        "- trainer\n",
        "  - \\_\\_init\\_\\_.py\n",
        "  - task.py\n",
        "\n",
        "The files `setup.cfg` and `setup.py` are the instructions for installing the package into the operating environment of the Docker image.\n",
        "\n",
        "The file `trainer/task.py` is the Python script for executing the custom hyperparameter tuning job. *Note*, when `trainer/task.py` is referred to in the worker pool specification, the directory slash is replaced with a dot and the file suffix (`.py`) is dropped (`trainer.task`).\n",
        "\n",
        "#### Package Assembly\n",
        "\n",
        "In the following cells, assemble the training package."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "examine_training_package"
      },
      "outputs": [],
      "source": [
        "# Make folder for Python hyperparameter tuning script\n",
        "! rm -rf custom\n",
        "! mkdir custom\n",
        "\n",
        "# Add package information\n",
        "! touch custom/README.md\n",
        "\n",
        "setup_cfg = \"[egg_info]\\n\\ntag_build =\\n\\ntag_date = 0\"\n",
        "! echo \"$setup_cfg\" > custom/setup.cfg\n",
        "\n",
        "setup_py = \"import setuptools\\n\\nsetuptools.setup(\\n\\n    install_requires=[\\n\\n        'tensorflow==2.5.0',\\n\\n        'tensorflow_datasets==1.3.0',\\n\\n    ],\\n\\n    packages=setuptools.find_packages())\"\n",
        "! echo \"$setup_py\" > custom/setup.py\n",
        "\n",
        "pkg_info = \"Metadata-Version: 1.0\\n\\nName: Boston Housing tabular regression\\n\\nVersion: 0.0.0\\n\\nSummary: Demostration hyperparameter tuning script\\n\\nHome-page: www.google.com\\n\\nAuthor: Google\\n\\nAuthor-email: aferlitsch@google.com\\n\\nLicense: Public\\n\\nDescription: Demo\\n\\nPlatform: Vertex\"\n",
        "! echo \"$pkg_info\" > custom/PKG-INFO\n",
        "\n",
        "# Make the training subfolder\n",
        "! mkdir custom/trainer\n",
        "! touch custom/trainer/__init__.py"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "taskpy_contents:hpt,boston"
      },
      "source": [
        "#### Task.py contents\n",
        "\n",
        "In the next cell, write the contents of the hyperparameter tuning script *task.py*. \n",
        "\n",
        "To summarize, perform the following steps in the script:\n",
        "\n",
        "- Parse the command line arguments to obtain hyperparameter settings for the current trial.\n",
        "- Get the directory for saving the model artifacts from the command line (`--model_dir`), and if not specified, use the environment variable `AIP_MODEL_DIR`.\n",
        "- Download and preprocess the Boston Housing dataset.\n",
        "- Build a DNN model.\n",
        "- The hyperparameter values for the number of units per dense layer and the learning rate are utilized during the model's build and compile process.\n",
        "- Define a `HPTCallback` which obtains the validation loss at the end of each epoch (`on_epoch_end()`) and reports it to the hyperparameter tuning service using `hpt.report_hyperparameter_tuning_metric()`.\n",
        "- Train the model with the `fit()` method and specify a callback which reports the validation loss back to the hyperparameter tuning service."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "taskpy_contents:hpt,boston"
      },
      "outputs": [],
      "source": [
        "%%writefile custom/trainer/task.py\n",
        "# Custom Training for Boston Housing\n",
        "\n",
        "import tensorflow_datasets as tfds\n",
        "import tensorflow as tf\n",
        "from tensorflow.python.client import device_lib\n",
        "from hypertune import HyperTune\n",
        "import numpy as np\n",
        "import argparse\n",
        "import os\n",
        "import sys\n",
        "tfds.disable_progress_bar()\n",
        "\n",
        "parser = argparse.ArgumentParser()\n",
        "parser.add_argument('--model-dir', dest='model_dir',\n",
        "                    default=os.getenv('AIP_MODEL_DIR'), type=str, help='Model dir.')\n",
        "parser.add_argument('--lr', dest='lr',\n",
        "                    default=0.001, type=float,\n",
        "                    help='Learning rate.')\n",
        "parser.add_argument('--decay', dest='decay',\n",
        "                    default=0.98, type=float,\n",
        "                    help='Decay rate')\n",
        "parser.add_argument('--units', dest='units',\n",
        "                    default=64, type=int,\n",
        "                    help='Number of units.')\n",
        "parser.add_argument('--epochs', dest='epochs',\n",
        "                    default=20, type=int,\n",
        "                    help='Number of epochs.')\n",
        "parser.add_argument('--steps', dest='steps',\n",
        "                    default=200, type=int,\n",
        "                    help='Number of steps per epoch.')\n",
        "parser.add_argument('--param-file', dest='param_file',\n",
        "                    default='/tmp/param.txt', type=str,\n",
        "                    help='Output file for parameters')\n",
        "parser.add_argument('--distribute', dest='distribute', type=str, default='single',\n",
        "                    help='distributed training strategy')\n",
        "args = parser.parse_args()\n",
        "\n",
        "print('Python Version = {}'.format(sys.version))\n",
        "print('TensorFlow Version = {}'.format(tf.__version__))\n",
        "print('TF_CONFIG = {}'.format(os.environ.get('TF_CONFIG', 'Not found')))\n",
        "\n",
        "\n",
        "def make_dataset():\n",
        "\n",
        "  # Scaling Boston Housing data features\n",
        "  def scale(feature):\n",
        "    max = np.max(feature)\n",
        "    feature = (feature / max).astype(np.float)\n",
        "    return feature, max\n",
        "\n",
        "  (x_train, y_train), (x_test, y_test) = tf.keras.datasets.boston_housing.load_data(\n",
        "    path=\"boston_housing.npz\", test_split=0.2, seed=113\n",
        "  )\n",
        "  params = []\n",
        "  for _ in range(13):\n",
        "    x_train[_], max = scale(x_train[_])\n",
        "    x_test[_], _ = scale(x_test[_])\n",
        "    params.append(max)\n",
        "\n",
        "  # store the normalization (max) value for each feature\n",
        "  with tf.io.gfile.GFile(args.param_file, 'w') as f:\n",
        "    f.write(str(params))\n",
        "  return (x_train, y_train), (x_test, y_test)\n",
        "\n",
        "# Build the Keras model\n",
        "def build_and_compile_dnn_model():\n",
        "  model = tf.keras.Sequential([\n",
        "      tf.keras.layers.Dense(args.units, activation='relu', input_shape=(13,)),\n",
        "      tf.keras.layers.Dense(args.units, activation='relu'),\n",
        "      tf.keras.layers.Dense(1, activation='linear')\n",
        "  ])\n",
        "  model.compile(\n",
        "      loss='mse',\n",
        "      optimizer=tf.keras.optimizers.RMSprop(learning_rate=args.lr, decay=args.decay))\n",
        "  return model\n",
        "\n",
        "\n",
        "model = build_and_compile_dnn_model()\n",
        "\n",
        "# Instantiate the HyperTune reporting object\n",
        "hpt = HyperTune()\n",
        "\n",
        "# Reporting callback\n",
        "class HPTCallback(tf.keras.callbacks.Callback):\n",
        "\n",
        "    def on_epoch_end(self, epoch, logs=None):\n",
        "        global hpt\n",
        "        hpt.report_hyperparameter_tuning_metric(\n",
        "        hyperparameter_metric_tag='val_loss',\n",
        "        metric_value=logs['val_loss'],\n",
        "        global_step=epoch)\n",
        "\n",
        "# Train the model\n",
        "BATCH_SIZE = 16\n",
        "(x_train, y_train), (x_test, y_test) = make_dataset()\n",
        "model.fit(x_train, y_train, epochs=args.epochs, batch_size=BATCH_SIZE, validation_split=0.1, callbacks=[HPTCallback()])\n",
        "model.save(args.model_dir)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tarball_training_script"
      },
      "source": [
        "#### Store hyperparameter tuning script on your Cloud Storage bucket\n",
        "\n",
        "Next, package the hyperparameter tuning folder into a compressed tar ball, and then store it in your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tarball_training_script"
      },
      "outputs": [],
      "source": [
        "! rm -f custom.tar custom.tar.gz\n",
        "! tar cvf custom.tar custom\n",
        "! gzip custom.tar\n",
        "! gsutil cp custom.tar.gz $BUCKET_URI/trainer_boston.tar.gz"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "train_custom_job_machine_specification"
      },
      "source": [
        "### Prepare your machine specification\n",
        "\n",
        "Now define the machine specification for your custom hyperparameter tuning job. This tells Vertex AI about the type of machine instance to provision for the hyperparameter tuning.\n",
        "  - `machine_type`: The type of GCP instance to provision -- e.g., n1-standard-8.\n",
        "  - `accelerator_type`: The type, if any, of hardware accelerator. In this tutorial if you previously set the variable `TRAIN_GPU != None`, you're using a GPU; otherwise you're using a CPU.\n",
        "  - `accelerator_count`: The number of accelerators."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "train_custom_job_machine_specification"
      },
      "outputs": [],
      "source": [
        "if TRAIN_GPU:\n",
        "    machine_spec = {\n",
        "        \"machine_type\": TRAIN_COMPUTE,\n",
        "        \"accelerator_type\": TRAIN_GPU,\n",
        "        \"accelerator_count\": TRAIN_NGPU,\n",
        "    }\n",
        "else:\n",
        "    machine_spec = {\"machine_type\": TRAIN_COMPUTE, \"accelerator_count\": 0}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "train_custom_job_disk_specification"
      },
      "source": [
        "### Prepare your disk specification\n",
        "\n",
        "(**optional**) Now define the disk specification for your custom hyperparameter tuning job. This tells Vertex AI about the type and size of disk to provision in each machine instance for the hyperparameter tuning.\n",
        "\n",
        "  - `boot_disk_type`: Either SSD or Standard. SSD is faster, and Standard is less expensive. Defaults to SSD.\n",
        "  - `boot_disk_size_gb`: Size of disk in GB."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "train_custom_job_disk_specification"
      },
      "outputs": [],
      "source": [
        "DISK_TYPE = \"pd-ssd\"  # [ pd-ssd, pd-standard]\n",
        "DISK_SIZE = 200  # GB\n",
        "\n",
        "disk_spec = {\"boot_disk_type\": DISK_TYPE, \"boot_disk_size_gb\": DISK_SIZE}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "train_custom_job_worker_pool_specification:prebuilt_container"
      },
      "source": [
        "### Define the worker pool specification\n",
        "\n",
        "Next, define the worker pool specification for your custom hyperparameter tuning job. The worker pool specification consists of the following parameters:\n",
        "\n",
        "- `replica_count`: The number of instances of this machine type to provision.\n",
        "- `machine_spec`: The hardware specification.\n",
        "- `disk_spec` : (optional) The disk storage specification.\n",
        "- `python_package`: The Python training package to install on the VM instance(s) and which Python module to invoke, along with command line arguments for the Python module.\n",
        "\n",
        "    - `executor_image_spec`: This is the docker image which is configured for your custom hyperparameter tuning job.\n",
        "\n",
        "    - `package_uris`: This is a list of the locations (URIs) of your Python training packages to install on the provisioned instance. The locations need to be in a Cloud Storage bucket. These can be either individual Python files or a zip (archive) of an entire package. In the later case, the job service unzips (unarchives) the contents into the docker image.\n",
        "\n",
        "    - `python_module`: The Python module (script) to invoke for running the custom hyperparameter tuning job. In this example, invoke the `trainer.task.py` -- note that it isn't neccessary to append the `.py` suffix.\n",
        "\n",
        "    - `args`: The command line arguments to pass to the corresponding Python module. In this example, you're going to configure\n",
        "    \n",
        "        - `--model-dir` : The Cloud Storage location for where to store the model artifacts. There are two ways to tell the hyperparameter tuning script where to save the model artifacts:\n",
        "          \n",
        "          - **method-1**(set `DIRECT` to `True`): Pass the Cloud Storage location as a command line argument to your training script.\n",
        "          - **method-2**(set `DIRECT` to `False`): The service passes the Cloud Storage location as the environment variable `AIP_MODEL_DIR` to your training script. In this case, tell the service the model artifact location in the job specification.\n",
        "          \n",
        "    - `--epochs`: The number of epochs for training.\n",
        "    - `--steps`: The number of steps (batches) per epoch.\n",
        "    - `--distribute` : The hyperparameter tuning distribution strategy to use for single or distributed hyperparameter tuning.\n",
        "         - `\"single\"`: single device.\n",
        "         - `\"mirror\"`: all GPU devices on a single compute instance.\n",
        "         - `\"multi\"`: all GPU devices on all compute instances."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "train_custom_job_worker_pool_specification:prebuilt_container"
      },
      "outputs": [],
      "source": [
        "JOB_NAME = \"custom_job\"\n",
        "MODEL_DIR = \"{}/{}\".format(BUCKET_URI, JOB_NAME)\n",
        "\n",
        "if not TRAIN_NGPU or TRAIN_NGPU < 2:\n",
        "    TRAIN_STRATEGY = \"single\"\n",
        "else:\n",
        "    TRAIN_STRATEGY = \"mirror\"\n",
        "\n",
        "EPOCHS = 20\n",
        "STEPS = 100\n",
        "\n",
        "DIRECT = False\n",
        "if DIRECT:\n",
        "    CMDARGS = [\n",
        "        \"--model-dir=\" + MODEL_DIR,\n",
        "        \"--epochs=\" + str(EPOCHS),\n",
        "        \"--steps=\" + str(STEPS),\n",
        "        \"--distribute=\" + TRAIN_STRATEGY,\n",
        "    ]\n",
        "else:\n",
        "    CMDARGS = [\n",
        "        \"--epochs=\" + str(EPOCHS),\n",
        "        \"--steps=\" + str(STEPS),\n",
        "        \"--distribute=\" + TRAIN_STRATEGY,\n",
        "    ]\n",
        "\n",
        "worker_pool_spec = [\n",
        "    {\n",
        "        \"replica_count\": 1,\n",
        "        \"machine_spec\": machine_spec,\n",
        "        \"disk_spec\": disk_spec,\n",
        "        \"python_package_spec\": {\n",
        "            \"executor_image_uri\": TRAIN_IMAGE,\n",
        "            \"package_uris\": [BUCKET_URI + \"/trainer_boston.tar.gz\"],\n",
        "            \"python_module\": \"trainer.task\",\n",
        "            \"args\": CMDARGS,\n",
        "        },\n",
        "    }\n",
        "]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_custom_job:mbsdk"
      },
      "source": [
        "## Create a custom job\n",
        "\n",
        "Use the class `CustomJob` to create a custom job, such as for hyperparameter tuning, with the following parameters:\n",
        "\n",
        "- `display_name`: A human readable name for the custom job.\n",
        "- `worker_pool_specs`: The specification for the corresponding VM instances."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "create_custom_job:mbsdk"
      },
      "outputs": [],
      "source": [
        "job = aiplatform.CustomJob(display_name=\"boston\", worker_pool_specs=worker_pool_spec)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_hpt_job:mbsdk"
      },
      "source": [
        "## Create a hyperparameter tuning job\n",
        "\n",
        "Use the class `HyperparameterTuningJob` to create a hyperparameter tuning job, with the following parameters:\n",
        "\n",
        "- `display_name`: A human readable name for the custom job.\n",
        "- `custom_job`: The worker pool spec from this custom job applies to the CustomJobs created in all the trials.\n",
        "- `metrics_spec`: The metrics to optimize. The dictionary key is the metric_id, which is reported by your training job, and the dictionary value is the optimization goal of the metric('minimize' or 'maximize').\n",
        "- `parameter_spec`: The parameters to optimize. The dictionary key is the metric_id, which is passed into your training job as a command line key word argument, and the dictionary value is the parameter specification of the metric.\n",
        "- `search_algorithm`: The search algorithm to use: `grid`, `random` and `None`. If `None` is specified, the `Vizier` service (Bayesian) is used.\n",
        "- `max_trial_count`: The maximum number of trials to perform."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "create_hpt_job:mbsdk"
      },
      "outputs": [],
      "source": [
        "from google.cloud.aiplatform import hyperparameter_tuning as hpt\n",
        "\n",
        "hpt_job = aiplatform.HyperparameterTuningJob(\n",
        "    display_name=\"boston\",\n",
        "    custom_job=job,\n",
        "    metric_spec={\n",
        "        \"val_loss\": \"minimize\",\n",
        "    },\n",
        "    parameter_spec={\n",
        "        \"lr\": hpt.DoubleParameterSpec(min=0.001, max=0.1, scale=\"log\"),\n",
        "        \"units\": hpt.IntegerParameterSpec(min=4, max=128, scale=\"linear\"),\n",
        "    },\n",
        "    search_algorithm=\"random\",\n",
        "    max_trial_count=6,\n",
        "    parallel_trial_count=1,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "run_hpt_job:mbsdk"
      },
      "source": [
        "## Run the hyperparameter tuning job\n",
        "\n",
        "Use the `run()` method to execute the hyperparameter tuning job."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "run_hpt_job:mbsdk"
      },
      "outputs": [],
      "source": [
        "hpt_job.run()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hpt_trial_results:mbsdk"
      },
      "source": [
        "### Display the hyperparameter tuning job trial results\n",
        "\n",
        "Once the hyperparameter tuning job completes, the `trials` property provides the results for each individual trial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hpt_trial_results:mbsdk"
      },
      "outputs": [],
      "source": [
        "print(hpt_job.trials)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "best_trial:mbsdk"
      },
      "source": [
        "### Best trial\n",
        "\n",
        "Now identify which trial is the best."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "best_trial:mbsdk"
      },
      "outputs": [],
      "source": [
        "best = (None, None, None, 0.0)\n",
        "for trial in hpt_job.trials:\n",
        "    # Keep track of the best outcome\n",
        "    if float(trial.final_measurement.metrics[0].value) > best[3]:\n",
        "        try:\n",
        "            best = (\n",
        "                trial.id,\n",
        "                float(trial.parameters[0].value),\n",
        "                float(trial.parameters[1].value),\n",
        "                float(trial.final_measurement.metrics[0].value),\n",
        "            )\n",
        "        except:\n",
        "            best = (\n",
        "                trial.id,\n",
        "                float(trial.parameters[0].value),\n",
        "                None,\n",
        "                float(trial.final_measurement.metrics[0].value),\n",
        "            )\n",
        "\n",
        "print(best)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "get_best_model"
      },
      "source": [
        "## Get the Best Model\n",
        "\n",
        "If you're using the method where the service informs the tuning script where to save the model artifacts (`DIRECT = False`), then the model artifacts for the best model are saved at:\n",
        "\n",
        "    MODEL_DIR/<best_trial_id>/model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "get_best_model"
      },
      "outputs": [],
      "source": [
        "BEST_MODEL_DIR = MODEL_DIR + \"/\" + best[0] + \"/model\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "delete_hpt_job"
      },
      "source": [
        "### Delete the hyperparameter tuning job\n",
        "\n",
        "The method `delete()` deletes the hyperparameter tuning job."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "delete_hpt_job"
      },
      "outputs": [],
      "source": [
        "hpt_job.delete()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hpt_vizier_intro"
      },
      "source": [
        "## Vertex AI Hyperparameter Tuning and Vertex AI Vizer service combined\n",
        "\n",
        "The following example demonstrates how to setup, execute and evaluate trials using the Vertex AI hyperparameter tuning service with Vizier search service."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_custom_job:mbsdk"
      },
      "source": [
        "## Create a custom job\n",
        "\n",
        "Use the class `CustomJob` to create a custom job, such as for hyperparameter tuning, with the following parameters:\n",
        "\n",
        "- `display_name`: A human readable name for the custom job.\n",
        "- `worker_pool_specs`: The specification for the corresponding VM instances.\n",
        "- `base_output_dir`: The Cloud Storage location for storing the model artifacts."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "create_custom_job:mbsdk"
      },
      "outputs": [],
      "source": [
        "job = aiplatform.CustomJob(\n",
        "    display_name=\"boston\",\n",
        "    worker_pool_specs=worker_pool_spec,\n",
        "    base_output_dir=MODEL_DIR,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_hpt_job:mbsdk,vizier"
      },
      "source": [
        "## Create a hyperparameter tuning job\n",
        "\n",
        "Use the class `HyperparameterTuningJob` to create a hyperparameter tuning job, with the following parameters:\n",
        "\n",
        "- `display_name`: A human readable name for the custom job.\n",
        "- `custom_job`: The worker pool spec from this custom job applies to the CustomJobs created in all the trials.\n",
        "- `metrics_spec`: The metrics to optimize. The dictionary key is the metric_id, which is reported by your training job, and the dictionary value is the optimization goal of the metric('minimize' or 'maximize').\n",
        "- `parameter_spec`: The parameters to optimize. The dictionary key is the metric_id, which is passed into your training job as a command line key word argument, and the dictionary value is the parameter specification of the metric.\n",
        "- `search_algorithm`: The search algorithm to use: `grid`, `random` and `None`. If `None` is specified, the `Vizier` service (Bayesian) is used.\n",
        "- `max_trial_count`: The maximum number of trials to perform.\n",
        "- `parallel_trial_count`: (Optional) It specifies the number of trials to run in parallel."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "create_hpt_job:mbsdk,vizier"
      },
      "outputs": [],
      "source": [
        "from google.cloud.aiplatform import hyperparameter_tuning as hpt\n",
        "\n",
        "hpt_job = aiplatform.HyperparameterTuningJob(\n",
        "    display_name=\"boston\",\n",
        "    custom_job=job,\n",
        "    metric_spec={\n",
        "        \"val_loss\": \"minimize\",\n",
        "    },\n",
        "    parameter_spec={\n",
        "        \"lr\": hpt.DoubleParameterSpec(min=0.0001, max=0.1, scale=\"log\"),\n",
        "        \"units\": hpt.IntegerParameterSpec(min=4, max=512, scale=\"linear\"),\n",
        "    },\n",
        "    search_algorithm=None,\n",
        "    max_trial_count=12,\n",
        "    parallel_trial_count=1,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "run_hpt_job:mbsdk"
      },
      "source": [
        "## Run the hyperparameter tuning job\n",
        "\n",
        "Use the `run()` method to execute the hyperparameter tuning job."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "run_hpt_job:mbsdk"
      },
      "outputs": [],
      "source": [
        "hpt_job.run()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hpt_trial_results:mbsdk"
      },
      "source": [
        "### Display the hyperparameter tuning job trial results\n",
        "\n",
        "Once the hyperparameter tuning job completes, the `trials` property provides the results for each individual trial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hpt_trial_results:mbsdk"
      },
      "outputs": [],
      "source": [
        "print(hpt_job.trials)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "best_trial:mbsdk"
      },
      "source": [
        "### Best trial\n",
        "\n",
        "Now look at which trial performed the best:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "best_trial:mbsdk"
      },
      "outputs": [],
      "source": [
        "best = (None, None, None, 0.0)\n",
        "for trial in hpt_job.trials:\n",
        "    # Keep track of the best outcome\n",
        "    if float(trial.final_measurement.metrics[0].value) > best[3]:\n",
        "        try:\n",
        "            best = (\n",
        "                trial.id,\n",
        "                float(trial.parameters[0].value),\n",
        "                float(trial.parameters[1].value),\n",
        "                float(trial.final_measurement.metrics[0].value),\n",
        "            )\n",
        "        except:\n",
        "            best = (\n",
        "                trial.id,\n",
        "                float(trial.parameters[0].value),\n",
        "                None,\n",
        "                float(trial.final_measurement.metrics[0].value),\n",
        "            )\n",
        "\n",
        "print(best)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "get_best_model"
      },
      "source": [
        "### Get the Best Model\n",
        "\n",
        "If you're using the method where the service informs the tuning script where to save the model artifacts (`DIRECT = False`), then the model artifacts for the best model are saved at:\n",
        "\n",
        "    MODEL_DIR/<best_trial_id>/model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "get_best_model"
      },
      "outputs": [],
      "source": [
        "BEST_MODEL_DIR = MODEL_DIR + \"/\" + best[0] + \"/model\"\n",
        "\n",
        "! gsutil ls {BEST_MODEL_DIR}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "delete_hpt_job"
      },
      "source": [
        "### Delete the hyperparameter tuning job\n",
        "\n",
        "The method `delete()` deletes the hyperparameter tuning job."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "delete_hpt_job"
      },
      "outputs": [],
      "source": [
        "hpt_job.delete()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vizier_intro"
      },
      "source": [
        "## Standalone Vertex AI Vizer service\n",
        "\n",
        "The Vizier service can be used as a standalone service for selecting the next set of parameters for a trial.\n",
        "\n",
        "**Note:** The service doesn't execute trials. You create your own trial and execution.\n",
        "\n",
        "Learn more about how to [Create Vertex AI Vizier studies](https://cloud.google.com/vertex-ai/docs/vizier/using-vizier)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vizier_client"
      },
      "source": [
        "### Specify the algorithm used to suggest trial parameters\n",
        "\n",
        "First, create a `StudyConfig`, and specify the algorithm to suggest the next trial.\n",
        "\n",
        "    GRID_SEARCH: grid search\n",
        "    RANDOM_SEARCH: random search\n",
        "    ALGORIGTHM_UNSPECIFIED: Vizier bayesian algorithm"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d7dd26490358"
      },
      "outputs": [],
      "source": [
        "problem = pyvizier.StudyConfig()\n",
        "problem.algorithm = pyvizier.Algorithm.RANDOM_SEARCH"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_vizier_study"
      },
      "source": [
        "### Create a study\n",
        "\n",
        "A study is a series of experiments, or trials, that help you optimize your hyperparameters or parameters.\n",
        "\n",
        "In the following example, the goal is to maximize y = x^2 with x in the range of \\[-10. 10\\]. This example has only one parameter and uses an easily calculated function to help demonstrate how to use Vizier.\n",
        "\n",
        "First, specify the metrics to minimize or maximize in the study as a list to the property `metric_information`. Then specify the parameters to the study using the `add_XXX_params()` method for the corresponding data type:\n",
        "\n",
        "    - add_bool_param\n",
        "    - add_categorical_param\n",
        "    - add_discrete_param\n",
        "    - add_float_param\n",
        "    - add_int_param\n",
        "\n",
        "You can create the study using the `create_or_load()` method."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "create_vizier_study"
      },
      "outputs": [],
      "source": [
        "STUDY_DISPLAY_NAME = \"xpow2\"\n",
        "\n",
        "problem.metric_information.append(\n",
        "    pyvizier.MetricInformation(name=\"y\", goal=pyvizier.ObjectiveMetricGoal.MAXIMIZE)\n",
        ")\n",
        "\n",
        "params = problem.search_space.select_root()\n",
        "params.add_float_param(\"x\", -10.0, 10.0, scale_type=pyvizier.ScaleType.LINEAR)\n",
        "\n",
        "study = Study.create_or_load(display_name=STUDY_DISPLAY_NAME, problem=problem)\n",
        "\n",
        "STUDY_NAME = study.name\n",
        "print(\"STUDY_NAME: {}\".format(STUDY_NAME))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "get_vizier_study"
      },
      "source": [
        "### Get Vizier study\n",
        "\n",
        "You can get a study using the `list()` method."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "get_vizier_study"
      },
      "outputs": [],
      "source": [
        "studies = Study.list()\n",
        "print(studies[0].gca_resource)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vizier_suggest_trial"
      },
      "source": [
        "### Get suggested trial\n",
        "\n",
        "Next, query the Vizier service for a suggested trial(s) using the `suggest()` method, with the following key/value pairs:\n",
        "\n",
        "- `count`: The number of trials to suggest.\n",
        "\n",
        "This call is a long running operation. The `result()` method from the response object waits until the call completes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "11ff2c4562cb"
      },
      "outputs": [],
      "source": [
        "SUGGEST_COUNT = 1\n",
        "\n",
        "trials = study.suggest(count=SUGGEST_COUNT)\n",
        "\n",
        "print(trials)\n",
        "\n",
        "# Get the trial ID of the first trial\n",
        "TRIAL_ID = trials[0].name"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "evaluate_vizier_trial"
      },
      "source": [
        "### Evaluate the results\n",
        "After receiving your trial suggestions, evaluate each trial and record each result as a measurement.\n",
        "\n",
        "For example, if the function you're trying to optimize is y = x^2, then evaluate the function using the trial's suggested value of x. Using a suggested value of 0.1, the function evaluates to y = 0.1 * 0.1, which results in 0.01."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vizier_add_measurement"
      },
      "source": [
        "### Add a measurement\n",
        "\n",
        "After evaluating your trial suggestion to get a measurement, add this measurement to your trial.\n",
        "\n",
        "Use the following commands to store your measurement and send the request. In this example, replace `RESULT` with the measurement. If the function you're optimizing is y = x^2, and the suggested value of x is 0.1, the result is 0.01."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vizier_add_measurement"
      },
      "outputs": [],
      "source": [
        "RESULT = 0.01\n",
        "\n",
        "measurement = pyvizier.Measurement()\n",
        "measurement.metrics[\"y\"] = RESULT\n",
        "\n",
        "trials[0].add_measurement(measurement)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cleanup:mbsdk"
      },
      "source": [
        "# Cleaning up\n",
        "\n",
        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "Otherwise, you can delete the individual resources you created in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cleanup:mbsdk"
      },
      "outputs": [],
      "source": [
        "# delete the Vizier study\n",
        "study.delete()\n",
        "\n",
        "# remove locally generated files\n",
        "! rm -rf custom.tar.gz custom\n",
        "\n",
        "# deelte cloud storage bucket\n",
        "delete_bucket = False  # set True for deletion\n",
        "\n",
        "if delete_bucket:\n",
        "    ! gsutil rm -r $BUCKET_URI"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "get_started_vertex_vizier.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}