notebooks/official/experiments/get_started_with_custom_training_autologging_local_script.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ur8xi4C7S06n"
      },
      "outputs": [],
      "source": [
        "# Copyright 2022 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JAPoU8Sm5E6e"
      },
      "source": [
        "# Vertex AI Experiments: Custom training autologging - Local script\n",
        "\n",
        "<table align=\"left\">\n",
        "\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/experiments/get_started_with_custom_training_autologging_local_script.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"><br> Run in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fofficial%2Fexperiments%2Fget_started_with_custom_training_autologging_local_script.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://cloud.google.com/ml-engine/images/colab-enterprise-logo-32px.png\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/experiments/get_started_with_custom_training_autologging_local_script.ipynb\">\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br>\n",
        "      Open in Vertex AI Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/experiments/get_started_with_custom_training_autologging_local_script.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br>\n",
        "      View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>\n",
        "<br/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "24743cf4a1e1"
      },
      "source": [
        "**_NOTE_**: This notebook has been tested in the following environment:\n",
        "\n",
        "* Python version = 3.9"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tvgnzT1CKxrO"
      },
      "source": [
        "## Overview\n",
        "\n",
        "As a Data Scientist experimenting with large models, you need a way to run experiments on a scalable training service to log parameters and metrics. This guarantees reproducibility.\n",
        "\n",
        "With Vertex AI training and Vertex AI Experiments autologging integration, you can run your ML experiments at scale and autolog their parameters and metrics\n",
        "with the `enable_autolog` argument.\n",
        "\n",
        "Learn more about [Vertex AI Experiments](https://cloud.google.com/vertex-ai/docs/experiments/intro-vertex-ai-experiments)\n",
        "and how to [Autolog data to an experiment run](https://cloud.google.com/vertex-ai/docs/experiments/autolog-data)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d975e698c9a4"
      },
      "source": [
        "### Objective\n",
        "\n",
        "In this tutorial, you learn how to autolog parameters and metrics of an ML experiment running on Vertex AI Training by leveraging the integration with Vertex AI Experiments.\n",
        "\n",
        "This tutorial uses the following Google Cloud ML services and resources:\n",
        "\n",
        "- Vertex AI Experiments\n",
        "- Vertex AI Training\n",
        "\n",
        "The steps performed include:\n",
        "\n",
        "- Formalize model experiment in a script\n",
        "- Run model traning using local script on Vertex AI Training\n",
        "- Check out ML experiment parameters and metrics in Vertex AI Experiments"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "08d289fa873f"
      },
      "source": [
        "### Dataset\n",
        "\n",
        "The [Glass Identification dataset](https://archive-beta.ics.uci.edu/dataset/42/glass+identification) is a dataset from USA Forensic Science Service with 6 types of glass defined in terms of their oxide content (for example, Na, Fe, K). The goal is to classify the types of glass based on oxide features."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aed92deeb4a0"
      },
      "source": [
        "### Costs\n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* Vertex AI\n",
        "* Cloud Storage\n",
        "\n",
        "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing),\n",
        "and [Cloud Storage pricing](https://cloud.google.com/storage/pricing),\n",
        "and use the [Pricing Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d1ea81ac77f0"
      },
      "source": [
        "## Get started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i7EUnXsZhAGF"
      },
      "source": [
        "### Install Vertex AI SDK for Python and other required packages\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2b4ef9b72d43"
      },
      "outputs": [],
      "source": [
        "# Install the packages\n",
        "USER = \"\"\n",
        "! pip3 install {USER} --upgrade google-cloud-aiplatform --quiet --no-warn-conflicts"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "58707a750154"
      },
      "source": [
        "### Restart runtime (Colab only)\n",
        "\n",
        "To use the newly installed packages, you must restart the runtime on Google Colab."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f200f10a1da3"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "\n",
        "    import IPython\n",
        "\n",
        "    app = IPython.Application.instance()\n",
        "    app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c87a2a5d7e35"
      },
      "source": [
        "<div class=\"alert alert-block alert-warning\">\n",
        "<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
        "</div>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5dccb1c8feb6"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "Authenticate your environment on Google Colab.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cc7251520a07"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c2fc3d7b6bfa"
      },
      "source": [
        "### Set Google Cloud project information and initialize Vertex AI SDK for Python\n",
        "\n",
        "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f02130bff721"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
        "LOCATION = \"us-central1\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zgPO1eR3CYjk"
      },
      "source": [
        "### Create a Cloud Storage bucket\n",
        "\n",
        "Create a storage bucket to store intermediate artifacts such as datasets.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MzGDU7TWdts_"
      },
      "outputs": [],
      "source": [
        "BUCKET_URI = f\"gs://your-bucket-name-{PROJECT_ID}-unique\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-EcIXiGsCePi"
      },
      "source": [
        "**If your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NIq7R4HZCfIc"
      },
      "outputs": [],
      "source": [
        "! gsutil mb -l $LOCATION -p $PROJECT_ID $BUCKET_URI"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3d5191a94246"
      },
      "source": [
        "### Initialize Vertex AI SDK for Python"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "de483dc2a7ee"
      },
      "outputs": [],
      "source": [
        "from google.cloud import aiplatform as vertex_ai\n",
        "\n",
        "vertex_ai.init(project=PROJECT_ID, location=LOCATION, staging_bucket=BUCKET_URI)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-_8nZXd7NqIj"
      },
      "source": [
        "### UUID\n",
        "If you’re in a live tutorial session, you may be using a shared test account or project. To avoid name collisions between users on resources created, create a Universal Unique Identifier (uuid) for each instance session. Append the UUID to the name of the resources you create in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dY-WpyyzNtS0"
      },
      "outputs": [],
      "source": [
        "import random\n",
        "import string\n",
        "\n",
        "\n",
        "# Generate a uuid of length 8\n",
        "def generate_uuid():\n",
        "    return \"\".join(random.choices(string.ascii_lowercase + string.digits, k=8))\n",
        "\n",
        "\n",
        "UUID = generate_uuid()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "set_service_account"
      },
      "source": [
        "### Set Service Account\n",
        "\n",
        "**If you don't know your service account**, try to get your service account using `gcloud` command by executing the second cell below.\n",
        "\n",
        "*Note:* The code for automatically finding your service account works on a user-managed Workbench AI noteboook.\n",
        "If you are using a fully-managed notebook, you'll need to manually enter your service account."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "M4WZi4CDJ39n"
      },
      "outputs": [],
      "source": [
        "SERVICE_ACCOUNT = \"[your-service-account]\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "autoset_service_account"
      },
      "outputs": [],
      "source": [
        "IS_COLAB = False\n",
        "if \"google.colab\" in sys.modules:\n",
        "    IS_COLAB = True\n",
        "\n",
        "\n",
        "if (\n",
        "    SERVICE_ACCOUNT == \"\"\n",
        "    or SERVICE_ACCOUNT is None\n",
        "    or SERVICE_ACCOUNT == \"[your-service-account]\"\n",
        "):\n",
        "    # Get your service account from gcloud\n",
        "    if not IS_COLAB:\n",
        "        shell_output = !gcloud auth list 2>/dev/null\n",
        "        SERVICE_ACCOUNT = shell_output[2].replace(\"*\", \"\").strip()\n",
        "\n",
        "    if IS_COLAB:\n",
        "        shell_output = ! gcloud projects describe  $PROJECT_ID\n",
        "        project_number = shell_output[-1].split(\":\")[1].strip().replace(\"'\", \"\")\n",
        "        SERVICE_ACCOUNT = f\"{project_number}-compute@developer.gserviceaccount.com\"\n",
        "\n",
        "    print(\"Service Account:\", SERVICE_ACCOUNT)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "set_service_account:pipelines"
      },
      "source": [
        "#### Set service account access for Vertex AI Training\n",
        "\n",
        "Run the following commands to grant your service account access to read and update metadata in Vertex AI ML Metadata while the custom training job is running -- you only need to run these once per service account. Check out the [documentation](https://cloud.google.com/vertex-ai/docs/experiments/tensorboard-training#create_a_service_account_with_required_permissions) to get more information."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mI3IJONMJ39n"
      },
      "outputs": [],
      "source": [
        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_URI\n",
        "\n",
        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_URI"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UlICvTdOtyLN"
      },
      "source": [
        "### Set up project template\n",
        "Set the folder you use in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rxRaeiQIt6kn"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "TUTORIAL_DIR = os.path.join(\n",
        "    os.getcwd(), \"custom_training_autologging_local_script_tutorial\"\n",
        ")\n",
        "os.makedirs(TUTORIAL_DIR, exist_ok=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-flLpWUeOX_n"
      },
      "source": [
        "### Get dataset\n",
        "Get the glass classification dataset from the public Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eRBPBlj1PBUK"
      },
      "outputs": [],
      "source": [
        "SOURCE_DATA_URL = \"gs://cloud-samples-data/vertex-ai/dataset-management/datasets/uci_glass_preprocessed/glass.csv\"\n",
        "DESTINATION_DATA_URL = f\"{BUCKET_URI}/data/glass.csv\"\n",
        "\n",
        "! gsutil cp $SOURCE_DATA_URL $DESTINATION_DATA_URL"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "960505627ddf"
      },
      "source": [
        "### Import libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PyQmSRbKA8r-"
      },
      "outputs": [],
      "source": [
        "import os"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rLzhLCJVKIHO"
      },
      "source": [
        "### Define constants"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sWa5mNJEKKEx"
      },
      "outputs": [],
      "source": [
        "# Training\n",
        "EXPERIMENT_NAME = f\"glass-classification-{UUID}\"\n",
        "TRAIN_SCRIPT_PATH = os.path.join(TUTORIAL_DIR, \"task.py\")\n",
        "JOB_DISPLAY_NAME = f\"sklearn-autologged-custom-job-{UUID}\"\n",
        "PRE_BUILT_TRAINING_CONTAINER_IMAGE_URI = f\"{LOCATION.split('-')[0]}-docker.pkg.dev/vertex-ai/training/tf-cpu.2-12.py310:latest\"\n",
        "MODEL_FILE_URI = f\"{BUCKET_URI}/models/model.joblib\"\n",
        "DESTINATION_DATA_PATH = DESTINATION_DATA_URL.replace(\"gs://\", \"/gcs/\")\n",
        "MODEL_FILE_PATH = MODEL_FILE_URI.replace(\"gs://\", \"/gcs/\")\n",
        "REPLICA_COUNT = 1\n",
        "TRAIN_MACHINE_TYPE = \"n1-standard-4\"\n",
        "TRAINING_JOBS_URI = f\"{BUCKET_URI}/jobs\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wJoFEV5VNWb3"
      },
      "source": [
        "### Create an experiment for tracking training parameters and metrics\n",
        "\n",
        "To start, initiate an experiment using the `init()` method."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8ZEB_90wNc2J"
      },
      "outputs": [],
      "source": [
        "vertex_ai.init(\n",
        "    project=PROJECT_ID,\n",
        "    location=LOCATION,\n",
        "    staging_bucket=BUCKET_URI,\n",
        "    experiment=EXPERIMENT_NAME,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ig4h3BP_EuqF"
      },
      "source": [
        "### Train a scikit-learn model with a prebuilt container\n",
        "\n",
        "Then, you train a custom model using a prebuilt container for scikit-learn models.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yeKHxs4TK0Es"
      },
      "source": [
        "#### Create scikit-learn training script"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zOKUe_J3JWBs"
      },
      "outputs": [],
      "source": [
        "task_script = f\"\"\"\n",
        "#!/usr/bin/env python3\n",
        "\n",
        "'''\n",
        "A simple module to train a classifier on the glass dataset.\n",
        "'''\n",
        "\n",
        "# Libraries\n",
        "import argparse\n",
        "from pathlib import Path\n",
        "import pandas as pd\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "from sklearn.metrics import accuracy_score\n",
        "import joblib\n",
        "\n",
        "\n",
        "# Variables\n",
        "DATA_PATH = '{DESTINATION_DATA_PATH}'\n",
        "MODEL_PATH = '{MODEL_FILE_PATH}'\n",
        "TEST_SIZE = 0.2\n",
        "SEED = 8\n",
        "\n",
        "# Helpers\n",
        "def read_data(path):\n",
        "    df = pd.read_csv(path)\n",
        "    return df\n",
        "\n",
        "\n",
        "def split_data(df):\n",
        "    y = df.pop('glass_type')\n",
        "    X = df\n",
        "    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=TEST_SIZE, random_state=SEED)\n",
        "    return X_train, X_test, y_train, y_test\n",
        "\n",
        "\n",
        "def train_model(X_train, y_train):\n",
        "    model = RandomForestClassifier(n_estimators=5)\n",
        "    model.fit(X_train, y_train)\n",
        "    return model\n",
        "\n",
        "\n",
        "def evaluate_model(model, X_test, y_test):\n",
        "    y_pred = model.predict(X_test)\n",
        "    accuracy = accuracy_score(y_test, y_pred)\n",
        "    return accuracy\n",
        "\n",
        "\n",
        "def save_model(model, path):\n",
        "    p = Path(path)\n",
        "    if not p.parent.exists():\n",
        "      p.parent.mkdir(parents=True)\n",
        "    joblib.dump(model, path)\n",
        "\n",
        "\n",
        "def main():\n",
        "\n",
        "    # Read data\n",
        "    df = read_data(DATA_PATH)\n",
        "\n",
        "    # Split data\n",
        "    X_train, X_test, y_train, y_test = split_data(df)\n",
        "\n",
        "    # Train model\n",
        "    model = train_model(X_train, y_train)\n",
        "\n",
        "    # Evaluate model\n",
        "    accuracy = evaluate_model(model, X_test, y_test)\n",
        "    print('Model accuracy:', accuracy)\n",
        "\n",
        "    # Save model\n",
        "    save_model(model, MODEL_PATH)\n",
        "\n",
        "\n",
        "\n",
        "if __name__ == '__main__':\n",
        "\n",
        "    # Run main\n",
        "    main()\n",
        "\"\"\"\n",
        "\n",
        "with open(TRAIN_SCRIPT_PATH, \"w\") as train_file:\n",
        "    train_file.write(task_script)\n",
        "train_file.close()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7tXILrH3Kr4p"
      },
      "source": [
        "#### Define custom training job\n",
        "\n",
        "Define a custom job with the prebuilt container image for training code packaged as Python script. In this case, you set `enable_autolog=True` to automatically track parameters and metrics after the training job completes. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jACJpf78LFoy"
      },
      "outputs": [],
      "source": [
        "job = vertex_ai.CustomJob.from_local_script(\n",
        "    project=PROJECT_ID,\n",
        "    staging_bucket=TRAINING_JOBS_URI,\n",
        "    display_name=JOB_DISPLAY_NAME,\n",
        "    script_path=TRAIN_SCRIPT_PATH,\n",
        "    container_uri=PRE_BUILT_TRAINING_CONTAINER_IMAGE_URI,\n",
        "    requirements=[\"pandas\", \"scikit-learn\"],\n",
        "    replica_count=REPLICA_COUNT,\n",
        "    machine_type=TRAIN_MACHINE_TYPE,\n",
        "    enable_autolog=True,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xfaE5rK-MYxO"
      },
      "source": [
        "### Run custom training job\n",
        "Next, you run the training job using the method `run`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4-Jn9rJeMUZa"
      },
      "outputs": [],
      "source": [
        "job.run(experiment=EXPERIMENT_NAME, service_account=SERVICE_ACCOUNT)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SYwi0B3cd515"
      },
      "source": [
        "### Get your autologged experiment\n",
        "\n",
        "After you train your model, you can get parameters and metrics of the autologged experiment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IQCXPih6d9nA"
      },
      "outputs": [],
      "source": [
        "experiment_df = vertex_ai.get_experiment_df(experiment=EXPERIMENT_NAME)\n",
        "experiment_df.T"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ffcedb5809e4"
      },
      "source": [
        "Also you can get custom training job metadata associated with the experiment you run. You use `job_spec` property of the `CustomJob` class to print custom job metadata such as the training python package, training resources and more. \n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fI2xd4DbJvHC"
      },
      "outputs": [],
      "source": [
        "job.job_spec"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TpV-iwP9qw9c"
      },
      "source": [
        "## Cleaning up\n",
        "\n",
        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "Otherwise, you can delete the individual resources you created in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sx_vKniMq9ZX"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "# delete flags\n",
        "delete_experiment = False\n",
        "delete_bucket = False\n",
        "\n",
        "# Delete experiment\n",
        "if delete_experiment or os.getenv(\"IS_TESTING\"):\n",
        "    experiment = vertex_ai.Experiment.get(experiment_name=EXPERIMENT_NAME)\n",
        "    experiment.delete(delete_backing_tensorboard_runs=True)\n",
        "\n",
        "# Delete Cloud Storage objects that were created\n",
        "if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
        "    ! gsutil -m rm -r $BUCKET_URI"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "get_started_with_custom_training_autologging_local_script.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}