notebooks/official/model_monitoring/get_started_with_model_monitoring_custom.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "ur8xi4C7S06n"
      },
      "outputs": [],
      "source": [
        "# @title Copyright & License (click to expand)\n",
        "# Copyright 2022 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fsv4jGuU89rX"
      },
      "source": [
        "# Vertex AI Model Monitoring for custom tabular models\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model_monitoring/get_started_with_model_monitoring_custom.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fofficial%2Fmodel_monitoring%2Fget_started_with_model_monitoring_custom.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://cloud.google.com/ml-engine/images/colab-enterprise-logo-32px.png\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
        "    </a>\n",
        "  </td>    \n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/model_monitoring/get_started_with_model_monitoring_custom.ipynb\">\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model_monitoring/get_started_with_model_monitoring_custom.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lA32H1oKGgpf"
      },
      "source": [
        "## Overview\n",
        "\n",
        "This tutorial demonstrates how to use Vertex AI Model Monitoring for custom tabular models.\n",
        "\n",
        "Learn more about [Vertex AI Model Monitoring](https://cloud.google.com/vertex-ai/docs/model-monitoring)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "de998a3953c9"
      },
      "source": [
        "### Objective\n",
        "\n",
        "In this notebook, you learn to use the Vertex AI Model Monitoring service to detect feature skewness and drift in the input predict requests, for custom tabular models.\n",
        "\n",
        "This tutorial uses the following Vertex AI services:\n",
        "\n",
        "- Vertex AI Model Monitoring\n",
        "- Vertex AI Prediction\n",
        "- Vertex AI model resource\n",
        "- Vertex AI endpoint resource\n",
        "\n",
        "The steps performed include:\n",
        "\n",
        "- Download a pre-trained custom tabular model.\n",
        "- Upload the pre-trained model to Vertex AI Model Registry.\n",
        "- Deploy the model resource to a Vertex AI endpoint resource.\n",
        "- Configure the endpoint resource for model monitoring.\n",
        "- Generate synthetic prediction requests to simulate skewness.\n",
        "- Wait for email alert notifications.\n",
        "- Generate synthetic prediction requests to simulate drift.\n",
        "- Wait for email alert notifications.\n",
        "\n",
        "Learn more about [Vertex AI Model Monitoring](https://cloud.google.com/vertex-ai/docs/model-monitoring/overview)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "edba71dc9840"
      },
      "source": [
        "### Model\n",
        "\n",
        "This tutorial uses a pre-trained model, where the model artifacts are stored in a public Cloud Storage bucket. \n",
        "\n",
        "The model is based on a [blog post about Churn prediction model](https://cloud.google.com/blog/topics/developers-practitioners/churn-prediction-game-developers-using-google-analytics-4-ga4-and-bigquery-ml). This model involves extensive log data describing how game users have interacted with a site. The raw data contains the following categories of information:\n",
        "\n",
        "- identity - unique player identitity numbers.\n",
        "- demographic features - information about the player, such as the geographic region in which a player is located.\n",
        "- behavioral features - counts of the number of times a player has triggered certain game events, such as reaching a new level.\n",
        "- churn propensity - this is the label or target feature. It provides an estimated probability that this player may churn, i.e., stop being an active player."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t6Cd51FkG09E"
      },
      "source": [
        "### Costs \n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* Vertex AI\n",
        "* BigQuery\n",
        "* Cloud Storage\n",
        "\n",
        "Learn about [Vertext AI\n",
        "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
        "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
        "Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4b331e2fd155"
      },
      "source": [
        "## Get Started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f3848df1e5b0"
      },
      "source": [
        "### Install Vertex AI SDK for Python and other required packages\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "b24b232ee039"
      },
      "outputs": [],
      "source": [
        "# Install required packages.\n",
        "! pip3 install --quiet --upgrade google-cloud-aiplatform \\\n",
        "                                 google-cloud-bigquery"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "restart"
      },
      "source": [
        "### Restart runtime (Colab only)\n",
        "\n",
        "To use the newly installed packages, you must restart the runtime on Google Colab."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "D-ZBOjErv5mM"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "\n",
        "    import IPython\n",
        "\n",
        "    app = IPython.Application.instance()\n",
        "    app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ee775571c2b5"
      },
      "source": [
        "<div class=\"alert alert-block alert-warning\">\n",
        "<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
        "</div>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "92e68cfc3a90"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "Authenticate your environment on Google Colab.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "46604f70e831"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "before_you_begin:nogpu"
      },
      "source": [
        "### Set Google Cloud project information \n",
        "\n",
        "Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "set_project_id"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
        "LOCATION = \"us-central1\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "42c8a7c56abd"
      },
      "source": [
        "### User email\n",
        "\n",
        "Set your user email address to receive monitoring alerts."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ce2589511bb6"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "USER_EMAIL = \"[your-email-address]\"  # @param {type:\"string\"}\n",
        "\n",
        "if os.getenv(\"IS_TESTING\"):\n",
        "    USER_EMAIL = \"noreply@google.com\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8RJ3_20etd31"
      },
      "source": [
        "### Notes about service account and permission\n",
        "\n",
        "**By default no configuration is required**, if you run into any permission related issue, please make sure the service accounts above have the required roles:\n",
        "\n",
        "|Service account email|Description|Roles|\n",
        "|---|---|---|\n",
        "|PROJECT_NUMBER-compute@developer.gserviceaccount.com|Compute Engine default service account|Dataflow Admin, Dataflow Worker, Storage Admin, BigQuery Admin, Vertex AI User|\n",
        "|service-PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com|AI Platform Service Agent|Vertex AI Service Agent|\n",
        "\n",
        "\n",
        "1. Go to [IAM console](https://console.cloud.google.com/iam-admin/iam).\n",
        "2. Check the **Include Google-provided role grants** checkbox.\n",
        "3. Find the above emails.\n",
        "4. Grant the corresponding roles."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1d8540ae3bb4"
      },
      "source": [
        "### Using data source from a different project\n",
        "\n",
        "If you're using data sources from a different project:\n",
        "\n",
        "- For BigQuery data source, grant the \"BigQuery Data Viewer\" role to both the service accounts.\n",
        "- For CSV data source, grant the \"Storage Object Viewer\" role to both the service accounts."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bucket:mbsdk"
      },
      "source": [
        "### Create a Cloud Storage bucket\n",
        "\n",
        "Create a storage bucket to store intermediate artifacts such as datasets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bucket"
      },
      "outputs": [],
      "source": [
        "BUCKET_URI = f\"gs://your-bucket-name-{PROJECT_ID}-unique\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "autoset_bucket"
      },
      "source": [
        "**If your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "91c46850b49b"
      },
      "outputs": [],
      "source": [
        "! gsutil mb -l $LOCATION -p $PROJECT_ID $BUCKET_URI"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a0d294ff6d10"
      },
      "source": [
        "### Import libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bd7a633296eb"
      },
      "outputs": [],
      "source": [
        "import google.cloud.aiplatform as aiplatform\n",
        "from google.cloud import bigquery\n",
        "from google.cloud.aiplatform import model_monitoring"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "daecf7b4cdc0"
      },
      "source": [
        "### Initialize Vertex AI SDK for Python\n",
        "\n",
        "To get started using Vertex AI, you must [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "init_aip:mbsdk,all"
      },
      "outputs": [],
      "source": [
        "aiplatform.init(project=PROJECT_ID, location=LOCATION, staging_bucket=BUCKET_URI)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "init_bq"
      },
      "source": [
        "### Create BigQuery client\n",
        "\n",
        "In this tutorial, you use data from the same public BigQuery table that was used to train the pre-trained model. You create a client interface, which you subsequently use to access the data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "init_bq"
      },
      "outputs": [],
      "source": [
        "bqclient = bigquery.Client(project=PROJECT_ID)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "accelerators:training,prediction"
      },
      "source": [
        "#### Set hardware accelerators\n",
        "\n",
        "You can set hardware accelerators for prediction (e.g., GPUs) or use only CPUs. Hardware accelertors lower the latency response for a prediction request. When choosing a hardware accelerator, consider the additional cost trade-off over latency.\n",
        "\n",
        "Set the variables `DEPLOY_GPU/DEPLOY_NGPU` to use a container image supporting a GPU and the number of GPUs allocated to the virtual machine (VM) instance. For example, to use a GPU container image with 4 Nvidia Tesla T4 GPUs allocated to each VM, you would specify:\n",
        "\n",
        "    (aiplatform.gapic.AcceleratorType.NVIDIA_TESLA_T4, 4)\n",
        "\n",
        "See the [locations where accelerators are available](https://cloud.google.com/vertex-ai/docs/general/locations#accelerators).\n",
        "\n",
        "Otherwise specify `(None, None)` to use a container image to run on a CPU."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xd5PLXDTlugv"
      },
      "outputs": [],
      "source": [
        "GPU = False\n",
        "if GPU:\n",
        "    DEPLOY_GPU, DEPLOY_NGPU = (aiplatform.gapic.AcceleratorType.NVIDIA_TESLA_T4, 1)\n",
        "else:\n",
        "    DEPLOY_GPU, DEPLOY_NGPU = (None, None)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "container:training,prediction"
      },
      "source": [
        "#### Set pre-built containers\n",
        "\n",
        "Set the pre-built Docker container image for prediction.\n",
        "\n",
        "For the latest list, see [Pre-built containers for prediction](https://cloud.google.com/ai-platform-unified/docs/predictions/pre-built-containers)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1u1mr18jlugv"
      },
      "outputs": [],
      "source": [
        "if GPU:\n",
        "    DEPLOY_VERSION = \"tf2-gpu.2-5\"\n",
        "else:\n",
        "    DEPLOY_VERSION = \"tf2-cpu.2-5\"\n",
        "\n",
        "DEPLOY_IMAGE = \"{}-docker.pkg.dev/vertex-ai/prediction/{}:latest\".format(\n",
        "    LOCATION.split(\"-\")[0], DEPLOY_VERSION\n",
        ")\n",
        "\n",
        "print(\"Deployment:\", DEPLOY_IMAGE, DEPLOY_GPU, DEPLOY_NGPU)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "machine:training,prediction"
      },
      "source": [
        "#### Set machine types\n",
        "\n",
        "Next, set the machine types to use for training and prediction.\n",
        "\n",
        "- Set the variable `DEPLOY_COMPUTE` to configure your compute resources for prediction.\n",
        " - Set a `machine type`:\n",
        "     - `n1-standard`: 3.75GB of memory per vCPU\n",
        "     - `n1-highmem`: 6.5GB of memory per vCPU\n",
        "     - `n1-highcpu`: 0.9 GB of memory per vCPU\n",
        " - `vCPUs`: number of \\[2, 4, 8, 16, 32, 64, 96 \\]\n",
        "\n",
        "**Note**: You may also use n2 and e2 machine types for training and deployment, but they don't support GPUs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YAXwbqKKlugv"
      },
      "outputs": [],
      "source": [
        "MACHINE_TYPE = \"n1-standard\"\n",
        "\n",
        "VCPU = \"4\"\n",
        "TRAIN_COMPUTE = MACHINE_TYPE + \"-\" + VCPU\n",
        "print(\"Train machine type\", TRAIN_COMPUTE)\n",
        "\n",
        "MACHINE_TYPE = \"n1-standard\"\n",
        "\n",
        "VCPU = \"4\"\n",
        "DEPLOY_COMPUTE = MACHINE_TYPE + \"-\" + VCPU\n",
        "print(\"Deploy machine type\", DEPLOY_COMPUTE)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b890e02cb4b0"
      },
      "source": [
        "## Introduction to Vertex AI Model Monitoring\n",
        "\n",
        "Vertex AI Model Monitoring is supported for AutoML tabular models and custom tabular models. You can monitor for skewness and drift detection of the features in the inbound prediction requests or the feature attributions (Explainable AI) in the outbound prediction responses. In other words, you monitor the distribution of the attributions that quantify feature contributions to the output (predictions).\n",
        "\n",
        "The following are the basic steps to enable model monitoring:\n",
        "\n",
        "1. Deploy a Vertex AI AutoML or custom tabular model to a Vertex AI endpoint.\n",
        "2. Configure a model monitoring specification.\n",
        "3. Upload the model monitoring specification to the Vertex AI endpoint.\n",
        "4. Upload schema or use automatic generation of the *input schema* for parsing.\n",
        "5. For feature skewness detection, upload the training data. This enables automatic generation of the feature distributions.\n",
        "6. For feature attributions, upload the corresponding Vertex Explainable AI specification.\n",
        "\n",
        "Once configured, you can enable/disable monitoring, change alerts and update the model monitoring configuration. \n",
        "\n",
        "When model monitoring is enabled, the sampled incoming prediction requests are logged into a BigQuery table. The input feature values contained in the logged requests are then analyzed for skewness or drift on a specified interval basis. You set a sampling rate to monitor a subset of the production inputs to the model, and the monitoring interval.\n",
        "\n",
        "The model monitoring service needs to know how to parse the feature values, which is referred to as the input schema. For AutoML tabular models, the input schema is automatically generated. For custom tabular models, the service attempts to automatically derive the input schema from the first 1000 prediction requests. Alternatively, one can upload the input schema.\n",
        "\n",
        "For skewness detection, the monitoring service requires a baseline for the statistical distribution of values in the training data. For AutoML tabular models this is automatically derived. For custom tabular models, you upload the training data to the service, and have the service automatically derive the distribution.\n",
        "\n",
        "For skewness and drift detection in feature attributions, you're required to enable Vertex Explainable AI feature for your deployed custom tabular models. For AutoML models, Vertex Explainable AI is automatically enabled.\n",
        "\n",
        "Learn more about [Introduction to Vertex AI Model Monitoring](https://cloud.google.com/vertex-ai/docs/model-monitoring/overview)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9bf06cd476e9"
      },
      "source": [
        "### Upload the model artifacts to Vertex AI Model Registry\n",
        "\n",
        "First, upload the pre-trained custom tabular model artifacts as a Vertex AI model resource using the `upload()` method, with the following parameters:\n",
        "\n",
        "- `display_name`: The human readable name for the model resource.\n",
        "- `artifact_uri`: The Cloud Storage location of the model artifacts.\n",
        "- `serving_container_image`: The serving container image to use when the model is deployed to a Vertex AI endpoint resource.\n",
        "- `sync`: Whether to wait for the process to complete, or return immediately (async)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0193f247e216"
      },
      "outputs": [],
      "source": [
        "MODEL_ARTIFACT_URI = \"gs://mco-mm/churn\"\n",
        "\n",
        "model = aiplatform.Model.upload(\n",
        "    display_name=\"churn\",\n",
        "    artifact_uri=MODEL_ARTIFACT_URI,\n",
        "    serving_container_image_uri=DEPLOY_IMAGE,\n",
        "    sync=True,\n",
        ")\n",
        "\n",
        "print(model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1069c6f0eba8"
      },
      "source": [
        "### Deploy the model to an endpoint\n",
        "\n",
        "Next, deploy your Vertex AI model resource to a Vertex AI endpoint resource using the `deploy()` method, with the following parameters:\n",
        "\n",
        "- `deploy_model_display`: The human reable name for the deployed model.\n",
        "- `machine_type`: The machine type for each VM node instance.\n",
        "- `min_replica_count`: The minimum number of nodes to provision for auto-scaling.\n",
        "- `max_replica_count`: The maximum number of nodes to provision for auto-scaling.\n",
        "- `accelerator_type`: The type, if any, of GPU accelators per provisioned node.\n",
        "- `accelrator_count`: The number, if any, of GPU accelators per provisioned node."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "b883402dda8c"
      },
      "outputs": [],
      "source": [
        "MIN_NODES = 1\n",
        "MAX_NODES = 1\n",
        "\n",
        "if GPU:\n",
        "    endpoint = model.deploy(\n",
        "        deployed_model_display_name=\"churn\",\n",
        "        machine_type=DEPLOY_COMPUTE,\n",
        "        min_replica_count=MIN_NODES,\n",
        "        max_replica_count=MAX_NODES,\n",
        "        accelerator_type=DEPLOY_GPU.name,\n",
        "        accelerator_count=DEPLOY_NGPU,\n",
        "    )\n",
        "else:\n",
        "    endpoint = model.deploy(\n",
        "        deployed_model_display_name=\"churn\",\n",
        "        machine_type=DEPLOY_COMPUTE,\n",
        "        min_replica_count=MIN_NODES,\n",
        "        max_replica_count=MAX_NODES,\n",
        "    )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ba4a2924aec6"
      },
      "source": [
        "## Configure a monitoring job\n",
        "\n",
        "Configuring the monitoring job consists of the following specifications:\n",
        "\n",
        "- `alert_config`: The email address(es) that are supposed to receive the monitoring alerts.\n",
        "- `schedule_config`: The time window to analyze predictions.\n",
        "- `logging_sampling_strategy`: The rate for sampling prediction requests. \n",
        "- `drift_config`: The features and drift thresholds to monitor.\n",
        "- `skew_config`: The features and skewness thresholds to monitor."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "87abd93294c4"
      },
      "source": [
        "### Configure the alerting specification\n",
        "\n",
        "Configure the `alerting_config` specification with the following settings:\n",
        "\n",
        "- `user_emails`: A list of one or more emails that should receive the alerts.\n",
        "- `enable_logging`: Streams detected anomalies to Cloud Logging. Default is False."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "be6137df0086"
      },
      "outputs": [],
      "source": [
        "# Create alerting configuration.\n",
        "alerting_config = model_monitoring.EmailAlertConfig(\n",
        "    user_emails=[USER_EMAIL], enable_logging=True\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "66c7283276bc"
      },
      "source": [
        "### Configure the monitoring interval specification\n",
        "\n",
        "Next, you configure the `schedule_config` specification with the following settings:\n",
        "\n",
        "- `monitor_interval`:  Sets the model monitoring job scheduling interval in hours. Minimum time interval is 1 hour."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1230b92d1eed"
      },
      "outputs": [],
      "source": [
        "# Monitoring Interval\n",
        "MONITOR_INTERVAL = 1  # @param {type:\"number\"}\n",
        "\n",
        "# Create schedule configuration\n",
        "schedule_config = model_monitoring.ScheduleConfig(monitor_interval=MONITOR_INTERVAL)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3bd8dd304191"
      },
      "source": [
        "### Configure the sampling specification\n",
        "\n",
        "Now, you configure the `logging_sampling_strategy` specification with the following settings:\n",
        "\n",
        "- `sample_rate`: The rate as a percentage (between 0 and 1) to randomly sample prediction requests for monitoring. Selected samples are logged to a BigQuery table."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3688c245a0e9"
      },
      "outputs": [],
      "source": [
        "# Sampling rate (optional, default=.8)\n",
        "SAMPLE_RATE = 0.5  # @param {type:\"number\"}\n",
        "\n",
        "# Create sampling configuration\n",
        "logging_sampling_strategy = model_monitoring.RandomSampleConfig(sample_rate=SAMPLE_RATE)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c1bc3bc8992f"
      },
      "source": [
        "### Configure the drift detection specification\n",
        "\n",
        "Then, you configure the `drift_config` specification with the following settings:\n",
        "\n",
        "- `drift_thresholds`: A dictionary of key/value pairs where the keys are the input features for monitoring the drift. The value represents the detection threshold. When not specified, the default drift threshold for a feature is 0.3 (30%).\n",
        "\n",
        "**Note:** Enabling drift detection is optional."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "128bdc05cb5b"
      },
      "outputs": [],
      "source": [
        "DRIFT_THRESHOLD_VALUE = 0.05\n",
        "\n",
        "DRIFT_THRESHOLDS = {\n",
        "    \"country\": DRIFT_THRESHOLD_VALUE,\n",
        "    \"cnt_user_engagement\": DRIFT_THRESHOLD_VALUE,\n",
        "}\n",
        "\n",
        "drift_config = model_monitoring.DriftDetectionConfig(drift_thresholds=DRIFT_THRESHOLDS)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "651b576aa037"
      },
      "source": [
        "### Configure the skew detection specification\n",
        "\n",
        "Next, you configure the `skew_config` specification with the following settings:\n",
        "\n",
        "- `data_source`: The source of the dataset of the original training data. The format of the source defaults to a BigQuery table. Otherwise the setting `data_format` must be set to one of the values below. The location of the data must be a Cloud Storage location.\n",
        "  - `csv`\n",
        "  - `jsonl`\n",
        "  - `tf-record`\n",
        "- `skew_thresholds`: A dictionary of key/value pairs where the keys are the input features for monitoring the skewness. The value represents the detection threshold. When not specified, the default skew threshold for a feature is 0.3 (30%).\n",
        "- `target_field`: The target label for the training dataset\n",
        "\n",
        "**Note:** Enabling skewness detection is optional."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4cb098c74d2c"
      },
      "outputs": [],
      "source": [
        "# URI to training dataset.\n",
        "DATASET_BQ_URI = \"bq://mco-mm.bqmlga4.train\"  # @param {type:\"string\"}\n",
        "# Prediction target column name in training dataset.\n",
        "TARGET = \"churned\"\n",
        "\n",
        "SKEW_THRESHOLD_VALUE = 0.5\n",
        "\n",
        "SKEW_THRESHOLDS = {\n",
        "    \"country\": SKEW_THRESHOLD_VALUE,\n",
        "    \"cnt_user_engagement\": SKEW_THRESHOLD_VALUE,\n",
        "}\n",
        "\n",
        "skew_config = model_monitoring.SkewDetectionConfig(\n",
        "    data_source=DATASET_BQ_URI, skew_thresholds=SKEW_THRESHOLDS, target_field=TARGET\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8b6d5bd71ac9"
      },
      "source": [
        "### Assemble the objective specification\n",
        "\n",
        "Finally, you assemble the objective specification (`objective_config`) with the following settings:\n",
        "\n",
        "- `skew_detection_config`: (Optional) The specification for the skewness detection configuration.\n",
        "- `drift_detection_config`: (Optional) The specification for the drift detection configuration.\n",
        "- `explanation_config`: (Optional) The specification for explanations when enabling monitoring for feature attributions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d353205a8aec"
      },
      "outputs": [],
      "source": [
        "objective_config = model_monitoring.ObjectiveConfig(\n",
        "    skew_detection_config=skew_config,\n",
        "    drift_detection_config=drift_config,\n",
        "    explanation_config=None,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8ac30ffa72b5"
      },
      "source": [
        "### Create the input schema\n",
        "\n",
        "The monitoring service needs to know the features and data types for the the feature inputs to the model, which is referred to as the *input schema*. The *input schema* can either be \n",
        " - Preloaded to the monitoring service.\n",
        " - Automatically generated by the monitoring service after receiving first 1000 prediction instances.\n",
        " \n",
        "In this tutorial, you preload the *input schema*.\n",
        "\n",
        "#### Create the predefined input schema\n",
        "\n",
        "The predefined *input schema* is specified as a YAML file. In this example, you retrieve the BigQuery schema for the training data, which includes the feature names and data types, to generate the YAML specification. The predefined *input schema* must be loaded to a Cloud Storage location.\n",
        "\n",
        "Learn more about [Custom instance schemas for parsing input](https://cloud.google.com/vertex-ai/docs/model-monitoring/overview#custom-input-schemas)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "59becb56ad34"
      },
      "outputs": [],
      "source": [
        "# Get the BQ table\n",
        "\n",
        "table = bigquery.TableReference.from_string(DATASET_BQ_URI[5:])\n",
        "bq_table = bqclient.get_table(table)\n",
        "\n",
        "yaml = \"\"\"type: object\n",
        "properties:\n",
        "\"\"\"\n",
        "\n",
        "schema = bq_table.schema\n",
        "for feature in schema:\n",
        "    if feature.name == TARGET:\n",
        "        continue\n",
        "    if feature.field_type == \"STRING\":\n",
        "        f_type = \"string\"\n",
        "    else:\n",
        "        f_type = \"integer\"\n",
        "    yaml += f\"\"\"  {feature.name}:\n",
        "    type: {f_type}\n",
        "\"\"\"\n",
        "\n",
        "yaml += \"\"\"required:\n",
        "\"\"\"\n",
        "for feature in schema:\n",
        "    if feature.name == TARGET:\n",
        "        continue\n",
        "    yaml += f\"\"\"- {feature.name}\n",
        "\"\"\"\n",
        "\n",
        "print(yaml)\n",
        "\n",
        "with open(\"schema.yaml\", \"w\") as f:\n",
        "    f.write(yaml)\n",
        "\n",
        "! gsutil cp schema.yaml {BUCKET_URI}/schema.yaml"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "82c7e7af7163"
      },
      "source": [
        "### Create the monitoring job\n",
        "\n",
        "Create a monitoring job, with your monitoring specifications, using Vertex AI's [`ModelDeploymentMonitoringJob.create()`](https://cloud.google.com/python/docs/reference/aiplatform/1.48.0/summary_method#google_cloud_aiplatform_ModelDeploymentMonitoringJob_create_summary) method, with the following parameters:\n",
        "\n",
        "- `display_name`: The human readable name for the monitoring job.\n",
        "- `project`: The project ID.\n",
        "- `region`: The region.\n",
        "- `endpoint`: The fully qualified resource name of the Vertex AI endpoint to enable monitoring.\n",
        "- `logging_sampling_strategy`: The specification for the sampling configuration.\n",
        "- `schedule_config`: The specification for the scheduling configuration.\n",
        "- `alert_config`: The specification for the alerting configuration.\n",
        "- `objective_configs`: The specification for the objectives configuration.\n",
        "- `analysis_instance_schema_uri`: The location of the YAML file containing the *input schema*."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4b8dd381c5c3"
      },
      "outputs": [],
      "source": [
        "monitoring_job = aiplatform.ModelDeploymentMonitoringJob.create(\n",
        "    display_name=\"churn\",\n",
        "    project=PROJECT_ID,\n",
        "    location=LOCATION,\n",
        "    endpoint=endpoint,\n",
        "    logging_sampling_strategy=logging_sampling_strategy,\n",
        "    schedule_config=schedule_config,\n",
        "    alert_config=alerting_config,\n",
        "    objective_configs=objective_config,\n",
        "    analysis_instance_schema_uri=f\"{BUCKET_URI}/schema.yaml\",\n",
        ")\n",
        "\n",
        "print(monitoring_job)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3c6d3b620264"
      },
      "source": [
        "#### Email notification of the monitoring job.\n",
        "\n",
        "An email notification is sent to the email address in the alerting configuration, notifying that the model monitoring job is now enabled.\n",
        "\n",
        "The contents of the email appear as below:\n",
        "\n",
        "<blockquote>\n",
        "Hello Vertex AI Customer,\n",
        "\n",
        "You are receiving this mail because you are using the Vertex AI Model Monitoring service.\n",
        "This mail is to inform you that we received your request to set up drift or skew detection for the Prediction Endpoint listed below. Starting from now, incoming prediction requests will be sampled and logged for analysis.\n",
        "Raw requests and responses will be collected from prediction service and saved in bq://[your-project-id].model_deployment_monitoring_[endpoint-id].serving_predict .\n",
        "</blockquote>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dcc4aae9e20f"
      },
      "source": [
        "#### Monitoring Job State\n",
        "\n",
        "After you start the Vertex AI Model Monitoring job, it stays in a **PENDING** state until `skew distribution baseline` is calculated. The monitoring service initiates a batch job to generate the distribution baseline from the training data. \n",
        "\n",
        "Once the baseline distribution is generated, then the monitoring job changes to **OFFLINE** state. On a per interval basis, for example, once an hour, the monitoring job enters **RUNNING** state while analyzing the sampled data. Once completed, it returns to the **OFFLINE** state while awaiting the next scheduled analysis."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f640fb7f10cd"
      },
      "outputs": [],
      "source": [
        "jobs = monitoring_job.list(filter=\"display_name=churn\")\n",
        "job = jobs[0]\n",
        "print(job.state)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e385d103aba6"
      },
      "source": [
        "### Automatic generation of the baseline distribution\n",
        "\n",
        "Next, the monitoring service creates a batch job to analyze the training data to generate the baseline distribution. Once completed, the monitoring service starts monitoring on the specified intervals."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "670b5bc98c2a"
      },
      "outputs": [],
      "source": [
        "import time\n",
        "\n",
        "# Pause a bit for the baseline distribution to be calculated\n",
        "if os.getenv(\"IS_TESTING\"):\n",
        "    time.sleep(180)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "26bac245f6c5"
      },
      "source": [
        "### Generate synthetic prediction requests for skew detection\n",
        "\n",
        "Now, you extract the first 1000 instances from the BigQuery training table for creating prediction requests. Then, modify the data (synthetic) to trigger the skewness detection in the prediction requests by making the follwing updates:\n",
        "\n",
        "- `country`: Set all values to Canada."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cb26c3dea306"
      },
      "outputs": [],
      "source": [
        "# Download the table.\n",
        "table = bigquery.TableReference.from_string(DATASET_BQ_URI[5:])\n",
        "\n",
        "rows = bqclient.list_rows(table, max_results=1000)\n",
        "\n",
        "instances = []\n",
        "for row in rows:\n",
        "    instance = {}\n",
        "    for key, value in row.items():\n",
        "        if key == TARGET:\n",
        "            continue\n",
        "        if value is None:\n",
        "            value = \"\"\n",
        "        if key == \"country\":\n",
        "            value = \"Canada\"\n",
        "        instance[key] = value\n",
        "    instances.append(instance)\n",
        "\n",
        "print(len(instances))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6d002569dadc"
      },
      "source": [
        "### Make prediction requests\n",
        "\n",
        "Next, you send the 1000 prediction requests to your Vertex AI endpoint resource using the `predict()` method."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "b2d69d89cd66"
      },
      "outputs": [],
      "source": [
        "for instance in instances:\n",
        "    response = endpoint.predict(instances=[instance])\n",
        "\n",
        "prediction = response[0]\n",
        "\n",
        "# print the prediction for the first instance\n",
        "print(prediction[0])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2b5859ea4ae9"
      },
      "source": [
        "### Logging sampled requests\n",
        "\n",
        "Once the monitoring service has started, the sampled prediction requests are logged to Cloud Storage. On the next monitoring interval, the sampled predictions are copied to the BigQuery logging table. Once the entries are logged, the monitoring service analyzes the sampled data.\n",
        "\n",
        "Next, you wait for the first logged entries to appear in the BigQuery logging table for prediction samples. Since you sent 1000 prediction requests, with 50% sampling, you should see around 500 entries."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bd177a8decbb"
      },
      "outputs": [],
      "source": [
        "while True:\n",
        "    time.sleep(180)\n",
        "\n",
        "    ENDPOINT_ID = endpoint.resource_name.split(\"/\")[-1]\n",
        "\n",
        "    table = bigquery.TableReference.from_string(\n",
        "        f\"{PROJECT_ID}.model_deployment_monitoring_{ENDPOINT_ID}.serving_predict\"\n",
        "    )\n",
        "    rows = bqclient.list_rows(table)\n",
        "    print(rows.total_rows)\n",
        "    if rows.total_rows > 0:\n",
        "        break"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aeaea3a7a194"
      },
      "source": [
        "### Skewness detection during monitoring\n",
        "\n",
        "The skewness detection for feature inputs occurs at the next monitoring interval. In this tutorial, you set the monitoring interval to one hour. So, in about an hour your monitoring job goes from **OFFLINE** to **RUNNING**. While running, it analyzes the logged sampled tables from the predictions during this interval and compares them to the baseline distribution.\n",
        "\n",
        "Once the analysis is completed, the monitoring job sends email notifications on the detected skewness, in this case `country`. Further, the monitoring job goes into **OFFLINE** state until the next interval.\n",
        "\n",
        "#### Wait for monitoring interval\n",
        "\n",
        "It can take up to 40 minutes or more from the moment the analysis on the monitoring interval is done until you receive an email alert.\n",
        "\n",
        "The contents of the email appear as below:\n",
        "\n",
        "<blockquote>\n",
        "   Hello Vertex AI Customer,\n",
        "\n",
        "You are receiving this mail because you are subscribing to the Vertex AI Model Monitoring service.\n",
        "This mail is just to inform you that there are some anomalies detected in your deployed models and may need your attention.\n",
        "\n",
        "\n",
        "Basic Information:\n",
        "\n",
        "Endpoint Name: projects/[your-project-id]/locations/us-central1/endpoints/3315907167046860800\n",
        "Monitoring Job: projects/[your-project-id]/locations/us-central1/modelDeploymentMonitoringJobs/8672170640054157312\n",
        "Statistics and Anomalies Root Path(Google Cloud Storage): gs://cloud-ai-platform-773884b1-2a32-48d6-8b83-c03cde416b68/model_monitoring/job-8672170640054157312\n",
        "BigQuery Command: SELECT * FROM `bq://[your-project-id].model_deployment_monitoring_3315907167046860800.serving_predict`\n",
        "\n",
        "\n",
        "Training Prediction Skew Anomalies (Raw Feature):\n",
        "\n",
        "Anomalies Report Path(Google Cloud Storage): gs://cloud-ai-platform-773884b1-2a32-48d6-8b83-c03cde416b68/model_monitoring/job-8672170640054157312/serving/2022-08-25T00:00/stats_and_anomalies/<deployed-model-id>/anomalies/training_prediction_skew_anomalies\n",
        "\n",
        "For more information about the alert, please visit the model monitoring alert page.\n",
        "\n",
        "Deployed model id: <deployed-model-id>\n",
        "\n",
        "Feature name\tAnomaly short description\tAnomaly long description\n",
        "country\tHigh Linfty distance between training and serving\tThe Linfty distance between training and serving is 0.947563 (up to six significant digits), above the threshold 0.5. The feature value with maximum difference is: Canada\n",
        "<blockquote>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "b91a0e19ff8b"
      },
      "outputs": [],
      "source": [
        "if os.getenv(\"IS_TESTING\"):\n",
        "    time.sleep(60 * 45)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "555642c341e1"
      },
      "source": [
        "### Generate synthetic prediction requests for drift detection\n",
        "\n",
        "Next, you extract the same first 1000 instances from the BigQuery training table to use for prediction requests. Then, modify the data (synthetic) to trigger the drift detection in the prediction requests by making the following updates:\n",
        "\n",
        "- `cnt_user_engagement`: increase the value 4x."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cb26c3dea306"
      },
      "outputs": [],
      "source": [
        "# Download the table.\n",
        "table = bigquery.TableReference.from_string(DATASET_BQ_URI[5:])\n",
        "\n",
        "rows = bqclient.list_rows(table, max_results=1000)\n",
        "\n",
        "instances = []\n",
        "for row in rows:\n",
        "    instance = {}\n",
        "    for key, value in row.items():\n",
        "        if key == TARGET:\n",
        "            continue\n",
        "        if value is None:\n",
        "            value = \"\"\n",
        "        elif key == \"cnt_user_engagement\":\n",
        "            value = int(value * 4)\n",
        "        instance[key] = value\n",
        "    instances.append(instance)\n",
        "\n",
        "print(len(instances))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6d002569dadc"
      },
      "source": [
        "### Make the prediction requests\n",
        "\n",
        "Next, you send the 1000 prediction requests to your Vertex AI endpoint resource using the `predict()` method."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "b2d69d89cd66"
      },
      "outputs": [],
      "source": [
        "for instance in instances:\n",
        "    response = endpoint.predict(instances=[instance])\n",
        "\n",
        "prediction = response[0]\n",
        "\n",
        "# print the prediction for the first instance\n",
        "print(prediction[0])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5184c68e4c99"
      },
      "source": [
        "### Logging sampled requests\n",
        "\n",
        "On the next monitoring interval, the sampled predictions are copied to the BigQuery logging table. Once the entries are logged, the monitoring service analyzes the sampled data.\n",
        "\n",
        "Next, you wait for the first logged entries to appear in the BigQuery logging table for prediction samples. Since you sent 1000 prediction requests, with 50% sampling, you should see around 1000 entries."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d9a2f9342b06"
      },
      "outputs": [],
      "source": [
        "while True:\n",
        "    time.sleep(180)\n",
        "\n",
        "    ENDPOINT_ID = endpoint.resource_name.split(\"/\")[-1]\n",
        "\n",
        "    table = bigquery.TableReference.from_string(\n",
        "        f\"{PROJECT_ID}.model_deployment_monitoring_{ENDPOINT_ID}.serving_predict\"\n",
        "    )\n",
        "    rows = bqclient.list_rows(table)\n",
        "    print(rows.total_rows)\n",
        "    if rows.total_rows > 550:\n",
        "        break"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "683ed0ba4ccd"
      },
      "source": [
        "### Drift detection during monitoring\n",
        "\n",
        "The drift detection for feature inputs occurs at the next monitoring interval. In this tutorial, you set the monitoring interval to one hour. So, in about an hour your monitoring job goes from **OFFLINE** to **RUNNING**. While running, it analyzes the logged sampled tables from the predictions during this interval and compares them to the previous monitoring interval distribution.\n",
        "\n",
        "Once the analysis is completed, the monitoring job sends email notifications on the detected drift, in this case `cnt_user_engagement`. Then, the monitoring job goes into **OFFLINE** state until the next interval.\n",
        "\n",
        "#### Wait for monitoring interval\n",
        "\n",
        "It can take up to 40 minutes or more from the moment the analyis on the monitoring interval is done until you receive an email alert."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2e64ffaae2de"
      },
      "outputs": [],
      "source": [
        "if os.getenv(\"IS_TESTING\"):\n",
        "    time.sleep(60 * 45)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "39cdad8fdcc5"
      },
      "source": [
        "### Delete the monitoring job\n",
        "\n",
        "Once you've received the email alerts and verified the content, you can:\n",
        "- pause the monitoring job using the `pause()` method.\n",
        "- delete the monitoring job using the `delete()` method. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ef1ddc1d6017"
      },
      "outputs": [],
      "source": [
        "# Pause the job\n",
        "monitoring_job.pause()\n",
        "# Delete the job\n",
        "monitoring_job.delete()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "18889460bd33"
      },
      "source": [
        "## Cleaning up\n",
        "\n",
        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "Otherwise, you can delete the individual resources you created in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c736a6bf1428"
      },
      "outputs": [],
      "source": [
        "# Undeploy the model from endpoint before deletion\n",
        "endpoint.undeploy_all()\n",
        "\n",
        "# Delete the endpoint\n",
        "endpoint.delete()\n",
        "\n",
        "# Delete the model\n",
        "model.delete()\n",
        "\n",
        "# Delete the Cloud Storage bucket\n",
        "delete_bucket = False  # Set True for deletion\n",
        "if delete_bucket:\n",
        "    ! gsutil rm -rf {BUCKET_URI}\n",
        "\n",
        "# Delete the locally generated files\n",
        "! rm -f schema.yaml\n",
        "\n",
        "# Delete the BigQuery table\n",
        "! bq rm -f {PROJECT_ID}.model_deployment_monitoring_{ENDPOINT_ID}"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "get_started_with_model_monitoring_custom.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}