notebooks/official/migration/sdk-custom-image-classification-prebuilt-container.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "copyright"
      },
      "outputs": [],
      "source": [
        "# Copyright 2021 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "title:migration,new"
      },
      "source": [
        "# Vertex AI Migration: Custom image classification with a pre-built training container\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/migration/sdk-custom-image-classification-prebuilt-container.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fofficial%2Fmigration%2Fsdk-custom-image-classification-prebuilt-container.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://cloud.google.com/ml-engine/images/colab-enterprise-logo-32px.png\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
        "    </a>\n",
        "  </td>    \n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/migration/sdk-custom-image-classification-prebuilt-container.ipynb\">\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/migration/sdk-custom-image-classification-prebuilt-container.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7a8a13b86a8b"
      },
      "source": [
        "## Overview\n",
        "\n",
        "This tutorial demonstrates how to use the Vertex AI SDK for Python to train using a pre-built container and deploy a custom image classification model for online and batch prediction.\n",
        "\n",
        "Learn more about [Migrate to Vertex AI](https://cloud.google.com/vertex-ai/docs/start/migrating-to-vertex-ai) and [Custom training overview](https://cloud.google.com/vertex-ai/docs/training/custom-training)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f1ae7d54ad29"
      },
      "source": [
        "### Objective\n",
        "\n",
        "In this tutorial, you learn how to train a tensorflow image classification model using a prebuilt container and Vertex AI training. After training, you also deploy the model to Vertex AI using a pre-built container and generate both batch and online predictions on it. \n",
        "\n",
        "This tutorial uses the following Google Cloud ML services and resources:\n",
        "\n",
        "- Vertex AI Training\n",
        "- Vertex AI Model Registry\n",
        "- Vertex AI batch predictions\n",
        "- Vertex AI endpoints\n",
        "\n",
        "\n",
        "The steps performed include:\n",
        "\n",
        "- Package the training code into a python application.\n",
        "- Containerize the training application using Cloud Build and Artifact Registry.\n",
        "- Create a custom container training job in Vertex AI and run it.\n",
        "- Evaluate the model generated from the training job.\n",
        "- Create a model resource for the trained model in Vertex AI Model Registry.\n",
        "- Run a Vertex AI batch prediction job.\n",
        "- Deploy the model resource to a Vertex AI endpoint.\n",
        "- Run a online prediction job on the model resource.\n",
        "- Clean up the resources created."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dataset:custom,cifar10,icn"
      },
      "source": [
        "### Dataset\n",
        "\n",
        "The dataset used for this tutorial is the [CIFAR10 dataset](https://www.tensorflow.org/datasets/catalog/cifar10) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The version of the dataset you use in this tutorial is the one that's available from TensorFlow SDK. The trained model classifies images into one of ten categories: *airplane*, *automobile*, *bird*, *cat*, *deer*, *dog*, *frog*, *horse*, *ship*, or *truck*."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "costs"
      },
      "source": [
        "### Costs\n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* Vertex AI\n",
        "* Cloud Storage\n",
        "\n",
        "Learn about [Vertex AI\n",
        "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
        "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
        "Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "61RBz8LLbxCR"
      },
      "source": [
        "## Get started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "No17Cw5hgx12"
      },
      "source": [
        "### Install Vertex AI SDK for Python and other required packages\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "install_aip:mbsdk"
      },
      "outputs": [],
      "source": [
        "! pip3 install --upgrade --quiet google-cloud-aiplatform google-cloud-storage\n",
        "\n",
        "! apt-get update && apt-get install -y python3-opencv-headless\n",
        "! apt-get install -y libgl1-mesa-dev\n",
        "! pip3 install --upgrade opencv-python-headless  \n",
        "! pip3 install --upgrade tensorflow==2.15.1  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R5Xep4W9lq-Z"
      },
      "source": [
        "### Restart runtime (Colab only)\n",
        "\n",
        "To use the newly installed packages, you must restart the runtime on Google Colab."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XRvKdaPDTznN"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "\n",
        "    import IPython\n",
        "\n",
        "    app = IPython.Application.instance()\n",
        "    app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SbmM4z7FOBpM"
      },
      "source": [
        "<div class=\"alert alert-block alert-warning\">\n",
        "<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
        "</div>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dmWOrTJ3gx13"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "Authenticate your environment on Google Colab.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NyKGtVQjgx13"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DF4l8DTdWgPY"
      },
      "source": [
        "### Set Google Cloud project information\n",
        "\n",
        "To get started using Vertex AI, you must have an existing Google Cloud project. Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "set_project_id"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
        "LOCATION = \"us-central1\"  # @param {type:\"string\"}\n",
        "\n",
        "# Set the project id\n",
        "! gcloud config set project {PROJECT_ID}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bucket:mbsdk"
      },
      "source": [
        "### Create a Cloud Storage bucket\n",
        "\n",
        "Create a storage bucket to store intermediate artifacts such as datasets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bucket"
      },
      "outputs": [],
      "source": [
        "BUCKET_URI = f\"gs://your-bucket-name-{PROJECT_ID}-unique\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_bucket"
      },
      "source": [
        "**If your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "create_bucket"
      },
      "outputs": [],
      "source": [
        "! gsutil mb -l {LOCATION} -p {PROJECT_ID} {BUCKET_URI}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "setup_vars"
      },
      "source": [
        "### Import libraries and define constants"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "import_aip:mbsdk"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "from google.cloud import aiplatform\n",
        "from tensorflow.keras.datasets import cifar10"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "init_aip:mbsdk"
      },
      "source": [
        "### Initialize Vertex SDK for Python\n",
        "\n",
        "Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "init_aip:mbsdk"
      },
      "outputs": [],
      "source": [
        "aiplatform.init(project=PROJECT_ID, staging_bucket=BUCKET_URI)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "accelerators:training,cpu,prediction,cpu,mbsdk"
      },
      "source": [
        "#### Set hardware accelerators\n",
        "\n",
        "You can set hardware accelerators for training and prediction.\n",
        "\n",
        "Set the variables `TRAIN_GPU/TRAIN_NGPU` and `DEPLOY_GPU/DEPLOY_NGPU` to use a container image supporting a GPU and the number of GPUs allocated to the virtual machine (VM) instance. For example, to use a GPU container image with 4 Nvidia Telsa T4 GPUs allocated to each VM, you'd specify:\n",
        "\n",
        "    (aiplatform.gapic.AcceleratorType.NVIDIA_TESLA_T4, 4)\n",
        "\n",
        "\n",
        "Otherwise specify `(None, None)` to use a container image to run on a CPU.\n",
        "\n",
        "Learn more [here](https://cloud.google.com/vertex-ai/docs/general/locations#accelerators) hardware accelerator support for your location\n",
        "\n",
        "**Note**: TF releases before 2.3 for GPU support fail to load the custom model in this tutorial. It's a known issue and fixed in TF 2.3 -- which is caused by static graph ops that are generated in the serving function. If you encounter this issue on your own custom models, use a container image for TF 2.3 with GPU support."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "accelerators:training,cpu,prediction,cpu,mbsdk"
      },
      "outputs": [],
      "source": [
        "TRAIN_GPU, TRAIN_NGPU = (None, None)\n",
        "\n",
        "DEPLOY_GPU, DEPLOY_NGPU = (None, None)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "container:training,prediction"
      },
      "source": [
        "#### Set pre-built containers\n",
        "\n",
        "Set the pre-built Docker container image for training and prediction.\n",
        "\n",
        "\n",
        "For the latest list, see [Pre-built containers for training](https://cloud.google.com/ai-platform-unified/docs/training/pre-built-containers).\n",
        "\n",
        "\n",
        "For the latest list, see [Pre-built containers for prediction](https://cloud.google.com/ai-platform-unified/docs/predictions/pre-built-containers)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "container:training,prediction"
      },
      "outputs": [],
      "source": [
        "TF = \"2-5\"\n",
        "\n",
        "if TRAIN_GPU:\n",
        "    TRAIN_VERSION = \"tf-gpu.{}\".format(TF)\n",
        "else:\n",
        "    TRAIN_VERSION = \"tf-cpu.{}\".format(TF)\n",
        "if DEPLOY_GPU:\n",
        "    DEPLOY_VERSION = \"tf2-gpu.{}\".format(TF)\n",
        "else:\n",
        "    DEPLOY_VERSION = \"tf2-cpu.{}\".format(TF)\n",
        "\n",
        "TRAIN_IMAGE = \"us-docker.pkg.dev/vertex-ai/training/{}:latest\".format(TRAIN_VERSION)\n",
        "DEPLOY_IMAGE = \"us-docker.pkg.dev/vertex-ai/prediction/{}:latest\".format(DEPLOY_VERSION)\n",
        "\n",
        "print(\"Training:\", TRAIN_IMAGE, TRAIN_GPU, TRAIN_NGPU)\n",
        "print(\"Deployment:\", DEPLOY_IMAGE, DEPLOY_GPU, DEPLOY_NGPU)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "machine:training,prediction"
      },
      "source": [
        "#### Set machine type\n",
        "\n",
        "Next, set the machine type to use for training and prediction.\n",
        "\n",
        "- Set the variables `TRAIN_COMPUTE` and `DEPLOY_COMPUTE` to configure  the compute resources for the VMs used for training and prediction.\n",
        " - `machine type`\n",
        "     - `n1-standard`: 3.75GB of memory per vCPU.\n",
        "     - `n1-highmem`: 6.5GB of memory per vCPU\n",
        "     - `n1-highcpu`: 0.9 GB of memory per vCPU\n",
        " - `vCPUs`: number of \\[2, 4, 8, 16, 32, 64, 96 \\]\n",
        "\n",
        "**Note**: The following isn't supported for training:\n",
        "\n",
        " - `standard`: 2 vCPUs\n",
        " - `highcpu`: 2, 4 and 8 vCPUs\n",
        "\n",
        "**Note**: You may also use n2 and e2 machine types for training and deployment, but they don't support GPUs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "machine:training,prediction"
      },
      "outputs": [],
      "source": [
        "MACHINE_TYPE = \"n1-standard\"\n",
        "\n",
        "VCPU = \"4\"\n",
        "TRAIN_COMPUTE = MACHINE_TYPE + \"-\" + VCPU\n",
        "print(\"Train machine type\", TRAIN_COMPUTE)\n",
        "\n",
        "VCPU = \"4\"\n",
        "DEPLOY_COMPUTE = MACHINE_TYPE + \"-\" + VCPU\n",
        "print(\"Deploy machine type\", DEPLOY_COMPUTE)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "examine_training_package"
      },
      "source": [
        "## Tutorial\n",
        "\n",
        "\n",
        "### Package layout\n",
        "\n",
        "Before you start the training, take a look at how a Python package is assembled for a custom training job. When unarchived, the package contains the following directory/file layout.\n",
        "\n",
        "- PKG-INFO\n",
        "- README.md\n",
        "- setup.cfg\n",
        "- setup.py\n",
        "- trainer\n",
        "  - \\_\\_init\\_\\_.py\n",
        "  - task.py\n",
        "\n",
        "The files `setup.cfg` and `setup.py` are the instructions for installing the package into the operating environment of the Docker image.\n",
        "\n",
        "The file `trainer/task.py` is the Python script for executing the custom training job. *Note*: When `trainer/task.py` is referred to in the worker pool specification, the directory slash is replaced with a dot and the file suffix (`.py`) is dropped (`trainer.task`).\n",
        "\n",
        "#### Package Assembly\n",
        "\n",
        "In the following cells, assemble the training package."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "examine_training_package"
      },
      "outputs": [],
      "source": [
        "# Make folder for Python training script\n",
        "! rm -rf custom\n",
        "! mkdir custom\n",
        "\n",
        "# Add package information\n",
        "! touch custom/README.md\n",
        "\n",
        "setup_cfg = \"[egg_info]\\n\\ntag_build =\\n\\ntag_date = 0\"\n",
        "! echo \"$setup_cfg\" > custom/setup.cfg\n",
        "\n",
        "setup_py = \"import setuptools\\n\\nsetuptools.setup(\\n\\n    install_requires=[\\n\\n        'tensorflow_datasets==1.3.0',\\n\\n    ],\\n\\n    packages=setuptools.find_packages())\"\n",
        "! echo \"$setup_py\" > custom/setup.py\n",
        "\n",
        "pkg_info = \"Metadata-Version: 1.0\\n\\nName: CIFAR10 image classification\\n\\nVersion: 0.0.0\\n\\nSummary: Demostration training script\\n\\nHome-page: www.google.com\\n\\nAuthor: Google\\n\\nAuthor-email: aferlitsch@google.com\\n\\nLicense: Public\\n\\nDescription: Demo\\n\\nPlatform: Vertex\"\n",
        "! echo \"$pkg_info\" > custom/PKG-INFO\n",
        "\n",
        "# Make the training subfolder\n",
        "! mkdir custom/trainer\n",
        "! touch custom/trainer/__init__.py"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "taskpy_contents:cifar10"
      },
      "source": [
        "### Task.py contents\n",
        "\n",
        "In the next cell, write the contents of the training script *task.py*.\n",
        "\n",
        "To summarize, the script performs the following steps:\n",
        "\n",
        "- Gets the directory where to save the model artifacts from the command line (`--model_dir`), and if not specified, then from the environment variable `AIP_MODEL_DIR`.\n",
        "- Loads CIFAR10 dataset from TF Datasets (tfds).\n",
        "- Builds a model using TF.Keras model API.\n",
        "- Compiles the model (`compile()`).\n",
        "- Sets a training distribution strategy according to the argument `args.distribute`.\n",
        "- Trains the model (`fit()`) with epochs and steps according to the arguments `args.epochs` and `args.steps`\n",
        "- Saves the trained model (`save(args.model_dir)`) to the specified model directory."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "taskpy_contents:cifar10"
      },
      "outputs": [],
      "source": [
        "%%writefile custom/trainer/task.py\n",
        "# Single, Mirror and Multi-Machine Distributed Training for CIFAR-10\n",
        "\n",
        "import tensorflow_datasets as tfds\n",
        "import tensorflow as tf\n",
        "from tensorflow.python.client import device_lib\n",
        "import argparse\n",
        "import os\n",
        "import sys\n",
        "tfds.disable_progress_bar()\n",
        "\n",
        "parser = argparse.ArgumentParser()\n",
        "parser.add_argument('--model-dir', dest='model_dir',\n",
        "                    default=os.getenv(\"AIP_MODEL_DIR\"), type=str, help='Model dir.')\n",
        "parser.add_argument('--lr', dest='lr',\n",
        "                    default=0.01, type=float,\n",
        "                    help='Learning rate.')\n",
        "parser.add_argument('--epochs', dest='epochs',\n",
        "                    default=10, type=int,\n",
        "                    help='Number of epochs.')\n",
        "parser.add_argument('--steps', dest='steps',\n",
        "                    default=200, type=int,\n",
        "                    help='Number of steps per epoch.')\n",
        "parser.add_argument('--distribute', dest='distribute', type=str, default='single',\n",
        "                    help='distributed training strategy')\n",
        "args = parser.parse_args()\n",
        "\n",
        "print('Python Version = {}'.format(sys.version))\n",
        "print('TensorFlow Version = {}'.format(tf.__version__))\n",
        "print('TF_CONFIG = {}'.format(os.environ.get('TF_CONFIG', 'Not found')))\n",
        "print('DEVICES', device_lib.list_local_devices())\n",
        "\n",
        "# Single Machine, single compute device\n",
        "if args.distribute == 'single':\n",
        "    if tf.test.is_gpu_available():\n",
        "        strategy = tf.distribute.OneDeviceStrategy(device=\"/gpu:0\")\n",
        "    else:\n",
        "        strategy = tf.distribute.OneDeviceStrategy(device=\"/cpu:0\")\n",
        "# Single Machine, multiple compute device\n",
        "elif args.distribute == 'mirror':\n",
        "    strategy = tf.distribute.MirroredStrategy()\n",
        "# Multiple Machine, multiple compute device\n",
        "elif args.distribute == 'multi':\n",
        "    strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy()\n",
        "\n",
        "# Multi-worker configuration\n",
        "print('num_replicas_in_sync = {}'.format(strategy.num_replicas_in_sync))\n",
        "\n",
        "# Preparing dataset\n",
        "BUFFER_SIZE = 10000\n",
        "BATCH_SIZE = 64\n",
        "\n",
        "\n",
        "def make_datasets_unbatched():\n",
        "\n",
        "  # Scaling CIFAR10 data from (0, 255] to (0., 1.]\n",
        "  def scale(image, label):\n",
        "    image = tf.cast(image, tf.float32)\n",
        "    image /= 255.0\n",
        "    return image, label\n",
        "\n",
        "\n",
        "  datasets, info = tfds.load(name='cifar10',\n",
        "                            with_info=True,\n",
        "                            as_supervised=True)\n",
        "  return datasets['train'].map(scale).cache().shuffle(BUFFER_SIZE).repeat()\n",
        "\n",
        "\n",
        "# Build the Keras model\n",
        "def build_and_compile_cnn_model():\n",
        "  model = tf.keras.Sequential([\n",
        "      tf.keras.layers.Conv2D(32, 3, activation='relu', input_shape=(32, 32, 3)),\n",
        "      tf.keras.layers.MaxPooling2D(),\n",
        "      tf.keras.layers.Conv2D(32, 3, activation='relu'),\n",
        "      tf.keras.layers.MaxPooling2D(),\n",
        "      tf.keras.layers.Flatten(),\n",
        "      tf.keras.layers.Dense(10, activation='softmax')\n",
        "  ])\n",
        "  model.compile(\n",
        "      loss=tf.keras.losses.sparse_categorical_crossentropy,\n",
        "      optimizer=tf.keras.optimizers.SGD(learning_rate=args.lr),\n",
        "      metrics=['accuracy'])\n",
        "  return model\n",
        "\n",
        "\n",
        "# Train the model\n",
        "NUM_WORKERS = strategy.num_replicas_in_sync\n",
        "# Here the batch size scales up by number of workers since\n",
        "# `tf.data.Dataset.batch` expects the global batch size.\n",
        "GLOBAL_BATCH_SIZE = BATCH_SIZE * NUM_WORKERS\n",
        "train_dataset = make_datasets_unbatched().batch(GLOBAL_BATCH_SIZE)\n",
        "\n",
        "with strategy.scope():\n",
        "  # Creation of dataset, and model building/compiling need to be within\n",
        "  # `strategy.scope()`.\n",
        "  model = build_and_compile_cnn_model()\n",
        "\n",
        "model.fit(x=train_dataset, epochs=args.epochs, steps_per_epoch=args.steps)\n",
        "model.save(args.model_dir)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tarball_training_script"
      },
      "source": [
        "### Store training script on your Cloud Storage bucket\n",
        "\n",
        "Next, package the training folder into a compressed tar ball, and then store it in your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tarball_training_script"
      },
      "outputs": [],
      "source": [
        "! rm -f custom.tar custom.tar.gz\n",
        "! tar cvf custom.tar custom\n",
        "! gzip custom.tar\n",
        "! gsutil cp custom.tar.gz $BUCKET_URI/trainer_cifar10.tar.gz"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "train_a_model:migration"
      },
      "source": [
        "### Create and run custom training job\n",
        "\n",
        "Learn more about how to [Create a Python training application for a prebuilt container](https://cloud.google.com/vertex-ai/docs/training/create-python-pre-built-container)\n",
        "\n",
        "To train a custom model, you perform two steps:\n",
        "1) Create a custom training job.\n",
        "2) Specify your training parameters and run the job."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_custom_training_job:mbsdk,no_model"
      },
      "source": [
        "#### Create custom training job\n",
        "\n",
        "A custom training job is created using the `CustomTrainingJob` class, with the following parameters:\n",
        "\n",
        "- `display_name`: The human readable name for the custom training job.\n",
        "- `container_uri`: The training container image.\n",
        "- `requirements`: Package requirements for the training container image (e.g., pandas).\n",
        "- `script_path`: The relative path to the training script."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "create_custom_training_job:mbsdk,no_model"
      },
      "outputs": [],
      "source": [
        "job = aiplatform.CustomTrainingJob(\n",
        "    display_name=\"cifar10-unique\",\n",
        "    script_path=\"custom/trainer/task.py\",\n",
        "    container_uri=TRAIN_IMAGE,\n",
        "    requirements=[\"gcsfs==0.7.1\", \"tensorflow-datasets==4.4\"],\n",
        ")\n",
        "\n",
        "print(job)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_custom_training_job:mbsdk,no_model"
      },
      "source": [
        "*Example output:*\n",
        "\n",
        "    <google.cloud.aiplatform.training_jobs.CustomTrainingJob object at 0x7feab1346710>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "run_custom_job:mbsdk,no_model"
      },
      "source": [
        "#### Run the custom training job\n",
        "\n",
        "Next, run the custom job to start the training job by invoking the `run()` method, with the following parameters:\n",
        "\n",
        "- `args`: The command-line arguments to pass to the training script.\n",
        "- `replica_count`: The number of compute instances for training (replica_count = 1 is single node training).\n",
        "- `machine_type`: The machine type for the compute instances.\n",
        "- `accelerator_type`: The hardware accelerator type.\n",
        "- `accelerator_count`: The number of accelerators to attach to a worker replica.\n",
        "- `base_output_dir`: The Cloud Storage location to write the model artifacts to.\n",
        "- `sync`: Set **True** to wait until the completion of the job."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "run_custom_job:mbsdk,no_model"
      },
      "outputs": [],
      "source": [
        "MODEL_DIR = \"{}/{}\".format(BUCKET_URI, \"unique\")\n",
        "\n",
        "EPOCHS = 20\n",
        "STEPS = 100\n",
        "\n",
        "DIRECT = True\n",
        "if DIRECT:\n",
        "    CMDARGS = [\n",
        "        \"--model-dir=\" + MODEL_DIR,\n",
        "        \"--epochs=\" + str(EPOCHS),\n",
        "        \"--steps=\" + str(STEPS),\n",
        "    ]\n",
        "else:\n",
        "    CMDARGS = [\n",
        "        \"--epochs=\" + str(EPOCHS),\n",
        "        \"--steps=\" + str(STEPS),\n",
        "    ]\n",
        "\n",
        "if TRAIN_GPU:\n",
        "    job.run(\n",
        "        args=CMDARGS,\n",
        "        replica_count=1,\n",
        "        machine_type=TRAIN_COMPUTE,\n",
        "        accelerator_type=TRAIN_GPU.name,\n",
        "        accelerator_count=TRAIN_NGPU,\n",
        "        base_output_dir=MODEL_DIR,\n",
        "        sync=True,\n",
        "    )\n",
        "else:\n",
        "    job.run(\n",
        "        args=CMDARGS,\n",
        "        replica_count=1,\n",
        "        machine_type=TRAIN_COMPUTE,\n",
        "        base_output_dir=MODEL_DIR,\n",
        "        sync=True,\n",
        "    )\n",
        "\n",
        "model_path_to_deploy = MODEL_DIR"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "run_custom_job:mbsdk,no_model"
      },
      "source": [
        "#### Wait for completion of custom training job\n",
        "\n",
        "Next, wait for the custom training job to complete. Alternatively, you can set the parameter `sync` to `True` in the `run()` method to block until the custom training job is completed."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "load_saved_model"
      },
      "source": [
        "### Load the saved model\n",
        "\n",
        "Your model is stored in a TensorFlow SavedModel format in a Cloud Storage bucket. Now, load the model from the Cloud Storage bucket and run model evaluation, preparing it for prediction requests.\n",
        "\n",
        "To load, use the TF.Keras `model.load_model()` method passing it the Cloud Storage path where the model is saved -- specified by `MODEL_DIR`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "load_saved_model"
      },
      "outputs": [],
      "source": [
        "local_model = tf.keras.models.load_model(MODEL_DIR)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "evaluate_custom_model:image"
      },
      "source": [
        "### Evaluate the model\n",
        "\n",
        "Now find out how good the model is.\n",
        "\n",
        "#### Load evaluation data\n",
        "\n",
        "Load the CIFAR10 test (holdout) data from `tf.keras.datasets`, using the method `load_data()`. This returns the dataset as a tuple of two elements. The first element is the training data and the second is the test data. Each element is also a tuple of two elements: the image data, and the corresponding labels.\n",
        "\n",
        "You don't need the training data, and hence load it as `(_, _)`.\n",
        "\n",
        "Before you can run the data through evaluation, you need to preprocess it:\n",
        "\n",
        "`x_test`: Normalize (rescale) the pixel data by dividing each pixel by 255. This replaces each single byte integer pixel with a 32-bit floating point number between 0 and 1.\n",
        "\n",
        "`y_test`: The labels are currently scalar (sparse). If you look back at the `compile()` step in the `trainer/task.py` script, you can find that it was compiled for sparse labels. So we don't need to do anything more."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "evaluate_custom_model:image,cifar10"
      },
      "outputs": [],
      "source": [
        "(_, _), (x_test, y_test) = cifar10.load_data()\n",
        "x_test = (x_test / 255.0).astype(np.float32)\n",
        "\n",
        "print(x_test.shape, y_test.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "perform_evaluation_custom"
      },
      "source": [
        "#### Perform the model evaluation\n",
        "\n",
        "Use the model's `evaluate()` method to perform the evaluation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "perform_evaluation_custom"
      },
      "outputs": [],
      "source": [
        "local_model.evaluate(x_test, y_test)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "import_model:migration,new"
      },
      "source": [
        "### Import models to Vertex\n",
        "\n",
        "Learn more on how to [Import models to Vertex](https://cloud.google.com/vertex-ai/docs/general/import-model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "serving_function_image"
      },
      "source": [
        "#### Serving function for image data\n",
        "\n",
        "To pass images to the prediction service, encode the compressed (e.g., JPEG) image bytes into base 64 -- which makes the content safe from modification while transmitting binary data over the network. Since this deployed model expects input data as raw (uncompressed) bytes, you need to ensure that the base 64 encoded data gets converted back to raw bytes before it's passed as input to the deployed model.\n",
        "\n",
        "To resolve this, define a serving function (`serving_fn`) and attach it to the model as a preprocessing step. Add a `@tf.function` decorator so the serving function is used by the underlying model (instead of upstream on a CPU).\n",
        "\n",
        "When you send a prediction or explanation request, the content of the request is base 64 decoded into a Tensorflow string (`tf.string`), which is passed to the serving function (`serving_fn`). The serving function preprocesses the `tf.string` into raw (uncompressed) numpy bytes (`preprocess_fn`) to match the input requirements of the model:\n",
        "- `io.decode_jpeg`- Decompresses the JPG image which is returned as a Tensorflow tensor with three channels (RGB).\n",
        "- `image.convert_image_dtype` - Changes integer pixel values to float 32.\n",
        "- `image.resize` - Resizes the image to match the input shape for the model.\n",
        "- `resized / 255.0` - Rescales (normalization) the pixel data between 0 and 1.\n",
        "\n",
        "At this point, the data can be passed to the model (`m_call`)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "serving_function_image"
      },
      "outputs": [],
      "source": [
        "CONCRETE_INPUT = \"numpy_inputs\"\n",
        "\n",
        "\n",
        "def _preprocess(bytes_input):\n",
        "    decoded = tf.io.decode_jpeg(bytes_input, channels=3)\n",
        "    decoded = tf.image.convert_image_dtype(decoded, tf.float32)\n",
        "    resized = tf.image.resize(decoded, size=(32, 32))\n",
        "    rescale = tf.cast(resized / 255.0, tf.float32)\n",
        "    return rescale\n",
        "\n",
        "\n",
        "@tf.function(input_signature=[tf.TensorSpec([None], tf.string)])\n",
        "def preprocess_fn(bytes_inputs):\n",
        "    decoded_images = tf.map_fn(\n",
        "        _preprocess, bytes_inputs, dtype=tf.float32, back_prop=False\n",
        "    )\n",
        "    return {\n",
        "        CONCRETE_INPUT: decoded_images\n",
        "    }  # User needs to make sure the key matches model's input\n",
        "\n",
        "\n",
        "@tf.function(input_signature=[tf.TensorSpec([None], tf.string)])\n",
        "def serving_fn(bytes_inputs):\n",
        "    images = preprocess_fn(bytes_inputs)\n",
        "    prob = m_call(**images)\n",
        "    return prob\n",
        "\n",
        "\n",
        "m_call = tf.function(local_model.call).get_concrete_function(\n",
        "    [tf.TensorSpec(shape=[None, 32, 32, 3], dtype=tf.float32, name=CONCRETE_INPUT)]\n",
        ")\n",
        "\n",
        "tf.saved_model.save(\n",
        "    local_model, model_path_to_deploy, signatures={\"serving_default\": serving_fn}\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "serving_function_signature:image"
      },
      "source": [
        "#### Get the serving function signature\n",
        "\n",
        "You can get the signatures of your model's input and output layers by reloading the model into memory, and querying it for the signatures corresponding to each layer.\n",
        "\n",
        "For this purpose, you need the signature of the serving function. When the data is sent for prediction as an HTTP request packet, the image data is base64 encoded, and the TF.Keras model takes numpy input. Your serving function does the conversion from base64 to a numpy array.\n",
        "\n",
        "When making a prediction request, you need to route the request to the serving function instead of the model, so you need to know the input layer name of the serving function -- which is used later when you make a prediction request."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "serving_function_signature:image"
      },
      "outputs": [],
      "source": [
        "loaded = tf.saved_model.load(model_path_to_deploy)\n",
        "\n",
        "serving_input = list(\n",
        "    loaded.signatures[\"serving_default\"].structured_input_signature[1].keys()\n",
        ")[0]\n",
        "print(\"Serving function input:\", serving_input)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "upload_model:mbsdk"
      },
      "source": [
        "### Upload the model\n",
        "\n",
        "Next, upload your model to a model resource using `Model.upload()` method, with the following parameters:\n",
        "\n",
        "- `display_name`: The human readable name for the model resource.\n",
        "- `artifact`: The Cloud Storage location of the trained model artifacts.\n",
        "- `serving_container_image_uri`: The serving container image.\n",
        "- `sync`: Set **True** to wait until the completion of the job.\n",
        "\n",
        "If the `upload()` method is run asynchronously, you can subsequently block until completion with the `wait()` method."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "upload_model:mbsdk"
      },
      "outputs": [],
      "source": [
        "model = aiplatform.Model.upload(\n",
        "    display_name=\"cifar10-unique\",\n",
        "    artifact_uri=MODEL_DIR,\n",
        "    serving_container_image_uri=DEPLOY_IMAGE,\n",
        "    sync=False,\n",
        ")\n",
        "\n",
        "model.wait()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "upload_model:mbsdk"
      },
      "source": [
        "*Example output:*\n",
        "\n",
        "    INFO:google.cloud.aiplatform.models:Creating Model\n",
        "    INFO:google.cloud.aiplatform.models:Create Model backing LRO: projects/759209241365/locations/us-central1/models/925164267982815232/operations/3458372263047331840\n",
        "    INFO:google.cloud.aiplatform.models:Model created. Resource name: projects/759209241365/locations/us-central1/models/925164267982815232\n",
        "    INFO:google.cloud.aiplatform.models:To use this Model in another session:\n",
        "    INFO:google.cloud.aiplatform.models:model = aiplatform.Model('projects/759209241365/locations/us-central1/models/925164267982815232')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "make_batch_predictions:migration"
      },
      "source": [
        "### Generate batch predictions\n",
        "\n",
        "Here is the [Overview of getting predictions on Vertex AI](https://cloud.google.com/vertex-ai/docs/predictions/overview)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "get_test_items:test"
      },
      "source": [
        "#### Get test items\n",
        "\n",
        "Use examples out of the test (holdout) portion of the dataset as a test items."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "get_test_items:test"
      },
      "outputs": [],
      "source": [
        "test_image_1 = x_test[0]\n",
        "test_label_1 = y_test[0]\n",
        "test_image_2 = x_test[1]\n",
        "test_label_2 = y_test[1]\n",
        "print(test_image_1.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "prepare_test_items:test,image"
      },
      "source": [
        "#### Prepare the request content\n",
        "You're going to send the CIFAR10 images as compressed JPG image, instead of the raw uncompressed bytes:\n",
        "\n",
        "- `cv2.imwrite`: Use openCV to write the uncompressed image to disk as a compressed JPEG image.\n",
        " - Denormalize the image data from \\[0,1) range back to [0,255).\n",
        " - Convert the 32-bit floating point values to 8-bit unsigned integers."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "prepare_test_items:test,image"
      },
      "outputs": [],
      "source": [
        "import cv2\n",
        "\n",
        "cv2.imwrite(\"tmp1.jpg\", (test_image_1 * 255).astype(np.uint8))\n",
        "cv2.imwrite(\"tmp2.jpg\", (test_image_2 * 255).astype(np.uint8))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "copy_test_items:test"
      },
      "source": [
        "#### Copy test item(s)\n",
        "\n",
        "For the batch prediction, copy the test items over to your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "copy_test_items:test"
      },
      "outputs": [],
      "source": [
        "! gsutil cp tmp1.jpg $BUCKET_URI/tmp1.jpg\n",
        "! gsutil cp tmp2.jpg $BUCKET_URI/tmp2.jpg\n",
        "\n",
        "test_item_1 = BUCKET_URI + \"/tmp1.jpg\"\n",
        "test_item_2 = BUCKET_URI + \"/tmp2.jpg\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "make_batch_file:custom,image"
      },
      "source": [
        "#### Create batch input file\n",
        "\n",
        "Now create batch input file, and store it in your local Cloud Storage bucket. The batch input file can only be in JSONL format. For JSONL file, make one dictionary entry per line for each data item (instance). The dictionary contains the key/value pairs:\n",
        "\n",
        "- `input_name`: the name of the input layer of the underlying model.\n",
        "- `'b64'`: A key that indicates the content is base64 encoded.\n",
        "- `content`: The compressed JPG image bytes as a base64 encoded string.\n",
        "\n",
        "Each instance in the prediction request is a dictionary entry of the form:\n",
        "\n",
        "                        {serving_input: {'b64': content}}\n",
        "\n",
        "To pass the image data to the prediction service encode the bytes into base64 -- which makes the content safe from modification when transmitting binary data over the network.\n",
        "\n",
        "- `tf.io.read_file`: Read the compressed JPG images into memory as raw bytes.\n",
        "- `base64.b64encode`: Encode the raw bytes into a base64 encoded string."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "make_batch_file:custom,image"
      },
      "outputs": [],
      "source": [
        "import base64\n",
        "import json\n",
        "\n",
        "gcs_input_uri = BUCKET_URI + \"/\" + \"test.jsonl\"\n",
        "with tf.io.gfile.GFile(gcs_input_uri, \"w\") as f:\n",
        "    bytes = tf.io.read_file(test_item_1)\n",
        "    b64str = base64.b64encode(bytes.numpy()).decode(\"utf-8\")\n",
        "    data = {serving_input: {\"b64\": b64str}}\n",
        "    f.write(json.dumps(data) + \"\\n\")\n",
        "    bytes = tf.io.read_file(test_item_2)\n",
        "    b64str = base64.b64encode(bytes.numpy()).decode(\"utf-8\")\n",
        "    data = {serving_input: {\"b64\": b64str}}\n",
        "    f.write(json.dumps(data) + \"\\n\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "batch_request:mbsdk,jsonl,custom"
      },
      "source": [
        "#### Make the batch prediction request\n",
        "\n",
        "Now that your model resource is trained, you can make a batch prediction by invoking the `batch_predict()` method, with the following parameters:\n",
        "\n",
        "- `job_display_name`: The human readable name for the batch prediction job.\n",
        "- `gcs_source`: A list of one or more batch request input files.\n",
        "- `gcs_destination_prefix`: The Cloud Storage location for storing the batch prediction resuls.\n",
        "- `instances_format`: The format for the input instances, either *csv* or *jsonl*. Defaults to *jsonl*.\n",
        "- `predictions_format`: The format for the output predictions, either *csv* or *jsonl*. Defaults to *jsonl*.\n",
        "- `machine_type`: The type of machine to use for training.\n",
        "- `accelerator_type`: The hardware accelerator type.\n",
        "- `accelerator_count`: The number of accelerators to attach to a worker replica.\n",
        "- `sync`: Set **True** to wait until the completion of the job."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "batch_request:mbsdk,jsonl,custom"
      },
      "outputs": [],
      "source": [
        "MIN_NODES = 1\n",
        "MAX_NODES = 1\n",
        "\n",
        "batch_predict_job = model.batch_predict(\n",
        "    job_display_name=\"cifar10-unique\",\n",
        "    gcs_source=gcs_input_uri,\n",
        "    gcs_destination_prefix=BUCKET_URI,\n",
        "    instances_format=\"jsonl\",\n",
        "    predictions_format=\"jsonl\",\n",
        "    model_parameters=None,\n",
        "    machine_type=DEPLOY_COMPUTE,\n",
        "    accelerator_type=DEPLOY_GPU,\n",
        "    accelerator_count=DEPLOY_NGPU,\n",
        "    starting_replica_count=MIN_NODES,\n",
        "    max_replica_count=MAX_NODES,\n",
        "    sync=False,\n",
        ")\n",
        "\n",
        "print(batch_predict_job)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "batch_request:mbsdk,jsonl,custom"
      },
      "source": [
        "*Example output:*\n",
        "\n",
        "    INFO:google.cloud.aiplatform.jobs:Creating BatchPredictionJob\n",
        "    <google.cloud.aiplatform.jobs.BatchPredictionJob object at 0x7874ad735a80> is waiting for upstream dependencies to complete.\n",
        "    "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "batch_request_wait:mbsdk"
      },
      "source": [
        "#### Wait for completion of batch prediction job\n",
        "\n",
        "Next, wait for the batch job to complete. Alternatively, you can set the parameter `sync` to `True` in the `batch_predict()` method to block until the batch prediction job is completed."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "batch_request_wait:mbsdk"
      },
      "outputs": [],
      "source": [
        "batch_predict_job.wait()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "batch_request_wait:mbsdk"
      },
      "source": [
        "*Example Output:*\n",
        "\n",
        "    INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/759209241365/locations/us-central1/batchPredictionJobs/181835033978339328 current state:\n",
        "    JobState.JOB_STATE_RUNNING\n",
        "    INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/759209241365/locations/us-central1/batchPredictionJobs/181835033978339328 current state:\n",
        "    JobState.JOB_STATE_RUNNING\n",
        "    INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/759209241365/locations/us-central1/batchPredictionJobs/181835033978339328 current state:\n",
        "    JobState.JOB_STATE_RUNNING\n",
        "    INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/759209241365/locations/us-central1/batchPredictionJobs/181835033978339328 current state:\n",
        "    JobState.JOB_STATE_RUNNING\n",
        "    INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/759209241365/locations/us-central1/batchPredictionJobs/181835033978339328 current state:\n",
        "    JobState.JOB_STATE_RUNNING\n",
        "    INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/759209241365/locations/us-central1/batchPredictionJobs/181835033978339328 current state:\n",
        "    JobState.JOB_STATE_RUNNING\n",
        "    INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/759209241365/locations/us-central1/batchPredictionJobs/181835033978339328 current state:\n",
        "    JobState.JOB_STATE_RUNNING\n",
        "    INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/759209241365/locations/us-central1/batchPredictionJobs/181835033978339328 current state:\n",
        "    JobState.JOB_STATE_RUNNING\n",
        "    INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/759209241365/locations/us-central1/batchPredictionJobs/181835033978339328 current state:\n",
        "    JobState.JOB_STATE_SUCCEEDED\n",
        "    INFO:google.cloud.aiplatform.jobs:BatchPredictionJob run completed. Resource name: projects/759209241365/locations/us-central1/batchPredictionJobs/181835033978339328"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "get_batch_prediction:mbsdk,custom,icn"
      },
      "source": [
        "### Get the predictions\n",
        "\n",
        "Next, get the results from the completed batch prediction job.\n",
        "\n",
        "The results are written to the Cloud Storage output bucket specified in the batch prediction request. Call the method `iter_outputs()` to get a list of each Cloud Storage file generated with the results. Each file contains one or more prediction requests in a JSON format:\n",
        "\n",
        "- `instance`: The prediction request.\n",
        "- `prediction`: The prediction response."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "get_batch_prediction:mbsdk,custom,icn"
      },
      "outputs": [],
      "source": [
        "import json\n",
        "\n",
        "bp_iter_outputs = batch_predict_job.iter_outputs()\n",
        "\n",
        "prediction_results = list()\n",
        "for blob in bp_iter_outputs:\n",
        "    if blob.name.split(\"/\")[-1].startswith(\"prediction\"):\n",
        "        prediction_results.append(blob.name)\n",
        "\n",
        "tags = list()\n",
        "for prediction_result in prediction_results:\n",
        "    gfile_name = f\"gs://{bp_iter_outputs.bucket.name}/{prediction_result}\"\n",
        "    with tf.io.gfile.GFile(name=gfile_name, mode=\"r\") as gfile:\n",
        "        for line in gfile.readlines():\n",
        "            line = json.loads(line)\n",
        "            print(line)\n",
        "            break"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "get_batch_prediction:mbsdk,custom,icn"
      },
      "source": [
        "*Example Output:*\n",
        "\n",
        "    {'instance': {'bytes_inputs': {'b64': '/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAIBAQEBAQIBAQECAgICAgQDAgICAgUEBAMEBgUGBgYFBgYGBwkIBgcJBwYGCAsICQoKCgoKBggLDAsKDAkKCgr/2wBDAQICAgICAgUDAwUKBwYHCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgr/wAARCAAgACADASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwD570PxBpmp6nfaEl48lzpUqpewPCU8lpEDqMsOeD26Z55Fa+s3HhnR/Aj6xZjV7rWrW4ke/wBMtLRGRLTaux1cuPnLlhtIAAUEE5490/ao8E6F4b8P3NxZeGksNW1z4h62Iby2t1/eC3ZoozJxwSiKQOhEZJ5JrqZtI8MftFfs56j8YI/hvo/gq1u9C0ywlbTbFoLa+1SOFWlgPGRmNiQzNkiPOflyf1WHFdark0K8UlUbkvJWel15ppn5MuD6MM6qUJzbppRdrO8lJa2a7NNHyJoGheKvHngfUfGjXSaHHZX/ANmW2kQTsHIBXzDxgt1GMAcDPU1xI1xdS16/8FaxNA2o2kPmGS2OI51zyV65Izz0z1xg1718Ivhd4b8IfBX4qeItWuxql+2tW+n6dHPOEijt1s9xYgnaR50hw2dvygDrXz/4v+HWo6ha6X8R/C7iwv7CTy7YiRSLslGG3AzlGAGQenPTFfL4XiDMvr0ZVZuSk/ej66adj6bGcPZX/Z8oUoKHKtJemurP1H+OekS/tAeAvDmpfDjw/wDbL3W/FOlalpkNgqyhJrtgsqPg4ACyyK4J9c1418XP2X4P2ev2jNQ+C3x6+OnhbRfCtpJHfLp1p4klkD73kldkhRAYTKzoSkmSmxiNysDXK/stftQD9kn9oSx8aa3p0uq+GdN1drq70W3cAJKYmRLmINgbl35xwGAI4ODXiXxK+Mtp8W/G+v8Ajvxl4mn/ALW1TU5bq6u9Q+fzHZixG8dFyQB0wOOnFfjuH40f1GNSnG05P3o9F5r9D9dr8LReNdOs7wS0l19PwKPxZ8TeNNAkvPh/8GruO8BE9v8A8JHbaq8VrPA8h+aSBl5mKKiiYAlQowRnAh+H/gWTwx4MiTV52vdRUlTLPMJNgK/NsJxgEgnpwGxmtnSfDsOl6VH4nuLWG8glbCtHcb1bvjqD+PSu78SSXfwn8F2XjnxHo2n3smpSKdPsJCpW3iB+Z2VRl2VckA4HA6k1xf8AEQs9wOKVWjGN0rK8eZLp1/M2nwLkuOwsqNWUrN3dpWb620P/2Q=='}}, 'prediction': [0.0560616329, 0.122713037, 0.121289924, 0.109751239, 0.121320881, 0.0897410363, 0.145011798, 0.0976110101, 0.0394041203, 0.0970953554]}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "make_online_predictions:migration"
      },
      "source": [
        "### Generate online predictions \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "deploy_model:mbsdk,all"
      },
      "source": [
        "#### Deploy the model\n",
        "\n",
        "Next, deploy your model for online prediction. To deploy the model, invoke the `deploy()` method, with the following parameters:\n",
        "\n",
        "- `deployed_model_display_name`: A human readable name for the deployed model.\n",
        "- `traffic_split`: Percent of traffic at the endpoint that goes to this model, which is specified as a dictionary of one or more key/value pairs.\n",
        "If there is only one model, then specify `traffic_split` as { \"0\": 100 }, where \"0\" refers to this model being uploaded and 100 means 100% of the traffic.\n",
        "If there are existing models on the endpoint, for which the traffic will be split, then use model_id to specify as { \"0\": percent, model_id: percent, ... }, where model_id is the model id of an existing model to the deployed endpoint. The percents must add up to 100.\n",
        "- `machine_type`: The type of machine to use for training.\n",
        "- `accelerator_type`: The hardware accelerator type.\n",
        "- `accelerator_count`: The number of accelerators to attach to a worker replica.\n",
        "- `starting_replica_count`: The number of compute instances to initially provision.\n",
        "- `max_replica_count`: The maximum number of compute instances to scale to. In this tutorial, only one instance is provisioned."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "deploy_model:mbsdk,all"
      },
      "outputs": [],
      "source": [
        "DEPLOYED_NAME = \"cifar10-unique\"\n",
        "\n",
        "TRAFFIC_SPLIT = {\"0\": 100}\n",
        "\n",
        "MIN_NODES = 1\n",
        "MAX_NODES = 1\n",
        "\n",
        "if DEPLOY_GPU:\n",
        "    endpoint = model.deploy(\n",
        "        deployed_model_display_name=DEPLOYED_NAME,\n",
        "        traffic_split=TRAFFIC_SPLIT,\n",
        "        machine_type=DEPLOY_COMPUTE,\n",
        "        accelerator_type=DEPLOY_GPU,\n",
        "        accelerator_count=DEPLOY_NGPU,\n",
        "        min_replica_count=MIN_NODES,\n",
        "        max_replica_count=MAX_NODES,\n",
        "    )\n",
        "else:\n",
        "    endpoint = model.deploy(\n",
        "        deployed_model_display_name=DEPLOYED_NAME,\n",
        "        traffic_split=TRAFFIC_SPLIT,\n",
        "        machine_type=DEPLOY_COMPUTE,\n",
        "        accelerator_type=DEPLOY_GPU,\n",
        "        accelerator_count=0,\n",
        "        min_replica_count=MIN_NODES,\n",
        "        max_replica_count=MAX_NODES,\n",
        "    )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "deploy_model:mbsdk,all"
      },
      "source": [
        "*Example output:*\n",
        "\n",
        "    INFO:google.cloud.aiplatform.models:Creating Endpoint\n",
        "    INFO:google.cloud.aiplatform.models:Create Endpoint backing LRO: projects/759209241365/locations/us-central1/endpoints/4867177336350441472/operations/4087251132693348352\n",
        "    INFO:google.cloud.aiplatform.models:Endpoint created. Resource name: projects/759209241365/locations/us-central1/endpoints/4867177336350441472\n",
        "    INFO:google.cloud.aiplatform.models:To use this Endpoint in another session:\n",
        "    INFO:google.cloud.aiplatform.models:endpoint = aiplatform.Endpoint('projects/759209241365/locations/us-central1/endpoints/4867177336350441472')\n",
        "    INFO:google.cloud.aiplatform.models:Deploying model to Endpoint : projects/759209241365/locations/us-central1/endpoints/4867177336350441472\n",
        "    INFO:google.cloud.aiplatform.models:Deploy Endpoint model backing LRO: projects/759209241365/locations/us-central1/endpoints/4867177336350441472/operations/1691336130932244480\n",
        "    INFO:google.cloud.aiplatform.models:Endpoint model deployed. Resource name: projects/759209241365/locations/us-central1/endpoints/4867177336350441472"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "endpoints_predict:migration,new,mbsdk"
      },
      "source": [
        "Learn more about how to [Train and use your own models](https://cloud.google.com/vertex-ai/docs/training-overview)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "get_test_item:test"
      },
      "source": [
        "### Get test item\n",
        "\n",
        "Use an example from the test (holdout) portion of the dataset as a test item."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "get_test_item:test"
      },
      "outputs": [],
      "source": [
        "test_image = x_test[0]\n",
        "test_label = y_test[0]\n",
        "print(test_image.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "prepare_test_item:test,image"
      },
      "source": [
        "#### Prepare the request content\n",
        "You're going to send the CIFAR10 image as compressed JPG image, instead of the raw uncompressed bytes:\n",
        "\n",
        "- `cv2.imwrite`: Use openCV to write the uncompressed image to disk as a compressed JPEG image.\n",
        " - Denormalize the image data from \\[0,1) range back to [0,255).\n",
        " - Convert the 32-bit floating point values to 8-bit unsigned integers.\n",
        "- `tf.io.read_file`: Read the compressed JPG images back into memory as raw bytes.\n",
        "- `base64.b64encode`: Encode the raw bytes into a base 64 encoded string."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "prepare_test_item:test,image"
      },
      "outputs": [],
      "source": [
        "import base64\n",
        "\n",
        "import cv2\n",
        "\n",
        "cv2.imwrite(\"tmp.jpg\", (test_image * 255).astype(np.uint8))\n",
        "\n",
        "bytes = tf.io.read_file(\"tmp.jpg\")\n",
        "b64str = base64.b64encode(bytes.numpy()).decode(\"utf-8\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "predict_request:mbsdk,custom,icn"
      },
      "source": [
        "### Get the prediction\n",
        "\n",
        "Now that your model resource is deployed to an endpoint resource, you can get online predictions by sending prediction requests to the endpoint resource.\n",
        "\n",
        "#### Request\n",
        "\n",
        "Since in this example your test item is in a Cloud Storage bucket, open and read the contents of the image using `tf.io.gfile.Gfile()`. To pass the test data to the prediction service, you need to encode the bytes into base64 -- which makes the content safe from modification while transmitting binary data over the network.\n",
        "\n",
        "The format of each instance is:\n",
        "\n",
        "    { serving_input: { 'b64': base64_encoded_bytes } }\n",
        "\n",
        "Since the `predict()` method can take multiple items (instances), send your single test item as a list of one test item.\n",
        "\n",
        "#### Response\n",
        "\n",
        "The response from the `predict()` call is a Python dictionary with the following entries:\n",
        "\n",
        "- `ids`: The internal assigned unique identifiers for each prediction request.\n",
        "- `predictions`: The predicted confidence, between 0 and 1, per class label.\n",
        "- `deployed_model_id`: The Vertex AI identifier for the deployed model resource which did the predictions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "predict_request:mbsdk,custom,icn"
      },
      "outputs": [],
      "source": [
        "# The format of each instance should conform to the deployed model's prediction input schema.\n",
        "instances = [{serving_input: {\"b64\": b64str}}]\n",
        "\n",
        "prediction = endpoint.predict(instances=instances)\n",
        "\n",
        "print(prediction)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "predict_request:mbsdk,custom,icn"
      },
      "source": [
        "*Example output:*\n",
        "\n",
        "    Prediction(predictions=[[0.0560616292, 0.122713044, 0.121289924, 0.109751239, 0.121320873, 0.0897410288, 0.145011798, 0.0976110175, 0.0394041166, 0.0970953479]], deployed_model_id='4087166195420102656', explanations=None)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "undeploy_model:mbsdk"
      },
      "source": [
        "## Undeploy the model\n",
        "\n",
        "When you're done doing predictions, undeploy the model from the endpoint resouce. This deprovisions all compute resources and ends billing for the deployed model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "undeploy_model:mbsdk"
      },
      "outputs": [],
      "source": [
        "endpoint.undeploy_all()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cleanup:mbsdk"
      },
      "source": [
        "## Cleaning up\n",
        "\n",
        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "Otherwise, you can delete the individual resources you created in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cleanup:mbsdk"
      },
      "outputs": [],
      "source": [
        "# Delete the model using the Vertex model object\n",
        "model.delete()\n",
        "\n",
        "# Delete the endpoint using the Vertex endpoint object\n",
        "endpoint.delete()\n",
        "\n",
        "# Delete the training job\n",
        "job.delete()\n",
        "\n",
        "# Delete the batch prediction job using the Vertex batch prediction object\n",
        "batch_predict_job.delete()\n",
        "\n",
        "# delete locally generated files\n",
        "! rm -rf custom custom.tar.gz tmp1.jpg tmp2.jpg tmp.jpg\n",
        "\n",
        "# delete cloud storage bucket\n",
        "delete_bucket = False  # set True for deletion\n",
        "if delete_bucket:\n",
        "    ! gsutil rm -r $BUCKET_URI"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "sdk-custom-image-classification-prebuilt-container.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}