CHAPTER 6. CHANGE ORDER CLASSIFICATION USING AI
INTRODUCTION
This chapter describes the activities the research team completed to test AI models to classify change orders as UR or NUR. The main goal was to evaluate the feasibility of using AI algorithms to accelerate and potentially automate the extraction of UR change orders for further analysis. The research team used manually classified change order data (see Chapter 5) to train, test, and validate the AI models. This chapter describes the process to accomplish this goal. Appendix B includes a summary of fundamental AI concepts to assist with the reading and understanding of the methodology and results.
DISCLOSURE ON THE USE OF ARTIFICIAL INTELLIGENCE
The research team used AI tools in two different ways. First, the research team used natural language processing (NLP) techniques to extract UR records from a dataset that contained more than 100,000 change order records from several DOTs. This chapter includes a detailed description of the methods used and the corresponding results.
Second, the research team used large language model (LLM) tools to assist with the definition of specific AI terms in Appendix B. The research team used LLM tools as follows:
- The research team used Open AI™ ChatGPT™, Google™ Bard™, and Microsoft™ Bing AI™ to find definitions of AI terms. The research team also searched for definitions using traditional online searches. In most cases, the research team combined definitions from several sources to prepare draft definitions of terms. After preparing the draft definitions, a member of the research team used one of the LLM tools to check for grammatical errors, sentence construction, and improve readability. A different member of the research team then reviewed and revised the definitions manually to make sure they fitted well within the report.
- The research team evaluated several commercially available AI detector tools. None of the AI detector tools performed satisfactorily, confirming similar observations that have been reported in the literature (110). For example, sample text the research team wrote without any support of AI tools was flagged as having been written by AI, but in other cases, text the research team wrote with the assistance of AI tools was not flagged.
- Finally, the research team used Grammarly™ to check text for plagiarism. The Grammarly tool produced a similarity value of 6 percent, which is lower than what is normally considered acceptable (i.e., 10–15 percent). Nevertheless, the research team reviewed each flagged sentence and revised a handful of sentences as needed.
METHODOLOGY
Dataset Selection
The research team used the change order database from Case 9, which includes 104,540 records spanning over 20 years (See Chapter 5). For AI modeling purposes, the research team focused on the description and remarks columns. An initial review of these columns revealed that officials
had reused text from earlier change orders to create new change orders. Because duplicate text could potentially affect the modeling process, the research team removed records that contained duplicate description and remarks. The result was 102,302 change order records with unique description and remarks columns. Of this total, 95,290 (93 percent) were NUR records and 7012 (7 percent) were UR records.
The low percentage of UR records (7 percent) added complexity to the AI model training and testing process. After several attempts, the research team settled on a sampling approach that included 11,640 records for training and testing, of which 9,312 records (80 percent) were used for training and 2,328 records (20 percent) were used for testing (Figure 28). The 11,640-record sample included 7,760 NUR records (67 percent) that were randomly selected from the subset of 95,290 NUR records and 3,880 UR records (33 percent) that were randomly selected from the subset of 7,012 UR records.
The research team used the remaining 90,662 records for validation, including 86,992 NUR records (96 percent) and 3,670 UR records (4 percent). The percentage of UR records used for validation was lower than the percentage of UR records for the entire population, but it was critical to increase the number of UR records for training and testing as much as possible. For validation, the research team divided the 90,662 records into five validation datasets of similar size and distribution of NUR and UR records.
Preprocessing
After setting up the datasets for training, testing, and validation, the research team concatenated the description and remarks columns and then preprocessed the resulting text into a clean, standardized format. The research team preprocessed text from the 102,302 change order records by completing the following steps:
- Convert to lowercase. This step was necessary to avoid duplication of words having different cases.
- Remove special characters and specific word patterns. This step involved the removal of special characters and specific word patterns, such as single characters (e.g., a, b, c) and words ending with a number (e.g., word1, text2).
- Remove punctuation and numbers. This step involved the removal of punctuation and standalone numbers from the text.
- Tokenize words. This step involved splitting text using whitespaces to generate individual words (or tokens).
- Remove stop words and lemmatize text: This step involved removing stop words and lemmatizing words to reduce words to their base or root form.
- Join words back into a string: This step involved joining the preprocessed tokenized words back into a string to use as an input parameter for vectorization. Downstream NLP tasks (e.g., training a model) often work with sentences or paragraphs, rather than with individual words.
Here is an example of raw text before preprocessing (65 words):
drilling invoice. due to incomplete utility relocations at wall sb02, the drill rig boom was in conflict with existing overhead utilities on panel 3. in order to continue with the wall construction, a low profile drilling rig was brought in to drill three shafts on 6/30/12. verbal approval was obtained on 6/28/12.\n\npricing is based on subcontractor invoice.\n\nno revised plan sheets.\n\nno third party funding involved.
Here is the same text after preprocessing (41 words):
drilling invoice due incomplete utility relocation wall drill rig boom conflict exist overhead utility panel order continue wall construction low profile drilling rig brought drill three shaft verbal approval obtain pricing base subcontractor invoice revise plan sheet third party funding involved.
For the 102,302 change order records, the average word count per change order decreased from 151 to 86 words after preprocessing (Table 61). The maximum word count and standard deviation also decreased.
Table 61. Word Count Before and After Text Preprocessing.
| Word Count | Raw Text | Preprocessed Text |
|---|---|---|
| Average | 151 | 86 |
| Minimum | 1 | 1 |
| Maximum | 5,123 | 2,920 |
| Standard Deviation | 182 | 102 |
Vectorization
After preprocessing the data, the research team used three vectorization techniques to transform text into numerical data that could be used to train the AI models: CountVectorizer, term frequency-inverse document frequency (TF-IDF), and BERT.
Training
The research team used the vectorized training datasets to train six AI models, as follows: Logistic regression, k-nearest neighbors (kNN), multi-layer perceptron classifier, SVM, random forest, and deep learning.
The research team applied two vectorization techniques (CountVectorizer and TF-IDF) to the six AI models, for a total of 12 classification models. The research team also trained the deep learning model using the BERT vectorization technique. Table 62 lists the AI models and corresponding vectorization techniques used.
Table 62. Configuration of AI Models and Vectorization Techniques.
| AI Model | Vectorization Technique | Computing Environment | ||
|---|---|---|---|---|
| CountVectorizer | TF-IDF | BERT | ||
| Logistic regression | X | X | Desktop computer | |
| kNN | X | X | Desktop computer | |
| Multi-layer perceptron classifier | X | X | Desktop computer | |
| SVM | X | X | Desktop computer | |
| Random forest | X | X | Desktop computer | |
| Deep learning | X | X | X | Supercomputer |
The research team trained all AI models except deep learning on a standard workstation desktop with 64 Gigabyte (GB) random access memory (RAM) and an 8-core i7 processor. The research team initially used the workstation desktop’s graphics processing unit (GPU) to train the deep learning model. However, due to limited available memory on the desktop, the research team could not complete the deep learning model training. The research team ended up using the supercomputer at the Texas A&M High-Performance Research Computing facility. More specifically, the research team used a computing node consisting of 360 GB RAM, 40 GB A100
GPU, and a 48-core processor. As mentioned previously, deep learning models often involve heavy matrix computations, making it necessary to use substantial computing resources during training. The deep learning model could be parallelized to take advantage of the multiple cores available on the A100 GPU. This parallel processing capability accelerates training, as multiple calculations can be performed simultaneously.
Using BERT with the deep learning AI model was successful, but the research team experienced difficulties with the other AI models. As mentioned, BERT is a complex transformer-based model with many parameters, and handling this complexity requires more computing resources than CountVectorizer or TF-IDF. This limitation made the use of BERT in a workstation environment more challenging. The research team successfully trained the deep learning model on the supercomputer. The research team was able to parallelize the deep learning model on the desktop workstation but could not train it due to limited GPU memory. For the remaining five AI models, the research team was able to parallelize training on the desktop workstation. However, training AI models using the BERT vectorization technique remained slow due to the significant memory and processing power required to manage complex BERT parameters.
Testing and Validation
The research team used the datasets shown in Figure 28 to train, test, and validate all 13 AI model/vectorization combinations shown in Table 62. After training and testing the AI models, the research team predicted the labels for all five validation datasets. The research team used accuracy, precision, recall, and F1 metrics to evaluate the performance of each model.
RESULTS AND DISCUSSION
Table 63 shows the results of training, testing, and validating the 13 AI models using change order data from Case 9. The table shows classification results for NUR and UR records separately. This differentiation was critical because the number of NUR records was much higher than the number of UR records, making the dataset imbalanced. As Table 63 shows, the average classification accuracy of validation datasets for NUR change orders was more than 97 percent for all models (except kNN). For the kNN models, the average accuracy was more than 93 percent. By comparison, the average classification accuracy of validation datasets for UR change orders ranged from 52–88 percent. Figure 29 shows average accuracy results for UR change order records.
As mentioned, the research team trained the deep learning model on a supercomputer and the random forest model on a standard desktop workstation. Figure 30 shows the training time of all AI models. The deep learning model took 5,465,080 milliseconds (or 91 minutes) to train. In contrast, the random forest model took 7,887 milliseconds (or 7.9 seconds). The deep learning model’s higher accuracy came at a computational cost, taking significantly longer to train than the random forest model, highlighting the tradeoff between computational efficiency and AI model accuracy.
Table 63. Results of AI Models on Change Order Datasets.
| Model (Vectorization) | Dataset | NUR | UR | Time (ms) | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Accuracy Precision | Recall | F1 | Accuracy | Precision | Recall | F1 | ||||
| Logistic Regression (CountVectorizer) | Training | 97% | 93% | 97% | 95% | 85% | 93% | 97% | 95% | 195 |
| Testing | 96% | 93% | 97% | 95% | 83% | 91% | 83% | 87% | 2 | |
| Validation 1 | 97% | 99% | 97% | 98% | 76% | 55% | 76% | 64% | 10 | |
| Validation 2 | 97% | 99% | 97% | 98% | 77% | 56% | 77% | 65% | 3 | |
| Validation 3 | 97% | 99% | 97% | 98% | 77% | 54% | 77% | 64% | 2 | |
| Validation 4 | 97% | 99% | 97% | 98% | 76% | 99% | 97% | 98% | 3 | |
| Validation 5 | 97% | 99% | 97% | 98% | 78% | 55% | 78% | 65% | 3 | |
| Average | 97% | 99% | 97% | 98% | 77% | 64% | 81% | 71% | 4 | |
| Standard Deviation | 0.00 | 0.00 | 0.00 | 0.00 | 0.01 | 0.20 | 0.09 | 0.15 | 3 | |
| Logistic Regression (TF-IDF) | Training | 97% | 90% | 97% | 94% | 79% | 94% | 79% | 86% | 148 |
| Testing | 97% | 90% | 97% | 93% | 78% | 94% | 78% | 85% | 0 | |
| Validation 1 | 98% | 99% | 98% | 99% | 73% | 64% | 73% | 68% | 0 | |
| Validation 2 | 98% | 99% | 98% | 99% | 74% | 64% | 74% | 69% | 2 | |
| Validation 3 | 98% | 99% | 98% | 99% | 73% | 65% | 73% | 69% | 2 | |
| Validation 4 | 98% | 99% | 98% | 99% | 72% | 65% | 72% | 68% | 4 | |
| Validation 5 | 98% | 99% | 98% | 99% | 75% | 64% | 75% | 69% | 16 | |
| Average | 98% | 99% | 98% | 99% | 73% | 64% | 73% | 69% | 5 | |
| Standard Deviation | 0.00 | 0.00 | 0.00 | 0.00 | 0.01 | 0.01 | 0.01 | 0.01 | 6 | |
| kNN (CountVectorizer) | Training | 96% | 86% | 96% | 91% | 69% | 90% | 69% | 78% | 2 |
| Testing | 94% | 83% | 94% | 88% | 61% | 83% | 61% | 70% | 1,043 | |
| Validation 1 | 94% | 98% | 94% | 96% | 50% | 27% | 50% | 35% | 7,328 | |
| Validation 2 | 94% | 98% | 94% | 96% | 53% | 28% | 53% | 37% | 7,169 | |
| Validation 3 | 94% | 98% | 94% | 96% | 55% | 27% | 55% | 37% | 7,276 | |
| Validation 4 | 94% | 98% | 94% | 96% | 51% | 27% | 51% | 35% | 7,296 | |
| Validation 5 | 94% | 98% | 94% | 96% | 53% | 27% | 53% | 36% | 7,318 | |
| Average | 94% | 98% | 94% | 96% | 52% | 27% | 52% | 36% | 7,277 | |
| Standard Deviation | 0.00 | 0.00 | 0.00 | 0.00 | 0.02 | 0.00 | 0.02 | 0.01 | 64 | |
| Model (Vectorization) | Dataset | NUR | UR | Time (ms) | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Accuracy Precision | Recall | F1 | Accuracy | Precision | Recall | F1 | ||||
| kNN (TF-IDF) | Training | 95% | 87% | 95% | 91% | 72% | 89% | 72% | 80% | 3 |
| Testing | 93% | 85% | 93% | 89% | 68% | 83% | 68% | 75% | 1,001 | |
| Validation 1 | 93% | 98% | 93% | 96% | 61% | 27% | 61% | 38% | 7,255 | |
| Validation 2 | 93% | 98% | 93% | 95% | 60% | 26% | 60% | 36% | 7,238 | |
| Validation 3 | 93% | 98% | 93% | 95% | 61% | 26% | 61% | 37% | 7,332 | |
| Validation 4 | 93% | 98% | 93% | 96% | 61% | 27% | 61% | 38% | 7,205 | |
| Validation 5 | 93% | 98% | 93% | 95% | 59% | 26% | 59% | 36% | 7,301 | |
| Average | 93% | 98% | 93% | 95% | 61% | 26% | 60% | 37% | 7,266 | |
| Standard Deviation | 0.00 | 0.00 | 0.00 | 0.01 | 0.01 | 0.01 | 0.01 | 0.01 | 51 | |
| Multi-Layer Perceptron Classifier (CountVectorizer) | Training | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 17,741 |
| Testing | 95% | 92% | 95% | 94% | 84% | 90% | 84% | 87% | 11 | |
| Validation 1 | 96% | 99% | 96% | 98% | 78% | 48% | 78% | 59% | 35 | |
| Validation 2 | 97% | 99% | 97% | 98% | 79% | 49% | 79% | 60% | 35 | |
| Validation 3 | 97% | 99% | 97% | 98% | 79% | 49% | 79% | 61% | 35 | |
| Validation 4 | 97% | 99% | 97% | 98% | 76% | 48% | 76% | 59% | 33 | |
| Validation 5 | 96% | 99% | 96% | 98% | 80% | 49% | 80% | 60% | 47 | |
| Average | 97% | 99% | 97% | 98% | 78% | 49% | 78% | 60% | 37 | |
| Standard Deviation | 0.00 | 0.00 | 0.01 | 0.00 | 0.01 | 0.01 | 0.02 | 0.01 | 6 | |
| Multi-Layer Perceptron Classifier (TF-IDF) | Training | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 38,900 |
| Testing | 95% | 93% | 95% | 94% | 85% | 90% | 85% | 88% | 6 | |
| Validation 1 | 97% | 99% | 97% | 98% | 79% | 49% | 79% | 61% | 33 | |
| Validation 2 | 97% | 99% | 97% | 98% | 80% | 51% | 80% | 62% | 27 | |
| Validation 3 | 96% | 99% | 96% | 98% | 81% | 99% | 96% | 98% | 31 | |
| Validation 4 | 96% | 99% | 96% | 98% | 77% | 48% | 77% | 59% | 31 | |
| Validation 5 | 96% | 99% | 96% | 98% | 80% | 49% | 80% | 60% | 32 | |
| Average | 97% | 99% | 96% | 98% | 79% | 59% | 82% | 68% | 31 | |
| Standard Deviation | 0.00 | 0.00 | 0.01 | 0.00 | 0.01 | 0.22 | 0.08 | 0.17 | 2 | |
| Model (Vectorization) | Dataset | NUR | UR | Time (ms) | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Accuracy Precision | Recall | F1 | Accuracy | Precision | Recall | F1 | ||||
| SVM (CountVectorizer) | Training | 98% | 93% | 98% | 95% | 85% | 95% | 85% | 90% | 154,729 |
| Testing | 96% | 92% | 96% | 94% | 83% | 92% | 83% | 87% | 1,176 | |
| Validation 1 | 98% | 99% | 98% | 98% | 77% | 60% | 77% | 67% | 7,962 | |
| Validation 2 | 98% | 99% | 98% | 98% | 78% | 60% | 78% | 68% | 8,068 | |
| Validation 3 | 98% | 99% | 98% | 98% | 79% | 60% | 79% | 68% | 8,101 | |
| Validation 4 | 98% | 99% | 98% | 98% | 77% | 59% | 77% | 66% | 7,867 | |
| Validation 5 | 98% | 99% | 98% | 98% | 78% | 59% | 78% | 67% | 7,848 | |
| Average | 98% | 99% | 98% | 98% | 78% | 60% | 78% | 67% | 7,969 | |
| Standard Deviation | 0.00 | 0.00 | 0.00 | 0.00 | 0.01 | 0.01 | 0.01 | 0.01 | 115 | |
| SVM (TF-IDF) | Training | 98% | 92% | 98% | 95% | 82% | 95% | 82% | 88% | 44,245 |
| Testing | 97% | 91% | 97% | 94% | 81% | 94% | 81% | 87% | 1,853 | |
| Validation 1 | 98% | 99% | 98% | 99% | 76% | 66% | 76% | 71% | 11,997 | |
| Validation 2 | 98% | 99% | 98% | 99% | 77% | 67% | 77% | 72% | 11,907 | |
| Validation 3 | 98% | 99% | 98% | 99% | 77% | 67% | 77% | 72% | 11,927 | |
| Validation 4 | 98% | 99% | 98% | 99% | 74% | 67% | 74% | 71% | 11,943 | |
| Validation 5 | 98% | 99% | 98% | 99% | 77% | 66% | 77% | 71% | 12,085 | |
| Average | 98% | 99% | 98% | 99% | 76% | 67% | 76% | 71% | 11,972 | |
| Standard Deviation | 0.00 | 0.00 | 0.00 | 0.00 | 0.01 | 0.01 | 0.01 | 0.01 | 72 | |
| Random Forest (CountVectorizer) | Training | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 7,386 |
| Testing | 97% | 93% | 97% | 95% | 87% | 93% | 87% | 90% | 72 | |
| Validation 1 | 98% | 99% | 98% | 99% | 79% | 62% | 79% | 69% | 392 | |
| Validation 2 | 98% | 99% | 98% | 99% | 80% | 62% | 80% | 70% | 398 | |
| Validation 3 | 98% | 99% | 98% | 99% | 81% | 66% | 81% | 73% | 387 | |
| Validation 4 | 98% | 99% | 98% | 99% | 78% | 62% | 78% | 69% | 339 | |
| Validation 5 | 98% | 99% | 98% | 98% | 78% | 61% | 78% | 68% | 333 | |
| Average | 98% | 99% | 98% | 99% | 79% | 63% | 79% | 70% | 370 | |
| Standard Deviation | 0.00 | 0.00 | 0.00 | 0.00 | 0.01 | 0.02 | 0.01 | 0.02 | 31 | |
| Model (Vectorization) | Dataset | NUR | UR | Time (ms) | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Accuracy Precision | Recall | F1 | Accuracy | Precision | Recall | F1 | ||||
| Random Forest (TF-IDF) | Training | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 7,887 |
| Testing | 96% | 94% | 96% | 95% | 87% | 92% | 87% | 89% | 75 | |
| Validation 1 | 98% | 99% | 98% | 99% | 79% | 64% | 79% | 71% | 410 | |
| Validation 2 | 98% | 99% | 98% | 99% | 82% | 63% | 82% | 71% | 431 | |
| Validation 3 | 98% | 99% | 98% | 99% | 82% | 66% | 82% | 73% | 427 | |
| Validation 4 | 98% | 99% | 98% | 99% | 80% | 62% | 80% | 70% | 385 | |
| Validation 5 | 98% | 99% | 98% | 99% | 80% | 63% | 80% | 71% | 382 | |
| Average | 98% | 99% | 98% | 99% | 81% | 64% | 81% | 71% | 407 | |
| Standard Deviation | 0.00 | 0.00 | 0.00 | 0.00 | 0.01 | 0.02 | 0.01 | 0.01 | 23 | |
| Deep Learning (BERT) | Training | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 5,465,080 |
| Testing | 95% | 97% | 95% | 96% | 94% | 90% | 94% | 92% | 17,880 | |
| Validation 1 | 97% | 99% | 97% | 98% | 88% | 53% | 88% | 67% | 131,440 | |
| Validation 2 | 97% | 99% | 97% | 98% | 88% | 52% | 88% | 66% | 132,050 | |
| Validation 3 | 97% | 100% | 97% | 98% | 89% | 53% | 89% | 66% | 132,100 | |
| Validation 4 | 97% | 99% | 97% | 98% | 87% | 52% | 87% | 65% | 132,170 | |
| Validation 5 | 97% | 99% | 97% | 98% | 88% | 53% | 88% | 66% | 132,040 | |
| Average | 97% | 99% | 97% | 98% | 88% | 53% | 88% | 66% | 131,960 | |
| Standard Deviation | 0.00 | 0.00 | 0.00 | 0.00 | 0.01 | 0.01 | 0.01 | 0.01 | 295 | |
| Deep Learning (TF-IDF) | Training | 97% | 91% | 97% | 94% | 82% | 93% | 82% | 87% | 2,222,300 |
| Testing | 97% | 91% | 97% | 94% | 80% | 92% | 80% | 86% | 990 | |
| Validation 1 | 98% | 99% | 98% | 98% | 74% | 56% | 74% | 64% | 1,530 | |
| Validation 2 | 97% | 99% | 97% | 98% | 76% | 56% | 76% | 64% | 1,520 | |
| Validation 3 | 98% | 99% | 98% | 98% | 76% | 58% | 76% | 66% | 1,350 | |
| Validation 4 | 98% | 99% | 98% | 98% | 75% | 57% | 75% | 65% | 1,630 | |
| Validation 5 | 98% | 99% | 98% | 98% | 76% | 57% | 76% | 65% | 1,470 | |
| Average | 98% | 99% | 98% | 98% | 75% | 57% | 75% | 65% | 1,500 | |
| Standard Deviation | 0.00 | 0.00 | 0.00 | 0.00 | 0.01 | 0.01 | 0.01 | 0.01 | 102 | |
| Model (Vectorization) | Dataset | NUR | UR | Time (ms) | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Accuracy Precision | Recall | F1 | Accuracy | Precision | Recall | F1 | ||||
| Deep Learning (CountVectorizer) | Training | 97% | 92% | 97% | 95% | 83% | 94% | 83% | 88% | 720,090 |
| Testing | 97% | 91% | 97% | 94% | 82% | 92% | 82% | 87% | 1,290 | |
| Validation 1 | 98% | 99% | 98% | 98% | 76% | 61% | 76% | 68% | 1,480 | |
| Validation 2 | 98% | 99% | 98% | 98% | 76% | 59% | 76% | 66% | 1,540 | |
| Validation 3 | 98% | 99% | 98% | 98% | 76% | 60% | 76% | 67% | 1,580 | |
| Validation 4 | 98% | 99% | 98% | 98% | 76% | 60% | 76% | 67% | 1,430 | |
| Validation 5 | 98% | 99% | 98% | 98% | 77% | 61% | 77% | 68% | 1,440 | |
| Average | 98% | 99% | 98% | 98% | 76% | 60% | 76% | 67% | 1,494 | |
| Standard Deviation | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.01 | 0.00 | 0.01 | 65 | |
Deep learning with the BERT vectorization technique achieved an overall accuracy or 88 percent for UR change orders, followed by random forest with TF-IDF, which achieved 81 percent, and the multi-layer perceptron classifier with TF-IDF, which achieved 79 percent.
Results were more nuanced after considering all four metrics (i.e., accuracy, precision, recall, and F1). Deep learning with BERT achieved the highest accuracy (88 percent) among all models and showed the highest recall (88 percent), indicating its ability to identify a higher proportion of actual UR change orders. However, its precision (53 percent) was lower than other models. By comparison, the TF-IDF version achieved moderate precision (57 percent) and recall (75 percent). The CountVectorizer version had slightly better precision (60 percent) but similar recall (76 percent). Deep learning BERT attained an F1 of 66 percent.
Random forest with TF-IDF achieved an accuracy of 81 percent, a precision of 64 percent, a recall of 81 percent. By comparison, the CountVectorizer version achieved an accuracy of 78 percent, a precision of 63 percent, and a recall of 79 percent. Random forest with TF-IDF identified a higher proportion of true UR change orders from all actual UR change orders. Random forest with TF-IDF achieved an F1 of 71 percent.
Overall, the results point to the random forest model with TF-IDF vectorization as a promising choice for UR classification tasks, with slightly higher precision and recall and less computational demands than the deep learning model. Conversely, the deep learning model with the BERT vectorization had a higher accuracy and recall than other AI modeling strategies, making it suitable for scenarios where capturing most actual UR change orders is crucial.
Several reasons explain the difference in performance between NUR and UR records. First, the total number of UR records was 7,012, of which 3,100 UR records were used for training. In addition, the change order dataset was highly imbalanced, with only 7 percent of UR change orders. Although sample size rules for training AI models are not prescriptive, it is common to read in the literature that a sample size for training purposes should be at least 10,000. By comparison, the number of NUR records was 95,290, of which 7,760 records were used for training.
Second, UR change orders are not homogeneous because they cover a wide range of population segments (e.g., underground versus aboveground facilities, wet versus dry facilities, and transmission versus distribution lines). Each type of utility facility or industry segment (e.g., water, electric, communications) uses different terminology and keywords. Sometimes, the same keyword is used for utility and non-utility purposes. For example, the keyword conduit might refer to a traffic light installation in one change order (i.e., NUR) but refer to an electric or a communication utility relocation in another change order (i.e., UR). Change order descriptions are highly unstructured, with a wide range of styles, sentence construction, and even spelling.
Increasing the number of UR records significantly from a single DOT is not feasible in the short term. However, the research team explored the feasibility of using the trained AI models from Case 9 to classify change orders from Cases 1, 2, 5, 6, and 8. Figure 31 shows the UR accuracy for all six cases. For Case 9, UR accuracy values are the same as those shown in Figure 29. Using Case 9 as the basis for comparison, Figure 31 shows the following trends:
- UR accuracies for Cases 6 and 8 were higher than for Case 9. Deep learning with BERT, random forest with TF-IDF, and random forest with CountVectorizer produced similar accuracy results for Cases 6 and 8 (i.e., around 90 percent or higher). At first, this result was surprising because the research team did not use data from Cases 6 or 8 to train the AI models. However, a closer analysis revealed that certain characteristics of the change order data from Cases 6 and 8 made the data suitable for the use of AI models. First, change order descriptions for Cases 6 and 8 were typically long and similar to those from Case 9. Second, the research team noted that change order descriptions from Cases 6 and 8 were often well written, used complete sentences, and avoided abbreviations and acronyms. These characteristics also made the manual review of the change orders (as described in Chapter 5) easier.
- UR accuracies for Cases 1, 2, and 5 were much lower than for Case 9 (i.e., around 50 percent or lower). These results are not surprising. For Cases 1 and 2, change order descriptions were quite short (typically 3–7 words long). For Case 5, the change order database included short descriptions and long remarks, but the remarks column included mainly a list of modified items. Interestingly, UR accuracy for Case 5 while using the multi-layer perceptron classifier was close to 70 percent, but still not high enough for reliable prediction purposes. The trained AI models from Case 9 were not adequate for Cases 1, 2, and 5. A different AI strategy for these cases would be necessary.
The research team also trained and validated AI models for Case 6. UR accuracy for most AI models was slightly higher than those shown in Figure 31. For example, for deep learning with BERT, the UR accuracy was 97 percent. For random forest and TF-IDF, the UR accuracy was 95 percent. For random forest and CountVectorizer, UR accuracy was also 95 percent.
USING AI MODELS TO ASSIST WITH MANUAL REVIEW OF CHANGE ORDERS
As the AI modeling work progressed, the research team used partial AI modeling results to accelerate the manual review of change orders. This activity was critical because completing the manual review of change orders was also necessary to develop a ground truth dataset needed to train, test, and validate the AI models. The strategy to use partial AI modeling results involved using a scoring system for AI predictions. If an AI model classified a change order as NUR, the score was 1, but if the model classified a change order as UR, the score was 2. For visualization and ease of tracking, the research team multiplied the scores by a factor of 10.
The research team used this strategy when partial results were available for three AI models (logistic regression, multi-layer perceptron classifier, and random forest) and two vectorization techniques (TF-IDF and CountVectorizer). For each change order, the research team calculated a composite score by adding the scores across all six model combinations. The composite scores ranged from 60 (if all six models predicted a change order to be NUR) to 120 (if all six models predicted a change order to be UR). Once the composite scores were in place, the research team reviewed change orders with a total score of 120, then those with a total score of 110, and so on until reaching the change orders with a total score of 60. By focusing first on change orders with the highest scores for UR records, the research team was able to find true UR records more quickly, which in turn enabled the research team to conduct the AI model evaluation faster.
Figure 32 shows the percentage of UR and NUR records for each composite score. For example, for all change order records with a composite score of 70, 92 percent of the records were NUR and 8 percent were UR. Similarly, for all change order records with a composite score of 110, 2 percent of the records were NUR and 98 percent of the records were UR.
Records with intermediate composite scores (i.e., 80 or 90) were challenging because they typically included longer descriptions, confusing explanations, and more acronyms and typing errors, making the classification more time consuming. Some change order descriptions included UR keywords, but other descriptions did not.
In addition to the list of commonly used one-word and two-word UR terms (Table 43) and the classification results from the AI models, the research team used typical spreadsheet tools such as sorting and filtering. The research team also wrote a macro to colorize words within each cell. As Figure 33 shows, the macro opens a small dialog box where the user enters the text to highlight and the color. Running the macro multiple times enabled different words with different colors to be highlighted within each cell (Figure 34).
The research team reviewed each record thoroughly. In general, classifying records with high composite scores (i.e., 110 or 120) was straightforward and did not take much time to complete. Change order descriptions in this category contained a sufficient number of keywords. Many of the descriptions were relatively short and clear (although quite a few records also had longer descriptions). Records with a composite score of 100 involved more analysis than records with a higher composite score. Change order descriptions were relatively short, although some of them were long. Records with low composite scores (i.e., 60 or 70) were straightforward because they typically did not include UR keywords, and it was obvious from the context that the change order was not UR.
To assist further with the identification of UR records, the research team measured the frequency of keywords within each record. For example, Figure 34 shows that the first change order description had 10 UR keywords (from the list of keywords in Table 43). The second change order description had 6 UR keywords. In general, as the total number of UR keywords increased, the likelihood that a change order was utility-related also increased.
