Haiyuan Cao

This is a Kaggle like competition which used the transaction data to predict the active customer.

i. Understanding the data, clean the data and subset the data.
We are not provided with a background intro to the data so we spend some time looking into the each variable and tried to find some pattern. Luckily we finally found some correlation between variables and then grouped and reduced the number of variables.
We do not use the… Show more

This is a Kaggle like competition which used the transaction data to predict the active customer.

i. Understanding the data, clean the data and subset the data.
We are not provided with a background intro to the data so we spend some time looking into the each variable and tried to find some pattern. Luckily we finally found some correlation between variables and then grouped and reduced the number of variables.
We do not use the features directly, and we do the feature engineering carefully for each feature. For some feature has outliers ,we eliminate those outliers. For some feature has the range value is quite wide, we do the sqrt transform. For some data, we also found that the NAs occur in all the records about food so we decided to train separate models on the data containing food NAs and those without NAs, We do really a lot of work on feature transformation and engineering.

ii. Add new features. We tried with the variables from the data but cannot make progess when we hit approximately 68% in public leader board. One teammate found a paper describing some interesting features to be used in the customer classification using transaction data. In that paper the authors introduced the variable "number of NAs and number of 0 for each customers" are quite important for final prediction of the active customer, so we add these features to our result and the model give the result goes beyond 69%.

iii. Ensemble methods. We first tried a single model but stopped at around 69%. After that we tried to combine 13 kinds of models with both parametric and non-parametric machine learning method. Based on these prediction models, we use the 2-layer 5-fold stacking method ensemble the output of the first-layer models. Show less

1. Take two datasets from Foursquare and Locu that describe the same entities, and identify which entity in one dataset is the same as an entity in the other dataset.
2. We construct some features according to the input dataset. We construct the features hiversine_distances for the location information including longitude and lattitude. The 'name' and 'address' information are evaluated used the jaccard similarity score for both the whole entry and each character in the entry. The 'phone… Show more

1. Take two datasets from Foursquare and Locu that describe the same entities, and identify which entity in one dataset is the same as an entity in the other dataset.
2. We construct some features according to the input dataset. We construct the features hiversine_distances for the location information including longitude and lattitude. The 'name' and 'address' information are evaluated used the jaccard similarity score for both the whole entry and each character in the entry. The 'phone number' is evaluted through the simple matching. The missing values in 'phone number' and 'address' are also marked by the dummy variable feature.
3. In our algorithm, we combine the records in the locu train dataset and the foursquare train dataset, featurize the dataset and then add the tag that whether they are in the matched list or not. Then we use the training data to train the random forest classifier. The number of trees are chosen by the cross validation method and the number of features are used the general "sqrt" method. Finally we choose the random forest classifier with the 100 trees according to the cross-validation F1 score.
4. Here we set a threshold 0.53 which comes the cross validation used in the matching method. For several matched items in the test dataset through the random forest classifier, we use the matched item with the highest probability.
5. Our result has precision 100%, recall 98.33% and F1 score 99.16%.
Show less

1. Using the data set published by Audioscrobbler with 24.2 million records about user’s player of artists to build the music recommender engine.
2. Implementing the alternating least squares recommender algorithm through the MLLib on Spark to build the music recommener
3. Preprocessing the raw data set using python functional programming to correct the misspelled or nonstandard artist’s ID
4. Using cross validation on Spark to select the hyperparameters for the matrix factorization… Show more

1. Using the data set published by Audioscrobbler with 24.2 million records about user’s player of artists to build the music recommender engine.
2. Implementing the alternating least squares recommender algorithm through the MLLib on Spark to build the music recommener
3. Preprocessing the raw data set using python functional programming to correct the misspelled or nonstandard artist’s ID
4. Using cross validation on Spark to select the hyperparameters for the matrix factorization model
5. Implement the final model on AWS platform to handle the huge amount of data
Show less

1. Dealing with the data set contains Apache Logs gathered by NASA's server in the months of July-October, 1995, which is around 1 GB using the HDFS.
2. Create a schema for the dataset in Hive through the regular expression to describe a concrete structure describing all the required fields.
3. Make the plot to depicting the number of requests made in a day for every day in the month of October.
4. Write a MapReduce job to calculate total bandwidth add all the response bytes sent by… Show more

1. Dealing with the data set contains Apache Logs gathered by NASA's server in the months of July-October, 1995, which is around 1 GB using the HDFS.
2. Create a schema for the dataset in Hive through the regular expression to describe a concrete structure describing all the required fields.
3. Make the plot to depicting the number of requests made in a day for every day in the month of October.
4. Write a MapReduce job to calculate total bandwidth add all the response bytes sent by NASA webserver.
Show less

1. Processing real user data and metrics from Zynga platform with 1 million user records and 247 features.
2. Implemented Lasso, ridge regression with logistic regression and random forest method to select the important features in predicting whether the user would be a payer.
3. Ensemble the stochastic gradient descent classifier with perceptron, log and hinge loss function, the knn method and the decision tree method with the selected important features to predict the payer. The… Show more

1. Processing real user data and metrics from Zynga platform with 1 million user records and 247 features.
2. Implemented Lasso, ridge regression with logistic regression and random forest method to select the important features in predicting whether the user would be a payer.
3. Ensemble the stochastic gradient descent classifier with perceptron, log and hinge loss function, the knn method and the decision tree method with the selected important features to predict the payer. The precision, recall and F1 score all reach up to 95%.
4. Using Kmeans++ method with the important features to cluster the user patterns on Zynga platform. The number of cluster is determined by the elbow method. Using cluster method, we can correctly reveal the difference pattern between paying users, the risk-prefer user and the mature user.
5. Based on the user pattern, we propose the strategy to hold campaign between different group of users to improve the engagement of users.
Show less

1. Processed JPEG data from the USPS open handwriting datasets data into the matrix with R.
2. Implemented the non-linear SVM method and Adaboost with R to recognize the handwriting numbers.
3. Chosen the kernel and margin parameters through cross validation to improve the recognized rate to 90%.

1. Building an efficient Naïve Bayes classifier to classify the papers belonging to Hamilton or Madison with the help of natural language processing package of R
2. Implementing the Ridge regression, Lasso and mutual information selection, respectively, to remove the irrelevant features in the text documents to improve the efficiency of the Bayes classifier.

1. Cleaned, processed and selected a bunch of features to find correlation between the rate of vehicle collisions and the location, time and weather of the driving route with R script through the API of NYC open dataset.
2. Applied normalization and PCA for the features of data, then implementing the unsupervised K-means++ method on AWS Cloud Platform with Spark, obtain the heat map of high danger area in NYC with the inputting time and driving route.

1. Processed tweets from Twitter Streaming API to extract tweets with locations tags using Python and SQL
2. Done sentiment analysis by writing the classifiers with python: naive Bayes classifier, maximum entropy classifier and support vector machines. The NLTK package is used to parse and analyze each tweet.
3. Improved the accuracy of self-written machine learning classifier by using the bi-grams, tri-grams and word dictionaries. The accuracy is around 80%.

Developed C code to implement Bouchard-Sornette option pricing-model to predict the SSE Index

Implemented Wolff-Monte-Carlo method by C code to study the phase transition in hexagonal Ising model.

1. Proposed a new global optimization algorithm for functional material searching based on the differential evolution algorithm using python.
2. Utilized new algorithm to find the grain boundary structures with lower formation energy on Hadoop.
3. Design A/B test to select the components in the algorithm to make the optimization efficient.

1. Developed the efficient algorithm with python to accelerate parallel large-scale data-analysis on Hadoop.
2. Accelerating the efficiency of calculation 10 times without lost the major accuracy comparing to the previous.

1. Developed python code to simulate the thermal transport in graphene-based materials.
2. Using multivariate numerical method with R to analyze the datasets obtained from the experiments.
3. Designed a new kind of 2D thermal rectifier and publish in peer-reviewed paper (top cited paper in journal)