Model Pre-Processing

For these models to be as effective as possible, I needed to perform some pre-processing steps so that my models could understand the data. I used Scikit-Learn's “Column Transformer” to make the necessary modifications. For the “level” and “move” columns I encoded each category as a 1 if that category described the player and 0 otherwise. This led to the creation of eight new columns: one for each of the three categorizations of “level” and one for each of the five categorizations of “move”. For example, Justin Ahrens would have a value of 1 for the columns “high” and “down”, and a value of 0 for the remaining six categorization columns. This encoding was necessary because the models can only understand numerical data, and “level” and “move” are categorical columns.

The other preprocessing step I needed to do was to “standardize” the numerical value columns. What this means is that I transformed the columns to reflect how much more or less feature values were than their feature average, while taking into account the standard deviation. This is important because the scales of each column vary. For example, a player can play 40 minutes a game (MPG feature) while producing 100 points on the season (PProd feature). In a raw sense, 100 is much larger than 40; however, compared to the average, 40 minutes per game is well above average, while 100 points produced is well below average. Certain kinds of models are not able to account for different scales, so by comparing each feature value to its average value (and accounting for each feature's standard deviation), you force all the features onto the same scale. You can read here for more details on standardization.

Model Results

After creating the features “level” and “move” and applying the appropriate pre-processing steps to the data, I selected which features to use in the models. I performed several rounds of model testing and feature exploration to narrow down which features appeared to be the most important. I then explored different combinations of those features and examined their impact on model score to get to the final few possibilities. In the end, I decided to use the following features for my models: minutes per game, pre transfer BPM, points produced, true shooting percentage, low, mid, high, 2 down, down, same, up, and 2 up. The last eight of those features are the encoded category values of the “level” and “move” features. There were some different feature combinations that provided marginal model score improvements that only included “down”, “2 down”, and “high” out of the eight encoded features. I decided to include them all despite the very slight decrease in score as I believed that including all of them increased the interpretability and made for a more intuitive model. Figure 4 provides a visual of the feature importance of the final combination of features that I used for my model. Pre transfer BPM was clearly the most important feature, but as I will detail in the next paragraph, adding the other features led to a substantial improvement in model score. Another thing to note is that each of the eight encoded features by themselves are not very important, but taking them all together does make an impact on the model.

As I mentioned previously, I trained multiple different kinds of models as I wanted to see which one was most effective for this data set. I tested several kinds of regression models such as multiple linear regression and ridge regression as well as a couple ensemble methods such as random forest regression and gradient boosting regression. In the end, the multiple regression model proved to be the most effective method for modeling this data regardless of what features were used. For the final multiple linear regression model I used, I was able to achieve an R-squared value of .406. This means that my model was able to explain almost 41% of the variation in post transfer BPM. While obviously I would like that percentage to be as close to 100% as possible, 41% is quite good considering all the variables at play that are not quantifiable. Factors such as moving to a new place, playing in a new system, offseason skill improvement (or decline), and numerous other things all contribute to how well a player plays after transferring. Additionally, this model served as a clear improvement over the baseline model that I created which only used pre transfer BPM to predict post transfer BPM. This baseline model had an R-squared value of .255. My model was able to explain over 15% more of the variation in post transfer BPM than the baseline model could. The random forest model also outperformed the baseline model with an R-squared score of .313, but this model was not able to produce as effective results as the multiple regression model.

After the completion of the 2022-23 NCAA basketball season, it is time to go back and evaluate my transfer predictions. As a reminder, the goal of this model is to identify “effective” college basketball transfers. What exactly “effective” means is subjective, but for this case I've decided to evaluate an effective transfer as one who has matched or exceeded a BPM value of two. I'm choosing this benchmark because I believe that outscoring an opponent by two more points per 100 possessions compared to an average producer qualifies you as an “effective” player. Once again, there are many valid opinions on what makes an effective player, and many of those pay no consideration to BPM. With that being said, BPM provides a quality advanced statistic that can serve as a useful benchmark for evaluation.

Now for the question of how accurate my model was in predicting effective transfers. Of the 130 transfers that my model predicted to have a BPM value greater than or equal to 2, 93 of them actually did. In other words, for every four transfers my model predicted to be “effective”, about three of them actually were (~72% accuracy).

Additionally, of those 37 errors, only 17 of them ended up being negative BPM players, meaning that the team was worse with that player on the floor compared to the average player production. This means that approximately 87% of the time my model predicted a player to be an “effective” transfer, they were at least not a negative contributor. With how many variables go into a player's impact on their new team after transferring, I would say that the model did a strong job of accurately predicting effective transfers this year. In terms of errors going the other way around (predicted to NOT be effective but actually were) my model was also very accurate. Approximately 81% of the time a player was predicted to have a BPM less than 2 by my model, they actually did. With that being said, this kind of error is not as impactful as the first case. If you are expecting someone to be an effective player and they aren't, that is much more damaging than expecting someone to not be an effective player and being pleasantly surprised when they are.