Never Forget Another Line of Code

The snippet begins by declaring the hyperparameters to tune with ranges to select from, initializes an XGBoost base estimator and sets an evaluation set for validation.

Then a randomized search CV model is initialized, before fitting the model to the training data with early stopping and finally printing out the best parameters found during the search.

 1|  from xgboost import XGBRegressor
 2|  from sklearn.model_selection import RandomizedSearchCV
 3|  
 4|  # Step 1: Declare parameters to tune and values distributions to ranges to select from
 5|  PARAMS = {"subsample":[0.75, 1],
 6|            "colsample_bytree":[0.75, 1],
 7|            "max_depth":[2, 6],
 8|            "min_child_weight":[1, 5],
 9|            "learning_rate":[0.1, 0.01]}
10|  
11|  # Step 1: Initialise XGBoost estimator
12|  estimator = XGBRegressor(objective='reg:squarederror', 
13|                      n_estimators=1000,
14|                      random_state=101)
15|  
16|  # Step 2: Declare evaluation set
17|  eval_set = [(X_test, y_test)]
18|  
19|  # Step 3: Intialise randomized search CV model with 5 fold cross validation
20|  model = RandomizedSearchCV(estimator, 
21|                             PARAMS, 
22|                             cv=5, 
23|                             n_jobs=-1, 
24|                             random_state=101, 
25|                             scoring='neg_mean_squared_error')
26|  
27|  # Step 4: Fit model with early stopping rounds set to 10
28|  model.fit(X_train, 
29|            y_train,
30|            eval_set=eval_set,
31|            eval_metric='rmse',
32|            early_stopping_rounds=10)
33|  
34|  print('Best Parameters:', model.best_params_)