{"id":14644,"date":"2021-03-08T08:12:26","date_gmt":"2021-03-08T16:12:26","guid":{"rendered":"https:\/\/www.springboard.com\/?p=14644"},"modified":"2022-09-14T05:59:52","modified_gmt":"2022-09-14T12:59:52","slug":"xgboost-explainer","status":"publish","type":"post","link":"https:\/\/www.springboard.com\/blog\/data-science\/xgboost-explainer\/","title":{"rendered":"XGBoost Simply Explained (With an Example in Python)"},"content":{"rendered":"\n

Boosting, especially of decision trees, is among the most prevalent and powerful machine learning algorithms.<\/p>\n\n\n\n

There are many variants of boosting algorithms and frameworks implementing those algorithms. XGBoost\u2014short for the exciting moniker extreme gradient boosting\u2014is one of the most well-known algorithms with an accompanying, and even more popular, framework.<\/p>\n\n\n\n

This article will guide you through the nuances of XGBoost (the algorithm) and how to use XGBoost (the framework).<\/p>\n\n\n\n

*Looking for the Colab Notebook<\/strong> for this post? Find it right here.*<\/p>\n\n\n\n

What is XGBoost?<\/h2>\n\n\n\n

The term \u201cXGBoost\u201d can refer to both a gradient boosting algorithm for decision trees that solves many data science<\/a> problems in a fast and accurate way and an open-source framework implementing that algorithm.<\/p>\n\n\n\n

To disambiguate between the two meanings of XGBoost, we’ll call the algorithm “XGBoost the Algorithm<\/strong>” and the framework “XGBoost the Framework<\/strong>.”<\/p>\n\n\n\n

To understand XGBoost the Framework, we first have to understand XGBoost the Algorithm.<\/p>\n\n\n\n

What is XGBoost the Algorithm?<\/h2>\n\n\n\n

As the name may reveal, XGBoost the Algorithm is a gradient boosting algorithm, a common technique in ensemble learning.<\/p>\n\n\n\n

To unpack that new phrase, ensemble learning is a type of machine learning that enlists many models to make predictions together. Boosting algorithms are distinguished from other ensemble learning techniques by building a sequence of initially weak models into increasingly more powerful models. Gradient boosting algorithms choose how to build a more powerful model using the gradient of a loss function that captures the performance of a model.<\/p>\n\n\n\n

Gradient boosting is a foundational approach to many machine learning algorithms. XGBoost has solidified its name in the boosting game with its use in many competition-winning models<\/a> and prolific reference in research<\/a>.<\/p>\n\n\n\n

How does XGBoost the Algorithm work?<\/h2>\n\n\n\n

XGBoost the Algorithm operates on decision trees, models that construct a graph that examines the input under various “if” statements (vertices in the graph). Whether the “if” condition is satisfied influences the next “if” condition and eventual prediction. XGBoost the Algorithm progressively adds more and more “if” conditions to the decision tree to build a stronger model.<\/p>\n\n\n\n

\"xgboost<\/a><\/figure>\n\n\n\n

How does XGBoost the Algorithm work?<\/h2>\n\n\n\n

XGBoost the Framework implements XGBoost the Algorithm and other generic gradient boosting techniques for decision trees.<\/p>\n\n\n\n

XGBoost the Framework is maintained by open-source contributors\u2014it\u2019s available in Python, R, Java, Ruby, Swift, Julia, C, and C++ along with other community-built, non-official support in many other languages.<\/p>\n\n\n\n

XGBoost the Algorithm was first published by University of Washington researchers<\/a> in 2016 as a novel gradient boosting algorithm. Like other gradient boosting algorithms on decision trees, XGBoost considers the leaves of the current decision tree and questions whether turning that leaf into a new \u201cif\u201d statement with separate predictions would benefit the model. The benefit to the model depends on the \u201cif\u201d statement chosen and which leaf it\u2019s placed on\u2014this can be determined using the gradient of the loss. The loss includes a scoring function that measures algorithm performance.<\/p>\n\n\n

Get To Know Other\tData Science Students<\/h4>
\"Mengqin<\/a>

Mengqin (Cassie) Gong<\/p>

Data Scientist at Whatsapp<\/p><\/div>

<\/div>

Read Story<\/a><\/p><\/div><\/div>

\"Pizon<\/a>

Pizon Shetu<\/p>

Data Scientist at Whiterock AI<\/p><\/div>

Read Story<\/a><\/p><\/div><\/div>

\"Jonas<\/a>

Jonas Cuadrado<\/p>

Senior Data Scientist at Feedzai<\/p><\/div>

Read Story<\/a><\/p><\/div><\/div><\/div><\/div>\n\n\n\n

What sets XGBoost the Algorithm apart?<\/h2>\n\n\n\n

XGBoost the Algorithm sets itself apart from other gradient boosting techniques by using a second-order approximation of the scoring function. This approximation allows XGBoost to calculate the optimal \u201cif\u201d condition and its impact on performance. XGBoost The Algorithm can then store these in its memory the next decision tree to save recomputing it.<\/p>\n\n\n\n

XGBoost the Algorithm is powerful on its own but is also a great fixer-upper using the other tools from your machine learning toolbox. Consider feature engineering for instance, where the machine learning engineer preprocesses the raw inputs into new input features before letting the model get its hands dirty. XGBoost the Algorithm makes the most of engineered features and can produce a nicely interpretable and high performing model.<\/p>\n\n\n\n

How do you use XGBoost?<\/h2>\n\n\n\n

Together, XGBoost the Algorithm and XGBoost the Framework form a great pairing with many uses.<\/p>\n\n\n\n