|
|
{% extends "layout.html" %}
|
|
|
|
|
|
{% block content %}
|
|
|
<!DOCTYPE html>
|
|
|
<html lang="en">
|
|
|
<head>
|
|
|
<meta charset="UTF-8">
|
|
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
|
|
<title>Study Guide: Gradient Boosting Regression</title>
|
|
|
|
|
|
<script src="https://polyfill.io/v3/polyfill.min.js?features=es6"></script>
|
|
|
<script id="MathJax-script" async src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
|
|
|
<style>
|
|
|
|
|
|
body {
|
|
|
background-color: #ffffff;
|
|
|
color: #000000;
|
|
|
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
|
|
|
font-weight: normal;
|
|
|
line-height: 1.8;
|
|
|
margin: 0;
|
|
|
padding: 20px;
|
|
|
}
|
|
|
|
|
|
|
|
|
.container {
|
|
|
max-width: 800px;
|
|
|
margin: 0 auto;
|
|
|
padding: 20px;
|
|
|
}
|
|
|
|
|
|
|
|
|
h1, h2, h3 {
|
|
|
color: #000000;
|
|
|
border: none;
|
|
|
font-weight: bold;
|
|
|
}
|
|
|
|
|
|
h1 {
|
|
|
text-align: center;
|
|
|
border-bottom: 3px solid #000;
|
|
|
padding-bottom: 10px;
|
|
|
margin-bottom: 30px;
|
|
|
font-size: 2.5em;
|
|
|
}
|
|
|
|
|
|
h2 {
|
|
|
font-size: 1.8em;
|
|
|
margin-top: 40px;
|
|
|
border-bottom: 1px solid #ddd;
|
|
|
padding-bottom: 8px;
|
|
|
}
|
|
|
|
|
|
h3 {
|
|
|
font-size: 1.3em;
|
|
|
margin-top: 25px;
|
|
|
}
|
|
|
|
|
|
|
|
|
strong {
|
|
|
font-weight: 900;
|
|
|
}
|
|
|
|
|
|
|
|
|
p, li {
|
|
|
font-size: 1.1em;
|
|
|
border-bottom: 1px solid #e0e0e0;
|
|
|
padding-bottom: 10px;
|
|
|
margin-bottom: 10px;
|
|
|
}
|
|
|
|
|
|
|
|
|
li:last-child {
|
|
|
border-bottom: none;
|
|
|
}
|
|
|
|
|
|
|
|
|
ul {
|
|
|
list-style-type: none;
|
|
|
padding-left: 0;
|
|
|
}
|
|
|
|
|
|
li::before {
|
|
|
content: "•";
|
|
|
color: #000;
|
|
|
font-weight: bold;
|
|
|
display: inline-block;
|
|
|
width: 1em;
|
|
|
margin-left: 0;
|
|
|
}
|
|
|
|
|
|
|
|
|
pre {
|
|
|
background-color: #f4f4f4;
|
|
|
border: 1px solid #ddd;
|
|
|
border-radius: 5px;
|
|
|
padding: 15px;
|
|
|
white-space: pre-wrap;
|
|
|
word-wrap: break-word;
|
|
|
font-family: "Courier New", Courier, monospace;
|
|
|
font-size: 0.95em;
|
|
|
font-weight: normal;
|
|
|
color: #333;
|
|
|
border-bottom: none;
|
|
|
}
|
|
|
|
|
|
|
|
|
.story {
|
|
|
background-color: #f9f9f9;
|
|
|
border-left: 4px solid #4CAF50;
|
|
|
margin: 15px 0;
|
|
|
padding: 10px 15px;
|
|
|
font-style: italic;
|
|
|
color: #555;
|
|
|
font-weight: normal;
|
|
|
border-bottom: none;
|
|
|
}
|
|
|
|
|
|
|
|
|
table {
|
|
|
width: 100%;
|
|
|
border-collapse: collapse;
|
|
|
margin: 25px 0;
|
|
|
}
|
|
|
th, td {
|
|
|
border: 1px solid #ddd;
|
|
|
padding: 12px;
|
|
|
text-align: left;
|
|
|
}
|
|
|
th {
|
|
|
background-color: #f2f2f2;
|
|
|
font-weight: bold;
|
|
|
}
|
|
|
|
|
|
|
|
|
@media (max-width: 768px) {
|
|
|
body, .container {
|
|
|
padding: 10px;
|
|
|
}
|
|
|
h1 { font-size: 2em; }
|
|
|
h2 { font-size: 1.5em; }
|
|
|
h3 { font-size: 1.2em; }
|
|
|
p, li { font-size: 1em; }
|
|
|
pre { font-size: 0.85em; }
|
|
|
table, th, td { font-size: 0.9em; }
|
|
|
}
|
|
|
</style>
|
|
|
</head>
|
|
|
<body>
|
|
|
|
|
|
<div class="container">
|
|
|
<h1>📘 Study Guide: Gradient Boosting Regression (GBR)</h1>
|
|
|
|
|
|
|
|
|
|
|
|
<div>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<a
|
|
|
href="/gradient-boosting-three"
|
|
|
target="_blank"
|
|
|
onclick="playSound()"
|
|
|
class="
|
|
|
cursor-pointer
|
|
|
inline-block
|
|
|
relative
|
|
|
bg-blue-500
|
|
|
text-white
|
|
|
font-bold
|
|
|
py-4 px-8
|
|
|
rounded-xl
|
|
|
text-2xl
|
|
|
transition-all
|
|
|
duration-150
|
|
|
|
|
|
/* 3D Effect (Hard Shadow) */
|
|
|
shadow-[0_8px_0_rgb(29,78,216)]
|
|
|
|
|
|
/* Pressed State (Move down & remove shadow) */
|
|
|
active:shadow-none
|
|
|
active:translate-y-[8px]
|
|
|
">
|
|
|
Tap Me!
|
|
|
</a>
|
|
|
</div>
|
|
|
|
|
|
<script>
|
|
|
function playSound() {
|
|
|
const audio = document.getElementById("clickSound");
|
|
|
if (audio) {
|
|
|
audio.currentTime = 0;
|
|
|
audio.play().catch(e => console.log("Audio play failed:", e));
|
|
|
}
|
|
|
}
|
|
|
</script>
|
|
|
|
|
|
|
|
|
<h2>🔹 Core Concepts</h2>
|
|
|
<div class="story">
|
|
|
<p><strong>Story-style intuition:</strong></p>
|
|
|
<p>Imagine you are trying to predict the price of houses. Your first guess is just the average price of all houses—not very accurate. So, you look at your mistakes (<strong>residuals</strong>). You build a second, simple model that's an expert at fixing those specific mistakes. Then, you look at the remaining mistakes and build a third expert to fix those. You repeat this, adding a new expert each time to patch the leftover errors, until your predictions are very accurate.</p>
|
|
|
</div>
|
|
|
<h3>Definition:</h3>
|
|
|
<p>
|
|
|
<strong>Gradient Boosting Regression (GBR)</strong> is an <strong>ensemble</strong> machine learning technique that builds a strong predictive model by <strong>sequentially combining multiple weak learners</strong>, usually decision trees. Each new tree focuses on correcting the errors (<strong>residuals</strong>) of the previous trees.
|
|
|
</p>
|
|
|
|
|
|
<h3>Difference from Random Forest (Bagging vs. Boosting):</h3>
|
|
|
<ul>
|
|
|
<li><strong>Random Forest:</strong> Builds many trees in <strong>parallel</strong>. Each tree sees a random subset of data, and their predictions are averaged. It's like asking many independent experts for their opinion and taking the average.</li>
|
|
|
<li><strong>Gradient Boosting:</strong> Builds trees <strong>sequentially</strong>. Each tree learns from the errors of the previous ones. It's like a team of experts where each new member is trained to fix the mistakes of the one before them.</li>
|
|
|
</ul>
|
|
|
|
|
|
<h2>🔹 Mathematical Foundation</h2>
|
|
|
<div class="story">
|
|
|
<p><strong>Story example: The Improving Chef</strong></p>
|
|
|
<p>A chef is trying to create the perfect recipe (the model). Their first dish (<strong>initial prediction</strong>) is just a basic soup. They taste it and note the errors (<strong>residuals</strong>)—it's not salty enough. They don't throw it out; instead, they add a pinch of salt (the <strong>weak learner</strong>). Then they taste again. Now it's a bit bland. They add some herbs. This step-by-step correction, guided by tasting (calculating the gradient), is how GBR refines its predictions.</p>
|
|
|
</div>
|
|
|
<h3>Step-by-step algorithm:</h3>
|
|
|
<ol>
|
|
|
<li>Initialize model with a constant prediction: \( F_0(x) = \text{mean}(y) \)</li>
|
|
|
<li>For each step (tree) m = 1 to M:</li>
|
|
|
<ul>
|
|
|
<li>Compute residuals (errors): \( r_i = y_i - F_{m-1}(x_i) \)</li>
|
|
|
<li>Train a weak learner (a small decision tree \(h_m(x)\)) to predict these residuals.</li>
|
|
|
<li>Update the model by adding the new tree, scaled by a learning rate \( \nu \):<br>
|
|
|
\( F_m(x) = F_{m-1}(x) + \nu \cdot h_m(x) \)</li>
|
|
|
</ul>
|
|
|
</ol>
|
|
|
|
|
|
<h2>🔹 Key Parameters</h2>
|
|
|
<table>
|
|
|
<thead>
|
|
|
<tr>
|
|
|
<th>Parameter</th>
|
|
|
<th>Explanation & Story</th>
|
|
|
</tr>
|
|
|
</thead>
|
|
|
<tbody>
|
|
|
<tr>
|
|
|
<td><strong>n_estimators</strong></td>
|
|
|
<td>The number of boosting stages, or the number of "mini-experts" (trees) to add in the sequence. <strong>Story:</strong> How many times the chef is allowed to taste and correct the recipe.</td>
|
|
|
</tr>
|
|
|
<tr>
|
|
|
<td><strong>learning_rate</strong></td>
|
|
|
<td>Scales the contribution of each tree. Small values mean smaller, more careful correction steps. <strong>Story:</strong> How much salt or herbs the chef adds at each step. A small pinch is safer than a whole handful.</td>
|
|
|
</tr>
|
|
|
<tr>
|
|
|
<td><strong>max_depth</strong></td>
|
|
|
<td>The maximum depth of each decision tree. Controls complexity. <strong>Story:</strong> A shallow tree is an expert on one simple rule (e.g., "add salt"). A deep tree is a complex expert who considers many factors.</td>
|
|
|
</tr>
|
|
|
<tr>
|
|
|
<td><strong>subsample</strong></td>
|
|
|
<td>The fraction of data used to train each tree. Introduces randomness to prevent overfitting. <strong>Story:</strong> The chef tastes only a random spoonful of the soup each time, not the whole pot, to avoid over-correcting for one odd flavor.</td>
|
|
|
</tr>
|
|
|
</tbody>
|
|
|
</table>
|
|
|
|
|
|
<h2>🔹 Strengths & Weaknesses</h2>
|
|
|
<div class="story">
|
|
|
<p>GBR is like a master craftsman who builds something beautiful piece by piece. The final product is incredibly accurate (<strong>high predictive power</strong>), but the process is slow (<strong>slower training</strong>) and requires careful attention to detail (<strong>sensitive to hyperparameters</strong>). If not careful, the craftsman might over-engineer the product (<strong>overfitting</strong>).</p>
|
|
|
</div>
|
|
|
<h3>Advantages:</h3>
|
|
|
<ul>
|
|
|
<li>✅ High predictive accuracy, often state-of-the-art.</li>
|
|
|
<li>✅ Works well with non-linear and complex relationships.</li>
|
|
|
<li>✅ Handles mixed data types (categorical + numeric).</li>
|
|
|
</ul>
|
|
|
<h3>Disadvantages:</h3>
|
|
|
<ul>
|
|
|
<li>❌ Slower training than bagging methods (like Random Forest).</li>
|
|
|
<li>❌ Sensitive to hyperparameters (requires careful tuning).</li>
|
|
|
<li>❌ Can overfit if not tuned properly.</li>
|
|
|
</ul>
|
|
|
|
|
|
<h2>🔹 Python Implementation</h2>
|
|
|
<div class="story">
|
|
|
<p>Here, we are programming our "chef" (the `GradientBoostingRegressor`). We give it the recipe book (`X`, `y` data) and set the rules (`n_estimators`, `learning_rate`). The chef then `fit`s the recipe by training on the data. Finally, we `predict` how a new dish will taste and `evaluate` how good our final recipe is.</p>
|
|
|
</div>
|
|
|
<pre><code>
|
|
|
from sklearn.ensemble import GradientBoostingRegressor
|
|
|
from sklearn.model_selection import train_test_split
|
|
|
from sklearn.metrics import mean_squared_error
|
|
|
import numpy as np
|
|
|
|
|
|
# Example dataset
|
|
|
X = np.array([[1], [2], [3], [4], [5], [6], [7], [8]])
|
|
|
y = np.array([2, 5, 7, 9, 11, 13, 15, 17])
|
|
|
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
|
|
|
|
|
|
# Initialize GBR
|
|
|
gbr = GradientBoostingRegressor(n_estimators=100, learning_rate=0.1, max_depth=2, random_state=42)
|
|
|
|
|
|
# Train
|
|
|
gbr.fit(X_train, y_train)
|
|
|
|
|
|
# Predict
|
|
|
y_pred = gbr.predict(X_test)
|
|
|
|
|
|
# Evaluate
|
|
|
mse = mean_squared_error(y_test, y_pred)
|
|
|
print(f"Mean Squared Error: {mse:.2f}")
|
|
|
</code></pre>
|
|
|
|
|
|
<h2>🔹 Real-World Applications</h2>
|
|
|
<div class="story">
|
|
|
<p>A bank uses GBR to predict credit risk. The first model makes a simple guess based on average income. The next model corrects for age, the next for loan amount, and so on. By chaining these simple experts, the bank builds a highly accurate system to identify customers who are likely to default, saving millions.</p>
|
|
|
</div>
|
|
|
<ul>
|
|
|
<li><strong>Credit risk scoring</strong> → predict if someone will default on a loan.</li>
|
|
|
<li><strong>Customer churn prediction</strong> → identify customers likely to leave a service.</li>
|
|
|
<li><strong>Energy demand forecasting</strong> → predict daily energy consumption for a city.</li>
|
|
|
<li><strong>Medical predictions</strong> → predict patient outcomes or disease risk based on their data.</li>
|
|
|
</ul>
|
|
|
|
|
|
<h2>🔹 Best Practices</h2>
|
|
|
<div class="story">
|
|
|
<p>Treat tuning GBR like a skilled surgeon: be careful and precise. Use <strong>cross-validation</strong> to find the best settings. Always keep an eye on the patient's vitals (<strong>validation error</strong>) to make sure the procedure is going well and stop if things get worse (<strong>early stopping</strong>). Always confirm if such a complex surgery is needed by checking if a simpler method works first (<strong>compare to baseline models</strong>).</p>
|
|
|
</div>
|
|
|
<ul>
|
|
|
<li>Use <strong>cross-validation</strong> and grid search to find the optimal hyperparameters.</li>
|
|
|
<li>Balance <strong>learning_rate</strong> and <strong>n_estimators</strong>: a smaller learning rate usually requires more trees.</li>
|
|
|
<li>Monitor training vs. validation error to detect overfitting early and use <strong>early stopping</strong>.</li>
|
|
|
<li>Compare GBR's performance against simpler models (like Linear Regression or Random Forest) to justify its complexity.</li>
|
|
|
</ul>
|
|
|
|
|
|
<h2>🔹 Key Terminology Explained</h2>
|
|
|
<div class="story">
|
|
|
<p><strong>The Story: The Student, The Chef, and The Tailor</strong></p>
|
|
|
<p>These terms might sound complex, but they relate to everyday ideas. Think of them as tools and checks to ensure our model isn't just "memorizing" answers but is actually learning concepts it can apply to new, unseen problems.</p>
|
|
|
</div>
|
|
|
<h3>Cross-Validation</h3>
|
|
|
<p>
|
|
|
<strong>What it is:</strong> A technique to assess how a model will generalize to an independent dataset. It involves splitting the data into 'folds' and training/testing the model on different combinations of these folds.
|
|
|
</p>
|
|
|
<p>
|
|
|
<strong>Story Example:</strong> Imagine a student has 5 practice exams. Instead of studying from all 5 and then taking a final, they use one exam to test themselves and study from the other four. They repeat this process five times, using a different practice exam for the test each time. This gives them a much better idea of their true knowledge and how they'll perform on the <strong>real</strong> final exam, rather than just memorizing answers. This rotation is <strong>cross-validation</strong>.
|
|
|
</p>
|
|
|
|
|
|
<h3>Validation Error</h3>
|
|
|
<p>
|
|
|
<strong>What it is:</strong> The error of the model calculated on a set of data that it was not trained on (the validation set). It's a measure of how well the model can predict new, unseen data.
|
|
|
</p>
|
|
|
<p>
|
|
|
<strong>Story Example:</strong> A chef develops a new recipe in their kitchen (the <strong>training data</strong>). The "training error" is how good the recipe tastes to <strong>them</strong>. But the true test is when a customer tries it (the <strong>validation data</strong>). The customer's feedback represents the "validation error". A low validation error means the recipe is a hit with new people, not just the chef who created it.
|
|
|
</p>
|
|
|
|
|
|
<h3>Overfitting</h3>
|
|
|
<p>
|
|
|
<strong>What it is:</strong> A modeling error that occurs when a model learns the training data's noise and details so well that it negatively impacts its performance on new, unseen data.
|
|
|
</p>
|
|
|
<p>
|
|
|
<strong>Story Example:</strong> A tailor is making a suit. If they make it <strong>exactly</strong> to the client's current posture, including a slight slouch and the phone in their pocket (the "noise"), it's a perfect fit for that one moment. This is <strong>overfitting</strong>. The training error is zero! But the moment the client stands up straight, the suit looks terrible. A good model, like a good tailor, creates a fit that works well in general, ignoring temporary noise.
|
|
|
</p>
|
|
|
|
|
|
<h3>Hyperparameter Tuning</h3>
|
|
|
<p>
|
|
|
<strong>What it is:</strong> The process of finding the optimal combination of settings (hyperparameters like `learning_rate` or `max_depth`) that maximizes the model's performance.
|
|
|
</p>
|
|
|
<p>
|
|
|
<strong>Story Example:</strong> Think of a race car driver. The car's engine is the model, but the driver can adjust the tire pressure, suspension, and wing angle. These settings are the <strong>hyperparameters</strong>. The driver runs several practice laps (like cross-validation), trying different combinations to find the setup that results in the fastest lap time. This process of tweaking the car's settings is <strong>hyperparameter tuning</strong>.
|
|
|
</p>
|
|
|
</div>
|
|
|
|
|
|
</body>
|
|
|
</html>
|
|
|
|
|
|
|
|
|
{% endblock %} |
