| # Math Question Classifier - Quick Start Guide |
|
|
| ## Execution Order |
|
|
| ### Setup (Blocks 1-7) |
| **Run once to setup environment and define classes** |
|
|
| 1. **Block 1**: Install packages |
| 2. **Block 2**: Import libraries |
| 3. **Block 3**: Set data path |
| 4. **Block 4**: Convert JSON to Parquet (one-time data preparation) |
| 5. **Block 5**: Define MathDatasetLoader class |
| 6. **Block 6**: Define MathFeatureExtractor class |
| 7. **Block 7**: Define MathQuestionClassifier class |
|
|
| ### Training & Evaluation (Blocks 8-13) |
| **Run to train and evaluate models** |
|
|
| 8. **Block 8**: Load dataset from Parquet files |
| 9. **Block 9**: Extract features (text preprocessing + math symbols + numeric) |
| 10. **Block 10**: Vectorize features (TF-IDF + scaling) |
| 11. **Block 11**: Train 5 models and compare performance |
| 12. **Block 12**: Detailed evaluation of best model |
| 13. **Block 13**: Complete test set analysis with 6 visualizations |
|
|
| --- |
|
|
| ## What Each Block Does |
|
|
| ### Block 1-3: Environment Setup |
| - Installs scikit-learn, pandas, matplotlib, seaborn, nltk |
| - Imports all necessary libraries |
| - Sets path to data directory (`./math`) |
|
|
| ### Block 4: Data Consolidation |
| **Purpose**: Convert JSON files to Parquet format |
| - **Input**: `./math/train/` and `./math/test/` folders with JSON files |
| - **Output**: `train.parquet` and `test.parquet` |
| - **Benefit**: 10-100x faster loading than JSON |
| - **Run**: Only once (skip if Parquet files already exist) |
|
|
| ### Block 5-7: Class Definitions |
| Define three main classes: |
| - **MathDatasetLoader**: Loads Parquet files, shows statistics |
| - **MathFeatureExtractor**: Cleans LaTeX, extracts math symbols, preprocesses text |
| - **MathQuestionClassifier**: Trains models, evaluates performance |
|
|
| ### Block 8: Load Data |
| - Loads `train.parquet` and `test.parquet` |
| - Shows class distribution for train and test sets |
| - Displays 2 bar charts (train/test distribution) |
|
|
| ### Block 9: Feature Extraction |
| Extracts three types of features: |
| 1. **Text features**: Preprocessed text (LaTeX cleaning, lemmatization) |
| 2. **Math symbol features**: 10 binary indicators (has_fraction, has_sqrt, etc.) |
| 3. **Numeric features**: 5 statistical measures (num_count, avg_number, etc.) |
|
|
| ### Block 10: Vectorization |
| - Creates TF-IDF features (5000 dimensions, trigrams) |
| - Scales additional features to [0,1] using MinMaxScaler |
| - **Critical**: Fits ONLY on training data (prevents data leakage) |
| - Converts to CSR format for efficient operations |
|
|
| ### Block 11: Model Training |
| Trains 5 optimized models: |
| 1. **Naive Bayes** (baseline) |
| 2. **Logistic Regression** (linear classifier) |
| 3. **SVM** (maximum margin) |
| 4. **Random Forest** (ensemble) |
| 5. **Gradient Boosting** (sequential ensemble) |
|
|
| **Output**: |
| - Comparison table with Accuracy, F1-Score, Training Time |
| - 2 bar charts comparing performance and speed |
| - Selects best model automatically |
|
|
| ### Block 12: Detailed Evaluation |
| - Confusion matrix visualization |
| - Classification report (precision, recall, F1 per class) |
| - Feature importance (for tree-based models) |
|
|
| ### Block 13: Complete Analysis |
| **Comprehensive evaluation on entire test set** |
|
|
| **6 Visualizations**: |
| 1. Confusion Matrix (absolute counts) |
| 2. Normalized Confusion Matrix (proportions) |
| 3. F1-Score by Topic (horizontal bar chart) |
| 4. Precision vs Recall (scatter plot, size = support) |
| 5. Test Set Distribution (bar chart) |
| 6. Confidence Distribution (histogram: correct vs incorrect) |
|
|
| **Analysis Sections**: |
| - Overall performance (accuracy, F1-score) |
| - Per-class metrics table |
| - Confusion pair analysis |
| - Summary statistics |
|
|
| --- |
|
|
| ## Expected Results |
|
|
| ### Model Performance (F1-Score) |
| - **Gradient Boosting**: 86-90% |
| - **Logistic Regression**: 85-89% |
| - **SVM**: 84-88% |
| - **Naive Bayes**: 78-82% |
| - **Random Forest**: 75-82% (expected to underperform on sparse features) |
|
|
| ### Training Time |
| - **Naive Bayes**: ~10 seconds |
| - **Logistic Regression**: ~30 seconds |
| - **SVM**: ~2 minutes |
| - **Random Forest**: ~3 minutes |
| - **Gradient Boosting**: ~5 minutes |
|
|
| ### Per-Topic Performance |
| **High Performance** (F1 > 90%): |
| - counting_and_probability |
| - number_theory |
| |
| **Medium Performance** (F1: 85-90%): |
| - geometry |
| - precalculus |
| |
| **Challenging** (F1: 80-85%): |
| - algebra ↔ intermediate_algebra (similar concepts) |
| - prealgebra ↔ algebra (overlapping operations) |
|
|
| --- |
|
|
| ## Key Design Decisions |
|
|
| ### 1. Data Leakage Prevention |
| **Critical**: TF-IDF vectorizer fitted ONLY on training data |
| ``` |
| Train/Test Split → Fit Vectorizer on Train → Transform Both |
| ``` |
| Without this, test vocabulary leaks into training, inflating performance by 1-3%. |
|
|
| ### 2. Feature Engineering |
| **Hybrid approach**: |
| - TF-IDF (5000 features): Captures text content |
| - Math symbols (10 features): Topic indicators (e.g., integrals → calculus) |
| - Numeric features (5 features): Statistical properties |
|
|
| **Why no hand-crafted keywords?** |
| Avoided topic-specific keyword lists to prevent heuristic bias. Let the model learn discriminative vocabulary from data. |
|
|
| ### 3. Hyperparameter Optimization |
| All models use optimized parameters: |
| - **C=1.0** (SVM/Logistic): Balanced regularization |
| - **max_depth=30** (Random Forest): Sufficient complexity |
| - **subsample=0.8** (Gradient Boosting): Stochastic sampling prevents overfitting |
| |
| ### 4. Class Imbalance Handling |
| `class_weight='balanced'` automatically adjusts weights inversely proportional to class frequencies. |
| |
| --- |
| |
| ## Methodology |
| |
| ### Problem Type |
| **Supervised Multi-Class Text Classification** |
|
|
| **Why Classification (not Clustering)?** |
| - Categories are predefined and labeled |
| - Objective: Assign to known subtopic |
| - Not discovering latent groups |
| - Supervised learning with known labels |
|
|
| ### Pipeline |
| ``` |
| JSON Files |
| ↓ |
| Parquet Conversion (Block 4) |
| ↓ |
| Feature Extraction (Block 9) |
| ↓ |
| TF-IDF Vectorization (Block 10) |
| ↓ |
| Model Training (Block 11) |
| ↓ |
| Evaluation (Blocks 12-13) |
| ``` |
|
|
| ### Feature Vector |
| ``` |
| Total: 5015 dimensions |
| ├── TF-IDF: 5000 (unigrams, bigrams, trigrams) |
| ├── Math Symbols: 10 (binary indicators) |
| └── Numeric: 5 (scaled to [0,1]) |
| ``` |
|
|
| --- |
|
|
| ## Troubleshooting |
|
|
| ### "No data loaded" |
| **Solution**: Check data path in Block 3 |
| ```python |
| DATA_PATH = './math' # Adjust to your path |
| ``` |
|
|
| ### "NameError: name 'results' is not defined" |
| **Solution**: Run blocks in order. Block 12-13 need Block 11 first. |
|
|
| ### "ValueError: Negative values" |
| **Solution**: Block 10 should complete successfully. MinMaxScaler scales features to [0,1]. |
|
|
| ### "TypeError: coo_matrix not subscriptable" |
| **Solution**: Block 10 converts to CSR format. Ensure it runs completely. |
| |
| ### Model underperforms |
| **Check**: |
| 1. Data leakage prevented? (Vectorizer fitted on train only) |
| 2. Features extracted correctly? (Block 9 output) |
| 3. Class distribution balanced? (Block 8 charts) |
| |
| --- |
| |
| ## Performance Optimization |
| |
| ### Speed Up Training |
| ```python |
| # Reduce vocabulary |
| vectorizer_config = {'max_features': 2000} |
| |
| # Fewer trees |
| RandomForestClassifier(n_estimators=100) |
|
|
| # Fewer boosting rounds |
| GradientBoostingClassifier(n_estimators=50) |
| ``` |
| |
| ### Reduce Memory |
| ```python |
| # Smaller vocabulary |
| vectorizer_config = {'max_features': 3000} |
| |
| # Fewer n-grams |
| vectorizer_config = {'ngram_range': (1, 2)} |
| ``` |
| |
| --- |
| |
| ## Output Files |
| |
| After Block 13 completes, you'll have: |
| - **train.parquet**: Training data (consolidated) |
| - **test.parquet**: Test data (consolidated) |
| - Performance metrics and visualizations |
| - Model saved in memory (classifier.best_model) |
|
|
| --- |
|
|
| ## Next Steps |
|
|
| ### Save Model |
| Add after Block 13: |
| ```python |
| import pickle |
| model_data = { |
| 'model': classifier.best_model, |
| 'vectorizer': classifier.vectorizer, |
| 'scaler': classifier.scaler, |
| 'label_encoder': classifier.label_encoder |
| } |
| with open('model.pkl', 'wb') as f: |
| pickle.dump(model_data, f) |
| ``` |
|
|
| ### Batch Prediction |
| ```python |
| # Load model |
| with open('model.pkl', 'rb') as f: |
| model_data = pickle.load(f) |
| |
| # Predict |
| new_problems = ["Solve x^2 = 16", "Find area of circle"] |
| for problem in new_problems: |
| # Preprocess → Extract features → Predict |
| prediction = model.predict(...) |
| ``` |
|
|
| --- |
|
|
| ## Summary |
|
|
| **13 Blocks, 3 Stages**: |
| 1. **Setup** (Blocks 1-7): One-time environment setup |
| 2. **Training** (Blocks 8-11): Data loading and model training |
| 3. **Evaluation** (Blocks 12-13): Comprehensive analysis |
|
|
| **Key Features**: |
| - Data leakage prevention |
| - 5 optimized models |
| - 6 visualization types |
| - Probability predictions |
| - Error analysis |
|
|
| **Expected Time**: 10-15 minutes total (including training) |
|
|
| **Expected Performance**: 85-90% F1-score on test set |
|
|
