| # Math Question Classifier - Quick Start Guide | |
| ## Execution Order | |
| ### Setup (Blocks 1-7) | |
| **Run once to setup environment and define classes** | |
| 1. **Block 1**: Install packages | |
| 2. **Block 2**: Import libraries | |
| 3. **Block 3**: Set data path | |
| 4. **Block 4**: Convert JSON to Parquet (one-time data preparation) | |
| 5. **Block 5**: Define MathDatasetLoader class | |
| 6. **Block 6**: Define MathFeatureExtractor class | |
| 7. **Block 7**: Define MathQuestionClassifier class | |
| ### Training & Evaluation (Blocks 8-13) | |
| **Run to train and evaluate models** | |
| 8. **Block 8**: Load dataset from Parquet files | |
| 9. **Block 9**: Extract features (text preprocessing + math symbols + numeric) | |
| 10. **Block 10**: Vectorize features (TF-IDF + scaling) | |
| 11. **Block 11**: Train 5 models and compare performance | |
| 12. **Block 12**: Detailed evaluation of best model | |
| 13. **Block 13**: Complete test set analysis with 6 visualizations | |
| --- | |
| ## What Each Block Does | |
| ### Block 1-3: Environment Setup | |
| - Installs scikit-learn, pandas, matplotlib, seaborn, nltk | |
| - Imports all necessary libraries | |
| - Sets path to data directory (`./math`) | |
| ### Block 4: Data Consolidation | |
| **Purpose**: Convert JSON files to Parquet format | |
| - **Input**: `./math/train/` and `./math/test/` folders with JSON files | |
| - **Output**: `train.parquet` and `test.parquet` | |
| - **Benefit**: 10-100x faster loading than JSON | |
| - **Run**: Only once (skip if Parquet files already exist) | |
| ### Block 5-7: Class Definitions | |
| Define three main classes: | |
| - **MathDatasetLoader**: Loads Parquet files, shows statistics | |
| - **MathFeatureExtractor**: Cleans LaTeX, extracts math symbols, preprocesses text | |
| - **MathQuestionClassifier**: Trains models, evaluates performance | |
| ### Block 8: Load Data | |
| - Loads `train.parquet` and `test.parquet` | |
| - Shows class distribution for train and test sets | |
| - Displays 2 bar charts (train/test distribution) | |
| ### Block 9: Feature Extraction | |
| Extracts three types of features: | |
| 1. **Text features**: Preprocessed text (LaTeX cleaning, lemmatization) | |
| 2. **Math symbol features**: 10 binary indicators (has_fraction, has_sqrt, etc.) | |
| 3. **Numeric features**: 5 statistical measures (num_count, avg_number, etc.) | |
| ### Block 10: Vectorization | |
| - Creates TF-IDF features (5000 dimensions, trigrams) | |
| - Scales additional features to [0,1] using MinMaxScaler | |
| - **Critical**: Fits ONLY on training data (prevents data leakage) | |
| - Converts to CSR format for efficient operations | |
| ### Block 11: Model Training | |
| Trains 5 optimized models: | |
| 1. **Naive Bayes** (baseline) | |
| 2. **Logistic Regression** (linear classifier) | |
| 3. **SVM** (maximum margin) | |
| 4. **Random Forest** (ensemble) | |
| 5. **Gradient Boosting** (sequential ensemble) | |
| **Output**: | |
| - Comparison table with Accuracy, F1-Score, Training Time | |
| - 2 bar charts comparing performance and speed | |
| - Selects best model automatically | |
| ### Block 12: Detailed Evaluation | |
| - Confusion matrix visualization | |
| - Classification report (precision, recall, F1 per class) | |
| - Feature importance (for tree-based models) | |
| ### Block 13: Complete Analysis | |
| **Comprehensive evaluation on entire test set** | |
| **6 Visualizations**: | |
| 1. Confusion Matrix (absolute counts) | |
| 2. Normalized Confusion Matrix (proportions) | |
| 3. F1-Score by Topic (horizontal bar chart) | |
| 4. Precision vs Recall (scatter plot, size = support) | |
| 5. Test Set Distribution (bar chart) | |
| 6. Confidence Distribution (histogram: correct vs incorrect) | |
| **Analysis Sections**: | |
| - Overall performance (accuracy, F1-score) | |
| - Per-class metrics table | |
| - Confusion pair analysis | |
| - Summary statistics | |
| --- | |
| ## Expected Results | |
| ### Model Performance (F1-Score) | |
| - **Gradient Boosting**: 86-90% | |
| - **Logistic Regression**: 85-89% | |
| - **SVM**: 84-88% | |
| - **Naive Bayes**: 78-82% | |
| - **Random Forest**: 75-82% (expected to underperform on sparse features) | |
| ### Training Time | |
| - **Naive Bayes**: ~10 seconds | |
| - **Logistic Regression**: ~30 seconds | |
| - **SVM**: ~2 minutes | |
| - **Random Forest**: ~3 minutes | |
| - **Gradient Boosting**: ~5 minutes | |
| ### Per-Topic Performance | |
| **High Performance** (F1 > 90%): | |
| - counting_and_probability | |
| - number_theory | |
| **Medium Performance** (F1: 85-90%): | |
| - geometry | |
| - precalculus | |
| **Challenging** (F1: 80-85%): | |
| - algebra ↔ intermediate_algebra (similar concepts) | |
| - prealgebra ↔ algebra (overlapping operations) | |
| --- | |
| ## Key Design Decisions | |
| ### 1. Data Leakage Prevention | |
| **Critical**: TF-IDF vectorizer fitted ONLY on training data | |
| ``` | |
| Train/Test Split → Fit Vectorizer on Train → Transform Both | |
| ``` | |
| Without this, test vocabulary leaks into training, inflating performance by 1-3%. | |
| ### 2. Feature Engineering | |
| **Hybrid approach**: | |
| - TF-IDF (5000 features): Captures text content | |
| - Math symbols (10 features): Topic indicators (e.g., integrals → calculus) | |
| - Numeric features (5 features): Statistical properties | |
| **Why no hand-crafted keywords?** | |
| Avoided topic-specific keyword lists to prevent heuristic bias. Let the model learn discriminative vocabulary from data. | |
| ### 3. Hyperparameter Optimization | |
| All models use optimized parameters: | |
| - **C=1.0** (SVM/Logistic): Balanced regularization | |
| - **max_depth=30** (Random Forest): Sufficient complexity | |
| - **subsample=0.8** (Gradient Boosting): Stochastic sampling prevents overfitting | |
| ### 4. Class Imbalance Handling | |
| `class_weight='balanced'` automatically adjusts weights inversely proportional to class frequencies. | |
| --- | |
| ## Methodology | |
| ### Problem Type | |
| **Supervised Multi-Class Text Classification** | |
| **Why Classification (not Clustering)?** | |
| - Categories are predefined and labeled | |
| - Objective: Assign to known subtopic | |
| - Not discovering latent groups | |
| - Supervised learning with known labels | |
| ### Pipeline | |
| ``` | |
| JSON Files | |
| ↓ | |
| Parquet Conversion (Block 4) | |
| ↓ | |
| Feature Extraction (Block 9) | |
| ↓ | |
| TF-IDF Vectorization (Block 10) | |
| ↓ | |
| Model Training (Block 11) | |
| ↓ | |
| Evaluation (Blocks 12-13) | |
| ``` | |
| ### Feature Vector | |
| ``` | |
| Total: 5015 dimensions | |
| ├── TF-IDF: 5000 (unigrams, bigrams, trigrams) | |
| ├── Math Symbols: 10 (binary indicators) | |
| └── Numeric: 5 (scaled to [0,1]) | |
| ``` | |
| --- | |
| ## Troubleshooting | |
| ### "No data loaded" | |
| **Solution**: Check data path in Block 3 | |
| ```python | |
| DATA_PATH = './math' # Adjust to your path | |
| ``` | |
| ### "NameError: name 'results' is not defined" | |
| **Solution**: Run blocks in order. Block 12-13 need Block 11 first. | |
| ### "ValueError: Negative values" | |
| **Solution**: Block 10 should complete successfully. MinMaxScaler scales features to [0,1]. | |
| ### "TypeError: coo_matrix not subscriptable" | |
| **Solution**: Block 10 converts to CSR format. Ensure it runs completely. | |
| ### Model underperforms | |
| **Check**: | |
| 1. Data leakage prevented? (Vectorizer fitted on train only) | |
| 2. Features extracted correctly? (Block 9 output) | |
| 3. Class distribution balanced? (Block 8 charts) | |
| --- | |
| ## Performance Optimization | |
| ### Speed Up Training | |
| ```python | |
| # Reduce vocabulary | |
| vectorizer_config = {'max_features': 2000} | |
| # Fewer trees | |
| RandomForestClassifier(n_estimators=100) | |
| # Fewer boosting rounds | |
| GradientBoostingClassifier(n_estimators=50) | |
| ``` | |
| ### Reduce Memory | |
| ```python | |
| # Smaller vocabulary | |
| vectorizer_config = {'max_features': 3000} | |
| # Fewer n-grams | |
| vectorizer_config = {'ngram_range': (1, 2)} | |
| ``` | |
| --- | |
| ## Output Files | |
| After Block 13 completes, you'll have: | |
| - **train.parquet**: Training data (consolidated) | |
| - **test.parquet**: Test data (consolidated) | |
| - Performance metrics and visualizations | |
| - Model saved in memory (classifier.best_model) | |
| --- | |
| ## Next Steps | |
| ### Save Model | |
| Add after Block 13: | |
| ```python | |
| import pickle | |
| model_data = { | |
| 'model': classifier.best_model, | |
| 'vectorizer': classifier.vectorizer, | |
| 'scaler': classifier.scaler, | |
| 'label_encoder': classifier.label_encoder | |
| } | |
| with open('model.pkl', 'wb') as f: | |
| pickle.dump(model_data, f) | |
| ``` | |
| ### Batch Prediction | |
| ```python | |
| # Load model | |
| with open('model.pkl', 'rb') as f: | |
| model_data = pickle.load(f) | |
| # Predict | |
| new_problems = ["Solve x^2 = 16", "Find area of circle"] | |
| for problem in new_problems: | |
| # Preprocess → Extract features → Predict | |
| prediction = model.predict(...) | |
| ``` | |
| --- | |
| ## Summary | |
| **13 Blocks, 3 Stages**: | |
| 1. **Setup** (Blocks 1-7): One-time environment setup | |
| 2. **Training** (Blocks 8-11): Data loading and model training | |
| 3. **Evaluation** (Blocks 12-13): Comprehensive analysis | |
| **Key Features**: | |
| - Data leakage prevention | |
| - 5 optimized models | |
| - 6 visualization types | |
| - Probability predictions | |
| - Error analysis | |
| **Expected Time**: 10-15 minutes total (including training) | |
| **Expected Performance**: 85-90% F1-score on test set | |