NFL Game Prediction Pipeline

The pipeline scrapes 1,673 NFL games from 2018 to 2024 across 28 scripts using BeautifulSoup, requests, and Selenium, handling paginated navigation, inconsistent table schemas, and rate limiting, and lands them as 30,000+ structured CSVs. A rolling-statistics engine then generates team, player, and momentum features with peak tracking, cached to Parquet for fast retraining.

The model is an XGBoost classifier reaching 71% accuracy against a 62.8% majority-class baseline, tuned through a staged randomized hyperparameter search with class-weight balancing, SMOTE oversampling, and decision-threshold optimization. It's validated with a walk-forward backtest across the 2024 season that retrains weekly on prior data only, to avoid look-ahead bias, and served through a Flask JSON API consumed by a Next.js frontend.

All of the data comes from Pro Football Reference, which hides most of its tables inside HTML comments. The scraper parses the visible page, then re-parses every comment node, and lands 19 tables per game: drives, snap counts, starters, advanced passing, rushing, receiving and defence, play-by-play, and game info. Ragged tables take their header from the longest row, relocated franchises are mapped to one name, and each boxscore request waits about three seconds.

Features are built in strict game order. Each game's row is assembled from the rolling state before that game's own stats are applied, and the state resets every season. That state holds season-to-date averages with a peak high and low per stat, last-three and last-five momentum windows, and 154 home-minus-away team differentials, plus per-player columns for 68 roster slots a side. For upcoming games, rosters are inferred from the top players per position by cumulative snap count. The cached matrix is 1,657 games by roughly 65,000 columns.