@geraldineram

Incorporating social media sentiment into backtests requires strict preprocessing. Data from X or Reddit often contains duplicates, spam, or bots, inflating perceived signals. Deduplication removes repeated posts, while noise filters exclude irrelevant chatter. Tokenization, lemmatization, and language normalization improve consistency. Weighting by verified user reputation mitigates manipulation. Time alignment ensures that only signals available before price moves are used, avoiding future leak. Sentiment features are then tested for persistence and orthogonality with other factors. Without preprocessing, models risk overfitting to noisy hype cycles, degrading live trading performance.

0 reply

0 recast

0 reaction