Backtests

Backtests are essential tools for evaluating forecasting model performance. They simulate how well different models would have predicted historical data, helping you choose the best algorithms for your specific use case.

What is Backtesting?

Backtesting is a method of checking how well forecasting models work using past data. Instead of waiting to see how a model performs in the future, backtesting lets you simulate how it would predict past demand and compare those predictions with what actually happened.

Key Benefits

• Model Comparison: Evaluate multiple algorithms side-by-side
• Performance Validation: Test models before using them for real forecasts
• Accuracy Metrics: Get quantitative measures of model performance
• Business Confidence: Make informed decisions about forecasting strategies

Creating Backtests

Step 1: Select Your Data

1. Navigate to the Backtests section in your workspace
2. Click "New Backtest" to start the creation process
3. Select your main dataset containing historical demand data
4. Optionally include categorical and external datasets

Step 2: Configure Backtest Parameters

Forecast Horizon

• Definition: How far ahead each model predicts
• Recommendation: Match your actual forecast horizon

Number of Windows

• Definition: How many times to repeat the backtest
• Purpose: Each window tests the model on a different time period
• Recommendation: 2-6 windows for reliable results

Step Size

• Definition: How far to move backward between windows
• Purpose: Determines overlap between test periods
• Recommendation: Usually set to match forecast horizon

Model Selection

Choose the same models available for forecasting:

• Traditional: ETS, ARIMA, Holt-Winters
• Machine Learning: LSTM, LightGBM, Linear Regression
• Neural Networks: N-BEATS, N-HITS, PatchTST, Autoformer

Step 3: Start the Backtest

1. Review your configuration
2. Click "Create Backtest" to start the process
3. Monitor progress in the backtest detail page
4. Access results when processing completes

Backtest Results

Performance Metrics

The backtest provides comprehensive accuracy metrics for each model:

Accuracy Measures

• 1 - WMAPE (Weighted Mean Absolute Percentage Error): Weighted percentage accuracy

Interpretation

• Higher Values: Indicate better model performance
• Consistent Performance: Model works well across different time periods
• Variable Performance: Model accuracy varies significantly

Model Comparison Table

The comparison table shows:

• Model Names: Each algorithm tested
• Horizon Columns: Performance at different time horizons (T1, T2, etc.)
• Accuracy Values: Percentage accuracy for each model/horizon combination
• Best Performance: Highlighted in green for easy identification

Overall Performance Summary

• Best Model: Identifies the top-performing algorithm
• Average Accuracy: Overall performance across all horizons
• Recommendation: Suggests the best model for your data

Understanding Backtest Results

Time Series Analysis

Historical vs Predicted Data

• Historical Line: Actual demand values (blue)
• Predicted Lines: Model predictions for each window
• Reference Areas: Highlight different forecast windows
• Item Selection: Analyze specific items or total demand

Chart Features

• Multiple Models: Compare several algorithms simultaneously
• Window Visualization: See how predictions change over time
• Model Toggle: Show/hide individual models

Advanced Features

Multi-Model Comparison

• Side-by-Side Analysis: Compare all selected models simultaneously
• Performance Ranking: See which models perform best overall
• Horizon Analysis: Understand how accuracy changes over time
• Item-Specific Performance: Identify best models for specific products

Detailed Metrics

• Time Series Metrics: Performance at each time period
• Aggregated Metrics: Overall performance across all windows
• Mean Metrics: Average performance for each model
• Statistical Significance: Confidence in performance differences

Export Capabilities

• CSV Download: Export detailed results for external analysis
• Performance Summary: Key metrics and recommendations
• Model Comparison: Side-by-side performance data
• Visualization Data: Chart data for custom analysis

Troubleshooting

Common Issues

Backtest Creation Failures

• Insufficient Data: Ensure enough historical data for all windows
• Invalid Parameters: Check that dates and horizons are reasonable
• Model Selection: Verify at least one model is selected
• System Resources: Large backtests may require more processing time

Poor Performance Results

• Data Quality: Review and clean historical data
• Model Selection: Try different algorithms for your data type
• Parameter Tuning: Adjust backtest configuration
• Business Context: Consider external factors affecting demand

Processing Delays

• Dataset Size: Larger datasets require more processing time
• Number of Windows: More windows increase processing time
• Model Complexity: Advanced models take longer to evaluate
• System Load: High system usage may slow processing

Performance Optimization

1. Reasonable Windows: Use 5-20 windows for most use cases
2. Efficient Models: Start with simpler models for quick results
3. Data Sampling: Use representative subsets for large datasets
4. Parallel Processing: Take advantage of system resources

Integration with Forecasting

From Backtest to Forecast

1. Model Selection: Choose the best performing models from backtests
2. Parameter Tuning: Use backtest insights to optimize forecast settings
3. Confidence Building: Gain confidence in forecast accuracy
4. Business Planning: Make informed decisions about forecasting strategy

Continuous Validation

• Regular Backtesting: Test models periodically as data patterns change
• Performance Monitoring: Track forecast accuracy over time
• Model Updates: Incorporate new algorithms and improvements
• Business Feedback: Align models with business requirements

---

Backtests are essential for building confidence in your forecasting approach. Regular testing and validation help ensure accurate predictions for business planning.