What Is Signal Accuracy and Why It Matters?
Signal accuracy is a historical statistic, not a guarantee.
Signal accuracy is not a promise. It is a statistical property, measured over a defined historical period and dataset, that describes how often a quantitative signal has been directionally correct in the past. Understanding this distinction precisely is the foundation for using signal-based analysis intelligently rather than either dismissing it or treating it as certainty it cannot provide.
In a quantitative investment context, signal accuracy refers to the proportion of signals that were followed by the predicted directional movement over a specified holding period and on a specified dataset. A system that reported 70% directional accuracy across a three-month out-of-sample validation dataset is making a historical statistical claim. It is not claiming that the next signal will be correct with 70% probability in the way that a coin is 50% likely to land heads. It is claiming that, over the range of conditions in that dataset, approximately seven out of ten signals were subsequently confirmed by price behaviour.
This distinction matters because the next market environment may differ in ways that affect the signal's accuracy. Understanding what drives that accuracy, and where it is likely to be maintained versus where it is likely to degrade, is the more important question.
In-sample versus out-of-sample accuracy: the only honest test
The most important distinction in evaluating any quantitative signal's accuracy is whether the measurement was taken in-sample or out-of-sample.
In-sample accuracy is measured on the same data the model was trained on. A complex model with many parameters can achieve very high in-sample accuracy simply by memorising the training data rather than identifying genuinely predictive patterns. This is overfitting, and in-sample accuracy provides no information about how the model will perform on data it has not seen.
Out-of-sample accuracy is measured on data that was genuinely held back from the model during training and used only for evaluation after the model was finalised. This is the credible test. A model that achieves 68% out-of-sample directional accuracy on a dataset spanning multiple market regimes has demonstrated a statistically meaningful predictive capability. A model that achieves 90% in-sample accuracy with unknown out-of-sample performance has demonstrated nothing about its production utility.
Walk-forward analysis extends this logic: the model is periodically retrained on an expanding window of historical data and evaluated on the next period, simulating how the system would actually have performed through time rather than treating the entire historical period as a single dataset. This approach provides the most realistic estimate of prospective performance.
Regime Sensitivity and why accuracy varies across market conditions
A signal's accuracy is not constant across all market conditions. It varies with the Market Regime, and understanding that variation is as important as the aggregate accuracy figure.
Trend-following signals, which generate positive classifications when instruments exhibit sustained directional movement, perform well in trending regimes by definition. In mean-reverting regimes, the same signals will tend to be wrong more often than right, because the market is doing precisely the opposite of what they are designed to capture. A 70% aggregate accuracy figure for a trend signal that was measured across a data period with predominantly trending conditions may overstate what to expect in a period with more mixed regime characteristics.
This is Regime Sensitivity in its most practical form: the signal's reliability varies systematically across regime conditions. A well-designed quantitative system does not report a single accuracy figure. It reports accuracy conditional on regime, and scales the Signal Confidence Score accordingly. When the current Market Regime is less favourable to the signal type, the confidence score should reflect that degraded expectation.
The Volatility-Adjusted Signal and calibrated confidence
A related consideration is the relationship between current volatility and signal reliability. In high-volatility environments, price movements are larger but also more random, in the sense that the ratio of signal to noise in price data typically decreases. A directional signal issued during a period of elevated volatility should carry different confidence weighting from the same signal issued in a stable, trending environment.
The Volatility-Adjusted Signal is a Trend Signal calibrated against the current volatility environment of the instrument, so that the Signal Confidence Score reflects not just the historical directional accuracy of the model but also the current conditions under which it is operating. During high-volatility regimes, confidence scores are adjusted downward to reflect the reduced signal-to-noise ratio. During stable trending conditions, confidence scores reflect the more favourable predictive environment.
What signal accuracy does not tell you
Signal accuracy, however carefully measured, does not tell you the magnitude of the correct signals versus the magnitude of the incorrect ones. A system with 60% accuracy could generate a strong positive expected return if its correct signals tend to precede larger moves than its incorrect signals, or a negative return if the reverse is true. Directional accuracy is one dimension of signal quality. Risk-adjusted performance over a defined period, expressed in terms that acknowledge the probabilistic nature of all historical evidence, is the more complete picture.
The intellectual honesty required here is uncomfortable but necessary. A well-calibrated signal system built on rigorous out-of-sample methodology, with appropriate regime and volatility adjustment, represents a genuine informational contribution. It is not a certainty machine. It is a systematic process for generating probabilistic assessments that are more reliable, on the historical evidence, than unaided human judgement under the conditions the model has been validated on. That is a meaningful and achievable claim. It should not be overstated into something it is not.
Key Terms:
Signal Accuracy: The proportion of signals from a quantitative system that were followed by the predicted directional price movement, measured over a defined historical period and dataset. A statistical property of historical performance, not a guarantee of future correctness.
Out-of-Sample Accuracy: Signal accuracy measured on data that was genuinely held back from the model during training. The credible test of whether a quantitative model has learned generalisable patterns rather than memorised historical noise.
Volatility-Adjusted Signal: A Trend Signal whose confidence score has been calibrated against the current volatility environment of the instrument. Reflects the reduced signal-to-noise ratio in high-volatility regimes through appropriately adjusted confidence weighting.
Regime Sensitivity: The degree to which a signal's directional accuracy varies across different Market Regimes. A signal type may perform significantly better in trending conditions than in mean-reverting ones, and accuracy estimates should be regime-conditional to be meaningful.
Overfitting: The failure mode in model training where the system achieves high in-sample accuracy by learning the specific noise of the training data, producing apparent performance that collapses on out-of-sample evaluation.
