Every signal looks good on its own. Gold rises when real yields fall, until the period when it does not. Miners outperform when the metal is in an uptrend, until the cost of energy eats the margin. A single indicator is a story with a good chart attached.
A scorecard is the attempt to be honest about that. It forces each observation into a fixed input, a defined transformation and a stated weight, so that the combination can be tested as a whole and the contribution of each part can be audited after the fact.
01The isolated-signal problem
There are three ways a single signal misleads. It can be right for the wrong reason, which means it will stop working when the reason changes. It can be right in one regime and useless in another, which means its full-sample statistics are an average of two different things. Or it can be redundant with a signal already in the model, which means adding it improves the story and not the portfolio.
A scorecard does not eliminate these problems. It makes them visible, because every input has to be declared and every weight has to be justified before the result is known.
02Anatomy of a scorecard
Each entry in a scorecard has five fields. If a candidate signal cannot fill all five, it is not ready to be tested.
- 01Raw inputThe exact series, its source and its publication lag. A macro series that is revised is used as it was first published, not as it looks today.
- 02TransformLevel, change, z-score, percentile rank, or a cross-sectional rank. The choice determines whether the signal is about states or about surprises.
- 03DirectionWhether a high value is bullish or bearish for the sleeve. Written as a signed coefficient, not as a sentence.
- 04WeightA fixed number, set from prior reasoning and capped so that no single signal dominates. Weights are reviewed, not continuously optimized.
- 05Failure modeThe specific condition under which this signal is expected to mislead. If this field is empty, the entry is not finished.
03Normalization decides everything
Most signal research fails here. A raw series with a trend, such as a price index or a yield, will produce a signal that is mostly a bet on the sample period. Transforming to a rolling z-score fixes the level problem and introduces a new one: the score now depends on the window, and a long window makes the signal slow while a short one makes it noisy.
The convention I use is a winsorized rolling z-score over a window long enough to contain more than one regime, with the extremes clipped at three standard deviations so that a single observation cannot dominate the composite. For cross-sectional signals, a percentile rank within the universe is preferred, because it is scale-free and does not assume normality.
Compute the same signal with two different windows. If the sign of its contribution flips, the signal is measuring the window rather than the market, and it is not ready for the composite.
04Weighting without overfitting
There are three defensible ways to set weights, and one indefensible one. Defensible: equal weights across signals within a theme, weights proportional to a prior belief about economic importance, or weights derived from a risk model that accounts for correlation. Indefensible: the weights that produced the best backtest.
I start equal-weighted because it is transparent and hard to overfit. If a signal cannot earn its place at an equal weight, a larger weight will not save it — it will only hide the problem in a better-looking curve. Once a scorecard is stable, weights can be tilted toward the signals with the clearest economic mechanism, never toward the ones with the best recent hit rate.
05A working inventory
The table below is the kind of inventory used for a multi-asset sleeve. It is intentionally short. A scorecard with fifteen inputs is usually a scorecard where five inputs are doing the work and ten are providing cover.
| Input | Transform | Direction | Weight | Known failure mode |
|---|---|---|---|---|
| Price vs 200-day average | Percentile rank | Positive | 0.30 | Choppy ranges generate repeated false starts |
| Slope of the average | Rolling z-score | Positive | 0.15 | Lags turning points by construction |
| Real yield change | 3-month z-score | Negative | 0.25 | Correlation with metals breaks during liquidity events |
| Dollar index | 3-month z-score | Negative | 0.10 | Overlaps the real-yield signal; weight kept small |
| Relative strength vs complex | Cross-sectional rank | Positive | 0.20 | Momentum crashes after sharp reversals |
Weights sum to one by construction. The composite is then mapped to a bounded score, and the score is compared against an entry threshold that is higher than the exit threshold — the same hysteresis principle used in the risk layer.
06Stability before significance
Statistical significance is a weak defense against overfitting when the search space is large. Stability is a better one. Three tests are applied to every scorecard before it is allowed near capital:
- Subsample stability. The contribution of each input is computed on disjoint periods. A signal that only works in one half is a candidate for removal, regardless of the full-sample result.
- Parameter plateau. The score is recomputed across neighboring windows and thresholds. If performance falls off a cliff one step away, the setting is a coincidence, not a finding.
- Ablation. Each input is removed in turn. If removing a signal does not change the results meaningfully, it should be removed permanently — every extra input is a new way to be wrong.
None of these tests tells you the strategy will work. They tell you whether the result you are looking at is a property of the market or a property of your search.
07Governance
A scorecard is a model, and models drift. Every version is dated and archived with its weights, its data source versions and the reason for the change. No weight is edited without a written rationale and a re-run of the stability tests. When a scorecard is retired, it is kept — the record of what was tried is part of the research, and it is the only defense against re-running the same failed idea two years later.
That sounds bureaucratic for a single researcher. It is also the difference between a research process and a collection of charts.
For discussion only. Not investment advice, and not a representation of actual or expected performance.
Back to work