aether_quant.evaluation¶
Evaluation statistics: rank IC with honest t-stats, Monte Carlo, Lean-exact Sharpe and PSR, deflated Sharpe and the cost-aware rank-book simulator.
Evaluation statistics: rank IC with honest t-stats, Monte Carlo, Lean-exact Sharpe and PSR, deflated Sharpe and the cost-aware rank-book simulator.
MarketPanel
dataclass
¶
Bars by calendar date (a NaN where the ticker has no bar that day). sessions are the equity sessions.
RankBookSimulationResult
dataclass
¶
One rank-book simulation: gross and net Sharpe and return, drawdown, turnover, cost drag and the daily series.
kill_switch_locked_date_list
class-attribute
instance-attribute
¶
kill_switch_locked_date_list: list = field(default_factory=list)
num_kill_switch_frozen_decisions
class-attribute
instance-attribute
¶
num_kill_switch_frozen_decisions: int = 0
num_kill_switch_locked_dates
class-attribute
instance-attribute
¶
num_kill_switch_locked_dates: int = 0
num_sleeve_guard_suspended_dates
class-attribute
instance-attribute
¶
num_sleeve_guard_suspended_dates: int = 0
per_date_component_returns
class-attribute
instance-attribute
¶
per_date_component_returns: dict = field(default_factory=dict)
per_date_net_return
class-attribute
instance-attribute
¶
per_date_net_return: list = field(default_factory=list)
rebalance_log
class-attribute
instance-attribute
¶
rebalance_log: list = field(default_factory=list)
apply_isotonic ¶
apply_isotonic(probability: ArrayLike, fit: dict[str, Any]) -> np.ndarray
Piecewise-linear interpolation through the fitted block means, flat outside the fitted range.
apply_temperature ¶
apply_temperature(probability: ArrayLike, temperature: float) -> np.ndarray
sigmoid(logit(p) / T): T > 1 softens an over-confident head, T < 1 sharpens it, T = 1 is the identity.
balanced_accuracy ¶
balanced_accuracy(probability: ArrayLike, outcome: ArrayLike, threshold: float = 0.5) -> float | None
Mean of the hit rates on up and down outcomes at threshold; None without both classes.
best_threshold_by_class ¶
best_threshold_by_class(probability: ArrayLike, outcome: ArrayLike, classes: ArrayLike, grid: Sequence[float] | None = None) -> dict[str, dict]
{class: {"threshold", "balanced_accuracy", "n"}}: the threshold on grid (default 0.30..0.70 step 0.05) maximising
balanced accuracy among that class's rows. A class lacking both outcomes, or with fewer than 20 rows, gets 0.5.
bet_starts ¶
bet_starts(positions_by_date: dict[str, dict[str, float]]) -> list[tuple[str, str, int]]
(date, ticker, side) each time a position leaves zero or flips sign (AFML 14.2's definition of a bet).
bootstrap_mean_pnl ¶
bootstrap_mean_pnl(pnls: ArrayLike, n: int = 10000, seed: int = 7, confidence: float = 0.95) -> dict[str, Any]
Mean P&L per trade with a percentile bootstrap interval; p_mean_le_0 = the share of resampled means <= 0.
brier_score ¶
brier_score(probability: ArrayLike, outcome: ArrayLike) -> float | None
Mean squared error of the probabilities against the 0/1 outcomes; None when empty.
calibration_report ¶
calibration_report(probability: ArrayLike, outcome: ArrayLike, classes: ArrayLike | None = None, bins: int = 10) -> dict[str, Any]
Everything the L2 decision needs for one head on one split: the raw measures, a reliability curve, both fitted maps with their cross-fitted log-loss, and the per-class thresholds.
classification_scores ¶
classification_scores(y_true: ArrayLike, y_pred: ArrayLike, probabilities: ArrayLike | None = None) -> dict[str, Any]
Accuracy, precision, recall, F1 of binary calls (1 = up / long), and the negative log-loss when probabilities of class 1 exist (higher is better; 0 is a perfect forecaster, -0.693 a coin flip).
coinbase_taker_bps ¶
coinbase_taker_bps(date: str) -> float
Taker fee of Lean's date-effective CoinbaseFeeModel for a market order on a USD pair, in bps, as its fills show.
compute_backtest_statistics ¶
compute_backtest_statistics(run: dict[str, Any], config: dict[str, Any], root: Path, *, run_name: str, book_history: list[dict] | None = None, trial_annual_sharpes: list[float] | None = None, offline_classification: dict[str, dict] | None = None, lean_basis: dict[str, Any] | None = None, attribution: dict[str, Any] | None = None) -> dict[str, Any]
Every section of the README's "Backtest Statistics" for one loaded run (backtest_audit.load_backtest_run).
lean_basis is lean_statistics.statistics_table output, attribution lean_attribution.attribute_run output; both
optional. Pure apart from reading the Lean data zips under root for closes and opens.
compute_for_run_dir ¶
compute_for_run_dir(run_dir: Path, config: dict[str, Any], root: Path) -> dict[str, Any]
aq evaluate --backtest-stats: load the run folder and its companions, then compute every section.
compute_participation_rate ¶
compute_participation_rate(*, trade_size_fraction: float, portfolio_value: float, avg_daily_dollar_volume: float) -> float
Fraction of the asset's ADV consumed by this trade (0 when degenerate).
compute_sqrt_impact_bps ¶
compute_sqrt_impact_bps(*, trade_size_fraction: float, daily_volatility: float, avg_daily_dollar_volume: float, portfolio_value: float, eta: float = 0.5) -> float
Computes temporary market impact in basis points.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
trade_size_fraction
|
float
|
absolute weight change (e.g. 0.05 = 5% of book). |
required |
daily_volatility
|
float
|
rolling daily return std (e.g. 0.02 = 2%). |
required |
avg_daily_dollar_volume
|
float
|
asset's ADV in dollars (e.g. 50_000_000). |
required |
portfolio_value
|
float
|
total book value in dollars. |
required |
eta
|
float
|
impact coefficient — higher for less liquid assets. Calibrated defaults: equity ~0.5, crypto ~1.0, bond ETF ~0.3. |
0.5
|
Returns:
| Type | Description |
|---|---|
float
|
Impact cost in basis points (always >= 0). Zero when any input is |
float
|
degenerate (zero size, zero ADV, zero vol). |
cross_fitted_log_loss ¶
cross_fitted_log_loss(probability: ArrayLike, outcome: ArrayLike, method: str) -> float | None
Honest validation log-loss of a calibration method: fit on the first half (time order) and score the second, then
swap, and average the two. method in {"raw", "temperature", "isotonic"}.
daily_returns ¶
daily_returns(equity: list[float]) -> list[float]
Period returns of an equity curve, skipping steps from a non-positive value.
default_fee_usd ¶
default_fee_usd(asset_class: str, notional: float, date: str, config: TargetWeightsConfig) -> float
The fee the target-weights simulator charges one fill: Coinbase taker on crypto, 0.2 bps (min $2) on forex, the equity terms otherwise.
deflated_sharpe_ratio ¶
deflated_sharpe_ratio(returns: ArrayLike, trial_annual_sharpes: list[float], periods_per_year: int = DAYS_PER_YEAR) -> dict[str, Any]
DSR (Bailey & López de Prado 2014): PSR against SR0, the best Sharpe expected from the number of configurations tried. Trial Sharpes are annual; they are moved to this series' period (÷ sqrt(periods_per_year)) first.
drawdown_episodes ¶
drawdown_episodes(dates: list[str], equity: list[float]) -> list[dict]
AFML 14.7 computeDD_TuW on an equity curve: one episode per high-watermark that is followed by a loss, with its
depth (fraction of the watermark), the watermark and recovery dates and the time under water in calendar days. The
last episode is open when the curve never regained its watermark.
engine_log_status ¶
engine_log_status(engine_log_text: str) -> dict[str, Any]
What a Lean engine log says about a run: whether teardown was reached, inference failures and a timed-out teardown.
excluded_alpha_sharpe ¶
excluded_alpha_sharpe(observed_t: float, years: float, confidence: float = 0.95) -> float
One-sided upper bound on the true alpha Sharpe, given the t observed: true values above this are rejected at confidence.
expected_calibration_error ¶
expected_calibration_error(probability: ArrayLike, outcome: ArrayLike, bins: int = 10) -> float | None
Sample-weighted mean |observed rate - mean probability| over the reliability bins.
expected_max_sharpe ¶
expected_max_sharpe(num_trials: int, trial_sharpe_variance: float) -> float | None
SR0 of the deflated Sharpe ratio: the expected maximum Sharpe of num_trials unskilled trials whose Sharpe ratios
have variance trial_sharpe_variance (same units as the returned value).
fit_isotonic ¶
fit_isotonic(probability: ArrayLike, outcome: ArrayLike) -> dict[str, Any] | None
Pool-adjacent-violators on outcomes sorted by probability: {"x": block means of p, "y": calibrated rates} (both non-decreasing). None for fewer than 10 usable rows or a single class.
fit_temperature ¶
fit_temperature(probability: ArrayLike, outcome: ArrayLike, low: float = 0.05, high: float = 20.0, iterations: int = 80) -> float | None
The temperature minimising log-loss: a golden-section search on log T (the loss is unimodal in it).
fundamental_law_ir ¶
fundamental_law_ir(ic: float, names: int, rebalances_per_year: float, transfer: float = 1.0) -> float
Grinold's IR = transfer * IC * sqrt(breadth), breadth = independent bets a year. Names aren't independent, so this is a ceiling.
hhi_concentration ¶
hhi_concentration(values: ArrayLike) -> float | None
AFML 14.6 getHHI: concentration of values (all one sign) normalised to [0, 1]; 0 = evenly spread, 1 = all in
one observation. None for two or fewer observations.
information_ratio ¶
information_ratio(returns: ArrayLike, benchmark_returns: ArrayLike, periods_per_year: int = DAYS_PER_YEAR) -> float | None
Annualised mean / std of the active return (strategy minus benchmark, same periods).
lean_book_calls ¶
lean_book_calls(book_history: Iterable[Mapping[str, Any]] | None, closes: Mapping[str, Mapping[str, float]], sorted_closes: Mapping[str, Sequence[str]], horizon: int = 20, source: str = LEAN_PROBABILITY_SOURCE) -> dict[str, Any]
The Lean run's book calls as aligned lists: side (1 = long) vs the sign of the ticker's own horizon-bar forward
move, plus the logged probability of "up" (source in the record's member_probabilities, V6.4.0) where there is one.
dates / tickers identify each call so an offline score can be matched to it.
lean_statistics ¶
lean_statistics(performance: Sequence[float], equity_dates: Iterable[date], *, trading_days_per_year: int, rate_changes: Sequence[tuple[date, float]]) -> dict[str, Any]
Sharpe, Sortino, PSR and their inputs exactly as Lean computes them; non-finite inputs give 0 like Lean's guards.
load_market_panel ¶
load_market_panel(root: Path, config: dict[str, Any], *, extra_equity: tuple[str, ...] = ('BIL',), data_dir: str = 'data') -> MarketPanel
Every universe asset plus the sweep ticker, from the Lean zips under root/data_dir.
load_trial_sharpes ¶
load_trial_sharpes(registry_path: Path) -> list[float]
Annual Sharpe ratios of every registered research trial (scripts/build_trial_registry.py).
log_loss ¶
log_loss(probability: ArrayLike, outcome: ArrayLike) -> float | None
Mean negative log-likelihood of the outcomes under the probabilities; None when empty.
max_drawdown ¶
max_drawdown(stats: dict[str, Any]) -> Any
The portfolio's maximum drawdown as the Lean report's statistics table prints it.
minimum_detectable_alpha_sharpe ¶
minimum_detectable_alpha_sharpe(years: float, t_bar: float = 3.0) -> float
The alpha Sharpe whose expected t just reaches t_bar over years.
monthly_counts ¶
monthly_counts(days: list[str], first: str, last: str) -> list[int]
Bets per calendar month from first to last, months without a bet included as zeros.
newey_west_t_stat ¶
newey_west_t_stat(values: ArrayLike, lags: int) -> float
HAC (Newey-West, Bartlett weights) t-statistic of the mean of a daily IC series. Consecutive daily ICs of an
h-day-horizon signal share most of their forward window, so the plain t-stat overstates significance; the
non-overlapping subsample is honest but throws away (h-1)/h of the dates (14 of 261 on the validation split).
lags ~ horizon - 1. Returns 0.0 when fewer than 2 finite values or no variance.
percentile ¶
percentile(values: ArrayLike, q: float) -> float | None
The q-th percentile of the finite values; None when there are none.
power_of_t_test ¶
power_of_t_test(alpha_sharpe: float, years: float, t_bar: float = 3.0) -> float
Chance that a true alpha Sharpe of alpha_sharpe clears t_bar over years.
power_table ¶
power_table(years_list: Sequence[float] = (3.3, 8.0, 15.8, 22.8), alpha_sharpes: Sequence[float] = (0.25, 0.5, 0.75, 1.0, 1.5), t_bar: float = 3.0) -> dict[str, Any]
The table ml/evaluation/v660/power.json holds, written before any score of the phase exists.
probabilistic_sharpe_ratio ¶
probabilistic_sharpe_ratio(returns: ArrayLike, benchmark_sharpe_per_period: float = 0.0) -> float | None
PSR (Bailey & López de Prado 2012): P(true Sharpe > benchmark) given the sample's length, skew and kurtosis. Both Sharpe ratios are per period (not annualised).
r_lean13 ¶
r_lean13(baseline: dict[str, Any], candidate: dict[str, Any], candidate_engine_log: str) -> dict[str, Any]
{"checks": {...}, "keep": bool, "values": {...}}; a value that cannot be read fails its clause.
r_lean14 ¶
r_lean14(baseline: dict[str, Any], candidate: dict[str, Any], candidate_engine_log: str, margin: float = MAX_SHARPE_LOSS) -> dict[str, Any]
R-LEAN13 with clause (a) turned into superiority: the strategy Sharpe must beat the baseline's by margin.
rank_ic_from_arrays ¶
rank_ic_from_arrays(predictions: ndarray, targets: ndarray, dates: ndarray, non_overlapping_stride: int = 1, min_names_per_date: int = 2) -> dict[str, Any]
Per-date Spearman rank correlation ("rank-IC") between a model's raw rank_5d/20d prediction score and the realized cross-sectional rank target (target_rank_5d/20d - already a per-date percentile rank of forward return, see train.py::build_cross_sectional_rank_targets()). This is the standard evaluation metric for a cross-sectional/long-short signal: a mean rank-IC of 0.02-0.05 with a real t-stat is a genuine, monetizable edge in this literature.
Spearman(A, B) = Pearson(rank(A), rank(B)); target_rank is already rank(realized return) rescaled to [0, 1] (a positive affine transform, which Pearson is invariant to), so only the model's own raw prediction needs an explicit per-date rank transform before a plain Pearson correlation against target_rank directly reproduces the true Spearman correlation between prediction and realized return.
non_overlapping_stride (e.g. 5 for the 5d horizon, 20 for the 20d
horizon) subsamples to every Nth unique date before computing -
consecutive daily rows share most of their forward-return window, so
the naive daily IC series is autocorrelated and its plain t-stat
overstates significance; the non-overlapping subsample gives a more
honest (if noisier, fewer-observation) significance check. Pass 1
(default) for the full, autocorrelated series.
min_names_per_date (V5.3.5, development/Problems.md #102) - default 2
reproduces this function's original behavior exactly (train.py's/
performance/rank_ic_monitor.py's existing callers are unaffected). A
date with exactly 2 names is mathematically FORCED to a Pearson
correlation of either +1 or -1 (two points always fit a line, tie
aside) - a legitimate computation, but statistically uninformative,
and it gets averaged into mean_ic with the exact same weight as a
date with 100 names. portfolio/rolling_ic_gate.py's offline
calibration/replay tools raise this well above 2 specifically because
a walk-forward rolling-IC distribution's early, thin-universe dates
were found (V5.3.5, real data) to produce spurious +-1 readings that
dominate the low end of the distribution.
Dates with fewer than min_names_per_date eligible (non-NaN target)
assets, or zero variance on either side (e.g. every asset tied),
contribute no IC value for that date (skipped, not a spurious 0.0 -
which would misrepresent "undefined" as "no correlation").
reliability_curve ¶
reliability_curve(probability: ArrayLike, outcome: ArrayLike, bins: int = 10) -> list[dict]
Equal-width bins on [0, 1]: {"low", "high", "n", "mean_probability", "observed_rate"} for each non-empty bin.
run_monte_carlo ¶
run_monte_carlo(per_date_net_returns: ArrayLike, *, n_runs: int = 1000, block_size: int = 20, seed: int = 42, method: str = 'block', n_plot_points: int = 500, periods_per_year: float = TRADING_DAYS_PER_YEAR) -> dict[str, Any]
Runs the bootstrap and returns a self-describing result dict:
{"config": {...},
"status": "OK" | "SKIPPED:
Equity curves are compounded from resampled daily returns and start at 1.0. The average curve is the element-wise MEAN over all runs (the red line), NOT the mean of final values.
sharpe_ratio ¶
sharpe_ratio(returns: ArrayLike, periods_per_year: int = DAYS_PER_YEAR) -> float | None
Annualised mean / sample std of per-period returns, no risk-free rate.
simulate_rank_book ¶
simulate_rank_book(frame: DataFrame, **kwargs: Any) -> RankBookSimulationResult
Public entry point - see _simulate_rank_book_core()'s docstring for the full algorithm. Discards held_days_by_ticker (only capacity_curve() needs it).
simulate_target_weights ¶
simulate_target_weights(weights_by_date: Mapping[str, Mapping[str, float]], panel: MarketPanel, config: TargetWeightsConfig, *, start: str, end: str, nav: float = 100000.0, sweep_config: dict[str, Any] | None = None, sweep_ticker: str = 'BIL', rf_dates: list[str] | None = None, rf_rates: list[float] | None = None, carry_fn: Callable[[str, str], float] | None = None, max_total_drawdown: float | None = None, max_drawdown_velocity: float | None = None, fee_fn: Callable[[str, float, str, TargetWeightsConfig], float] = default_fee_usd, lot_size: Mapping[str, float] | None = None) -> dict[str, Any]
One path over [start, end]. Returns {"nav", "strategy" (ex-sweep daily return), "strategy_excess" (that, less the risk-free return on the net capital invested), "total" (daily return), "gross", "fees", "financing", "turnover", "fills", "locked_on"}, the series indexed by calendar date.
"strategy" is what Lean reports. Ex-sweep, a long book pays nothing for the cash it ties up, which the sweep would have earned, so a long sleeve's Sharpe is raw and not excess unless it is scored on "strategy_excess". A forex position is self-financed and has no such cost.
A drawdown lock (on the strategy, ex sweep, as main.py reads it) closes everything at the next fills and stops trading.
strategy_sharpe ¶
strategy_sharpe(stats: dict[str, Any]) -> Any
The strategy's annualised Sharpe (no risk-free rate) as the Lean report's statistics table prints it.
summarize_metric_stability ¶
summarize_metric_stability(per_window_values: list[float], *, min_windows: int, max_sign_flip_fraction: float) -> dict[str, Any]
Cross-window stability summary (used by Phase 4's walk-forward evaluation and Phase 5's ablation) - mirrors train.py::assess_ranking_quality()'s era-sign-instability check, one level up: a window whose sign disagrees with the overall mean's sign is a "flip", same concept as an opposite-sign era. num_windows below min_windows or a sign-flip fraction above max_sign_flip_fraction both fail "stable".
verdict_for_run_dirs ¶
verdict_for_run_dirs(baseline_dir: Path, candidate_dir: Path, config: dict[str, Any], root: Path, rule: str = 'non_inferiority', margin: float = MAX_SHARPE_LOSS) -> dict[str, Any]
Statistics of both run folders (backtest_statistics.compute_for_run_dir, each with its own frozen config) and the
candidate's engine log.