# Repeated 25% held-out-gene ridge benchmark

Eligible genes: 2160; seeds: 5; test fraction: 0.25.

Each outer split is stratified by perturbation cell-count quintile. Ridge alpha is selected using training genes only. Multiplicity correction covers all model-by-seed primary tests.

## Per-split summary

| model | seed | n_test_genes | mean_model_rmse | mean_zero_rmse | mean_train_mean_rmse | mean_model_cosine | mean_train_mean_cosine | mean_perturbation_specific_residual_cosine | relative_rmse_improvement_vs_zero | relative_rmse_improvement_vs_train_mean | raw_cosine_difference_vs_train_mean |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ridge_v1 | 20260712 | 540 | 0.11128 | 0.122424 | 0.11662 | 0.350793 | 0.350142 | 0.255013 | 0.0910253 | 0.0457842 | 0.000651046 |
| ridge_v1 | 20260713 | 540 | 0.114361 | 0.125713 | 0.119704 | 0.347704 | 0.350854 | 0.242491 | 0.0902989 | 0.0446318 | -0.00315017 |
| ridge_v1 | 20260714 | 540 | 0.11221 | 0.123006 | 0.117724 | 0.340734 | 0.341558 | 0.264303 | 0.0877712 | 0.0468384 | -0.000823845 |
| ridge_v1 | 20260715 | 540 | 0.109462 | 0.119611 | 0.11551 | 0.331329 | 0.332757 | 0.26689 | 0.0848548 | 0.0523585 | -0.00142846 |
| ridge_v1 | 20260716 | 540 | 0.112585 | 0.125205 | 0.11844 | 0.354335 | 0.358483 | 0.248336 | 0.100788 | 0.0494344 | -0.00414783 |
| ridge_v2 | 20260712 | 540 | 0.112445 | 0.122424 | 0.11662 | 0.35227 | 0.350142 | 0.241648 | 0.0815094 | 0.0357948 | 0.00212732 |
| ridge_v2 | 20260713 | 540 | 0.115037 | 0.125713 | 0.119704 | 0.351307 | 0.350854 | 0.224254 | 0.0849217 | 0.0389847 | 0.000452762 |
| ridge_v2 | 20260714 | 540 | 0.113055 | 0.123006 | 0.117724 | 0.342917 | 0.341558 | 0.229507 | 0.0809043 | 0.0396633 | 0.00135933 |
| ridge_v2 | 20260715 | 540 | 0.1101 | 0.119611 | 0.11551 | 0.336984 | 0.332757 | 0.246984 | 0.0795228 | 0.0468372 | 0.0042261 |
| ridge_v2 | 20260716 | 540 | 0.113411 | 0.125205 | 0.11844 | 0.360066 | 0.358483 | 0.230058 | 0.094191 | 0.0424609 | 0.00158278 |

## Gate reproducibility

| model | n_splits | rmse_pass_splits | raw_composite_pass_splits | perturbation_specific_pass_splits |
| --- | --- | --- | --- | --- |
| ridge_v1 | 5 | 5 | 0 | 5 |
| ridge_v2 | 5 | 5 | 0 | 5 |

## Feature v2 versus v1

| seed | comparison | n_genes | mean_difference | bootstrap_ci95_low | bootstrap_ci95_high | direction_if_v2_better | paired_randomization_p_two_sided | observed_direction | holm_p_feature_version_family |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 20260712 | v2_minus_v1_rmse | 540 | 0.00116497 | 0.000260731 | 0.00231707 | less | 0.0153992 | v1_better | 0.0615969 |
| 20260712 | v2_minus_v1_residual_cosine | 540 | -0.0133655 | -0.0260836 | -0.000759931 | greater | 0.0375981 | v1_better | 0.0751962 |
| 20260713 | v2_minus_v1_rmse | 540 | 0.000675978 | 0.000133615 | 0.00121619 | less | 0.0188991 | v1_better | 0.0615969 |
| 20260713 | v2_minus_v1_residual_cosine | 540 | -0.0182374 | -0.0308964 | -0.00584441 | greater | 0.0040998 | v1_better | 0.0327984 |
| 20260714 | v2_minus_v1_rmse | 540 | 0.000844678 | 0.000247136 | 0.00146302 | less | 0.00709965 | v1_better | 0.0454977 |
| 20260714 | v2_minus_v1_residual_cosine | 540 | -0.0347955 | -0.0489716 | -0.0217428 | greater | 9.9995e-05 | v1_better | 0.00099995 |
| 20260715 | v2_minus_v1_rmse | 540 | 0.000637765 | -5.18013e-06 | 0.00128662 | less | 0.0525974 | v1_better | 0.0751962 |
| 20260715 | v2_minus_v1_residual_cosine | 540 | -0.0199062 | -0.0329326 | -0.00747291 | greater | 0.0019999 | v1_better | 0.0179991 |
| 20260716 | v2_minus_v1_rmse | 540 | 0.000825948 | 0.000227199 | 0.00147637 | less | 0.0115994 | v1_better | 0.0579971 |
| 20260716 | v2_minus_v1_residual_cosine | 540 | -0.0182784 | -0.0316476 | -0.00535271 | greater | 0.00649968 | v1_better | 0.0454977 |

The perturbation-specific residual cosine subtracts the training-set common delta from both prediction and observation. It is a common-effect sensitivity analysis, not an implementation of the complete Systema benchmark.