Benchmarks

Benchmark 001 / rung (a) / first internal diagnostic

Eigna TLM: a forecasting model fine-tuned on proprietary trading data

First internal diagnostic: 243 held-out days, BTC and ETH hourly, 24-hour forecasts.

CRPS vs base Kronos

18.1% lower

Paired test p 0.006, significant

MAE vs base Kronos

13.3% lower

Paired test p 0.017, significant

vs last value and GARCH(1,1)

About equal

CRPS gaps −3.5 and −2.0 bp, both 95% intervals include zero

Video / Video models / not run yet

Video benchmarks have not run yet

A planned Eigna video expert, to be compared with open video generation models on the same harness and ladder.

The benchmark ladder

Each rung adds one piece, base side against Eigna side

Every model type climbs the same six rungs, scored by one deterministic harness on a rented H200 GPU with pinned inputs. A rung shows a mean and a 95% interval only after it has run, N times per configuration.

Video ladder 6 rungs, none run yet

  1. Rung a: Base model vs Eigna expert

    Base video modelvsPlanned Eigna Video

    The model alone: the public base model against the planned Eigna expert, on the same pinned test set.

    N = 0 runsPlanned on: rented H200

    Not run yet

  2. Rung b: Base + tools vs expert + tools

    Base video model + toolsvsPlanned Eigna Video + tools

    Both sides get the same tools, and we compare the outcomes on the same test set.

    N = 0 runsPlanned on: rented H200

    Not run yet

  3. Rung c: Decision model vs tuned decision model

    Decision modelvsTuned decision model

    A model that decides whether to act, how much, or when to ask a person, before and after we tune it.

    N = 0 runsPlanned on: rented H200

    Not run yet

  4. Rung d: Base + decision vs expert + tuned decision

    Base video model + decision modelvsPlanned Eigna Video + tuned decision model

    The expert and the decision model together: the base pair against the tuned pair.

    N = 0 runsPlanned on: rented H200

    Not run yet

  5. Rung e: Base + decision + tools vs expert + tuned decision + tools

    Base video model + decision model + toolsvsPlanned Eigna Video + tuned decision model + tools

    The whole chain, from model to decision to action.

    N = 0 runsPlanned on: rented H200

    Not run yet

  6. Rung f: Rung (e) plus auto-learning

    Same as rung (e)vsRung (e), retrained on new data

    The whole chain again, retrained when new data arrives, to show what each retrain adds.

    N = 0 runsPlanned on: rented H200

    Not run yet

Public reference

Public models we plan to compare against

Each reference will run on the same harness as the planned Eigna expert. Names only for now: no numbers until our own runs finish and each licence is checked.

Hear when the first Video rung runs.

Join the waitlist

How it was made and tested

One sealed checkpoint, scored on days it never saw

We started from the public Kronos model, fine-tuned it on proprietary trading data, sealed the result, and only then opened the held-out days. The bench below runs that order.

Fig 1 The proving bench
Step 1 of 5 or hover, tap and use arrow keys

    Comparison

    Five forecasters, the same 243 days

    Every model forecast BTC and ETH 24 hours ahead from the same hourly history. Lower CRPS and MAE are better. Errors are a share of the price at forecast time.

    Scores for each forecaster over 243 held-out days
    Forecaster CRPS MAE 10 to 90 band coverage target 80% Direction accuracy Rank IC

    Last value, GARCH(1,1) and EWMA forecast no move, so direction accuracy and rank IC do not apply to them. Brackets show 95% intervals.

    Paired tests, day by day

    How does the TLM error compare?

    Mean daily difference in error, comparator minus Eigna TLM, in basis points of price, with 95% intervals. Right of zero means Eigna TLM had lower error; left means higher. The short upright mark is the 1% non-inferiority margin.

    Calibration

    Do the 10 to 90 bands hold 80% of outcomes?

    Share of real prices inside the forecast band, for each hour ahead. Fine-tuning moved coverage up, but both models stay below the 80% target: the bands are still too narrow.

    Eigna TLM Base Kronos Target 80%

    Paired tests

      Run record

      The harness is deterministic: the same pinned data, windows, scoring code, sample counts and seeds. The model is not. Inputs replay byte for byte; scores vary by run, so more seeds are coming.

      Public reference

      Two public suites, planned after the next ladder runs

      Next on the rented H200: rung (a) to N = 4 runs, then rungs (b) and (c). After those, we plan to run Eigna TLM and base Kronos on two public forecasting suites. Each table stays empty until our own run finishes and the licence check for the published rows is done.

      fev-bench

      Not run yet

      100 forecasting tasks, 22 of them hourly. Skill score and win rate against Seasonal Naive, with bootstrap intervals.

      fev-bench, not run yet
      Forecaster SQL skill MASE skill Leakage
      Eigna TLM Not run yet
      Base Kronos Not run yet

      Chronos Benchmark II

      Not run yet

      Zero-shot tasks, among them exchange rates and economic series. WQL and MASE relative to Seasonal Naive, geometric mean.

      Chronos Benchmark II, not run yet
      Forecaster WQL, relative MASE, relative
      Eigna TLM Not run yet
      Base Kronos Not run yet

      Not comparable with the hourly crypto scorecard above: different data, frequency, horizon and metrics. We never average across tables or give one overall score. Published rows join each table with attribution (name, link, licence and commit) after our own run.

      Public forecasters we plan to add

        Live benchmark

        Coming soon

        Once live, Eigna TLM will forecast the next 24 hours of BTC on each venue every hour. A fixed rule will turn each forecast into BUY, SELL or HOLD, and a paper account will track the running result with fees. No real orders.

        • Paper only
        • Not trading advice

        Eigna partner exchange (development)

        BTC-USD-PERP

        Paper only
        Price
        Waiting for data
        Forecast band, 24 hours
        Action
        No signal yet
        Paper PnL
        Not started

        Coinbase

        BTC-USD

        Paper only
        Price
        Waiting for data
        Forecast band, 24 hours
        Action
        No signal yet
        Paper PnL
        Not started

        Kraken

        BTC/USD

        Paper only
        Price
        Waiting for data
        Forecast band, 24 hours
        Action
        No signal yet
        Paper PnL
        Not started

        Method

        How to read these numbers

        CRPS

        The continuous ranked probability score compares the whole forecast distribution with what happened. It rewards forecasts that are both close and honest about their spread. Lower is better.

        MAE

        The mean absolute error of the central forecast after 24 hours, as a share of the price. Lower is better.

        Coverage

        Each forecast gives a band from its 10th to its 90th percentile. If the bands are right, about 80% of real prices land inside them. Less than 80% means the model is too sure of itself.

        Direction and rank IC

        Direction accuracy is how often the forecast called up or down correctly. Rank IC is the rank correlation between forecast and realised moves. Both are secondary.

        The test

        The model was sealed before scoring. It then forecast 243 whole UTC days it never saw in training, for BTC and ETH, from hourly history, 24 hours ahead. Every forecaster was scored on the same days.

        Significance

        We compare models day by day on the same days and resample whole blocks of days, 10,000 times, so that runs of calm or volatile days do not fool the test. A result counts at the 5% level.

        This is a first internal diagnostic. It did not meet our own acceptance rules, and it covers two assets only. Results on new data may differ. Nothing here is trading advice.