A better price prediction can still be a losing trade

The daily price classifier was right 57.43% of the time, against a 54.86% majority-class baseline. The tax-and-slippage-aware daily holding proxy lost 1,529,993 gp across 1,629 selected trades. Both results came from MMG's daily feasibility study. Keeping only the first would have made a weak trading product look like a promising one.

MMG is an Old School RuneScape market-analysis project. Its prices are measured in game currency, but the engineering question is familiar: when does a prediction justify an action? A model can predict whether a price rises and still be unable to identify trades whose gains cover entry, exit, and market costs. The study forced that distinction into the product decision.

A direction target turns a future price change into a yes-or-no label. A tiny increase and a large increase can both count as correct. Accuracy also treats a small wrong call and a catastrophic wrong call alike. A trading rule pays for the sizes of those outcomes, the prices at which it can transact, and the costs along the way.

The daily study used time-based evaluation with separation between training and testing. Its direction accuracy beat the majority baseline across all three reported walk-forward regimes, with lifts between 2.51 and 2.90 percentage points. Those reported lifts describe performance on the study’s historical time splits. They do not establish that every positive prediction is worth acting on.

The dataset was a historical public snapshot. The resulting daily holding proxy is also a simulation of trades under assumptions, not a ledger of filled live orders. That qualification belongs beside the loss as much as it belongs beside a hypothetical gain. The experiment measures the proposed rule on that proxy.

The study also trained a classifier to predict whether the next daily midpoint would rise by more than 2.1%, an approximate tax hurdle. That target measures a price move, not net profit after execution costs. It reached 0.727 ROC-AUC, and its selected calls had about 45.7% precision against a 17.2% base rate. The selection enriched the positive class. That result answers a ranking question: were hurdle-clearing labels more concentrated among the selected examples?

It leaves several economic questions open. How large were the gains and losses? Were both sides of the trade available at the assumed prices? What did tax and slippage remove? What capital was tied up at the same time? A classification metric can be useful while leaving those terms unresolved.

The holding proxy supplies a direct warning. Its selected trades had a 33.7% win rate and a mean trade ROI of -3.06%. The aggregate currency loss is a sum over that proxy, while mean trade ROI averages individual trade returns. They are different summaries, neither a measured portfolio return nor a promise about future fills.

The negative trading result did not make every forecasting task useless. In the same daily study, next-day volatility prediction improved mean absolute error by 30.4% over the rolling-volatility baseline. Next-day spread forecasting improved it by 6.2% over using today's spread. Those tasks ask about risk and transaction conditions rather than guaranteeing a profitable purchase.

That distinction changed what the research could support. A warning that an estimate is sensitive to volatile prices needs evidence of risk forecasting. A live buy recommendation needs substantially more, including execution and return evidence. Treating both features as generic ML would hide the different standards they must meet.

The decision from this study was to withhold profitability claims and automated trade recommendations while retaining the narrower risk and ranking direction for further work. These are historical exploratory daily results, separate from the later four-hour studies, and not current product validation. A September source audit subsequently found future-dependent features and serving defects in the later intraday pipeline. That audit does not quantify a correction to these daily scores, but it reinforces the need to validate each pipeline separately.

I would make the same separation in another prediction product: write down the user action, account for the consequences of being wrong, and evaluate that action directly before promoting a model score into a recommendation.