OSRS Market Tools (MMG)

MMG began with a practical market question: which Old School RuneScape money-making methods and item flips are worth considering at current prices? The browser tools calculate margins and compare methods from public Grand Exchange data. The research asks a harder question, whether predicting future prices adds enough value to justify a trade.

The evaluated trading strategies lost money after costs, even where historical studies reported gains over prediction baselines. A label audit found that many returns called four-hour returns covered a different elapsed time. Repairing that target redirected the research toward risk context, but did not finish the validity work: a September source audit found future-dependent features and incorrect per-item freshness. The calculator remains independently useful, while model-driven rankings still need revalidation.

The Grand Exchange is a shared marketplace with changing prices, limited liquidity, and a tax on completed sales. In the research cost model, the sale tax is 2%, rounded down and capped at 5,000,000 gp per item. A profitable round trip must also cover the bid-ask spread and slippage. Predicting the direction of the midpoint is therefore only part of the decision.

This separates the calculator from the forecaster. The calculator applies known arithmetic to observed prices, while the forecaster estimates an uncertain future outcome. Even a well-calibrated probability of a rise cannot promise that an order fills at the displayed price or that the rise exceeds the full cost of execution. I kept those questions separate in both evaluation and the public interface.

Five static pages cover money-making methods, GE flipping, trends, bankroll planning, and data/settings. The core calculations run in the visitor’s browser against public OSRS Wiki prices. That keeps the useful arithmetic independent of model serving or an account. An optional local-first FastAPI service adds collected history, a SQLite archive, feature and label builders, and research routes.

Selected research estimates and forecasts can load from published JSON snapshots without the local service. Method and GE history, planner profile sync, and Hiscores import have additional helper requirements. The frontend attempts loopback access when browser policy allows it and blocks remote plain-HTTP helpers from the HTTPS site. Browser policy, CORS, and reachability still determine whether that connection works.

A separate Postgres-backed accounts service implements registration, login, token refresh, browser-state sync, and subscription state. Live research routes have subscription checks, while static snapshots are ungated. Checkout remains a development test flow. This architecture gives the static product a usable core, but also creates separate questions about market freshness, snapshot age, helper availability, and account readiness.

The collector records five-minute, hourly, and daily price bars into SQLite. Keyed inserts avoid duplicate rows, and startup and hourly gap-repair attempts help recover missing history. The price archive is retained separately from operational audit records. These mechanisms support a growing history without proving continuous collection or complete recovery from every outage.

Completed days can be compacted into Zstandard Parquet through DuckDB, and research exports freeze an immutable dataset for each experiment. A run then reads the same data throughout its evaluation instead of a database changing underneath it. Invalid market observations are excluded rather than repaired with guessed prices.

The intraday features describe returns, spreads, trend, volume, cyclical time, price level, and liquidity. Direction models estimate the chance of a threshold rise, while quantile models estimate the range of possible returns. Calibration and residual-based widening make uncertainty part of the output. Walk-forward splits train on the past and test later periods, with an embargo at least as long as the prediction horizon.

The daily feasibility study used 1.55 million observations across 4,420 items from a public price snapshot. Direction accuracy reached 57.43% against a 54.86% majority-class baseline. The tax-hurdle classifier also found ranking signal, with 0.727 ROC-AUC and 45.7% precision among selected calls against a 17.2% base rate. Acting on those selections still lost 1,529,993 gp across 1,629 trades.

The original intraday study reported 74.0% direction accuracy against a 68.5% majority baseline and Brier skill of +0.19. A separate mechanical EMA crossover backtest on the intraday collection lost 75,099,719 gp across 47,203 trades. That rule was not a direct deployment of the classifier, so its loss does not by itself measure the classifier’s trading value. Alongside the daily proxy and later model-signal backtest, it showed why prediction metrics could not serve as the product’s trading acceptance gate.

The detailed experiments below retain the original setups and results. The intraday numbers also need the label correction that follows: their row-count horizon cannot be treated as an exact four-hour measurement.

The original intraday label builder shifted a price series forward by 48 rows. With uninterrupted five-minute bars that spans four hours. With missing bars, it reaches further into the future. The model was being asked a variable-horizon question under a fixed-horizon name.

The continuity audit of the top 100 items scanned 839,687 rows and found 22,830 interior gaps. Of 834,887 row-shift labels, only 580,076 landed exactly 14,400 seconds later, a correct share of 69.48%. The repair joined targets by the exact timestamp offset and restarted rolling features at gaps. The resulting research dataset contains 802,443 rows across 789 items. That broader dataset and the continuity audit cover different item groups, so their row counts are not a before-and-after size comparison.

This changed how the earlier evidence could be read. A walk-forward split does not repair a mislabeled horizon, and the old and corrected model scores are not a controlled before-and-after comparison of algorithm quality. The rebuild establishes what each target means before asking whether a model predicts it.

The historical reports motivated a shift toward risk and ranking, with distinct measurements. In the daily study, next-day volatility MAE improved by 30.4% over a rolling baseline, and next-day spread MAE improved by 6.2% over today’s spread, with 0.582 rank correlation. The corrected four-hour study reported 12.40% volatility MAE skill and 15.29% future-spread MAE skill against calibration-selected naive baselines. These belong to different horizons and datasets, and the later source audit means they remain exploratory evidence rather than validation of the displayed rankings.

The corrected study still did not validate trading recommendations. Only about 2.2% of its examples cleared the conservative tax, spread, and slippage hurdle, and its combined execution proxy remained negative. The product direction became risk context and margin-compression estimates, with trade signals parked until conservative, liquidity-aware backtests across multiple regimes and paper trading support them.

Abstention is part of that boundary. In one historical scoring run, 725 predictions narrowed to 18 candidate signals and all were suppressed by freshness checks. That successful check in the legacy scoring path does not validate the newer serving path, whose item-level freshness was later found to be wrong. A model confidence score cannot make an old observation current.

The September 18 check opened all five public pages, and the analyzer and scanner displayed live-price data. On September 20, the published research manifest was still dated July 11. The accounts health endpoint responded, but database readiness timed out. These observations describe different parts of the product, so a successful page load cannot stand in for fresh research or a functioning account.

The later browser check found a more specific mismatch. Celastrus bark showed current prices beside the research labels Low risk and upward margin, even though the record was marked gated and the published snapshot was roughly 72 days old. Its tooltip still reported an age of 140 hours. No expired forecast cone was observed against the current price chart, but stale risk and direction labels were visible.

Tutorial dismissal worked in the September 20 check. Hosted login, session refresh, state sync, and planner editing remain unverified. The dated Trends capture below records the earlier helper-unavailable state and published-snapshot age.

The exact-horizon repair fixed the target, but future observations still affected the inputs. The builder joined future bars before calculating cross-item ranks. Removing one item’s future bar could therefore change another item’s present-time feature even when their observed histories stayed identical. An isolated before-and-after reproduction makes that dependency visible below.

The September 20 local repair builds current features and their comparison groups first. Training labels are attached afterward at the exact four-hour timestamp, while inference keeps a separate unlabelled artifact. Rolling features use the full available history before training rows are selected. Missing targets and unknown tax metadata remain unknown. Extreme future moves stay in the data with an explicit flag.

That changes the model’s input contract. The loader refuses historical weights and requires matching feature order, configuration, schema, and artifact hashes. Compatible datasets and models have not been rebuilt or trained. The synthetic checks establish specific software invariants, and the old performance scores cannot be carried forward to these new inputs. The exact displayed risk-adjusted ranking still needs its own evaluation.

The serving boundary asks whether inputs are valid and whether they are still current. Each daily and intraday estimate has its own source interval, issue time, target, and expiry. Publishing an old snapshot again cannot renew it. Expired records lose their current rank, risk label, direction, and forecast, and cannot change the comparison group for a current item’s relative risk. Earlier browser fixtures showed estimates disappearing at expiry while historical prices and deterministic calculations stayed usable.

The new intraday loader is deliberately strict about missing inputs. If any requested item’s latest row is ineligible or nonfinite, the whole requested batch is unavailable. It does not substitute an older row or invent an imputation rule. Because observed receipt times are absent, availability still uses explicit interval-end and daily-date-end assumptions. Later metadata snapshots cannot fill earlier gaps. These rules sacrifice availability rather than silently treating an unknown input as known.

The component evaluator checks the training-data contract before fitting and requires a new output file, so it cannot silently replace a historical result. Completed outputs bind data, configuration, source, and dependency identities. Synthetic checks verified those safeguards with fitting intercepted.

A separate retained-record evaluator now reuses the ranking function used by serving. It selects the top candidates before reading their later outcomes, and compares spread ordering on the same eligible items. Missing outcomes do not trigger replacement selections. This tests the ordering within the eligibility rules, while leaving the value of those rules as a separate question. The synthetic example below shows why evaluating probability alone would miss the displayed decision.

These repairs remain local source checkpoints. The public MMG application has not received them. The downloadable records distinguish synthetic backend checks, earlier browser fixtures with mocked services, and dated public observations. Real-data rebuilding, compatible model training, real evaluation of the displayed ranking, and hosted acceptance remain separate work. Historical numerical results retain their unresolved bias, and no model-quality improvement is claimed.