Your VWAP report is slow because it insists on reliving every trade.
Every morning, the same ritual: someone clicks a button, and the system decides to re-experience the entire day’s trading as if it’s the first time. Trades that arrived yesterday get a full replay today. You pay the hardware bill for this particular hobby.
The excuses are predictable. “We need raw trades for accuracy.” Fine. Keep them. That does not mean every report must rescan them from scratch. “We can cache the final VWAP.” Until a late trade arrives or someone asks for a different time bucket. “Just average the hourly VWAPs.” No. Averaging averages is how you invent the wrong answer faster.
Here’s how it actually works:
- Bucket by instrument and the report’s time interval.
- Store sum of price times quantity.
- Store sum of quantity.
- Divide once when reading; never merge finished averages.
The trades come in. You add price times quantity to one running total and quantity to another. At read time, you divide. If quantity is zero, there is no VWAP to divide out. The raw tape stays for audit, corrections, and questions the summary can’t answer. The report does not.
In Accumulo, you can do this with Combiners. A summing Combiner merges values sharing the same key—like your instrument and time bucket. Custom iterators handle logic a Combiner can’t. Configure them for scans, minor compactions, and major compactions. Compaction is not an instant promise that every bucket is already one stored value. Yes, that distinction matters.
A time-series database or incrementally maintained summary table follows the same idea. The sums wait for the report, not the report for the sums. Corrections need a deliberate policy. Bucket definitions must match the report. Pre-aggregation is not permission to forget what a trade means.
More hardware is a costly way to avoid addition. The raw tape is evidence, not a daily endurance test.
Make the sums wait for the report, not the report wait for the sums.