Executor Roadmap

Factor into two separate issues: handling positive deltas (within a tick and across ticks), and negative deltas (across ticks only).

Positive delta handling

We want to get from where we are to a fully push-based delta-handling mechanism. We can take this in stages

Phase 1: Improve current engine to do selective Rule triggering.

In this stage we avoid "pulling" from rules where all deltas are empty.

1. Extend depanalysis.rb to map from collection name to stratum/rule pair.
2. Upon delta to collection, push those stratum/rule pairs onto a priority queue by stratum #.
3. Change stratum eval to only evaluate rules on priority queue, and to push results onto the queue. Make sure tick_deltas works properly.

Phase 2: Implement selective delta handling at the BudCollection level

In this stage we only "pull" along branches of joins that have work to do.

1. Simple for base collections -- tick_delta and do_insert(o, @delta)
2. For joins, need to fiddle the arguments/mode on each appropriately.
3. Group/Argagg should behave as usual

Phase 3: Delta-rewrite rules into a dataflow

This is the full rewrite. I believe the right way to design this is a push-based eddy-style approach with an extensible scheduler mechanism. It will be easier to get this right once we know how to handle state across deltas -- i.e. once we've done the previous two steps.

Cross-tick delta handler

Reduce cross-tick re-evaluation via materialization

1. Modify expensive collection methods to keep their within-tick state across ticks. Includes joins (i.e. pairs/matches/lefts/rights/flatten/etc), group/argagg, etc.
2. Do naive invalidation of that state (e.g. empty Join cache on a deletion, empty group/argagg cache on any change, etc.)
3. Think about extensibility interfaces for subsequent improvements to invalidation -- e.g. consider how to support "differential" join logic to handle deletion in joins.