We are excited to announce the release of Hyperspace 0.3.0!

Notable new features / improvements:

• Mutable dataset support: Hyperspace v0.3.0 supports mutable dataset where users can append or delete the source data.
  • Hybrid scan: Prior to v0.3.0, any change in the original dataset content required a full refresh to make the index usable again, which could be a costly operation. With the Hybrid scan, the existing index can be utilized along with newly appended and/or deleted source files, without explicit refresh operation. Please check out the doc on Hybrid Scan for more detail.
  • Incremental refresh: v0.3.0 introduces a "incremental" mode to refresh indexes. In this mode, index files are created only for the newly appended source files; deleted source files are also handled by removing them from the existing index files. Please check out the doc on Incremental Refresh for more detail.
• Optimize index: The number of files for indexes can increase due to the incremental refreshes, degrading the performance. The new optimizeIndex API optimizes the existing indexes by merging index files to create an optimal number of files. Please check out the doc on Optimize Index for more detail.

Breaking changes:

In order to support features like Hybrid scan, incremental refresh, etc., the index metadata required unavoidable changes. Thus, the indexes generated with v0.2.0 are not compatible with v0.3.0 and need to be reconstructed.

Thank you for trying it out and we look forward to your feedback!

Credits
Andrew Fogarty, Apoorve Dave, Eunjin Song, Justin Breese, Pouria Pirzadeh, Rahul Potharaju, Tarun Rajput, Terry Kim, Veysi Ertekin, Yash Datta