What does it do better than parquet?
The compression ratio quoted is lower than parquet but I expect higher to be better in this context
looks like a cool project, but id say keep working on it since there seems to be some confusion on <i>why</i> someone would want to use this: no benchmarks and overall pretty vibe-codey (which id personally be very hesitant to use in production)<p>another comment already mentioned comparison to vortex, which is the same compression ratio and same speeds as youre claiming - but your compression is half of parquet. and if speed is the main goal youre going for, python is an interesting choice. no hate, but def keep working on it, and would love to see more concrete benchmarks with various columnar store types
> The fastest, most compressed columnar format for big data<p>How large a dataset can it tackle? I work with Parquet files spanning 300million+ records (~800MB files) using DuckDB and it works within seconds.<p>I might be interested to see benchmarks against Parquet and Vortex. A DuckDB extension would be great as well.
A binary file format built from first principles for modern data systems. Parse 100MB 50x faster than JSON, with 50-70% better compression. Full language support (Python, Java, JavaScript, Go, C#, Ruby). Includes a VS Code extension for viewing .kore files. 3 years of production testing before open source release.