wire / wire-format / orc-format

Apache ORC (Optimized Row Columnar) format

ORC is a columnar storage format for Hadoop/Spark. File structure: stripes (row groups), each with index, data, and footer; file footer (schema, statistics, stripe locations); postscript (compression codec, footer length, magic 'ORC'). Magic: 'ORC' (0x4F 0x52 0x43) at the end of the file.

wire-format kind wire-format status standard verification verified tier B wire-format@1

aka: Apache ORC (Optimized Row Columnar) format

wire format

spec: Apache ORC specification

ORC is a columnar storage format for Hadoop/Spark. File structure: stripes (row groups), each with index, data, and footer; file footer (schema, statistics, stripe locations); postscript (compression codec, footer length, magic 'ORC'). Magic: 'ORC' (0x4F 0x52 0x43) at the end of the file.

fieldsizemeaning
StripevariableIndex stream (row group offsets) + data stream (column data) + footer (column statistics).
File footervariableHeader (1 byte, version), content length, stripe statistics, row index strides, schema (ProtoBuf serialized), and column statistics.
PostscriptvariableCompression codec (0 none, 1 zlib, 2 snappy, 3 lzo, 4 lz4, 5 zstd), footer length (variable-length), magic 'ORC' (3 bytes).
Magic3 bytes'ORC' (0x4F 0x52 0x43) at the very end of the file.

example:

4f 52 43

ORC magic at end of file


provenance

agent: curl -H 'accept: application/json' wire.phall.io/wire-format/orc-format or /wire-format/orc-format.json