miamifere.blogg.se

If not, shouldn’t there be?Īnd yes I am aware of the relevant XKCD cartoon ( ).

Is there nothing like this already? Maybe there is and I just haven’t heard of it. Maybe a different extension could used when values are stored in column major order (column1, column 2 etc). It could be called something like “unicode separated value” (hat tip to on Twitter for the name) or “unit separated value” with file extension. Typing \u001F or \u001E in some editors might be a faff, but it is hardly a showstopper. You could write a fast parser for it in minutes. It would be reasonably compact, efficient to parse and easy to manually edit (Notepad++ shows the unit separator as a ‘US’ symbol). If you want to put \u001F or \u001E in your data – tough you can’t. Rows are separated by \u001E (ASCII record separator).Columns are separated by \u001F (ASCII unit separator).Values stored in row major order (row 1, row2 etc).But it is binary, so can’t be viewed or edited with standard tools, which is a pain.ĭon’t even get me started on Excel’s proprietary, ghastly binary format. It is very efficient with it’s columnar storage and compression. XML and JSON are tree structures and not suitable for efficiently storing tabular data (plus other issues). But can’t store tabs and still has issues with line endings, encodings etc. Microsoft Excel and Apple Numbers don’t even agree on how to interpret some edge cases for CSV. Writing a parser to handle all the different dialects is not at all trivial. Use of separators and line endings are inconsistent (sometimes comma, sometimes semicolon). It is difficult to parse efficiently using multiple cores, due to the quoting (you can’t start parsing from part way through a file).

One quote in the wrong place and the file is invalid. And they are all highly sub-optimal for the job.ĬSV is a mess. Most tabular data currently gets exchanged as: CSV, Tab separated, XML, JSON or Excel. There doesn’t seem to be anything that is reasonably space efficient, simple and quick to parse and text based (not binary) so you can view and edit it with a standard editor. It is an important part of my data transformation software. I support reading and writing tabular data in various formats in all 3 of my software application.