Ideally, we'd like to keep the exact same format we have and just write some glue code to hand records to the SQL layer. I can see a case for the creation of files that sit alongside each hourly data file to provide indexing, but we'd like to avoid fundamental changes to the existing data structure if possible. This is to keep compatibility with other tools that read/write our data, and to avoid having to convert many years of data that our customers have collected.