A SOFA file describes how a head hears. SOFA (Spatially Oriented Format for Acoustics, standardised as AES69) stores head-related transfer functions: for hundreds or thousands of directions, the impulse response from a loudspeaker at that direction to a microphone in each ear. With such a set you can place any sound anywhere around a listener on headphones. I wrote my first SOFA reader, PasSofa, in Object Pascal in 2017, and ported it to JavaScript as WebSofa. Now there is go-sofa, which reads and writes SOFA files in pure Go, on top of my fork of a pure-Go HDF5 library, go-hdf5. Both started in February. On the last weekend of September they got about 230 commits and 39,000 added lines. Every figure in this post was made by a Go program that reads the files with go-sofa or go-hdf5.
Listen first (on headphones)
The voice comes from go-pocket-tts, my Go port of Kyutai’s Pocket TTS. FFmpeg’s sofalizer filter rendered it at 72 directions, using the MIT KEMAR head-related impulse responses, and a few lines of Go crossfade between those renders while the angle turns. The KEMAR file was re-saved by go-sofa first. That detail matters: sofalizer loads SOFA files with libmysofa, the C library most players and plugins use, and a file saved by go-sofa v0.2.0, released on 26 September, gets only “Selected SOFA file is invalid”. Most of the weekend went into fixing that.
What a SOFA file holds
A SOFA file is a set of arrays named after their dimensions: M measurements, R receivers (the two ears), E emitters and N samples. Data.IR holds the impulse responses as [M][R][N], SourcePosition says where the loudspeaker was for each measurement, and global attributes name the convention, such as SimpleFreeFieldHRIR, plus the author, licence and the software that wrote the file. Besides impulse responses (FIR), go-sofa reads and writes transfer functions (TF and TF-E, including spherical-harmonics HRTFs) and second-order sections (SOS). Reading a set with go-sofa takes a few lines:
f, err := sofa.Open("MIT_KEMAR_normal_pinna.sofa")
if err != nil {
log.Fatal(err)
}
defer f.Close()
fmt.Println(f.SOFAConventions, f.M, f.R, f.N) // SimpleFreeFieldHRIR 710 2 512
src := f.SourcePositions[0] // azimuth, elevation, radius
ir, _ := f.IRAt(0, 0) // first direction, receiver 0How the directions are chosen depends on who measured them. These are the three databases I used in the figures below:
Looking at a head
Once the data is in Go, it is just arrays. Here are the 72 horizontal-plane measurements of the KEMAR left ear, one spectrum per direction:
The time difference between the ears is the other big cue. I computed it from the low-passed cross-correlation of the two impulse responses, for a dummy head, a human listener and a simulated head, and plotted Woodworth’s formula for a rigid sphere next to them:
HDF5 underneath
Under the hood a SOFA file is a netCDF-4 file, and that is an HDF5 file with some conventions. HDF5 is a small file system of its own, with object headers, B-trees, local, global and fractal heaps, free-space managers, checksums and several versions of most of them. I wrote a small tool against go-hdf5’s internals that records which structure occupies which bytes, and ran it on the original KEMAR file and on the copy go-sofa v0.3.0 saved:
ListenerPosition, which compresses to 11 bytes. go-sofa v0.3.0 writes the small variables uncompressed next to their headers and compresses only the impulse responses, in one chunk per ear, so its file ends up slightly smaller. go-sofa v0.2.0 did not compress at all, and its copy was five times larger.go-hdf5 started as a fork of scigolib/hdf5, a pure-Go HDF5 library that reads and writes. In February I added what SOFA needed first, namely attributes on the root group. Writing a file that some reader accepts is easy. Writing one that every reader accepts is the hard part.
What the weekend fixed
The weekend started with the HDF5 C library itself. On Thursday evening go-hdf5’s output became readable by libhdf5 and deterministic, byte for byte. Then came dimension scales, which netCDF uses to name the dimensions. With those, h5py, netCDF4-python and ncdump open go-sofa’s files and show M, R and N as real dimensions. My favourite bug of the weekend was in the name index of dense attribute storage. HDF5 hashes the names with Bob Jenkins’ lookup3, which processes 12 bytes at a time, and go-hdf5 handled names whose length is a multiple of 12 differently. In SOFA files, DateModified and Organization are both exactly 12 characters long, and libhdf5 could not find those two attributes.
libmysofa was harder, because it is not libhdf5. It implements just the part of HDF5 that SOFA files from MATLAB and Python use, and that part has limits. It follows at most 25 header continuation blocks per file. It resolves references only into a global heap that starts below 64 KiB. In dense storage it reads only string attributes, and no strings marked as UTF-8. It also expects the root group to be a new-style group with link messages. go-hdf5 v0.18.0 therefore writes the root group the way netCDF-C does, keeps dataset attributes in the object header, reserves room for each dataset’s DIMENSION_LIST attribute up front, and reserves a global heap collection for the dimension references right after the root group. That is the green block in the byte map. Strings are written in 8 bytes instead of 9, and scalar dataspaces as the 4-byte version-2 message. In go-sofa, ReceiverPosition and EmitterPosition got the three-dimensional layout the conventions require. Now SimpleFreeFieldHRIR files from go-sofa pass libmysofa’s mysofa_load and mysofa_check. CI builds libmysofa at a pinned commit and loads every generated and every re-saved file with it.
Along the way go-sofa learned more of the standard: the mandatory attributes and defaults of each convention from the SOFA Toolbox tables, and warnings for files that break a convention. On save it records its own provenance (APIName, APIVersion, DateModified, a History line), and it writes reproducible output when SOURCE_DATE_EPOCH is set.
Large files and hostile ones
Some databases are large, so OpenLazy reads the metadata and positions and leaves the audio in the file, to be read one measurement at a time. The first test with Kayser2009, eight receivers in 584 directions, took 78 seconds and allocated 75.7 GB. Each measurement touches 32 compressed chunks, more than the chunk cache held, so every chunk was decompressed again for each of the 146 measurements it spans. With a chunk cache sized to what one measurement touches, it now takes 0.6 seconds and 885 MB. In go-hdf5, repeated single-row reads of a compressed dataset dropped from 720 to 13 µs.
Files are input, so the readers must also survive garbage. go-hdf5 bounds every allocation that comes from a size stored in the file. A test feeds every header message parser every truncated prefix of valid messages, and the fuzzer found one panic in the dataspace parser. go-sofa refuses, before reading any data, a file that declares far more elements than its size makes plausible. Before that, a crafted file of a few kilobytes could make it allocate gigabytes. Every real file I tested stays at least 52 times below that limit.
Update, 4 October: go-sofa now compresses when it saves. The audio data is shuffled and deflated at level 4 by default, which brings the re-saved KEMAR file to 1.10 MB, slightly smaller than the 1.17 MB original and down from 5.85 MB. libmysofa had two more surprises. It reads chunk index keys until it finds one carrying the element size, so go-hdf5 v0.18.1 now writes the size into the last key too. And it reads only chunk indexes that fit in a single B-tree node, so go-sofa never splits a variable into more than 64 chunks. WithDeflate(0) still writes the old, uncompressed layout.
Trying it
go get github.com/CWBudde/go-sofa
go install github.com/CWBudde/go-sofa/cmd/sofainfo@latest
sofainfo MIT_KEMAR_normal_pinna.sofasofa2json dumps a file as JSON, and sofaprobe shows the HDF5 structure underneath. The libmysofa fixes and compression are in go-sofa v0.3.0, which needs go-hdf5 v0.18.1. Both are on GitHub: go-sofa and go-hdf5.