It’s true – processing data from software defined radios can be a bit
complex
👈😏👈 – which tends to keep all but the most grizzled experts and bravest
souls from playing with it. While I wouldn’t describe myself as either, I will
say that I’ve stuck with it for longer than most would have expected of me.
One of the biggest takeaways I have from my adventures with software defined
radio is that there’s a lot of cool crossover opportunity between RF and
nearly every other field of engineering.
Fairly early on, I decided on a very light metadata scheme to track SDR
captures, called rfcap. rfcap has withstood my test
of time, and I can go back to even my earliest captures and still make sense of
what they are – IQ format, capture frequencies, sample rates, etc. A huge
part of this was the simplicity of the scheme (fixed-lengh header, byte-aligned
to supported capture formats), which made it roughly as easy to work with as a
raw file of IQ samples.
However, rfcap has a number of downsides. It’s only a single, fixed-length
header. If the frequency of operation changed during the capture, that change
is not represented in the capture information. It’s not possible to easily
represent mulit-channel coherent IQ streams, and additional metadata is
condemned to adjacent text files.
ARF (Archive of RF)
A few years ago, I needed to finally solve some of these shortcomings and tried
to see if a new format would stick. I sat down and wrote out my design goals
before I started figuring out what it looked like.
First, whatever I come up with must be capable of being streamed and processed
while being streamed. This includes streaming across the network or merely
written to disk as it’s being created. No post-processing required. This is
mostly an artifact of how I’ve built all my tools and how I intereact with my
SDRs. I use them extensively over the network (both locally, as well
as remotely by friends across my wider
lan). This decision sometimes even
prompts me to do some crazy things from time
to time.
I need actual, real support for multiple IQ channels from my multi-channel SDRs
(Ettus, Kerberos/Kracken SDR, etc) for playing with things like
beamforming.
My new format must be capable of storing
multiple streams in a single capture file, rather than a pile of files in
a directory (and hope they’re aligned).
Finally, metadata must be capable of being stored in-band. The initial set of
metadata I needed to formalize in-stream were Frequency Changes and
Discontinuities. Since then, ARF has grown a few more.
After getting all that down, I opted to start at what I thought the simplest
container would look like,
TLV
(tag-length-value) encoded packets. This is a fairly well trodden path,
and used by a bunch of existing protocols
we
all
know
and
love.
Each ARF file (or stream) was a set of
encoded “packets” (sometimes called data units in other specs). This means that
unknown packet types may be skipped (since the length is included) and
additional data can be added after the existing fields without breaking
existing decoders.
Heads up!
Once this is posted, I'm not super likely to update this page. Once this
goes out, the latest stable copy of the ARF spec is maintained at
draft-tagliamonte-arf-00.txt.
This page may quickly become out of date, so if you're actually interested in
implementing this, I've put a lot of effort into making the draft
comprehensive, and I plan to maintain it as I edit the format.
Unlike a “traditional” TLV structure, I opted to add “flags” to the top-level
packet. This gives me a bit of wiggle room down the line, and gives me a
feature that I like from ASN.1 – a “critical” bit. The critical bit indicates
that the packet must be understood fully by implementers, which allows future
backward incompatible changes by marking a new packet type as critical. This
would only really be done if something meaningfully changed the interpretation
of the backwards compatible data to follow.
| Flag |
Description |
| 0x01 | Critical (tag must be understood) |
Within each Packet is a tag field. This tag indicates how the contents of the
value field should be interpreted.
In order to help with checking the basic parsing and encoding of this format,
the following is an example packet which should parse without error.
00, // tag (0; no subpacket is 0 yet)
00, // flags (0; no flags)
00, 00 // length (0; no data)
// data would go here, but there is none
Additionally, throughout the rest of the subpackets, there are a few unique and
shared datatypes. I document them all more clearly in the draft, but to quickly
run through them here too:
UUID
This field represents a globally unique idenfifer, as defined by RFC 9562, as
16 raw bytes.
Frequency
Data encoded in a Frequency field is stored as microhz (1 Hz is stored as
1000000, 2 Hz is stored as 2000000) as an unsigned 64 bit integer. This has a
minimum value of 0 Hz, and a maximum value of 18446744073709551615 uHz, or just
above 18.4 THz. This is a bit of a tradeoff, but it’s a set of issues that I
would gladly contend with rather than deal with the related issues with storing
frequency data as a floating point value downstream. Not a huge factor, but as
an aside, this is also how my current generation SDR processing code (sparky)
stores Frequency data internally, which makes conversion between the two
natural.
IQ samples
ARF supports IQ samples in a number of different formats. Part of the idea here
is I want it to be easy for capturing programs to encode ARF for a specific
radio without mandating a single iq format representation. For IQ types with
a scalar value which takes more than a single byte, this is always paired
with a Byte Order field, to indicate if the IQ scalar values are little or
big endian.
| ID |
Name |
Description |
| 0x01 | f32 | interleaved 32 bit floating point scalar values |
| 0x02 | i8 | interleaved 8 bit signed integer scalar values |
| 0x03 | i16 | interleaved 16 bit signed integer scalar values |
| 0x04 | u8 | interleaved 8 bit unsigned integer scalar values |
| 0x05 | f64 | interleaved 64 bit floating point scalar values |
| 0x06 | f16 | interleaved 16 bit floating point scalar values |
Each ARF file must start with a specific Header packet. The header contains
information about the ARF stream writ large to follow. Header packets are
always marked as “critical”.
magic
flags
start
guid
site guid
#st
In order to help with checking the basic parsing and encoding of this format,
the following is an example header subpacket (when encoded or decoded this
will be found inside an ARF packet as described above) which should parse
without error, with known values.
00, 00, 00, fa, de, dc, ab, 1e, // magic
00, 00, 00, 00, 00, 00, 00, 00, // flags
18, 27, a6, c0, b5, 3b, 06, 07, // start time (1740543127)
// guid (fb47f2f0-957f-4545-94b3-75bc4018dd4b)
fb, 47, f2, f0, 95, 7f, 45, 45,
94, b3, 75, bc, 40, 18, dd, 4b,
// site_id (ba07c5ce-352b-4b20-a8ac-782628e805ca)
ba, 07, c5, ce, 35, 2b, 4b, 20,
a8, ac, 78, 26, 28, e8, 05, ca
Immediately after the arf Header, some number of Stream Headers
follow. There must be exactly the same number of Stream Header packets as are
indicated by the num streams field of the Header. This has the nice effect of
enabling clients to read all the stream headers without requiring buffering of
“unread” packets from the stream.
id
flags
fmt
bo
rate
freq
guid
site
In order to help with checking the basic parsing and encoding of this format,
the following is an example stream header subpacket (when encoded or decoded
this will be found inside an ARF packet as described above) which should parse
without error, with known values.
00, 01, // id (1)
00, 00, 00, 00, 00, 00, 00, 00, // flags
01, // format (float32)
01, // byte order (Little Endian)
00, 00, 01, d1, a9, 4a, 20, 00, // rate (2 MHz)
00, 00, 5a, f3, 10, 7a, 40, 00, // frequency (100 MHz)
// guid (7b98019d-694e-417a-8f18-167e2052be4d)
7b, 98, 01, 9d, 69, 4e, 41, 7a,
8f, 18, 16, 7e, 20, 52, be, 4d,
// site_id (98c98dc7-c3c6-47fe-bc05-05fb37b2e0db)
98, c9, 8d, c7, c3, c6, 47, fe,
bc, 05, 05, fb, 37, b2, e0, db,
Samples
Block of IQ samples in the format indicated by this stream’s format and
byte_order field sent in the related Stream Header.
In order to help with checking the basic parsing and encoding of this format,
the following is an samples subpacket (when encoded or decoded
this will be found inside an ARF packet as described above). The IQ values
here are notional (and are either 2 8 bit samples, or 1 16 bit sample,
depending on what the related Stream Header was).
01, // id
ab, cd, ab, cd, // iq samples
Frequency Change
The center frequency of the IQ stream has changed since the
Stream Header or last Frequency Change
has been sent. This is useful to capture IQ streams that are jumping
around in frequency during the duration of the capture, rather than
starting and stopping them.
In order to help with checking the basic parsing and encoding of this format,
the following is a frequency change subpacket (when encoded or decoded
this will be found inside an ARF packet as described above).
01, // id
00, 00, b5, e6, 20, f4, 80, 00 // frequency (200 MHz)
Discontinuity
Since the last Samples packet for this stream, samples have been dropped
or not encoded to this stream. This can be used for a stream that has
dropped samples for some reason, a large gap (radio was needed for something
else), or communicating “iq snippits”.
In order to help with checking the basic parsing and encoding of this format,
the following is a discontinuity subpacket (when encoded or decoded this will
be found inside an ARF packet as described above).
Location
Up-to-date location as of this moment of the IQ stream, usually from a GPS.
This allows for in-band geospatial information to be marked in the IQ stream.
This can be used for all sorts of things (detected IQ packet snippits aligned
with a time and location or a survey of rf noise in an area)
flags
lat
long
el
accuracy
The sys field indicates the Geodetic system to be used for the provided
latitude, longitude and elevation fields. The full list of supported
geodetic systems is currently just WGS84, but in case something meaningfully
changes in the future, it’d be nice to migrate forward.
Unfortunately, being a bit of a coward here, the accuracy field is a bit of a
cop-out. I’d really rather it be what we see out of kinematic state estimation
tools like a kalman filter, or at minimum, some sort of ellipsoid. This is
neither of those - it’s a perfect sphere of error where we pick the largest
error in any direction and use that. Truthfully, I can’t be bothered to model
this accurately, and I don’t want to contort myself into half-assing something
I know I will half-ass just because I know better.
In order to help with checking the basic parsing and encoding of this format,
the following is a location subpacket (when encoded or decoded this will be
found inside an ARF packet as described above).
00, 00, 00, 00, 00, 00, 00, 00, // flags
01, // system (wgs84)
3f, f3, be, 76, c8, b4, 39, 58, // latitude (1.234)
40, 02, c2, 8f, 5c, 28, f5, c3, // longitude (2.345)
40, 59, 00, 00, 00, 00, 00, 00, // elevation (100)
40, 24, 00, 00, 00, 00, 00, 00 // accuracy (10)
Vendor Extension
In addition to the fields I put in the spec, I expect that I may need custom
packet types I can’t think of now. There’s all sorts of useful data that could
be encoded into the stream, so I’d rather there be an officially sanctioned
mechanism that allows future work on the spec without constraining myself.
Just an example, I’ve used a custom subpacket to create test vectors, the data
is encoded into a Vendor Extension, followed by the IQ for the modulated
packet. If the demodulated data and in-band original data don’t match, we’ve
regressed. You could imagine in-band speech-to-text, antenna rotator azimuth
information, or demodulated digital sideband data (like FM HDR data) too. Or
even things I can’t even think of!
In order to help with checking the basic parsing and encoding of this format,
the following is a vendor extension subpacket (when encoded or decoded this
will be found inside an ARF packet as described above).
// extension id (b24305f6-ff73-4b7a-ae99-7a6b37a5d5cd)
b2, 43, 05, f6, ff, 73, 4b, 7a,
ae, 99, 7a, 6b, 37, a5, d5, cd,
// data (0x01, 0x02, 0x03, 0x04, 0x05)
01, 02, 03, 04, 05
Tradeoffs
The biggest tradeoff that I’m not entirely happy with is limiting the length
of a packet to u16 – 65535 bytes. Given the u8 sample header, this limits us
to 8191 32 bit sample pairs at a time. I wound up believing that the overhead in
terms of additional packet framing is worth it – because always encoding 4
byte lengths felt like overkill, and a dynamic length scheme ballooned
codepaths in the decoder that I was trying to keep as easy to change as
possible as I worked with the format.
.
Image of the estimated qualiy of articles of the four articles in the second mixed-methods paper. Extreme dips reflect periods of frequent vandalism.
![[RSS 1.0 Feed]](common/rss10.png){kind=small}
![[RSS 2.0 Feed]](common/rss20.png){kind=small}
![[Atom Feed]](common/atom.png){kind=small}
![[FOAF Subscriptions]](common/foaf.png){kind=small}
![[OPML Subscriptions]](common/opml.png){kind=small}
![[Hacker]](common/hacker.png){kind=small}
![[Planet]](common/planet.png){kind=small}