Thanks! Yeah, nice to see so many groups helping out.

BTW, I'm looking for people experienced in bioacoustic research to provide critical feedback on the quality of my device for these purposes. I don't have the experience to judge, but I'm expecting it to be very suitable.

The open acoustic devices devices are very cool. There's are low power, my approach is not.

My approach could be useful if you wanted to network the devices, such as if you detected a gunshot you could potentially relay this information via networking to a central server that collects and localizes the notifications. Also, if you need some devices on short notice, raspberry pi's are nice and available again.

Is the gunshot detection code from open acoustic devices open? That would be interesting to play with in conjunction with my system for me.

Hi Kim! 

Yes, for a networked real time system your approach seems promising!

I do not know, but most of what these guys do will be open source sooner or later. If you cannot locate what you need at

Open Acoustic Devices | AudioMoth

Home for all information on AudioMoth, a low-cost, open-source acoustic monitoring device.

openacousticdevices

I would try to contact [email protected]. He is very responsive. 

Cheers,

Lars

Using a full fledge OS make's some things easier and some things harder. Obviously, networking is easier, but the registration of the received audio buffers to the actual system time is more complex.

In addition, I've made this potentially harder for myself because I'm using jackd real-time network based audio distribution under the covers. This has the advantage though that I can have multiple sinks to the stream. In this case, one sink for recording the audio and another sink could feed a real-time gunshot detector model for example and even real time monitoring of the audio stream (Maybe useful for anti-poaching).

So testing is revealing some challenges in reliably determining the latency through the stack. But once these have been solved I can formally test it and write up the report and publish the code.

I've been able to see sub 1ms alignment on an external sound event on two different devices to the same sound event. But I saw some outliers, so I have to investigate further to eliminate these.

Hi Kim,

Have you measured the lag of the microphone? Most USB microphones will typically buffer a couple of millisecond frames so you might have to account for that delay. 

They will typically also skip or duplicate samples within a 1 ms frame as the clock controlling sample acquisition  isn’t exactly synced with the USB frame rate.

To get 1us precision on AudioMoth we respond to the PPS time signal and also measure the clock counter value that is controlling the ADC capture. That way we can measure the precise time that each ADC sample is acquired. Description here - 

This is easier to do on an embedded device with control over all the low-level factors.

That said, with sounds that aren’t impulsive, correlating waveforms accurately to within 1 ms is still quite challenging, so the accuracy of the synchronisation seems less significant to the overall accuracy of the system.

Alex

I don't measure the lag of the USB microphone because I have no way of ascertaining that. However, I would expect it to be a somewhat repeatable value and as all of my sound acquisition is through USB sound cards I expect that the start times will be in effect normalized amongst the different values to the start of the sound as far as the jackd daemon gets it. Technically speaking because I am working backwards from the sound event in the file to the effective start, so this also minimizes the effective of lag in the USB device itself. The only exception to this is the seeed studios microphone hat, which uses i2c, however that has shown to be difficult to use in practical terms because the microphone is attached to the circuit board.

I think because I work backwards from the event to the time that jackd receives it identically between all devices that the start time will effectively be "normalized" between all the devices and any lag that happens within the USB device itself is thus irrelevant in terms of correlating the TDOAs. If the USB device took 5 minutes of buffering the start time will still be calculated backwards from the event to the start time of the package receipt by jackd.

Actually, I maintain a tracking file that contains the "in this way calculated start time relative to it's buffer index in the file and as the system clock has been proven to always have less than 1 us of error I can't see how the buffer time of the USB itself actually factors into the equation at all? I'm pretty sure that this approach bypasses that. Please correct me if my logic is incorrect here, but it seems correct to me.

The jackd provides a call to ascertain the current microsecond counter that it considers time, called get_jack_time() and the get_last_frame_time() is returned relative to this. I substract the difference between the system clock and the instance in time that the get_jack_time() call is made which then minimizes the jitter. Additionally, jackd provides a call to get the latency at that moment, which is always in terms of a fixed number of buffers. So I work back to this time of the start of the current buffer and add the latency.

In testing, the results show that so long as I setup the microphones in a way that the event start can be well ascertained between both devices that I do seem to get less than 1ms between the times calculated that way.

Indeed, microcontrollers should give you greater control over that.

There's another new real time streamer called pipewire which appears to allow timestamping close to the capture source, at some time I'll take a look at that as well.

The accuracy of the overall system is consistently tracking at less than 1 us of error according to chronyc tracking and chronyc sourcestats.

Thanks for the document reference! I'll give that a read this weekend.

Whoops! No, I don't think it make's the delay in the USB device irrelevant. However surely that's got to be really small, a cheap USB device simply will not have very much memory inside of it.

I would hope that any lag due to that will be consistent lag and not variable lag :) If so, then the TDOAs will still be normalized amongst the same device type in any case.

Yep it depends how it’s implemented. Our AudioMoth USB Microphone firmware uses a few 1 ms buffers in a ring which uses a few kilobytes of SRAM.

If the same USB microphones are used throughout it’s probably reasonably consistent with 0-2 ms delay between audio capture and it appearing in the USB packet.

AudioMoth GPS Sync gives better than 1us absolute accuracy on the timestamp of each audio sample, but that’s overkill for most sound source localisation problems. The typical error in measuring the time delay between non-impulsive sounds seems to be closer to 1 ms in practice once multiple paths and echos get mixed in.

The key thing is to build a few, co-locate the microphones, and measure the difference in the time of arrival of acoustic events. Hard to predict without actually measuring it.

Being able to stream timestamped audio would be really nice. There was a really interesting citizen science project that was detecting meteors using infrasound frequency detectors connected to Raspberry Pi.

Homebf

Detect Earthquakes & Watch the Earth Move Professional grade seismograph and infrasound

Raspberry Shake

 We’ve been keen to try something similar with weather events using synchronised AudioMoth recordings.

I asked chatgpt about this. She said that she would wager a guess that between 10-20 microbuffers of sound could possibly be a ballpark guess about an old chipset like the cm108 or the general plus. However, she also thought that it would likely be the case that the delay for such buffering if present would likely be somewhat consistent given that there are no other USB devices on the bus to cause it to wait. Each microbuffer was for high speed USB 125 us. I suspect that 10 microbuffers is likely on the high side due to the device being very cheap, old and having so special functions. But in any case, so long as the delay was consistent, it would effectively normalize itself out for TDOA sound localization.

I did test it with co-locating the microphones and clapping. It was consistently independently calculating the time of arrival with less than 1 ms difference.

I will be adding as a flag the ability to add to the tracking file the jack_time value. I have to test it but I suspect that the same frames arriving in different streams would have the same jack time value, I suspect that's the whole point somewhat. Then in essence you have time stamped streams because I would be logging the jack time along with the buffer number and the exact time with each period, which I'm currently making 2048 samples, so that's aligning the time to the system clock around 20 times a second and the system clock is super accurate at less than 1us of error. The chrony daemon is constantly making micro adjustments to the system clock to ensure that.

What I don't know is what happens to the jack_time values when you relay to another host. Jack audio allows you to pipe all these things together in chains. As many hops as you like. You setup a dummy server on a node and a relay connection to that other host.

I suspect that when you do this the dummy server has it's own jack time values, but I don't know without asking or testing. If it maintained the original jack_time throughout the hops that would be cool. In any case, if jack audio doesn't do it pipewire states that it does timestamp the streams. Pipewire state that they maintain the API api as jack audio so as to make it easy for apps to be compatible so some stage it should be quite trival to implement it for pipewire as well.

Yeah, the accuracy here is way overkill, but it's easy to do so why not. What I don't have experience in is how much timing errors then translate to the localized co-ordinates, I expect the 34cm of distance error in the tdoa would translate to a potentially a larger distance error in the result. raven allows me to examine at sub millisecond resolution, so if I can provide that, maybe it can help in the closer distance studies. But I really did it because I could and it appeals to the geek in me.

Actually, I'm hoping this project will appeal to several usecases. Around here in Limburg, the Netherlands we have a lot of unexplained explosions. I want to localize those out of curiosity. But I've love it if the wildlife folks can benefit from it.

What is in the realms of possibility with respect to sounds with harder to identify starting points I hope to find out. The starting points are harder to identify but the distances and hence tdoas should be bigger, I don't know what to expect yet. But by making this stuff super cheap hopefully a lot more people will experiment.

Mostly in rooms they tends to go with phase difference approaches, but I do believe that so long as the location of the microphones were such that they can be echo free that this approach would work as well. There's enough accuracy.

Yeah, the thing about rtk GPSes though is that they need to be able to receive a stream of network provided correction data as I understand it. However, I'm not using the GPS for position, just time. It would be handy to have one of those things though for setting up the data for the experiments though. Now I use google maps, which to be fair is also effective.

I did reach out to the elephant people in their call for submissions. However, I'm just one guy, with not much free time and so as a choice for a party to help out not sure if that would appeal to them. Helping some other partner however, might be an option. Certainly interesting and highly applicable I think.

Super nice the sparkfun pages :)

BTW, on the subject of phased arrays, seeed studios have a lot of gear of that. I solved a compile bug in one of the sound kernel modules that I still have to make a pull request for. It's for a module that's not used for all sound applications, but the one I had wouldn't work without it. I also tested with the 2-mic hat from seeed, but it's a little inconvenient that the mics on on the board itself. But I got that working as well.

BTW, I would love to help the elephant people! My other project would be very good here as well, the one that triggers video capture events using AI computer vision. My system is using the very latest, fastest and most accurate computer vision algorithm (yolov6) and so should perform perfectly! It's highly configurable as well as it has a web interface to a flexible state machine for controlling how the system can react to the events.