Group

Data management and processing tools / Feed

Conservation tech work doesn't stop after data is collected in the field. Equally as important to success is navigating data management and processing tools. For the many community members who deal with enormous datasets, this group will be an invaluable resource to trade advice, discuss workflows and tools, and share what works for you.

discussion

Data mgmt for Passive Acoustic Monitoring best practices?

Hello!I'm running a small passive acoustic monitoring project for terrestrial species, using audiomoths and swifts. How do people and organizations manage the ballooning datasets...

4 0

Hi Alex--

The first thing I'd suggest you think through is how much data you have vs how much data you are currently working on. Because if you have data from previous years that you want to ensure you're storing securely and reliably but don't need immediate access to in order to run analysis on, that opens up some options. You can compress data using lossless algorithms like FLAC, where the compression ratio varies but 50% is a pretty good margin, and then convert back to WAV if necessary for reanalysis. Compressing using MP3, OGG, AIFF, or other compression algorithms is an option that saves even more storage space but you will lose information in ways you wouldn't with FLAC--it depends on your specific needs.

I'd also recommend setting up a RAID array (RAID = "Redundant Array of Inexpensive Disks"). This offers some additional security in event of a drive failure. A lot of folks who do video editing, probably the most similar use case to people working with acoustic data who also lack the institutional support of a large company or university IT department use a local NAS enclosure like https://www.qnap.com/en-us/product/ts-433 that are designed for just this purpose. Some higher initial startup costs than just buying individual USB hard drives but that does come with some perks including additional reliability and can be faster to read data depending on the exact drive specs and your local networking setup.

There are also low-cost cloud storage services like Amazon's Glacier. However, getting these set up can be a little bit tricky and they are not particularly responsive (for example, if you upload data to Glacier, it will be very safe, but getting it back if you need to use it again can take a few days depending on the dataset size).

Hello Alex,

   My information might not be that helpful to you, still, our organisation have an Enterprise license of AWS cloud and we store all our media files (video, pictures, audio etc.) there. We are also using a media management solution, Piction, thru which we upload the files into the S3 bucket and in the process it also captures the file metadata (some of the metadata values needs to be entered manually). This is useful to search the files if someone wants to view or process the file later. We are soon deciding on the file storage configuration so that old files will move to cheap storage like AWS Glacier, which will take a maximum of a week time to retrieve it.  

Jitendra 

Hi Alex,

I'd go much further along the lines that David @dtsavage sets out. Before jumping to implementations, better think through why you want to keep all that data, and for who? From your question, it appears you have at least three purposes:

1- for yourself to do your research

2- for others to re-use.

3- for yourself to have a back-up

For 1) you should do what works best for you.

For 3) use your organization's back-up system or whatever comes close to that

For 2 and 3) As you are indicating yourself : deposit your data at your nation's repository or zenodo.org if your nation doe not have one. It may be some documentation work ( which is what you should do anyways, right? ), but then you can stop worrying about holding on to it. Someone else is doing that for you and they do a much better job - because it is their job. Moreover, you increase the chance that other will actually become aware of all that data that you are sitting on by putting it into a repository. Who is otherwise going to find out and how that you have those disks on your desk? Lastly, depositing your data can also serve as a back-up. If you don't want to share it before you've published about it, there is likely the option of depositing under time-embargo or of depositing while requiring your consent for any re-use. 

You ask how many people actually do this? You can find the answers at the repository, but I suggest that what matters most is whether you want to for your own reasons, and whether your funders, or organization's funders require it.

See full post
careers

Shark Lab Data Analyst

This position (at California State University, Long Beach) provides data management & analysis support to Shark Lab research operations including shark tagging, active tracking, receiver data, AUV & UAV data...

0
See full post
discussion

Software to aid acoustic sound files visualization/labelling + Software to syncronize video/acoustic sonograms

Hi everybody!I am currently trying to figure out if there is any open-source software that could improve our citizen science project on bat monitoring in Europe (and potentially...

4 1

I would also recommend Arbimon. It is well set up to handle Audiomoth recordings. Being cloud based, you will need a good internet connection for sound file upload. I'm just starting to investigate its use for Song Scope recordings. Setting up the call recognisers will be a slow process, but they can be made available to all users once done.

You could try using a video editor like DaVinci for looking at your video and audio together. I don't think DaVinci displays sonograms by default (just waveform) but I think it will open your selected audio in an external editor which would allow you to see the sonograms and make measurements with something like Audacity or Kaleidoscope.

The open-source program Audacity can show the spectrograms and histograms and has quite a lot of other useful features, e.g. playing ultrasound calls slower, so it can be heard by people. 

See full post
discussion

Multi-SD card reading hubs

Hi everyone,I'm writing to see if anyone has any experience with using/building hubs that can read multiple SD cards at once? I'm trying to explore ways to speed up the manual...

6 1

Good point and nice link. My USB 3.0 (errr USB 3.1 Gen 1) portable HDD gets a real read speed of 133 MBps or ~1Gbps. The theoretical max is 5 Gbps but I think the speed is limited by processor used by the HDD/USB bridge controller. Most SD card readers may be more limited even if the SD card can handle higher data rates. That may be why it could be best to just use the multi-port device from SanDisk. 

Jamie,

I have been using a 12-card Lexar Professional Workflow system for several years to process images from 24 trail cameras.  The system has 4 bays, with 3 card readers per bay.  I wrote software to automatically download and rename the images using metadata read from the info banner burned into the images by the camera.  (Alternatively, the software can use the Exif metadata attached to the image.)  The software is described in a recent Wildlife Society Bulletin: 

The software is available on github:  

Don't be put off by "time-lapse recordings" - it works on still images also.

Regards,

Mike Hilton

Have a look at @tessa_rhinehart 's TechTutors on scaling up acoustic surveys. This is addressed at 11 min: 

The "hexadecapus" is the hardware, but transfer is automated by naming the SDCards prior to use and scripts take care of the transfer. The scripts may be written up in the audiomoth guide but the kitzes lab have this on github (although it may be mac specific) 

See full post
discussion

Interest in an "IT for Ecologists" webinar/manual?

I've seen a few posts around Wildlabs but also on Twitter, the bioacoustics stackexchange, and a few other places asking for recommendations for things like hard drive models, sd...

2 6

Next WILDLABS virtual course?!? :) 

Agreed that this would be super useful! It definitely would have served me well in undergrad when I was trying out some AI camera trap image analysis and ended up having to build my own system from scrap pieces in a professor's gear closet, ha. Just some basic understanding of what would be required from a system for that kind of work would have saved me a lot of time and chaos. Some kind of guide or course that covers what to be thinking about depending on your data and goals would be awesome

See full post
discussion

Conservation Tech Directory - new update!

New directory update from @gracieermi & I! And an extra special one as we've just passed our 1-year anniversary! Super exciting to see how far this has come! Check it out:...

2 2

Congrats on the milestone Carly and Gracie!

See full post
discussion

Mangrove soundscape

Hello everyone, I am currently working on the restoration of mangroves in an area of Colombia. I want to do an acoustic monitoring inside the mangrove. I want to record...

4 0

You might check out the devices listed in the Conservation Tech Directory - you can search for 'acoustic recorder' or 'ARU' or something like that.

The most common off-the-shelf models (other than Frontier Labs' BARLTs mentioned above😊) are Open Acoustic Devices Audiomoths & Hydromoths (which may be particularly useful for you since they come in completely watertight cases that can be stuck underwater for deployments), Wildlife Acoustics' SongMeter series, and Cornell Lab of Ornithology's Swift units.  

 

Camilo--

Are you interested only in airborne sounds or do you want to deploy hydrophones? That decision would inform a lot of other decisions about your purchase of equipment, as would having a clearer picture of your budget.

Hi there Camilo, 

What an interesting project! If you are looking for a lower cost, but effective tools for acoustic monitoring you might want to look into two options: 

SoundTraps - are very commonly used and perform quite well: 

SonarPoints - these are also a great instrument option: 

See full post
funding

HeroX Challenge: Better Call Trawl

NASA Tournament Lab
Explore BOEM data, build analytic tools, and recommend solutions for sea turtle relocation trawling approaches to reduce incidental risk. Prize: $40,000

0
See full post
event

Data Beyond Borders Symposium

Join Biodiversa+and GBIF: The Global Biodiversity Information Facility on 6 October 2022 for their joint symposium: "Data beyond borders: collaborating to support biodiversity information needs in Europe and beyond...

0
See full post
careers

Join the BirdNET team! 

The BirdNet team are seeking to fill three roles, including an Ecologist, Data scientist and an Embedded Systems Engineer based in Germany. 

4
See full post
article

Mara Predators Hackathon 

APPLY NOW! The Sovereign Nature Initiative has partnered with the Kenya Wildlife Trust to experiment with emerging technologies to support their predators' conservation work.Challenges will focus on:1. Lion...

0
See full post