CV4Animals: Computer Vision for Animal Behavior Tracking and Modeling

In conjunction with Computer Vision and Pattern Recognition 2023. June 18, 2023 full day, Room 'East 4'

Online Event
18 Jun 2023
9:00 am ~ 5:45 pm America/Vancouver

Stephanie O'Donnell

@StephODonnell
| She / Her

Fauna & Flora

I'm the Community Manager at WILDLABS


Times (in PDT)	Sessions
9:20~9:30am	Welcome
Room 'East 4'
9:30 - 10:00	Keynote: Kostas Daniilidis	Title: Multi-view Tracking in an Outdoor Aviary
Room 'East 4'		Abstract: The ability to capture detailed interactions among individuals in a social group is foundational to our study of animal behavior and neuroscience. Recent advances in deep learning and computer vision are driving rapid progress in methods that can record the actions and interactions of multiple individuals simultaneously. Many social species, such as birds, however, live deeply embedded in a three-dimensional world. This world introduces additional perceptual challenges such as occlusions, orientation-dependent appearance, large variation in apparent size, and poor sensor coverage for 3D reconstruction, that are not encountered by applications studying animals that move and interact only on 2D planes. In this talk, I will present a system for studying the behavioral dynamics of a group of songbirds as they move throughout a 3D aviary. We study the complexities that arise when tracking a group of closely interacting animals in three dimensions and introduce a novel dataset for evaluating multi-view trackers. We analyze captured ethogram data and demonstrate that social context affects the distribution of sequential interactions between birds in the aviary. Finally, we present preliminary results of tracking with event-based cameras.
10:00 - 10:30	Keynote: Tanya Berger-Wolf	Title: Imageomics: Images as the Source of Information about Life
Room 'East 4'		Abstract: Introducing the new field of imageomics: from images to biological traits using biology-structured machine learning. Images are the most abundant, readily available source for documenting life on the planet. Coming from natural history collections, laboratory scans, field studies, camera traps, wildlife surveys, autonomous vehicles on the land, water, and in the air, as well as tourists’ cameras, citizen scientists’ platforms, and posts on social media, there are millions of images of living organisms. But their power is yet to be harnessed for science and conservation. Even the traits of organisms cannot be readily extracted from images. The analysis of traits, the integrated products of genes and environment, is critical for biologists to predict effects of environmental change or genetic manipulation and to understand the significance of patterns in the four billion year evolutionary history of life. Knowledge-guided machine learning and computer vision can turn massive collections of images into high resolution information database about living organisms, enabling scientific discovery, conservation, and policy decisions. This is our vision of the new scientific field of imageomics.
10:30 - 11:00	Orals	Oral 1 by Mohannad Elhamod Discovering Novel Biological Traits From Images Using Phylogeny-Guided Neural Networks
Room 'East 4'		Oral 2 by Valentin Gabeff Scene and animal attributes retrieval from camera trap data with domain-adapted language-vision models Oral 3 by Otto Brookes PanAf20K: A Large Video Dataset for Wild Ape Detection & Behaviour Analysis
11:00 - 12:30	Poster Session 1	see poster assignment below
Room TBD
12:30 - 1:30pm	Lunch

1:30 - 2:00 pm	Keynote: Katija Kakani	Title: Developing imaging technologies to search for, discover, and understand life in the deep sea
Room 'East 4'		Abstract: The deep sea is the largest habitable ecosystem on the planet and remains one of the least explored. Very little is known about deep sea inhabitants, their behavior, and the limits and drivers for their survival. Ecomechanics, the multidisciplinary research field of the mechanisms that underlie organismal interactions and survival within their environment, has proven largely successful in terrestrial fields and lab-based organismal systems, but has had limited applicability to deep sea animals. The reason for this deficit is largely due to the technological challenges to access this environment for study, and the typically short duration of any observations, precluding documentation of many critical behaviors. Here we present a number of developments from MBARI’s Bioinspiration Lab that can be used to quantify biodiversity, biomechanics, and behavior of animals in the deep sea. All powered by computer vision, these developments include: two 4000 m-rated, ROV-based imaging systems DeepPIV and EyeRIS used to quantify time-resolved particle fields and structures in 2D and 3D, a 1500 m-rated AUV-based plenoptic imaging system Chiton, an underwater labeled image database called FathomNet that can be used to fuel machine learning algorithm development, and machine learning-integrated vehicle control algorithms called ML-Tracking that enable long-duration observations of individual animals. These instruments and approaches can be applied to a wide range of science use cases, both in the water column and on the seafloor.
2:00 - 2:30pm	Keynote: Devis Tuia	Title: Remote sensing enables monitoring life under water
Room 'East 4'		Abstract: The oceans are a major source of biodiversity and food. The preservation of the quality of the foodchain, as well as of life underwater in general is a primary concern for society. However, observing water media routinely is a complex task that remote sensing technology empowered by AI can unlock. In this talk, I will present some recent researches, where a deep learning system, based on consumer grade cameras, is deployed to 3D reconstruct and map coral ecosystems at scale.
2:30 - 3:00pm	Keynote: Albert Ali Salah	Title: AI and Animal Wellbeing
Room 'East 4'		Abstract: Rapid progress in computer based automation approaches creates new opportunities and challenges for animal wellbeing. On the one hand, innovative applications are being implemented to assist pet owners, people who work with animals, veterinarians, and for tasks that require monitoring of animals in different settings. On the other hand, we see challenges similar to those in automatic human behavior analysis, where automation may cause us to lose the human touch, and ethical issues and risks ensue from the potential usage of such applications, both in the short and in the long run. In this talk, I would like to draw parallels between computer based human and non-human animal behavior analysis, providing examples from our interdisciplinary work in equine and canine pain estimation, as well as other work at the newly established AI and Animal Wellbeing Laboratory at Utrecht University, to raise a number of issues and to discuss both technical and humanity challenges of the field.
3:00 - 3:20 pm	Orals	Oral 4 by Luke Meyers Towards Automatic Honey Bee Flower-Patch Assays with Paint Marking Re-Identification
Room 'East 4'		Oral 5 by Medha Sawhney Detecting and Tracking Hard-to-Detect Bacteria in Dense Porous Backgrounds
3:20 - 3:45 pm	Afternoon coffee

3:45 - 4:15 pm	Panel discussion
Room 'East 4'
4:15 - 5:45 pm	Poster Session 2	see poster assignment below
Room TBD

Times (in PDT)

Sessions

9:20~9:30am

Welcome

Room 'East 4'

9:30 - 10:00

Keynote: Kostas Daniilidis

Title: Multi-view Tracking in an Outdoor Aviary

Room 'East 4'

Abstract: The ability to capture detailed interactions among individuals in a social group is foundational to our study of animal behavior and neuroscience. Recent advances in deep learning and computer vision are driving rapid progress in methods that can record the actions and interactions of multiple individuals simultaneously. Many social species, such as birds, however, live deeply embedded in a three-dimensional world. This world introduces additional perceptual challenges such as occlusions, orientation-dependent appearance, large variation in apparent size, and poor sensor coverage for 3D reconstruction, that are not encountered by applications studying animals that move and interact only on 2D planes. In this talk, I will present a system for studying the behavioral dynamics of a group of songbirds as they move throughout a 3D aviary. We study the complexities that arise when tracking a group of closely interacting animals in three dimensions and introduce a novel dataset for evaluating multi-view trackers. We analyze captured ethogram data and demonstrate that social context affects the distribution of sequential interactions between birds in the aviary. Finally, we present preliminary results of tracking with event-based cameras.

10:00 - 10:30

Keynote: Tanya Berger-Wolf

Title: Imageomics: Images as the Source of Information about Life

Room 'East 4'

Abstract: Introducing the new field of imageomics: from images to biological traits using biology-structured machine learning.
Images are the most abundant, readily available source for documenting life on the planet. Coming from natural history collections, laboratory scans, field studies, camera traps, wildlife surveys, autonomous vehicles on the land, water, and in the air, as well as tourists’ cameras, citizen scientists’ platforms, and posts on social media, there are millions of images of living organisms. But their power is yet to be harnessed for science and conservation. Even the traits of organisms cannot be readily extracted from images. The analysis of traits, the integrated products of genes and environment, is critical for biologists to predict effects of environmental change or genetic manipulation and to understand the significance of patterns in the four billion year evolutionary history of life.
Knowledge-guided machine learning and computer vision can turn massive collections of images into high resolution information database about living organisms, enabling scientific discovery, conservation, and policy decisions. This is our vision of the new scientific field of imageomics.

10:30 - 11:00

Orals

Oral 1 by Mohannad Elhamod
Discovering Novel Biological Traits From Images Using Phylogeny-Guided Neural Networks

Room 'East 4'

Oral 2 by Valentin Gabeff
Scene and animal attributes retrieval from camera trap data with domain-adapted language-vision models
Oral 3 by Otto Brookes
PanAf20K: A Large Video Dataset for Wild Ape Detection & Behaviour Analysis

11:00 - 12:30

Poster Session 1

see poster assignment below

Room TBD

12:30 - 1:30pm

Lunch

1:30 - 2:00 pm

Keynote: Katija Kakani

Title: Developing imaging technologies to search for, discover, and understand life in the deep sea

Room 'East 4'

Abstract: The deep sea is the largest habitable ecosystem on the planet and remains one of the least explored. Very little is known about deep sea inhabitants, their behavior, and the limits and drivers for their survival. Ecomechanics, the multidisciplinary research field of the mechanisms that underlie organismal interactions and survival within their environment, has proven largely successful in terrestrial fields and lab-based organismal systems, but has had limited applicability to deep sea animals. The reason for this deficit is largely due to the technological challenges to access this environment for study, and the typically short duration of any observations, precluding documentation of many critical behaviors. Here we present a number of developments from MBARI’s Bioinspiration Lab that can be used to quantify biodiversity, biomechanics, and behavior of animals in the deep sea. All powered by computer vision, these developments include: two 4000 m-rated, ROV-based imaging systems DeepPIV and EyeRIS used to quantify time-resolved particle fields and structures in 2D and 3D, a 1500 m-rated AUV-based plenoptic imaging system Chiton, an underwater labeled image database called FathomNet that can be used to fuel machine learning algorithm development, and machine learning-integrated vehicle control algorithms called ML-Tracking that enable long-duration observations of individual animals. These instruments and approaches can be applied to a wide range of science use cases, both in the water column and on the seafloor.

2:00 - 2:30pm

Keynote: Devis Tuia

Title: Remote sensing enables monitoring life under water

Room 'East 4'

Abstract: The oceans are a major source of biodiversity and food. The preservation of the quality of the foodchain, as well as of life underwater in general is a primary
concern for society. However, observing water media routinely is a complex task that remote sensing technology empowered by AI can unlock. In this talk, I will present some recent researches, where a deep learning system, based on consumer grade cameras, is
deployed to 3D reconstruct and map coral ecosystems at scale.

2:30 - 3:00pm

Keynote: Albert Ali Salah

Title: AI and Animal Wellbeing

Room 'East 4'

Abstract: Rapid progress in computer based automation approaches creates new opportunities and challenges for animal wellbeing. On the one hand, innovative applications are being implemented to assist pet owners, people who work with animals, veterinarians, and for tasks that require monitoring of animals in different settings. On the other hand, we see challenges similar to those in automatic human behavior analysis, where automation may cause us to lose the human touch, and ethical issues and risks ensue from the potential usage of such applications, both in the short and in the long run. In this talk, I would like to draw parallels between computer based human and non-human animal behavior analysis, providing examples from our interdisciplinary work in equine and canine pain estimation, as well as other work at the newly established AI and Animal Wellbeing Laboratory at Utrecht University, to raise a number of issues and to discuss both technical and humanity challenges of the field.

3:00 - 3:20 pm

Orals

Oral 4 by Luke Meyers
Towards Automatic Honey Bee Flower-Patch Assays with Paint Marking Re-Identification

Room 'East 4'

Oral 5 by Medha Sawhney
Detecting and Tracking Hard-to-Detect Bacteria in Dense Porous Backgrounds

3:20 - 3:45 pm

Afternoon coffee

3:45 - 4:15 pm

Panel discussion

Room 'East 4'

4:15 - 5:45 pm

Poster Session 2

see poster assignment below

Room TBD

Event / 18 Jun 2023