Roadmap for online image and video annotation software
New and improved online and in-field platforms for exploration, management and annotation of georeferenced images & video
Overview of the current annotation workflow:
Currently only a tiny proportion of the images and video that are collected in marine surveys are analysed. Transforming underwater imagery into quantitative information for science and policy decisions, requires substantial effort by human experts. Different groups tend to handle the collected data in different ways, using different sampling techniques, different annotation tools and even referring to the same taxa by different names. The lack of established channels for data processing often results in significant lags between data collection and scientific discovery. The decentralised, unstandardised nature of existing methods makes comparison across disparate sites difficult, resulting little data re-use and limited collaboration. Having a global repository of data that has been annotated in a consistent format opens up the possibility of answering ‘big picture’ questions and provides opportunities to access large data sets for training machine learning algorithms. These algorithms ultimately have the potential to provide a scalable, cost-effective and collaborative environment for dealing with huge volumes of seafloor imagery and video through input from ecologists, biologists, experts, citizen scientists and machine learning algorithms.
Development & Support
The information below summarises the core features of the base platform:
Flexible data storage
Requiring that all data is available/uploaded to the centralised web server poses a barrier for adding new data from other sources, duplicating data that is often already available elsewhere online. Many national marine observing programs (for example IMOS through the Australian Ocean Data Network (AODN), or the Marine Geoscience Data System (MGDS) in the USA), are mandated to put data online in an openly accessible location. These distributed data storage facilities should be leveraged to reduce data duplication and inconsistencies, and will also mean that data can be made readily available much more quickly. Using the a framework for interpreting flexible meta data formats, it takes minutes (instead of days) to import datasets and get them ready for detailed annotation.
Flexible annotation schemes
There is no one size fits all when it comes to annotation schemes. Many have tried and failed to create an annotation scheme that suites the needs of all potential users, and now we have a variety of competing standardised schemes that exist. Our approach is to provide a flexible annotation system which allows users to annotate data using their annotation scheme of choice (whether selected from existing standard schemes or a new custom one). And instead of enforcing a single scheme, we will provide the capability to translate between different annotation schemes, which means that data labeled under one scheme can be viewed under another making sure that all annotated data is maintained in a consistent format. In addition, it will allow multiple labels per point as well as apply additional tags and free-form comments per annotation.
Collaborative / automated labeling
Much of our scientific understanding of benthic environments ultimately depends on human interpretation of seafloor imagery. Traditional approaches, largely reliant on manual annotation of a small subsets by human experts, will not scale to the increasing demand for quantitative understanding of marine habitats for science and regulatory compliance, nor to the increasing volume of seafloor images. This platform seeks to enable and manage collaborative human-machine labelling invoking human experts, citizen scientists and machine learning algorithms to achieve validated accuracy and predictable time frames.
"Media object" annotation
The new system will enable the same consistent labels to be applied to different media objects (images, video and large-scale mosaics). It will also offer the capability for defining validation sets based on other annotation sets for assessing annotation quality. There is a widespread global need for a flexible web-based video annotation tool. Using pre-existing video hosting technology, this is a relatively straightforward extension to the current platform.
In-field data annotation
Unannotated data from the field can be considered to be a liability in the sense that it often results in huge repositories of images and video that need to be assessed at a later time. Real-time annotation for video and stills using the same annotation interface running a local caching database that can be easily synchronised with the online system post cruise, would make it possible to better leverage time and resources in the field and help to reduce the “post processing debt”.
Education & outreach
With the advances in high bandwidth communications and social media, education & outreach activities have become commonplace on ocean-bound research cruises. It is possible to leverage the development effort in creating science tools to facilitate outreach goals, opening up the potential to acquire large volumes of crowd-sourced data that can compliment science objectives and engage the general public. We have already had some successes in this area.
More software features
Data importing and distributed data storage
- Streamlined ingesting of data from a variety of different platforms and sources.
- Distributed data storage system allows remotely hosted data to be annotated in the system without requiring copying of data to server.
Multiple media types with flexible metadata formats
- Generic media object including capabilities for images, videos and map-based visual mosaics.
- Support for metadata including georeferenced information, as well as any other related sensor information.
- Simplified, responsive map-based data exploration interface.
- Map-based dataset selection and querying tools
Dataset management, sharing and collaboration
- Enhanced functionality for managing projects, collaboration and data sharing.
- Ability to control privacy and sharing with groups / individuals.
- Ability to define embargo periods on sensitive data.
Tag-based annotation API
- Implement customisable tag-based annotation scheme that encourages standardisation and enforces consistency without being restrictive.
- Manages configurable / standardised annotation schemes.
- Sub frame and whole frame annotations with freeform comments.
- Ability to apply additional tags per annotation.
- Multi-label annotation capability including the ability to perform label quality validation.
- Annotation scheme translation capability (data labeled using one scheme can be viewed under another).
Data visualisation, summaries and reporting tools
- Visualisation of summarised annotation data and statistics on a map.
- Detailed summaries and reports of annotation data for selected datasets.
- Aggregated data views and visualisations by translating annotated data across different projects into consistent high-level summaries.
Automated and crowd-sourced annotation
- Add an interface to machine learning algorithms.
- Manage training and provide machine-generated labels for new data.
- Implement active learning to request specific annotations from human experts.
- Implement a market for paid, validated annotations by workers (initially can be through something like Mechanical Turk)
- Model accuracy, speed and cost of the range of labelling resources available (experts, algorithms and general public acting as workers).
- Optimise allocation of tasks to labelling resources based on objectives and constraints (trade off time, accuracy and cost).
- Extend annotation system to enable annotations at timestamps within video.
- Initially integrate with existing video hosting and streaming services.
Citizen science projects
- Setup system and simplified UI for citizen science projects with tutorial interface.
In-field data storage and annotation system
- Offline database / local caching for in-field annotations
- Interfacing with local network storage/streaming for images and video.
- Easy syncing of offline data to online system upon cruise completion.