Research

[Haptic Audio Production Tools]

Digital Audio Workstations (DAWs) are highly visual interfaces, and I’m currently working on leveraging widely available haptic interfaces (e.g. smartphones) to make audio production tools for people with visual impairments.

Deep Learning Tools for Audacity

We provide a software framework that lets deep learning practitioners easily integrate their own PyTorch models into the open-source Audacity DAW. This lets ML audio researchers put tools in the hands of sound artists without doing DAW-specific development work.

Invited talk at Bay Innovative Signal Hackers Meetup (2021)

Leveraging Hierarchical Structures for Few-Shot Musical Instrument Recognition

In this work, we exploit hierarchical relationships between instruments in a few-shot learning setup to enable classification of a wider set of musical instruments, given a few examples at inference. See the supplementary code on github.

update: this work won the Best Paper Award at ISMIR 2021! :)

ISMIR 2021 Poster Video

Software

Audacity with Deep Learning

I am contributing a deep learning framework and a deep model manager that connects to HuggingFace to Audacity. This project was funded by a Google Summer of Code grant. Read the Work Product Summary.

audacitorch

PyTorch wrappers for using your deep model in Audacity, and sharing it with the community!

torchopenl3

A PyTorch port of the openl3 audio embedding (ported from the marl implementation).

Philharmonia Dataset

PyTorch dataset bindings for 14,000 sound samples of the Philharmonia Orchestra, retrieved from their website.  [github]