Animal calls expose what audio embeddings learn by accident
Audio foundation models appear to capture evolutionary structure in animal vocalizations without being trained for phylogeny, and the domain-specific models did not clearly win. The useful lesson is practical: test general embeddings before paying the tax for narrow pretraining work.
TL;DR: General audio foundation models can encode biological structure they were never trained to predict, so domain-specific pretraining should be treated as a hypothesis to test, not an automatic upgrade.
Can an audio model hear the tree of life?
The paper to read is “Phylogenetic signal in marine mammal and bird vocalizations captured by audio foundation models: the limited benefit of domain-specific pretraining”, posted on arXiv cs.AI and cs.LG.
The setup is clean. Take pretrained audio models, embed animal vocalizations, then ask whether distances in embedding space line up with evolutionary distances between species. Not “can the model classify a dolphin.” Not “can it detect a bird call.” A stranger question: does the representation carry phylogenetic signal that nobody explicitly asked it to learn?
The answer, at least in these datasets, is yes.
Across 32 marine mammal species from the Watkins Marine Mammal Sound Database, 1,754 recordings total, the models recovered strong signal inside the 26 cetaceans. CLAP and BEATs-bio both hit Mantel r = 0.82, AST hit r = 0.74, with p < 0.001. That is not subtle.
The control matters. Hand-crafted MFCC features, the old standby 105-dimensional acoustic feature set, found basically nothing: r = 0.040, p = 0.338. The result also survived projecting every model embedding down to 105 dimensions with PCA, so the win was not just “the neural model had more numbers.” And it survived a partial Mantel test controlling for dominant frequency, with partial Mantel r = 0.404 while keeping 97% of the variance explained. Translation: this is not only pitch wearing a lab coat.
Why didn’t the specialist models win?
This is the part I find more useful for builders.
The researchers repeated the analysis on 20 bird species using the Jetz et al. 2012 phylogeny, then added BirdNET, a classifier trained end-to-end on around 6,000 bird species. If domain matching were the whole game, BirdNET should have crushed the general models.
It did not.
AST reached r = 0.55. CLAP reached r = 0.52. BirdNET and BEATs-bio landed around r = 0.32 to 0.36.
That does not mean BirdNET is bad. It means “trained on the right domain” and “best representation for this downstream geometry task” are different claims. BirdNET is optimized for bird species classification. BEATs-bio is bioacoustic. Those objectives may encourage features that separate labels well, but not necessarily features that preserve the smooth evolutionary distances between species.
This is a recurring foundation model lesson. The pretraining target leaves fingerprints. Sometimes broad, messy training creates representations that transfer better than a narrow model with the right logo on the box.
What does this change for applied audio AI?
For me, the takeaway is not “general models always win.” That would be too easy, and wrong.
The takeaway is that embedding quality is task-shaped. If you care about retrieval, clustering, anomaly detection, similarity search, weak labeling, or scientific structure discovery, you should test the representation directly against that shape. Classification accuracy on a neighboring task is not enough.
This paper also gives a nice pattern for evaluation. The researchers did not ask a model to produce an impressive demo. They compared distance matrices. Embedding distance versus phylogenetic distance. Then they ran controls against simpler features, dimensionality, and dominant frequency. That is the bar I want more applied AI teams to copy: make the hidden claim measurable, then try to kill it.
There are limits. The datasets are not huge. The bird set is 20 species. The marine mammal result is strongest within cetaceans. Mantel tests can show correlation in distance structure, not causal understanding. Nobody should walk away saying these models “understand evolution.” They encode acoustic patterns that correlate with evolutionary relatedness. That is interesting enough without anthropomorphizing it.
Practitioner’s Take: If you are building an audio workflow, start with two or three general embedding models before commissioning a domain-specific model or fine-tune. Build a small evaluation set around the actual structure you need: nearest neighbors, cluster purity, temporal drift, expert-ranked similarity, or some external ground truth like taxonomy. The catch most teams miss is that the best classifier is not always the best embedding engine. Test the geometry, not the badge.