Ava: Turning Text into Synth Presets

Building a differentiable synthesizer and teaching a neural network to control it.

What if, instead of endlessly scrolling through synthesizer presets, you could simply describe the sound you wanted?

Ava is my attempt to make that possible: a subtractive synthesizer that turns text descriptions into editable presets. Under the hood, I built a differentiable synthesizer in PyTorch and trained a small neural network to predict its parameters from audio. I then used a pretrained text-to-audio model to generate training data, and brought everything together in a C++ plugin.

In this post, I'll walk through how I built it, the experiments that worked (and those that didn't), and some interactive demos you can play with yourself.

A Quick Listen

Before getting into the technical details, here's a taste of what Ava can do. For each text description, you can compare the generated reference audio with Ava's attempt to reproduce it using a synthesizer preset.

You'll find more examples and a playable synth further down the page.

Lo-fi piano

D♯5

“a warm lo-fi piano with a soft muted touch, gently swelling.”

Generated reference

Ava

Full prompt

A single isolated piano-inspired subtractive synthesizer note at D#5 (MIDI 75, 622.25 Hz). Desired timbre: a warm lo-fi piano with a soft muted touch, gently swelling. Dry and monophonic, triggered once, within a 6-second clip.

Concert flute

G♯4

“a clear bright concert flute with a smooth flowing tone, gently wavering.”

Generated reference

Ava

Full prompt

A single isolated flute-inspired subtractive synthesizer note at G#4 (MIDI 68, 415.30 Hz). Desired timbre: a clear bright concert flute with a smooth flowing tone, gently wavering. Dry and monophonic, triggered once, within a 6-second clip.

Explore all examples and try the synth →

1. Preamble

A few years ago, I got into music production as a hobby. I spent quite some time learning about sound design, although I never became particularly good at it. My usual workflow went something like this: I'd have a sound in mind, try to recreate it on my favorite synthesizer, fail, and start scrolling endlessly through presets. I'd find plenty of interesting sounds, but rarely the one I was looking for. Eventually, I'd get distracted, forget my original idea, and give up.

Around the same time, I became curious about how synthesizers actually worked. I started reading The Audio Programmer's book on building synthesizer plugins, and realized that much of a synthesizer is essentially a sequence of mathematical operations on audio signals, many of which can be made differentiable.

That got me thinking: what if I implemented a synthesizer in PyTorch, and used backpropagation to learn its parameters?

The original idea was sound-to-synth: given an audio sample, a neural network would predict the synthesizer parameters needed to reproduce it. By rendering audio through a differentiable synth and comparing it with the original, I could train the network end-to-end, using the synthesizer itself as the decoder.

But then I realized this wasn't quite the problem I wanted to solve. How often do you have a recording of the exact sound you're imagining? Most of the time, you can at least try to describe it: a warm, slightly detuned pad with a slow attack, or a bright, punchy pluck. And even if the result isn't perfect, having a reasonably close preset to start tweaking would already make sound design much easier.

So the idea evolved into text-to-synth. I initially considered using a joint audio-text embedding model like CLAP[1], but eventually decided to use pretrained text-to-audio models to generate paired training data.

Most of the project was developed on my MacBook M1 Pro, with A100 GPUs for the more expensive experiments.

And why Ava? It's named after the robot protagonist of Ex Machina, the movie that inspired me to study AI many years ago.

2. The Synth

For this project, I wanted something relatively simple, but still expressive enough to generate a good variety of sounds. So I went with a subtractive synthesizer: a synthesizer that starts with one or more waveforms and shapes the resulting sound using filters, envelopes, and modulation.

Ava follows a fairly standard architecture, illustrated in the interactive diagram below. You can click on each component to see what it does.

The signal path Open a block to see what it does.
Oscillator A

A sine, triangle, square or sawtooth wave at the played note's pitch.

Oscillator B

The same four waveform choices, with a detuning offset relative to oscillator A.

Oscillator mix

The mix parameter crossfades between the two oscillators.

White
noise
Noise blend

Noise mix blends white noise into the oscillator signal before filtering.

Resonant low-pass

Cutoff controls brightness; resonance emphasizes frequencies near the cutoff. The cutoff follows the played pitch. The LFO moves it over time, with its rate and depth setting the speed and amount.

LFO
rate, depth
Drive

Soft distortion adds harmonic content before the amplitude envelope is applied.

ADSR envelope

Attack, decay and sustain shape the held note's amplitude. Note-off starts the release.

Output gain

Gain sets the preset's level. MIDI velocity also scales each note's amplitude, independently of the predicted preset.

Audio output

The synth combines two oscillators, optionally mixed with white noise, and passes the signal through a resonant low-pass filter, drive, an ADSR envelope, and output gain. An LFO can also modulate the filter cutoff. The neural network predicts the controls for these components, including waveform selection, detuning, filter settings, and envelope parameters.

One important design decision was to exclude pitch and velocity from the predicted parameters. These are normally provided through MIDI, independently of the synth preset. Besides, estimating pitch from audio is already a challenging problem on its own, and I didn't want the model to have to solve both tasks simultaneously.

Instead, the network predicts only the preset parameters, while the synth receives pitch, velocity, and note timing separately. This simplifies the learning problem, but requires the corresponding MIDI information for each training sample. I'll explain later how I handled that.

I intentionally kept the synthesis engine fairly limited. The goal wasn't to build a competitor to commercial synthesizers, but to see how much a simple synth controlled by a neural network could achieve.

Deep Dive: Differentiable Synthesis in PyTorchOptional · open to read

The idea behind this project is to use a synthesizer as a differentiable decoder. Given a set of preset parameters \theta and MIDI information m, the synth produces a waveform:

\hat{x}=S(\theta,m)

Here, \theta and m are inputs to the synth; the synthesizer itself has no trainable weights. It's a collection of signal-processing operations. Implementing them in PyTorch allows gradients to propagate through the entire signal chain.

These gradients can be used to optimize the parameters of a single sound directly, or train a neural network to predict them from audio. Most synth operations are straightforward to differentiate, but some, particularly waveform selection and recursive filtering, require a bit more care.

Oscillators and waveform selection

Let's start with the oscillators. Each produces a periodic waveform at a frequency determined by the MIDI pitch. Ava supports sine, triangle, square, and sawtooth waves.

The simplest is a sine wave:

x(t)=\sin(2\pi ft)

where f is the frequency in hertz and t is time in seconds.

For digital audio, this function is evaluated at discrete time steps. A sample rate of 24 kHz means 24,000 samples per second, so consecutive samples are separated by 1/24000 seconds.

For a recording containing N samples, the time vector is:

t_n=\frac{n}{f_s},\qquad n=0,1,\ldots,N-1

where f_s is the sample rate.

From synth/synth.py, with explanatory comments

# Sample indices 0, 1, ..., N-1 become times in seconds after division.
time = torch.arange(round(duration * self.sample_rate),
                    device=freq.device, dtype=torch.float32) / self.sample_rate

Evaluating the equation at every time step produces the sine wave:

From synth/oscillators/sine.py, with explanatory comments

def __call__(self, freq, time):
    # freq: [batch, 1]; time: [samples].
    # Broadcasting evaluates every time point at each batch element's frequency.
    # 2*pi*freq*time is the phase in radians; initial phase is zero.
    return torch.sin(2 * torch.pi * freq * time)  # [batch, samples]

Multiplying time by frequency gives the number of completed cycles, while multiplying by 2\pi converts that into radians. At 200 Hz and 24 kHz, each cycle takes 120 samples.

PyTorch broadcasting also makes it possible to generate several waveforms at different frequencies in parallel, using the same time vector.

Building more complex waveforms

Square, triangle, and sawtooth waves can be constructed by adding sine waves at multiples of the fundamental frequency, known as harmonics.

For example, a square wave contains only odd harmonics, with amplitudes decreasing inversely with their harmonic number:

x_{\mathrm{square}}(t)\approx\frac{4}{\pi} \left[ \sin(2\pi ft) +\frac{\sin(2\pi 3ft)}{3} +\frac{\sin(2\pi 5ft)}{5} +\cdots \right]

Starting with a sine wave, adding the third harmonic begins to flatten the peaks. Adding the fifth, seventh, and subsequent odd harmonics makes the transitions progressively sharper, approaching the familiar square-wave shape.

Start with the fundamental, f
Square-wave approximation using odd harmonics through 1
Add the third harmonic, 3f
Square-wave approximation using odd harmonics through 3
Add the fifth and seventh
Square-wave approximation using odd harmonics through 7
Continue through the 31st
Square-wave approximation using odd harmonics through 31
Solid green shows the running sum; dashed brown shows the new components in the middle two panels. All panels use the same amplitude scale and the synth's 4/π normalization. Two cycles at 200 Hz, sampled at 24 kHz.

In Ava, the square and triangle oscillators use odd harmonics up to the 31st, while the sawtooth uses all harmonics up to the 31st. The implementation also excludes harmonics at or above the Nyquist frequency (half the sample rate) to reduce aliasing.

These are finite approximations, which explains the small ripples visible in the square wave.

The harmonics can all be calculated at once using PyTorch broadcasting. For a batch of B sounds, N audio samples, and H harmonics, the calculation produces a tensor of shape [B, H, N], which is then summed along the harmonic dimension.

From synth/oscillators/square.py, with explanatory comments

# Odd harmonic numbers: 1, 3, ..., 31. Initially [H].
harmonics = torch.arange(1, 32, 2, device=freq.device, dtype=freq.dtype)
harmonics = harmonics.view(1, -1, 1)  # [1, H, 1]
freq = freq.unsqueeze(1)             # [B, 1, 1]
time = time.unsqueeze(-2)            # [1, N], broadcast as [1, 1, N]
allowed = freq * harmonics < self.sample_rate / 2  # [B, H, 1]
# One sine wave per harmonic, evaluated at every sample time.
waves = torch.sin(2 * torch.pi * freq * harmonics * time)  # [B, H, N]
# Mask, weight by 1/harmonic, then sum H to produce [B, N].
return 4 / torch.pi * (allowed * waves / harmonics).sum(dim=1)

Here are the four resulting waveforms generated by Ava:

Sine
Two cycles of the sine oscillator at 200 Hz
Triangle
Two cycles of the triangle oscillator at 200 Hz
Square
Two cycles of the square oscillator at 200 Hz
Sawtooth
Two cycles of the sawtooth oscillator at 200 Hz
Two cycles at 200 Hz, rendered by the actual oscillator classes at 24 kHz. The non-sine waves use finite harmonic sums.

Making waveform selection differentiable

With all four waveforms available, how can a neural network choose between them?

Waveform selection is discrete: each oscillator must select one of four possibilities. A hard selection using argmax would prevent gradients from flowing back through the choice.

A softmax-weighted combination of the four waveforms provides a differentiable alternative:

x(t)=\sum_{k=1}^{4}p_kx_k(t), \qquad p_k=\operatorname{softmax}(z)_k

where z contains the predicted logits, p_k the selection probabilities, and x_k(t) the candidate waveforms.

From synth/oscillators/bank.py

# Keep the four candidate waveforms on a separate axis: [batch, 4, samples].
waves = torch.stack([osc(freq, time) for osc in self.oscillators], dim=1)
# selection: [batch, 4] -> [batch, 4, 1], sharing each weight across time.
# Sum the waveform axis to return [batch, samples].
return (selection.unsqueeze(-1) * waves).sum(dim=1)

In the code, the order is sine, square, sawtooth, triangle. For example, weights of [1, 0, 0, 0] select a pure sine wave, while [0.5, 0.5, 0, 0] mix sine and square equally.

This makes selection differentiable, but introduces a mismatch: during training, the oscillator can blend several waveforms, while the final plugin uses a single discrete selection.

I also experimented with a straight-through estimator (STE), which uses a hard selection during the forward pass but approximates its gradients using the softmax.

# Turn four logits into probabilities, then make a one-hot hard choice.
probability = raw.softmax(dim=-1)
hard = torch.nn.functional.one_hot(probability.argmax(-1), 4).to(probability)
# The bracket is zero in the forward pass, but carries softmax gradients.
controls[name] = hard + (probability - probability.detach())

While the STE better matches the behavior of the plugin, I found it considerably less stable during optimization. Soft selection generally worked better, so I kept it for training.

Detuning and mixing

Both oscillators start from the MIDI note's frequency, but the second can be detuned slightly. Detuning is measured in cents, with 100 cents per semitone and 1,200 per octave.

A detuning of c cents changes the frequency according to:

f_B=f_A\,2^{c/1200}

When two oscillators have slightly different frequencies, their interference creates a slow beating effect, making the sound feel richer and more animated.

The oscillator mix parameter then crossfades between them: zero selects only oscillator A, one selects only B, and 0.5 gives equal contributions.

From synth/synth.py, with explanatory comments

# Convert the cents offset to a frequency ratio for oscillator B.
freq_b = freq * 2 ** (detune_cents / 1200)
wave_b = self.oscillators(freq_b, time, oscillator_b)
# wave is oscillator A here; mix controls their relative contribution.
wave = (1 - mix) * wave + mix * wave_b

Finally, white noise can be mixed into the oscillator signal before filtering.

ADSR envelope

So far, the oscillators generate continuous periodic signals. Turning these into musical notes requires control over how their amplitude changes over time.

This is the job of the ADSR envelope: attack raises the amplitude, decay lowers it toward the sustain level, and release brings it back to zero after note-off.

Shape the envelopeThe dashed line marks note-off. Try releasing the note before the attack or decay finishes.
ADSR amplitude over a six-second clip00.510123456 s

Ava uses exponential curves for these transitions. The following function maps progress through a stage, from zero to one, into a curved transition:

From synth/envelopes/adsr.py, with explanatory comments

def curve(self, progress):
    # Before a stage starts, progress is 0; after it ends, progress is 1.
    progress = progress.clamp(0, 1)
    # self.curvature = 4.0. Normalize the exponential to run from 0 to 1.
    curve = -torch.expm1(-self.curvature * progress) / -math.expm1(-self.curvature)
    # Make completed transitions exactly 1, including in finite precision.
    return torch.where(progress >= 1, torch.ones_like(curve), curve)

The curvature is fixed at 4, giving a relatively fast initial transition that gradually slows toward its endpoint. At halfway through the stage, the curve has already completed about 88% of the transition.

The same curve handles both attack and decay, with different durations and amplitude ranges. During attack, the level rises from zero to one. During decay, it falls from one toward the sustain level.

From synth/envelopes/adsr.py, with explanatory comments

def held_level(self, time, attack, decay, sustain):
    attack_level = self.curve(time / attack)  # Progress through attack.
    # Decay begins at time = attack, and drops by (1 - sustain).
    decay_level = 1 - (1 - sustain) * self.curve((time - attack) / decay)
    return torch.where(time < attack, attack_level, decay_level)

Release requires a little more care. A note might end while the attack or decay is still in progress, so release doesn't always start from the sustain level.

Instead, the envelope is evaluated at the exact note-off time and fades from that value to zero.

From synth/envelopes/adsr.py, with explanatory comments

# Amplitude while the key is held, evaluated at every audio sample.
held = self.held_level(time, attack, decay, sustain)
# Evaluate the same function at note-off, even if attack/decay is unfinished.
level_at_release = self.held_level(note_press_duration, attack, decay, sustain)
# Fade that level to zero over the release duration.
tail = level_at_release * (1 - self.curve((time - note_press_duration) / release))
return torch.where(time < note_press_duration, held, tail)

This avoids discontinuities when releasing notes early, while keeping the envelope differentiable with respect to its parameters.

The MIDI input determines when the note starts and stops. The predicted envelope parameters determine how the amplitude evolves around those events.

Low-pass filter and LFO modulation

The filter was probably the most interesting, and challenging, component to implement.

Ava uses a second-order resonant low-pass filter, also called a biquad filter. It attenuates frequencies above a cutoff, while the resonance parameter Q controls the emphasis around that cutoff.

A filter, one sample at a time

Unlike the oscillators, a biquad filter is recursive: its output depends not only on the current and previous input samples, but also on previous output samples.

Let's call the input waveform x[n] and the filtered waveform y[n], where n identifies the audio sample.

The filter follows this equation:

\begin{aligned} a_0y[n]={}&b_0x[n]+b_1x[n-1]+b_2x[n-2]\\ &-a_1y[n-1]-a_2y[n-2] \end{aligned}

The b_i coefficients weight the current and previous input samples. The coefficients a_1 and a_2 weight previous outputs, while a_0 scales the current output.

For example, x[n-1] is the input one sample ago, while y[n-2] is the filtered output two samples ago.

The coefficients determine the filter's behavior and are calculated from the cutoff frequency and resonance.

Calculating and applying the filter

In Ava, the cutoff follows the MIDI pitch, expressed as an octave offset from the note frequency. This helps preserve a similar filtering effect when playing different notes.

Given a cutoff frequency f_c, sample rate f_s, and resonance Q, the calculation starts with:

\omega=\frac{2\pi f_c}{f_s}, \qquad \alpha=\frac{\sin\omega}{2Q}

The low-pass filter coefficients are then:

\begin{aligned} b_0=b_2&=\frac{1-\cos\omega}{2}, &b_1&=1-\cos\omega\\ a_0&=1+\alpha, &a_1&=-2\cos\omega, &a_2&=1-\alpha \end{aligned}

All coefficients are normalized by a_0, so the recurrence can be written with a_0=1.

From synth/filters/lowpass.py, with explanatory comments

# Convert cutoff Hz to radians per sample.
omega = 2 * torch.pi * cutoff_hz / self.sample_rate
cosine = torch.cos(omega)
alpha = torch.sin(omega) / (2 * q)
# Pack [a0, a1, a2] and [b0, b1, b2] for each cutoff/Q pair.
a = torch.cat([1 + alpha, -2 * cosine, 1 - alpha], dim=-1)
b = torch.cat([(1 - cosine) / 2, 1 - cosine, (1 - cosine) / 2], dim=-1)
# Divide every coefficient by a0, so the new a0 is 1.
return a / a[:, :1], b / a[:, :1]

When the cutoff and resonance remain constant, these coefficients only need to be calculated once. The recursive filter can then be applied using torchaudio.functional.lfilter.

From synth/filters/lowpass.py, with explanatory comments

# Compute one coefficient set per batch element from the fixed cutoff and Q.
batch = max(audio.shape[0], cutoff_hz.shape[0])
a, b = self.coefficients(cutoff_hz.expand(batch, 1), q.expand(batch, 1))
# lfilter evaluates the recurrence using these coefficients.
filtered = lfilter(audio.double().expand(batch, -1), a, b,
                   clamp=False, batching=True)
return filtered.to(audio.dtype)

The interesting part is that PyTorch can still propagate gradients through the filtering operation, even though each output sample depends on previous outputs.

Adding LFO modulation

The cutoff can also move over time.

An LFO (low-frequency oscillator) adds a periodic modulation to the cutoff. Its rate determines how quickly the cutoff changes, while its depth controls how far it moves.

With the cutoff expressed in octaves relative to the MIDI pitch, the formula becomes:

f_c(t)=f_{\mathrm{MIDI}}\, 2^{o_{\mathrm{cutoff}}+d\sin(2\pi rt)}

where o_{\mathrm{cutoff}} is the base cutoff offset, d the modulation depth in octaves, and r the LFO rate.

This means the filter coefficients are no longer constant: they need to change as the LFO moves the cutoff.

The Python synth defaults to coefficient updates at 200 Hz, once every 5 milliseconds. At 24 kHz, that corresponds to one update every 120 samples. The rate is configurable: for the datasets and audio-training experiments in this post, I used sample-wise updates at 24 kHz, as does the C++ plugin.

Between updates, the filter continues processing audio at the full sample rate using the current coefficients.

At each update, the modulated cutoff is calculated and converted into a new set of filter coefficients:

Simplified from synth/synth.py and synth/filters/lowpass.py; batching omitted

# For one update time; the full implementation batches all update times.
offset = torch.as_tensor(cutoff_octaves, device=freq.device, dtype=torch.float64)
# Add the LFO's octave offset to the preset's base offset.
offset = offset + self.lfo(time, rate, depth)
cutoff_hz = freq.double() * 2 ** offset
# Recalculate a and b using exactly the same coefficient function as before.
a, b = self.lowpass.coefficients(cutoff_hz, q)

torchaudio.functional.lfilter expects fixed coefficients for each signal. For the changing cutoff, Ava first expands each coefficient set over the samples where it applies. It computes the input-weighted part with vectorized PyTorch operations, then uses TorchLPC's sample_wise_lpc for the recursive part:

From synth/filters/lowpass.py, with the numerator split across lines

# a and b contain one coefficient set per sample: [batch, samples, 3].
padded = torch.nn.functional.pad(x, (2, 0))
numerator = (b[..., 0] * x
             + b[..., 1] * padded[:, 1:-1]
             + b[..., 2] * padded[:, :-2])
filtered = sample_wise_lpc(
    numerator.contiguous(), a[..., 1:].contiguous())

This is one filtering pass over the whole clip, so the filter's history continues across coefficient updates rather than restarting at each one.

The important part is that the complete process remains differentiable, allowing gradients to flow back through the filter coefficients to the cutoff, resonance, LFO rate, and depth.

Drive, gain, and putting everything together

The final components are relatively simple.

Drive adds harmonic distortion using a normalized tanh nonlinearity:

y=\frac{\tanh(dx)}{\tanh(d)}

where d is the drive amount. The zero-drive case is handled separately as y=x, which is also the limit as d approaches zero.

The ADSR envelope then shapes the amplitude, while output gain and MIDI velocity control the final level.

With everything connected, the synthesizer is just a differentiable function mapping MIDI information and preset parameters to audio. Rendering a sound and computing a loss makes it possible to backpropagate through the entire signal chain.

Of course, being differentiable doesn't guarantee that the parameters are easy to optimize. Different parameter combinations can produce similar sounds, and the loss landscape may contain local minima. That's what I wanted to investigate next.

3. Optimizing Synth Parameters Directly

Before training a neural network, I wanted to check whether the synth parameters could actually be recovered through gradient descent.

The experiment was simple: I chose a preset, generated a reference sound, and then tried to recover its parameters by minimizing the difference between the reference and reconstructed audio. I started by optimizing one parameter at a time, keeping the others fixed at their ground-truth values, before moving to multiple parameters.

For now, I used the mean squared error (MSE) between the two waveforms:

\mathcal{L}_{\mathrm{MSE}}=\frac{1}{T}\sum_{t=1}^{T}\left(x_t-S(\theta,m)_t\right)^2

Here, x is the reference audio, S(\theta,m) is the reconstructed waveform, and T is the number of audio samples.

Why use MSE for audio?

Waveform MSE is generally not a great measure of perceptual similarity. It's particularly sensitive to phase: two signals can sound almost identical but have a large MSE if their waveforms are slightly misaligned.

Two sine waves of amplitude 0.5 at 1 kHz, with a 90-degree phase difference corresponding to six samples at 24 kHz.

Sine · 0°

Sine · 90°

Same pitch and level, different phase: at 1 kHz, six samples at 24 kHz correspond to a 90° offset. The waveform MSE is about 0.25, although the tones should sound the same when played separately.

This experiment is a little different, though. Both sounds come from the same deterministic synthesizer, with identical MIDI inputs, oscillator starting phases, and noise realization. A parameter configuration producing exactly the same waveform therefore exists.

This doesn't make optimization easy, but it makes MSE a reasonable starting point for testing whether the gradients are useful.

Recovering the parameters

The results were encouraging: for the tested preset, every parameter could be recovered in at least one run. However, convergence depended on the parameter and its initialization.

The following plots show three representative examples, each tested with different random seeds.

Seed 123 · solid Seed 456 · dashed Seed 789 · dotted

Gain

Target: 0.5

Gain-only fitting: all three seed groups converge to near-zero waveform MSE.

Recovered:
0.5 / 0.5 / 0.5

Detuning

Target: 12 cents

Detuning-only fitting: seeds 123 and 789 recover 12 cents, while seed 456 stalls at 15.592 cents with MSE about 0.02235.

Recovered (cents):
12 / 15.592 / 12

LFO rate

Target: 2 Hz

LFO-rate-only fitting: seed 123 recovers 2 Hz; seeds 456 and 789 stall around 1.708 Hz with MSE about 0.001084.

Recovered (Hz):
2 / 1.708 / 1.708

All three seed groups are plotted, including the failures. Some curves overlap: the two failed LFO-rate runs settle at the same loss. Each curve shows the lowest MSE among four random restarts at each step; recovered values come from the best saved iterate.

Gain was relatively easy to recover, with all three seed groups converging to the correct value of 0.5.

Detuning was harder. Two seed groups recovered the target of 12 cents, while the third got stuck around 15.6 cents.

The LFO rate was even more challenging: only one seed group recovered the correct 2 Hz, while the other two converged to approximately 1.708 Hz.

In fact, detuning and LFO rate were the only parameters that failed in the single-parameter tests; all the others were recovered across all three seed groups. A separate scan of the LFO-rate loss confirmed a local minimum around 1.708 Hz, explaining why optimization repeatedly got stuck there.

Things become even more complicated when optimizing several parameters together, since changing one parameter can sometimes compensate for changes in another.

I also compared soft waveform selection with the straight-through estimator introduced earlier. Soft selection was considerably easier to optimize, although the final saved presets still use discrete oscillator choices.

What did I learn?

These experiments confirmed that gradients through the synthesizer are useful, but also showed that differentiability doesn't guarantee successful optimization. Even when the target sound can be reproduced exactly, gradient descent can get stuck in local minima.

There's also an important distinction between recovering the original parameters and reproducing the original sound. Different parameter combinations can produce similar audio, so the two objectives aren't necessarily equivalent.

Rather than optimizing a new preset from scratch for every recording, the next step was to train a neural network to predict the parameters directly from audio.

4. Learning from Synth-Generated Audio

Before trying to reconstruct sounds from a text-to-audio model, I wanted to start with something simpler: can a neural network recover the parameters of sounds generated by Ava?

Ava can generate as much training data as needed, with the original parameters available as ground truth. This makes it possible to train the network directly on the parameters, without the computational cost of rendering audio at every training step.

More importantly, it's a useful first experiment to see whether the audio representation contains enough information to recover the synth controls.

Generating and Encoding the Data

I generated 100,000 training examples, 10,000 validation examples, and 10,000 test examples, each using a randomly sampled preset. Pitches were sampled uniformly from MIDI 21–78, with fixed velocity. Each recording lasts six seconds at 24 kHz: four seconds with the key held, followed by two seconds for the release tail.

The parameters were sampled within predefined ranges, using log-uniform distributions for envelope times, resonance, and LFO rate, and linear-uniform distributions for the remaining continuous controls. I also balanced the 16 possible oscillator waveform combinations across the dataset.

Rather than training a model directly on raw waveforms, I decided to use a pretrained audio encoder to obtain more compact representations. Besides reducing the training cost, this would also let me reuse an existing text-to-audio generative model later on.

I initially experimented with EnCodec[2] and Music2Latent[3], before settling on SAME-S, the autoencoder used by Stable Audio 3 Small SFX[4]. These weren't controlled comparisons, but SAME-S had one particularly useful advantage: an available text-conditioned model that generates audio in the same latent space.

For these six-second recordings, SAME-S produces a sequence of 66 latent vectors, each containing 256 values. I encoded the dataset once using the frozen encoder and saved the latents alongside the original presets.

Preset + MIDISample controls and pitch
PyTorch synthRender the note
Audio6 seconds · 24 kHz
SAME-SFrozen encoder
Latents66 frames × 256 values

Original preset → 13 continuous targets + two waveform labels

The latents become the network input; the preset supplies the supervision. MIDI is saved for rendering and target conversion, but isn't an additional network input.

I ran this entire stage, from data generation to supervised training, on my MacBook M1 Pro.

Predicting the Synth Parameters

For the prediction network, I used a small Transformer with around 440,000 parameters. It processes the sequence of audio latents and predicts 13 continuous controls and two discrete waveform selections.

The continuous outputs are bounded between 0 and 1 using sigmoid activations, then mapped to their actual parameter ranges by the synthesizer. For the two oscillators, the network predicts four logits each, corresponding to the available waveforms.

For training, I used MSE for the normalized continuous parameters and cross-entropy for the waveform selections. The two losses are combined as:

\begin{aligned} \mathcal{L}_{\mathrm{param}} &= \frac{1}{13}\sum_{i=1}^{13}(\hat{u}_i-u_i)^2\\ \mathcal{L} &= \mathcal{L}_{\mathrm{param}}+ 0.01\,\frac{\mathrm{CE}_A+\mathrm{CE}_B}{2} \end{aligned}

where u_i and \hat{u}_i are the ground-truth and predicted normalized parameters. All continuous parameters are weighted equally, while the waveform classification term receives a weight of 0.01. Both loss terms are also averaged over the batch.

No audio needs to be rendered during training: the network simply learns to map audio latents to the parameters that originally generated them.

How Well Does It Work?

I trained the model for 200 epochs, using an exponential moving average (EMA) of the weights for evaluation, and selected the checkpoint with the lowest validation parameter MSE.

On the 10,000 test examples, the model reached a normalized parameter RMSE of 0.0488. Waveform classification accuracy was 92.7% for oscillator A and 93.2% for B, with both correctly predicted in 86.6% of the examples.

Interestingly, around 84% of waveform classification errors involved confusing sine and triangle waves.

Training and validation losses over 200 epochs. Parameter MSE continues to improve; validation cross-entropy reaches its minimum around epoch 99 and rises later. A dotted line marks the selected checkpoint at epoch 196.
Training uses live weights with dropout; validation uses EMA weights in evaluation mode. Classification loss rises later in training, even as parameter MSE improves.

Not all parameters were equally easy to recover, as shown below.

Test normalized parameter RMSE: LFO rate 0.1012, Decay 0.0839, Resonance (Q) 0.0585, Detuning 0.0458, LFO depth 0.0445, Oscillator mix 0.0418, Noise mix 0.0372, Drive 0.0333, Sustain 0.0282, Cutoff 0.0209, Attack 0.0196, Gain 0.0153, Release 0.0114.
Test-set RMSE in the network's normalized control coordinates, before conversion to physical units. Lower is better. Results use the EMA checkpoint selected at epoch 196.

LFO rate and decay were the hardest parameters, while release, gain, attack, and cutoff were recovered more accurately.

The scatter plots below show how predictions compare with their targets across the entire test set.

LFO rate: target versus predicted normalized values for all 10,000 test examples. The dashed diagonal marks exact recovery.
Choose a control to see all 10,000 test predictions. Both axes use normalized values; the dashed diagonal marks exact recovery. Some target ranges are narrower than [0,1] because of the bounds used in data generation.

These results are encouraging, although it's difficult to tell how much of the remaining error comes from the audio representation, the prediction network, or the ambiguity of the parameters themselves.

For example, a high filter cutoff might have very little effect on a sound with almost no high-frequency content, making its exact value difficult to infer. This is one reason why recovering the original preset isn't necessarily the same as reproducing the sound accurately.

Listening to the Reconstructions

But how do these parameter errors translate into audio?

Below are four examples from the held-out test set, comparing the original sounds with reconstructions generated using the predicted parameters. Both versions use the same MIDI note and timing.

Example 1 · A0

Original

Predicted preset

Compare preset values
ParameterOriginalPredicted
Oscillator ATriangleTriangle
Oscillator BSquareSquare
Gain0.3380.345
Mix0.5920.572
Detuning (cents)9.477.43
Noise mix0.0170.105
Cutoff (octaves)1.632.02
Resonance (Q)1.791.4
LFO rate (Hz)11.18
LFO depth (octaves)0.3590.344
Drive2.262.49
Attack (s)0.1920.196
Decay (s)0.07730.0864
Sustain0.8330.822
Release (s)0.6230.642

Example 2 · D3

Original

Predicted preset

Compare preset values
ParameterOriginalPredicted
Oscillator ASineSine
Oscillator BSawtoothSawtooth
Gain0.3510.334
Mix0.6740.688
Detuning (cents)19.318.9
Noise mix0.120.133
Cutoff (octaves)1.821.8
Resonance (Q)1.521.36
LFO rate (Hz)2.372.66
LFO depth (octaves)0.6320.632
Drive2.352.96
Attack (s)0.02260.0233
Decay (s)0.1610.303
Sustain0.730.745
Release (s)0.2110.242

Example 3 · C5

Original

Predicted preset

Compare preset values
ParameterOriginalPredicted
Oscillator ATriangleTriangle
Oscillator BTriangleTriangle
Gain0.2860.238
Mix0.7480.729
Detuning (cents)16.116.9
Noise mix0.03430.0239
Cutoff (octaves)2.062.06
Resonance (Q)0.7120.72
LFO rate (Hz)1.231.2
LFO depth (octaves)0.6650.573
Drive0.4550.931
Attack (s)0.08910.0782
Decay (s)0.4390.509
Sustain0.6670.744
Release (s)0.1430.128

Example 4 · F3

Original

Predicted preset

Compare preset values
ParameterOriginalPredicted
Oscillator ASineSine
Oscillator BSawtoothSawtooth
Gain0.7210.743
Mix0.3420.372
Detuning (cents)24.123.1
Noise mix0.1590.122
Cutoff (octaves)2.232.12
Resonance (Q)0.630.69
LFO rate (Hz)3.134.68
LFO depth (octaves)0.1190.237
Drive3.983.24
Attack (s)0.1120.114
Decay (s)0.7410.872
Sustain0.4250.426
Release (s)1.11.22

The reconstructions are generally close, although the parameter tables reveal some noticeable differences. In the fourth example, for instance, both oscillator waveforms are predicted correctly, but the LFO rate is estimated at 4.68 Hz instead of 3.13 Hz.

Overall, this experiment suggests that the pretrained audio representation contains enough information to recover the synth controls reasonably well. But direct parameter supervision is only possible because I generated the data myself.

Sounds produced by an external model have no known synth parameters. Instead, training needs to be based on how closely the synthesizer reproduces the target audio.

5. Training with an Audio Loss

So far, I've trained the network to recover the original synth parameters. But sounds that weren't generated by Ava have no ground-truth parameters to supervise the model with.

Instead, training needs to compare the audio produced by the predicted preset with the target recording.

I first tested this on the synth-generated dataset, where an exact reconstruction is possible. This provided a controlled setting for evaluating audio-loss training before moving to text-generated sounds.

Moving to a Spectral Loss

As discussed earlier, waveform MSE is very sensitive to phase and temporal alignment. For reconstructing arbitrary audio, comparing sounds in the frequency domain is often more useful.

The short-time Fourier transform (STFT) splits a waveform into overlapping windows and measures its frequency content over time. Comparing the magnitudes of these spectrograms reduces sensitivity to phase differences while retaining information about how the sound evolves.

The window size introduces a trade-off: short windows capture transients more precisely, while longer windows provide better frequency resolution, which is particularly useful for distinguishing nearby frequencies. A multi-resolution STFT loss combines both.

A 500 Hz band-limited square wave and a quarter-cycle-shifted copy. Waveform samples differ, but their harmonic patterns remain similar in 128-sample and 2,048-sample STFTs and a 32-band log-mel spectrogram. Each row includes the measured distance to the reference.
The same 500 Hz square wave, shifted by 90°. The waveforms differ, but their harmonic patterns stay similar in the spectral views, particularly with longer windows. Waveforms are zoomed in; spectrograms show one second. Each spectral row uses a shared color scale. The distances use different scales and aren't directly comparable; finite windows and clip boundaries keep the spectral losses from being exactly phase-invariant.

For training, I used auraloss[5], which combines two measures: spectral convergence, measuring the relative difference between magnitude spectrograms, and the L1 distance between their log magnitudes.

\mathcal{L}_{\mathrm{STFT}} = \frac{1}{R}\sum_{r=1}^{R} \left( \frac{\lVert A_r-\hat A_r\rVert_F}{\lVert A_r\rVert_F} + \operatorname{mean} \left|\log A_r-\log\hat A_r\right| \right)

Here, A_r and \hat A_r are the target and reconstructed STFT magnitudes at resolution r. The logarithm gives quieter frequency components more influence, while a small magnitude floor prevents numerical issues.

After experimenting with different losses on a small subset of synth-generated sounds, I settled on a multi-resolution STFT loss with relatively long windows, combined with a log-mel loss.

The latter groups frequencies into perceptually motivated bands using mel filterbanks:

B_k=F_k|\mathrm{STFT}_k(x)|

The loss compares the log-mel representations of the target and reconstruction, averaging over K resolutions:

\mathcal{L}_{\mathrm{logmel}}= \frac{1}{K}\sum_{k=1}^{K} \operatorname{mean} \left|\log B_k-\log\hat B_k\right|

The final objective combines the two losses with equal weight:

\mathcal{L}_{\mathrm{audio}}= \mathcal{L}_{\mathrm{STFT}}+ \mathcal{L}_{\mathrm{logmel}}

Importantly, this is the only training objective. I still log waveform MSE and parameter errors for evaluation, but don't use them to compute gradients.

Training Through the Synth

Training with an audio loss is considerably more expensive, since every forward pass now requires rendering audio through the differentiable synthesizer. For these experiments, I moved from my MacBook to NVIDIA A100 40 GB GPUs.

I compared two approaches: training the same 440k-parameter Transformer from scratch, or initializing it with the weights obtained from parameter-space pretraining.

Both models were trained on the same 100,000 synth-generated sounds, using AdamW with a learning rate of 10^{-3}, batch size 128, and 50 epochs. Each run took about 16 hours, including validation.

The network still predicts all 13 continuous controls and both oscillator selections. During training, waveform selection uses soft mixtures to allow gradients to flow. Validation and testing use discrete oscillator choices and EMA weights, matching the final preset behavior.

Validation results over 50 epochs. Parameter-pretrained initialization gives lower audio loss than training from scratch, but its waveform classification accuracy declines during fine-tuning. Diamonds show the held-out test results at epoch 50.
Validation curves use EMA weights and discrete oscillator choices. Diamonds show test results from the selected checkpoints, both at epoch 50. Waveform accuracy is a diagnostic, not part of the training objective.

On the 10,000 held-out test sounds, the model trained from scratch achieved an audio loss of 1.150, compared with 0.910 for the fine-tuned model: roughly a 21% improvement from pretraining.

Interestingly, the two models recovered both original oscillator waveforms correctly in only 24.3% and 67.6% of examples, respectively. Waveform classification accuracy also declined during fine-tuning, even as reconstruction loss improved.

This highlights an important distinction: the parameters that best reproduce a sound aren't necessarily the ones that originally generated it. With audio reconstruction as the objective, the network is free to find alternative presets that produce similar results.

The difference can also be heard in the reconstructions. Below are two held-out examples, comparing the original recording, the parameter-pretrained model before fine-tuning, and both models trained with the audio loss.

C5

Original

Pretrained (before fine-tuning)

From scratch

Fine-tuned

F3

Original

Pretrained (before fine-tuning)

From scratch

Fine-tuned

To my ears, the fine-tuned reconstructions sound a little closer to the originals, which is consistent with the lower audio loss.

Overall, these results suggest that parameter-space pretraining provides a useful starting point for audio reconstruction, at least with synth-generated audio and within the training budget tested. The next challenge is to move beyond sounds generated by Ava.

6. From Text to Synth Presets

So far, I've only trained on sounds generated by Ava itself. But a text-to-synth system needs to learn from a wider variety of sounds, including those Ava might not be able to reproduce exactly.

Rather than collecting recordings and descriptions by hand, I decided to use a pretrained text-to-audio model to generate the training data.

Generating Audio from Text

I used Stable Audio 3 Small SFX[4], a text-conditioned latent diffusion model operating in the SAME-S latent space. Starting from random noise, it generates an audio latent in just eight sampling steps. This latent can be passed directly to the parameter prediction network, or decoded into audio to use as a reconstruction target.

Since the generator would eventually run locally in the plugin, I also quantized its weights to reduce memory usage. Specifically, the linear layers of the text encoder and latent generator use INT8 weights, while the remaining operations stay in floating point.

Importantly, I used the same quantized models and eight-step sampler for both data generation and deployment, so the prediction network learns from the same type of latents it will receive in the final plugin.

Text passes through an eight-step INT8 latent generator and SAME decoder. CREPE checks the audio pitch; accepted raw latents, decoded audio and note settings are saved together.
Each accepted latent is paired with its decoded audio and the note settings needed to render the synth.

For the dataset, I started with eight instrument families: bass, piano, guitar, trumpet, flute, strings, synth lead, and pad. I wrote a collection of descriptions for each family and combined them with optional timbre modifiers and notes sampled from the corresponding instrument's range.

Here are four examples from the test set:

  • Guitar · D♯4: "a fuzzy garage guitar with a thick buzzing tone, hollow and woody."
  • Piano · E3: "a rounded vintage electric piano with a gentle buzz, hollow and woody."
  • Trumpet · A♯3: "a dark mellow muted trumpet, thin and delicate."
  • Synth Lead · C4: "a gritty distorted rave lead with a sharp biting sound, warm and fuzzy."

Each description is placed inside a prompt template that specifies the instrument, pitch, and desired recording characteristics. For example:

A single isolated guitar-inspired subtractive synthesizer note at D#4 (MIDI 63, 311.13 Hz). Desired timbre: a fuzzy garage guitar with a thick buzzing tone, hollow and woody. Dry and monophonic, triggered once, within a 6-second clip.

This encourages the model to generate isolated, dry, monophonic sounds rather than complete musical passages. Full prompts are unique across the dataset splits, although individual descriptions and modifiers can be reused.

I generated 10,000 training examples, along with 1,000 validation and 1,000 test examples.

Pitch and Note Timing

There's still one problem: the generative model doesn't always get the pitch right, even when the prompt asks for a specific note.

To check this, I used CREPE[6], through torchcrepe, to estimate the fundamental frequency of each generated recording. After excluding quiet sections and unreliable predictions, I kept sounds whose estimated pitch stayed close to the requested note.

For accepted recordings, the synthesizer uses the median detected frequency directly in hertz, without rounding it to the nearest MIDI note.

When a sample fails the pitch check, I regenerate it with a different random seed. After 20 unsuccessful attempts, I replace the prompt entirely.

I also kept the synth's note timing fixed: note-on at the beginning of the six-second clip and note-off after four seconds. This is a simplification, since the generated audio doesn't necessarily follow those timings. A better approach would be to include note duration in the prompt and estimate the actual onset and release.

One important distinction: CREPE is only used to prepare the training dataset. In the final plugin, pitch, velocity, and note timing come directly from MIDI, so neither pitch estimation nor audio decoding is needed to play the generated presets.

Training the Parameter Mapper

With the dataset ready, I could finally train the text-to-synth model.

Each example contains the generated latent, its decoded audio, and the corresponding pitch and note timing. The prediction network receives the original SAME-S latent directly, without additional encoding or normalization.

As in the previous experiment, it predicts the synthesizer parameters, renders the corresponding audio through the differentiable synth, and minimizes the STFT and log-mel reconstruction losses against the decoded reference.

There are no ground-truth synth parameters this time. The network simply learns to find the preset that best approximates each generated sound.

The saved latent enters the parameter mapper. Predicted controls and separate pitch and timing drive the synth. STFT and log-mel losses compare rendered audio with the decoded target, and gradients update the mapper through the synth.
The mapper learns to predict a preset by comparing its rendered sound with the decoded target. Pitch and timing enter the synth separately.

Does Pretraining Help?

In the previous experiment, initializing the network with parameter-pretrained weights improved audio reconstruction on synth-generated sounds. But will that still help here?

The text-generated sounds are much more varied, and many of them can't be reproduced exactly by Ava. Pretraining might provide a useful starting point, but it could also bias the network towards the distribution of randomly generated synth presets.

To investigate this, I compared training from scratch with fine-tuning from the parameter-pretrained model, using two learning rates: 10^{-3} and 10^{-4}.

All four runs used the same 440k-parameter Transformer, AdamW optimizer, batch size of 128, and 200 training epochs. Each took approximately 5.5–6.2 hours on an NVIDIA A100 40 GB GPU.

I selected checkpoints based on validation loss and evaluated them on the same 1,000 held-out test sounds.

InitializationLearning rateSelected epochTest audio loss
Fine-tuned10^{-3}831.912
From scratch10^{-3}1971.888
Fine-tuned10^{-4}1561.901
From scratch10^{-4}1981.918

The best model trained from scratch achieved a test audio loss of 1.888, compared with 1.901 for the best fine-tuned model, a difference of less than 1%.

Pretraining helped at the lower learning rate, but not at the higher one. The differences are small, and with only one seed per configuration, it's difficult to draw strong conclusions. Still, pretraining seems to offer less of an advantage here than it did when reconstructing the synth-generated data.

Of course, numerical losses only tell part of the story. What really matters is how the resulting presets sound.

7. Results: Let's Have a Listen!

Enough about training and losses. Let's finally hear what Ava can do!

Pick an instrument below to compare the audio generated from a text description with Ava's attempt to recreate it. All reconstructions in the main gallery use the best model trained from scratch.

Click Try this preset to load the selected sound into the interactive synth below.

Comparing Training Strategies

The two training strategies can also be compared by ear. Below are four examples showing the original generated sound alongside reconstructions from the best fine-tuned and scratch-trained models.

Guitar · D♯4

“a fuzzy garage guitar with a thick buzzing tone, hollow and woody.”

Generated audio

Fine-tuned

From scratch

Piano · D♯5

“a warm lo-fi piano with a soft muted touch, gently swelling.”

Generated audio

Fine-tuned

From scratch

Trumpet · A♯3

“a dark mellow muted trumpet, thin and delicate.”

Generated audio

Fine-tuned

From scratch

Flute · G♯4

“a clear bright concert flute with a smooth flowing tone, gently wavering.”

Generated audio

Fine-tuned

From scratch

The fine-tuned model uses a learning rate of 10^{-4}, while the scratch model uses 10^{-3}. Both use discrete oscillator selections. The piano reference was re-decoded from its saved test latent, and the recordings haven't been loudness-matched.

Where Things Go Off Track

Some sounds work considerably better than others, and the problems don't always come from Ava.

For example, the text-to-audio model sometimes generates sounds that don't quite match the prompt. To my ears, the guitars and pianos often sound surprisingly similar, and one of the bamboo flute examples even slipped through the pitch check with an incorrect note. Better prompts and more reliable data generation would already help.

On the reconstruction side, Ava seems to struggle particularly with plucky sounds, complex distortion, and evolving dynamics. Some of this is probably due to the limited expressiveness of the synthesizer, but my earlier experiments also showed that optimization can be difficult even when the synth is capable of reproducing the target.

Still, several of the bass, piano, flute, and trumpet examples sound pretty convincing to me. And even when the reconstruction isn't particularly faithful, the resulting preset can be interesting in its own right.

Play and Tweak the Presets

And that's probably my favorite part of the project: these aren't just generated audio samples, but actual synthesizer presets.

Try loading an example from the gallery, playing it at a different pitch, or adjusting a few parameters to see how the sound changes.

Ava.TEXT TO SYNTH

01   SOURCE

02   FILTER + MOTION

03   AMPLITUDE ENVELOPE

04   OUTPUT

Drag knobs up/down · double-click to restore

Loading the presets…

The browser demo uses the C++ synthesizer compiled to WebAssembly, with presets generated in advance. Click Enable audio to start playing. Text generation is only available in the desktop plugin, where you can enter your own descriptions and generate new presets.

8. Bringing Ava to C++

The final step was bringing everything together in a working plugin.

I had already started a C++ synthesizer back in 2024, but it needed updating to match the PyTorch implementation. I kept the audio engine independent of JUCE, which handles the plugin interface and MIDI integration.

Unlike the PyTorch version, which renders complete audio clips in batches, the C++ synth needs to generate audio continuously, preserving oscillator phases, envelopes, and filter states between buffers. To check that both implementations behaved consistently, I compared six-second recordings and tested note timing and continuity across audio buffers. The same C++ engine also powers the browser demo above.

For inference, I exported the neural networks to ONNX and used ONNX Runtime to run everything locally on the CPU. The text encoder and latent generator use the INT8 versions introduced earlier, while the smaller parameter prediction network converts the generated latent into a preset. No Python runtime, external service, audio decoder, or pitch estimator is needed.

To avoid interrupting audio playback, text generation runs in a separate helper process and the parameter mapper runs on a background thread. Once the preset is ready, it behaves like any regular synth patch: you can play it through MIDI and adjust every parameter manually.

Ava's desktop interface with a preset loaded, text generation controls, two oscillators, filter and LFO, envelope controls and a piano keyboard.
The desktop plugin, with a generated preset loaded.

The plugin currently supports Apple Silicon Macs (sorry, Windows users!), with Audio Unit, VST3, and standalone versions.

Both the PyTorch synthesizer and training code and the C++ plugin are available on GitHub, along with the trained mapper weights. You can find build and installation instructions in the C++ repository.

9. Conclusions

So, did it work? Well, sort of!

Some sounds came surprisingly close to the reference, while others were considerably less convincing. Still, I'm quite happy with what Ava can already do, especially considering how simple the synthesizer is.

There's plenty of room for improvement. The most obvious direction is to make the synth more expressive, with additional oscillators, effects, and more flexible modulation. But the synth itself isn't the only limitation. My experiments showed that some parameters were difficult to optimize even when an exact match was possible, suggesting that better losses, training strategies, and models could also improve the results. And, of course, more reliable text-to-audio generation would provide better training data.

Beyond the results, this was a really fun project. I got to combine two things I've been interested in for a long time, music production and generative modeling, and learned quite a bit about both along the way.

Ava hasn't quite solved the endless preset-scrolling problem that originally motivated the project. But having something that can turn a sound description into a playable, editable synth preset feels like a pretty good start.

Afterword: A Note on Using LLMs

I started this project in 2024, when coding assistants weren't nearly as capable as they are today. Back then, I built the C++ synthesizer myself, following The Audio Programmer's book, with a few modifications and a rather ugly interface. I also implemented a way to load an audio sample and predict its synth parameters using a PyTorch model, and wrote most of the differentiable synthesizer and training pipeline. I eventually stopped while working on the low-pass filter and LFO, and the project sat unfinished for almost two years.

When I picked it up again in 2026, things had changed quite a bit. I relied heavily on Codex to finish the PyTorch synth, run experiments and ablations, and help analyze the results. Once I was happy with the implementation, I went through the Python code file by file, refactoring and reviewing everything before moving on.

I was a little less rigorous with the C++ side, delegating much of the interface, integration, and installation work to Codex. I still tested the plugin myself, going through several iterations to identify and fix bugs.

I also used ChatGPT to help write this post. I started with a rough draft, focusing on what I wanted to say rather than polished English, and worked through it section by section to improve the writing and structure. I also used Codex to help create the diagrams and visualizations.

Overall, these tools made it much easier to revisit an old project, experiment with different ideas, and finally bring everything together. They didn't replace the need to understand the code, evaluate the experiments, or make design decisions, but they certainly made the process faster and more enjoyable.

References

  1. Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail and Huaming Wang. CLAP: Learning Audio Concepts From Natural Language Supervision. arXiv:2206.04769, 2022.
  2. Alexandre Défossez et al. High Fidelity Neural Audio Compression. 2022.
  3. Marco Pasini, Stefan Lattner and George Fazekas. Music2Latent: Consistency Autoencoders for Latent Audio Compression. 2024.
  4. Zach Evans et al. Stable Audio 3. 2026.
  5. Christian J. Steinmetz and Joshua D. Reiss. auraloss: Audio focused loss functions in PyTorch. DMRN+15, 2020.
  6. torchcrepe. PyTorch implementation of the CREPE pitch tracker. GitHub repository.