MusicMCP - An experimental, agent-controlled music production tool

Resources Community Resources
, , , ,
1

Hi everyone,

I have been working on MusicMCP, a local music composition and rendering system that an AI coding agent such as Codex can control through MCP.

The important distinction is that the AI is not generating a finished audio waveform directly. It writes and edits structured musical data, including notes, chords, drum patterns, instruments, arrangement, automation, and mixer settings; a deterministic engine then renders that project into playable audio.

GitHub: github.com/gmoddev/MusicMCP

MusicMCP is free to use under the MIT License. You can modify it, build on it, and repost or redistribute your version as long as you retain the copyright and license notice.

This is still experimental and has not been thoroughly battle-tested. I have made two complete tracks with it so far, but neither track is currently used in a released Roblox game. I plan to update this post once the music has been used and tested in an actual game environment.

Update: MusicMCP now has experimental headless VST3 and CLAP instrument support. An agent can scan an explicitly trusted plugin folder, inspect an instrument, capture or restore its state, use it on a project track, and render it through the same master, stem, cache, manifest, and analysis pipeline as the built-in instruments. Piano-style controls such as sustain, sostenuto, soft pedal, expression, modulation, pitch bend, and channel pressure are represented as structured automation. I have successfully rendered the Steinberg SDK MDA Piano fixture with sustain and repeatable output, but broad VST3 compatibility is not yet certified. The project will keep this feature opt-in until at least three independent Windows instruments pass the full compatibility checklist.

Why I built it

AI tools can help with code, UI, writing, and asset workflows, but music is often treated as a one-shot generation problem: enter a prompt, receive an audio file, and hope it fits.

I wanted a workflow that is easier to inspect and revise. With MusicMCP, the agent can make targeted production changes such as:

  • replace the drums in a chorus without rewriting the whole song;
  • change a chord progression while preserving the arrangement;
  • add swing, accents, ghost notes, flams, or hi-hat ratchets;
  • change an instrument without changing the underlying composition;
  • adjust track volume, panning, effects, sends, or automation;
  • render a new master and individual stems;
  • inspect measurable issues such as clipping, excessive peaks, silence, or stereo imbalance.

The saved project remains structured and readable instead of becoming an opaque audio result.

Why this could be useful for Roblox developers

The intended Roblox workflow is offline production, not runtime AI generation. You create and iterate on the music locally, export the result, and then upload the approved audio to Roblox through the normal asset workflow.

Some potential benefits are:

  • Faster iteration: Ask for a specific revision instead of regenerating an entire track from scratch.
  • Game-ready exports: Render a master plus aligned track or bus stems, which can help with layered or adaptive music systems.
  • Repeatable results: The project, renderer settings, assets, seeds, and outputs are hashed in a render manifest. The same pinned setup can reproduce the same result.
  • Version-controlled music: Projects are human-readable JSON and can be reviewed or stored in Git alongside a game repository.
  • Local-first operation: Composition data, samples, SoundFonts, and renders can remain on your machine.
  • Renderer independence: A song can move from a basic built-in synth to SoundFonts or layered samples without rewriting its notes and arrangement.
  • Agent-friendly editing: High-level MCP tools can modify many events in one call instead of requiring one tool call per note.
  • Objective feedback: The analysis pipeline can measure peaks, clipping, loudness-related values, silence, stereo balance, frequency distribution, and onset density. These measurements are useful checks, but they do not pretend to decide whether a track is artistically good.

Stems are probably the most interesting Roblox-specific possibility. A track could be exported as compatible musical layers, for example ambience, percussion, bass, and lead; a game could control those uploaded layers based on combat, location, round state, or intensity. MusicMCP does not provide that Roblox runtime system; it produces the aligned source material that such a system could use.

What it currently supports

  • structured song projects with tempo, meter, sections, tracks, patterns, notes, drums, automation, routing, and mixer data;
  • deterministic WAV rendering, individual stems, MIDI export, and optional FLAC/MP3 support;
  • built-in synths and a wavetable synth that require no external instrument assets;
  • FluidSynth rendering with SF2/SF3 SoundFonts;
  • layered WAV sample instruments with velocity layers, round robin, articulations, choke groups, tuning, envelopes, and per-hit expression;
  • high-level beat writing with swing, accents, ghost notes, velocity cycles, seeded probability, flams, grooves, and ratchets;
  • MIDI, MusicXML, and DAWproject interchange within documented limits;
  • persistent render jobs, caching, validation, manifests, and objective audio analysis;
  • a local stdio MCP server for Codex and other compatible agents.

What it is not

MusicMCP is not a traditional DAW, a live music system, or a replacement for an experienced composer or audio engineer. It currently has no piano-roll interface, recording workflow, time stretching, or complete VST3/CLAP processing support.

It also does not make samples or SoundFonts legally safe to use. You are responsible for the source and license of any instrument or sample content you provide, as well as Roblox’s audio-upload and moderation requirements.

MusicMCP’s code and documentation are released under the MIT License. You may use, modify, publish, and redistribute them, including commercially, as long as the copyright and license notice is retained. Credit through that notice is required; a visible mention or repository link is also appreciated.

The MIT License covers MusicMCP itself, not third-party samples, SoundFonts, imported assets, or external tools. Those may have separate usage and attribution requirements.

Current status

The full local loop works:

Describe or revise a composition
        ↓
Agent edits structured project data
        ↓
Validate and render locally
        ↓
Inspect the master and objective analysis
        ↓
Make targeted changes and render again
        ↓
Export the approved master or stems

As mentioned above, I have produced two tracks with the system, but this is still early testing. They have not yet shipped in any Roblox experiences, so I do not want to overstate how proven the workflow is. I will update this when I have real in-game results, especially around looping, compression after upload, stem synchronization, and how well the tracks hold up during actual gameplay.

If you are interested in agent-assisted music tools, deterministic asset pipelines, or adaptive Roblox music, I would appreciate feedback on the architecture and on which production features would be most useful next.

Repository and setup instructions: github.com/gmoddev/MusicMCP

3 Likes
2

I get that AI is evolving to the point where we have the capabilities to produce AI generated music. And I understand that is not your intention here, however, why are we moving towards agent-assisted music creation? Doesn’t that take away from the entire point of the creativity of music? And additionally, if used for commissions, customers more often than not ask for changes to be made once a composer makes a track. Would this allow for adjustments to be made? I rarely have a customer not make any changes to a track I create. But the best part of it is that I created it, so it is always super easy to re-adjust.

2 Likes
3

You can use AI to assist with human creativity too, which I believe is the purpose of this. Additionally, about the “profit” aspect since you mentioned it, I heard of a few artists who use AI in their works to speed up their workflow; it’s not one-sided.

4

The boring answer to why I made it is that I’m a programmer, not a composer, and I can’t afford to commission someone every time I need music for a project.

I still wanted something more controllable than “prompt an AI and get an audio file back,” though. That’s actually why MusicMCP stores the composition as structured data instead of generating the finished audio directly.

And yes, adjustments are possible, and they’re one of the main reasons I designed it this way. If I don’t like the drums, arrangement, instrumentation, chord progression, mix, etc., the agent can modify that specific part while leaving the rest of the project alone. The underlying composition isn’t locked inside a generated waveform, so individual parts can be changed and rendered again without starting over.

I also don’t really see MusicMCP as answering “why should composers use agents?” In fact, in its current form, it probably isn’t nearly as useful to an experienced composer. There’s only a very basic UI because the agent is intended to be the primary interface. A composer already has a DAW, a piano roll, mixer, plugin interfaces, and most importantly the knowledge to directly make the changes they want. Their existing workflow is probably going to be faster and more natural for them.

It’s more answering: “How can someone like me, who needs custom game music but isn’t a composer and can’t afford to continually commission it, get something editable and iterative rather than one-shot generated audio?”

It certainly doesn’t replace what an experienced composer can do, nor is that really the goal. If I had the budget to hire a composer whenever I needed music, I’d probably do that. I don’t, which is basically why this project exists.

5

So does it generate a midi track and configures VST of sort?
I really like the progress towards non destructive workflow

6

Sort of, but MIDI isn’t actually the internal format.

MusicMCP stores the song as its own structured project data, which are: notes, chords, drum hits, sections, tempo, automation, mixer settings, etc. The agent edits that directly.

MIDI is one of the formats it can import/export, so you can generate a MIDI file from MusicMCP or bring one into it, but the workflow isnt basically “AI generates a MIDI file and then hands that to a VST.”

For VST3/CLAP instruments, MusicMCP can actually host the plugin headlessly, configure its parameters/state, send the authored note/performance data to it, and render the instrument to audio. It can also use plugin effects on tracks/buses/master.

So conceptually it’s closer to:

Structured MusicMCP project
    ↓
notes / drums / automation / arrangement
    ↓
selected instrument backend
    ├── built-in synth
    ├── SoundFont
    ├── sampler
    └── VST3 / CLAP
    ↓
rendered audio

MIDI can come in or go out along the way, but it isn’t required.

The non-destructive part is the entire reason I went in this direction. It remains editable instead of just being a . wav.

1 Like
7

I do a significant amount of free work without requiring anything in return. If you are interested, reach out to me. It is a hobby of mine and I enjoy helping people.