syskiller/ITOSMPiano-BachToBasics
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add audio-to-MIDI import via Basic Pitch (MP3, WAV, M4A, OGG, FLAC, AAC), fix piano remount on view change

Browse Source
This commit is contained in:
Fabricio Giglio
2026-05-17 08:10:03 +02:00
parent 9d2991a916
commit 6de3c8d98e
6 changed files with 134 additions and 47 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+14 -3
View File
@@ -29,13 +29,15 @@ A browser-based piano practice tool. Drop in a **MIDI file**, a **MusicXML score
- **MIDI** (`.mid`, `.midi`): parsed in a Web Worker, converted to MusicXML on the backend for sheet rendering
- **MusicXML** (`.xml`, `.mxl`): both uncompressed and compressed (zip-style) variants
- **PDF sheet music** (`.pdf`): optical music recognition via [Audiveris](https://github.com/Audiveris/audiveris) to MusicXML, then to MIDI (optional setup, see below)
- **Audio recordings** (`.mp3`, `.wav`, `.m4a`, `.ogg`, `.flac`, `.aac`): audio-to-MIDI transcription via [Basic Pitch](https://github.com/spotify/basic-pitch) (optional setup, see below) - drop a recording of yourself improvising, get a playable piano roll
Drag-and-drop anywhere on the window, or use the import button.
> ⚠️ **A note on import accuracy.** Both PDF-to-MIDI (optical music recognition) and MIDI-to-sheet-music (notation reconstruction) are inherently lossy conversions. Expect some inconsistencies:
> ⚠️ **A note on import accuracy.** PDF-to-MIDI (optical music recognition), MIDI-to-sheet-music (notation reconstruction), and audio-to-MIDI (pitch detection) are all inherently lossy conversions. Expect some inconsistencies:
>
> - **PDF imports** can mis-read notes, dynamics, ornaments, articulation, and voicing, especially with low-resolution scans, handwritten scores, or complex layouts. Audiveris is the best open-source OMR engine available, but no engine matches a careful human transcription.
> - **MIDI imports** produce sheet music by re-deriving notation from raw note timings. MIDI doesn't encode key signatures, beaming, voicing, articulation, or enharmonic spelling, so the rendered sheet is an interpretation, not a faithful reproduction of the composer's original score.
> - **Audio imports** detect pitches in the mixed signal - this is pitch-detection, not source separation. Quality is excellent on solo instruments (piano, voice, guitar) and degrades on complex mixes (full-band recordings with drums + bass + vocals). For multi-track sources, expect a rough sketch rather than a clean transcription. Solo piano recordings give the best results.
>
> For the cleanest sheet-music experience, import a **MusicXML** file (or `.mxl`) when you have one, which preserves full notation semantics end-to-end.
@@ -131,6 +133,16 @@ PDF export uses [LilyPond](https://lilypond.org/) for typesetting. It's not bund
Restart the backend after installing. Without LilyPond, MIDI and MusicXML export still work; only PDF export is unavailable.
### Optional: Audio-to-MIDI via Basic Pitch
Audio-to-MIDI transcription uses [Basic Pitch](https://github.com/spotify/basic-pitch), Spotify's open-source pitch-detection model. It's not bundled by default because it pulls in TensorFlow (~500 MB) and has tight Python-version constraints.
- **macOS**: requires Python **3.11** specifically. `brew install python@3.11`, then create a 3.11 venv at `backend/.venv` and `pip install -r backend/requirements-transcribe.txt`. (Python 3.12+ doesn't work on macOS because `basic-pitch` requires `tensorflow-macos<2.15.1`, and no compatible wheel exists for 3.12.)
- **Linux**: Python 3.11 or 3.12. Activate the venv and `pip install -r backend/requirements-transcribe.txt`.
- **Docker**: edit `backend/Dockerfile` to also `pip install -r requirements-transcribe.txt` (or use a separate build target), then `docker compose build backend`. The image already uses Python 3.12.
Restart the backend after installing. Without Basic Pitch, MIDI / MusicXML / PDF import still works; only audio import (MP3, WAV, etc.) is unavailable.
## Using the app on iPad / iPhone (same network)
1. In `frontend/vite.config.ts`, add `host: true` to the `server` block (or run `pnpm --filter frontend dev --host`)
@@ -189,9 +201,8 @@ Requires a Chromium-based browser (Chrome, Edge, Brave, Arc) for **Web MIDI API*
The backend already exposes endpoints for several features that don't yet have UI hooks:
- **Audio-to-MIDI transcription** via [Basic Pitch](https://github.com/spotify/basic-pitch) (`/transcribe/mp3`)
- **YouTube-to-MIDI** via yt-dlp + Basic Pitch (`/youtube/extract`), sync data model already in place
- **Export** to PDF / MP3 / MIDI / MusicXML (`/export/pdf`, `/export/mp3`)
- **MP3 export** (MIDI / MusicXML / PDF export already shipped; MP3 needs an OfflineAudioContext WAV-render pipeline)
These will become user-facing in upcoming releases. PRs welcome.
backend/requirements-transcribe.txt
+20 -5
View File
@@ -1,11 +1,26 @@
# Optional audio-transcription dependencies (Basic Pitch).
# Requires Python 3.11 or 3.12, tensorflow-macos doesn't yet support 3.13.
#
# To enable MP3 / audio-to-MIDI transcription:
# brew install python@3.12
# /opt/homebrew/bin/python3.12 -m venv .venv
# Requires Python 3.11 on macOS. (basic-pitch's tensorflow-macos<2.15.1
# constraint excludes 3.12+ - no compatible tensorflow-macos wheel exists.)
# Linux is more flexible; 3.11 or 3.12 both work there.
#
# To enable MP3 / audio-to-MIDI transcription on macOS:
# brew install python@3.11
# /opt/homebrew/bin/python3.11 -m venv .venv
# source .venv/bin/activate
# pip install -r requirements-transcribe.txt
#
-r requirements.txt
basic-pitch>=0.3.6
# setuptools < 81 is required because basic-pitch's dependency `resampy`
# still imports `pkg_resources`, which setuptools 81 (released 2025) removed
# after years of deprecation. Until upstream resampy migrates to
# `importlib.metadata`, we pin to the last setuptools major that ships
# pkg_resources (80.x).
setuptools<81
# `[tf]` extra is REQUIRED since basic-pitch 0.4+ made the model backend opt-in.
# Without it, basic-pitch installs but `predict()` fails (no TensorFlow to load
# the ICASSP_2022_MODEL_PATH saved model). The `[onnx]` extra would be lighter
# (~10 MB vs TF's ~500 MB) but our service uses the TF saved-model format.
basic-pitch[tf]>=0.3.6
backend/routers/transcribe.py
+23 -2
View File
@@ -158,6 +158,27 @@ async def transcribe_mp3(file: UploadFile, background_tasks: BackgroundTasks):
)
except HTTPException:
raise
except Exception:
except Exception as e:
# Surface actionable runtime errors (e.g. missing basic-pitch) to the
# client so the user knows what to install. Match by message substring
# since exceptions may be wrapped on the way back.
logger.exception("transcribe_mp3 failed")
raise HTTPException(status_code=500, detail="Transcription failed. Please try again.")
msg = str(e)
# Only match the specific "not installed" sentinel - any other error
# that happens to mention basic-pitch (e.g. file paths inside the
# package, model-load failures) should NOT be wrongly reported as
# "needs install".
if "basic-pitch is not installed" in msg.lower():
raise HTTPException(
status_code=503,
detail=(
"Audio-to-MIDI transcription needs Basic Pitch on the backend "
"(not bundled by default).\n\n"
"Quick install: with a Python 3.11 venv active on macOS (or 3.11/3.12 "
"on Linux), run pip install -r backend/requirements-transcribe.txt. "
"For Docker, see the README section 'Optional: Audio-to-MIDI via Basic Pitch'."
),
)
raise HTTPException(
status_code=500, detail=f"Transcription failed: {msg or 'Unknown error'}"
)
frontend/src/components/Transport/TransportControls.tsx
+5 -3
View File
@@ -30,6 +30,7 @@ export function TransportControls() {
loadMidiFile,
loadMusicXmlFile,
loadPdfFile,
loadAudioFile,
isLoadingDocument,
loadingMessage,
exportMidi,
@@ -87,6 +88,7 @@ export function TransportControls() {
if (file.name.match(/\.midi?$/i)) loadMidiFile(file);
else if (file.name.match(/\.(xml|mxl)$/i)) loadMusicXmlFile(file);
else if (file.name.match(/\.pdf$/i)) loadPdfFile(file);
else if (file.name.match(/\.(mp3|wav|m4a|ogg|flac|aac)$/i)) loadAudioFile(file);
};
const total = doc?.totalDuration ?? 0;
@@ -209,8 +211,8 @@ export function TransportControls() {
>
{/* ── Zone 1: File import - accent-styled to stand out ── */}
<label
title="Open MIDI, MusicXML, or PDF sheet music"
aria-label="Import MIDI, MusicXML, or PDF file"
title="Open MIDI, MusicXML, PDF sheet music, or audio recording (MP3/WAV/etc.)"
aria-label="Import MIDI, MusicXML, PDF, or audio file"
style={{
display: "inline-flex",
alignItems: "center",
@@ -243,7 +245,7 @@ export function TransportControls() {
)}
<input
type="file"
accept=".mid,.midi,.xml,.mxl,.pdf"
accept=".mid,.midi,.xml,.mxl,.pdf,.mp3,.wav,.m4a,.ogg,.flac,.aac"
className="hidden"
onChange={(e) => {
const f = e.target.files?.[0];
frontend/src/store/useAppStore.ts
+45
View File
@@ -140,6 +140,7 @@ export interface AppState {
loadMidiFile: (file: File) => Promise<void>;
loadMusicXmlFile: (file: File) => Promise<void>;
loadPdfFile: (file: File) => Promise<void>;
loadAudioFile: (file: File) => Promise<void>;
generateFingering: () => Promise<void>;
exportMidi: () => void;
exportMusicXml: () => void;
@@ -436,6 +437,50 @@ export const useAppStore = create<AppState>((set, get) => {
}
},
loadAudioFile: async (file: File) => {
const id = crypto.randomUUID();
const rawTitle = file.name.replace(/\.(mp3|wav|m4a|ogg|flac|aac)$/i, "");
const title = rawTitle.replace(/[\x00-\x1f\x7f/\\]/g, "").slice(0, 200) || "Untitled";
set({
isLoadingDocument: true,
loadingMessage: "Transcribing audio…",
loadError: null,
});
try {
const form = new FormData();
form.append("file", file, file.name);
// Basic Pitch can take 10-60 s on the backend; no client-side fetch
// timeout - the server's subprocess timeout is the effective limit.
const res = await fetch("/api/transcribe/mp3", { method: "POST", body: form });
if (!res.ok) {
const body = (await res.json().catch(() => ({}))) as { detail?: string };
throw new Error(body.detail ?? `Server error ${res.status}`);
}
// Endpoint returns a raw MIDI file (audio/midi) - parse it through the
// same Web Worker pipeline as a regular .mid import.
const midiBuffer = await res.arrayBuffer();
const bufferForDoc = midiBuffer.slice(0);
const doc = await parseMidiInWorker(midiBuffer, id, title);
set({ document: { ...doc, midiBuffer: bufferForDoc } });
await syncEngine.loadDocument({ ...doc, midiBuffer: bufferForDoc });
// Kick off MusicXML transcription so the sheet-music view eventually
// populates, mirroring loadMidiFile's pattern. Fire-and-forget.
fetchMusicXml(bufferForDoc.slice(0), doc, id).catch(console.warn);
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
set({ loadError: msg });
} finally {
set({ isLoadingDocument: false, loadingMessage: null });
}
},
generateFingering: async () => {
const { document: doc } = get();
if (!doc?.musicXml) return;
frontend/src/views/PracticeView.tsx
+27 -34
View File
@@ -32,6 +32,7 @@ export function PracticeView() {
loadMidiFile,
loadMusicXmlFile,
loadPdfFile,
loadAudioFile,
settings,
updateSettings,
status,
@@ -117,8 +118,9 @@ export function PracticeView() {
if (file.name.match(/\.midi?$/i)) loadMidiFile(file);
else if (file.name.match(/\.(xml|mxl)$/i)) loadMusicXmlFile(file);
else if (file.name.match(/\.pdf$/i)) loadPdfFile(file);
else if (file.name.match(/\.(mp3|wav|m4a|ogg|flac|aac)$/i)) loadAudioFile(file);
},
[loadMidiFile, loadMusicXmlFile, loadPdfFile]
[loadMidiFile, loadMusicXmlFile, loadPdfFile, loadAudioFile]
);
const handleDrop = useCallback(
@@ -265,40 +267,31 @@ export function PracticeView() {
</div>
</header>
{/* ── Main content - layout-aware ────────────────────────────────────── */}
{/* ── Main content - layout-aware ──────────────────────────────────────
PianoKeyboardView is a SINGLETON at the bottom of the main column so
PixiJS doesn't re-init on every view switch (skeleton flash + perf cost).
Same for FallingNotesView in falling/all modes - kept mounted across
the two modes that show it. Only the conditional top-content elements
(PianoModeBackground / SheetMusicView / sheet sidebar) mount and
unmount on layoutMode change. */}
<main className="flex flex-1 overflow-hidden min-h-0 relative">
{layoutMode === "piano" && (
<div className="flex flex-col flex-1 min-h-0 overflow-hidden">
<PianoModeBackground />
<PianoKeyboardView />
</div>
)}
{layoutMode === "falling" && (
<div className="flex flex-col flex-1 min-h-0 overflow-hidden">
<FallingNotesView />
<PianoKeyboardView />
</div>
)}
{layoutMode === "sheet" && (
<div className="flex flex-col flex-1 min-h-0 overflow-hidden">
<SheetMusicView />
<PianoKeyboardView />
</div>
)}
{/* Optional sheet sidebar (only in "all" mode) */}
{layoutMode === "all" && (
<>
<div
className="flex flex-col flex-shrink-0 min-h-0 overflow-hidden relative"
style={{ flex: "0 0 42%", minWidth: 260, maxWidth: 440 }}
>
<SheetMusicView />
</div>
<div className="flex flex-col flex-1 min-h-0 overflow-hidden">
<FallingNotesView />
<PianoKeyboardView />
</div>
</>
<div
className="flex flex-col flex-shrink-0 min-h-0 overflow-hidden relative"
style={{ flex: "0 0 42%", minWidth: 260, maxWidth: 440 }}
>
<SheetMusicView />
</div>
)}
{/* Main column: top content varies, piano persists across all modes */}
<div className="flex flex-col flex-1 min-h-0 overflow-hidden">
{layoutMode === "piano" && <PianoModeBackground />}
{(layoutMode === "falling" || layoutMode === "all") && <FallingNotesView />}
{layoutMode === "sheet" && <SheetMusicView />}
<PianoKeyboardView />
</div>
</main>
{/* ── Empty state overlay ────────────────────────────────────────────── */}
@@ -362,7 +355,7 @@ function EmptyState({ onImport }: { onImport: (file: File) => void }) {
No file loaded
</div>
<div style={{ fontSize: 13, color: "var(--color-text-muted)", lineHeight: 1.6 }}>
Import a MIDI, MusicXML, or PDF sheet music file to start practicing
Import a MIDI, MusicXML, PDF sheet music, or audio recording to start practicing
</div>
</div>
@@ -406,7 +399,7 @@ function EmptyState({ onImport }: { onImport: (file: File) => void }) {
<input
ref={fileInputRef}
type="file"
accept=".mid,.midi,.xml,.mxl,.pdf"
accept=".mid,.midi,.xml,.mxl,.pdf,.mp3,.wav,.m4a,.ogg,.flac,.aac"
className="hidden"
onChange={(e) => {
const f = e.target.files?.[0];