Add audio-to-MIDI import via Basic Pitch (MP3, WAV, M4A, OGG, FLAC, AAC), fix piano remount on view change

README.md

@@ -29,13 +29,15 @@ A browser-based piano practice tool. Drop in a **MIDI file**, a **MusicXML score
 - **MIDI** (`.mid`, `.midi`): parsed in a Web Worker, converted to MusicXML on the backend for sheet rendering
 - **MusicXML** (`.xml`, `.mxl`): both uncompressed and compressed (zip-style) variants
 - **PDF sheet music** (`.pdf`): optical music recognition via [Audiveris](https://github.com/Audiveris/audiveris) to MusicXML, then to MIDI (optional setup, see below)
+- **Audio recordings** (`.mp3`, `.wav`, `.m4a`, `.ogg`, `.flac`, `.aac`): audio-to-MIDI transcription via [Basic Pitch](https://github.com/spotify/basic-pitch) (optional setup, see below) - drop a recording of yourself improvising, get a playable piano roll
 
 Drag-and-drop anywhere on the window, or use the import button.
 
-> ⚠️ **A note on import accuracy.** Both PDF-to-MIDI (optical music recognition) and MIDI-to-sheet-music (notation reconstruction) are inherently lossy conversions. Expect some inconsistencies:
+> ⚠️ **A note on import accuracy.** PDF-to-MIDI (optical music recognition), MIDI-to-sheet-music (notation reconstruction), and audio-to-MIDI (pitch detection) are all inherently lossy conversions. Expect some inconsistencies:
 >
 > - **PDF imports** can mis-read notes, dynamics, ornaments, articulation, and voicing, especially with low-resolution scans, handwritten scores, or complex layouts. Audiveris is the best open-source OMR engine available, but no engine matches a careful human transcription.
 > - **MIDI imports** produce sheet music by re-deriving notation from raw note timings. MIDI doesn't encode key signatures, beaming, voicing, articulation, or enharmonic spelling, so the rendered sheet is an interpretation, not a faithful reproduction of the composer's original score.
+> - **Audio imports** detect pitches in the mixed signal - this is pitch-detection, not source separation. Quality is excellent on solo instruments (piano, voice, guitar) and degrades on complex mixes (full-band recordings with drums + bass + vocals). For multi-track sources, expect a rough sketch rather than a clean transcription. Solo piano recordings give the best results.
 >
 > For the cleanest sheet-music experience, import a **MusicXML** file (or `.mxl`) when you have one, which preserves full notation semantics end-to-end.
 
@@ -131,6 +133,16 @@ PDF export uses [LilyPond](https://lilypond.org/) for typesetting. It's not bund
 
 Restart the backend after installing. Without LilyPond, MIDI and MusicXML export still work; only PDF export is unavailable.
 
+### Optional: Audio-to-MIDI via Basic Pitch
+
+Audio-to-MIDI transcription uses [Basic Pitch](https://github.com/spotify/basic-pitch), Spotify's open-source pitch-detection model. It's not bundled by default because it pulls in TensorFlow (~500 MB) and has tight Python-version constraints.
+
+- **macOS**: requires Python **3.11** specifically. `brew install python@3.11`, then create a 3.11 venv at `backend/.venv` and `pip install -r backend/requirements-transcribe.txt`. (Python 3.12+ doesn't work on macOS because `basic-pitch` requires `tensorflow-macos<2.15.1`, and no compatible wheel exists for 3.12.)
+- **Linux**: Python 3.11 or 3.12. Activate the venv and `pip install -r backend/requirements-transcribe.txt`.
+- **Docker**: edit `backend/Dockerfile` to also `pip install -r requirements-transcribe.txt` (or use a separate build target), then `docker compose build backend`. The image already uses Python 3.12.
+
+Restart the backend after installing. Without Basic Pitch, MIDI / MusicXML / PDF import still works; only audio import (MP3, WAV, etc.) is unavailable.
+
 ## Using the app on iPad / iPhone (same network)
 
 1. In `frontend/vite.config.ts`, add `host: true` to the `server` block (or run `pnpm --filter frontend dev --host`)
@@ -189,9 +201,8 @@ Requires a Chromium-based browser (Chrome, Edge, Brave, Arc) for **Web MIDI API*
 
 The backend already exposes endpoints for several features that don't yet have UI hooks:
 
-- **Audio-to-MIDI transcription** via [Basic Pitch](https://github.com/spotify/basic-pitch) (`/transcribe/mp3`)
 - **YouTube-to-MIDI** via yt-dlp + Basic Pitch (`/youtube/extract`), sync data model already in place
-- **Export** to PDF / MP3 / MIDI / MusicXML (`/export/pdf`, `/export/mp3`)
+- **MP3 export** (MIDI / MusicXML / PDF export already shipped; MP3 needs an OfflineAudioContext WAV-render pipeline)
 
 These will become user-facing in upcoming releases. PRs welcome.
 

backend/requirements-transcribe.txt

@@ -1,11 +1,26 @@
 # Optional audio-transcription dependencies (Basic Pitch).
-# Requires Python 3.11 or 3.12, tensorflow-macos doesn't yet support 3.13.
 #
-# To enable MP3 / audio-to-MIDI transcription:
+# Requires Python 3.11 on macOS. (basic-pitch's tensorflow-macos<2.15.1
-#   brew install python@3.12
+# constraint excludes 3.12+ - no compatible tensorflow-macos wheel exists.)
-#   /opt/homebrew/bin/python3.12 -m venv .venv
+# Linux is more flexible; 3.11 or 3.12 both work there.
+#
+# To enable MP3 / audio-to-MIDI transcription on macOS:
+#   brew install python@3.11
+#   /opt/homebrew/bin/python3.11 -m venv .venv
 #   source .venv/bin/activate
 #   pip install -r requirements-transcribe.txt
 #
 -r requirements.txt
-basic-pitch>=0.3.6
+
+# setuptools < 81 is required because basic-pitch's dependency `resampy`
+# still imports `pkg_resources`, which setuptools 81 (released 2025) removed
+# after years of deprecation. Until upstream resampy migrates to
+# `importlib.metadata`, we pin to the last setuptools major that ships
+# pkg_resources (80.x).
+setuptools<81
+
+# `[tf]` extra is REQUIRED since basic-pitch 0.4+ made the model backend opt-in.
+# Without it, basic-pitch installs but `predict()` fails (no TensorFlow to load
+# the ICASSP_2022_MODEL_PATH saved model). The `[onnx]` extra would be lighter
+# (~10 MB vs TF's ~500 MB) but our service uses the TF saved-model format.
+basic-pitch[tf]>=0.3.6

backend/routers/transcribe.py

@@ -158,6 +158,27 @@ async def transcribe_mp3(file: UploadFile, background_tasks: BackgroundTasks):
         )
     except HTTPException:
         raise
-    except Exception:
+    except Exception as e:
+        # Surface actionable runtime errors (e.g. missing basic-pitch) to the
+        # client so the user knows what to install. Match by message substring
+        # since exceptions may be wrapped on the way back.
         logger.exception("transcribe_mp3 failed")
-        raise HTTPException(status_code=500, detail="Transcription failed. Please try again.")
+        msg = str(e)
+        # Only match the specific "not installed" sentinel - any other error
+        # that happens to mention basic-pitch (e.g. file paths inside the
+        # package, model-load failures) should NOT be wrongly reported as
+        # "needs install".
+        if "basic-pitch is not installed" in msg.lower():
+            raise HTTPException(
+                status_code=503,
+                detail=(
+                    "Audio-to-MIDI transcription needs Basic Pitch on the backend "
+                    "(not bundled by default).\n\n"
+                    "Quick install: with a Python 3.11 venv active on macOS (or 3.11/3.12 "
+                    "on Linux), run pip install -r backend/requirements-transcribe.txt. "
+                    "For Docker, see the README section 'Optional: Audio-to-MIDI via Basic Pitch'."
+                ),
+            )
+        raise HTTPException(
+            status_code=500, detail=f"Transcription failed: {msg or 'Unknown error'}"
+        )

frontend/src/components/Transport/TransportControls.tsx

@@ -30,6 +30,7 @@ export function TransportControls() {
     loadMidiFile,
     loadMusicXmlFile,
     loadPdfFile,
+    loadAudioFile,
     isLoadingDocument,
     loadingMessage,
     exportMidi,
@@ -87,6 +88,7 @@ export function TransportControls() {
     if (file.name.match(/\.midi?$/i)) loadMidiFile(file);
     else if (file.name.match(/\.(xml|mxl)$/i)) loadMusicXmlFile(file);
     else if (file.name.match(/\.pdf$/i)) loadPdfFile(file);
+    else if (file.name.match(/\.(mp3|wav|m4a|ogg|flac|aac)$/i)) loadAudioFile(file);
   };
 
   const total = doc?.totalDuration ?? 0;
@@ -209,8 +211,8 @@ export function TransportControls() {
       >
         {/* ── Zone 1: File import - accent-styled to stand out ── */}
         <label
-          title="Open MIDI, MusicXML, or PDF sheet music"
+          title="Open MIDI, MusicXML, PDF sheet music, or audio recording (MP3/WAV/etc.)"
-          aria-label="Import MIDI, MusicXML, or PDF file"
+          aria-label="Import MIDI, MusicXML, PDF, or audio file"
           style={{
             display: "inline-flex",
             alignItems: "center",
@@ -243,7 +245,7 @@ export function TransportControls() {
           )}
           <input
             type="file"
-            accept=".mid,.midi,.xml,.mxl,.pdf"
+            accept=".mid,.midi,.xml,.mxl,.pdf,.mp3,.wav,.m4a,.ogg,.flac,.aac"
             className="hidden"
             onChange={(e) => {
               const f = e.target.files?.[0];

frontend/src/store/useAppStore.ts

@@ -140,6 +140,7 @@ export interface AppState {
   loadMidiFile: (file: File) => Promise<void>;
   loadMusicXmlFile: (file: File) => Promise<void>;
   loadPdfFile: (file: File) => Promise<void>;
+  loadAudioFile: (file: File) => Promise<void>;
   generateFingering: () => Promise<void>;
   exportMidi: () => void;
   exportMusicXml: () => void;
@@ -436,6 +437,50 @@ export const useAppStore = create<AppState>((set, get) => {
       }
     },
 
+    loadAudioFile: async (file: File) => {
+      const id = crypto.randomUUID();
+      const rawTitle = file.name.replace(/\.(mp3|wav|m4a|ogg|flac|aac)$/i, "");
+      const title = rawTitle.replace(/[\x00-\x1f\x7f/\\]/g, "").slice(0, 200) || "Untitled";
+
+      set({
+        isLoadingDocument: true,
+        loadingMessage: "Transcribing audio…",
+        loadError: null,
+      });
+
+      try {
+        const form = new FormData();
+        form.append("file", file, file.name);
+
+        // Basic Pitch can take 10-60 s on the backend; no client-side fetch
+        // timeout - the server's subprocess timeout is the effective limit.
+        const res = await fetch("/api/transcribe/mp3", { method: "POST", body: form });
+
+        if (!res.ok) {
+          const body = (await res.json().catch(() => ({}))) as { detail?: string };
+          throw new Error(body.detail ?? `Server error ${res.status}`);
+        }
+
+        // Endpoint returns a raw MIDI file (audio/midi) - parse it through the
+        // same Web Worker pipeline as a regular .mid import.
+        const midiBuffer = await res.arrayBuffer();
+        const bufferForDoc = midiBuffer.slice(0);
+        const doc = await parseMidiInWorker(midiBuffer, id, title);
+
+        set({ document: { ...doc, midiBuffer: bufferForDoc } });
+        await syncEngine.loadDocument({ ...doc, midiBuffer: bufferForDoc });
+
+        // Kick off MusicXML transcription so the sheet-music view eventually
+        // populates, mirroring loadMidiFile's pattern. Fire-and-forget.
+        fetchMusicXml(bufferForDoc.slice(0), doc, id).catch(console.warn);
+      } catch (err) {
+        const msg = err instanceof Error ? err.message : String(err);
+        set({ loadError: msg });
+      } finally {
+        set({ isLoadingDocument: false, loadingMessage: null });
+      }
+    },
+
     generateFingering: async () => {
       const { document: doc } = get();
       if (!doc?.musicXml) return;

frontend/src/views/PracticeView.tsx

@@ -32,6 +32,7 @@ export function PracticeView() {
     loadMidiFile,
     loadMusicXmlFile,
     loadPdfFile,
+    loadAudioFile,
     settings,
     updateSettings,
     status,
@@ -117,8 +118,9 @@ export function PracticeView() {
       if (file.name.match(/\.midi?$/i)) loadMidiFile(file);
       else if (file.name.match(/\.(xml|mxl)$/i)) loadMusicXmlFile(file);
       else if (file.name.match(/\.pdf$/i)) loadPdfFile(file);
+      else if (file.name.match(/\.(mp3|wav|m4a|ogg|flac|aac)$/i)) loadAudioFile(file);
     },
-    [loadMidiFile, loadMusicXmlFile, loadPdfFile]
+    [loadMidiFile, loadMusicXmlFile, loadPdfFile, loadAudioFile]
   );
 
   const handleDrop = useCallback(
@@ -265,40 +267,31 @@ export function PracticeView() {
         </div>
       </header>
 
-      {/* ── Main content - layout-aware ────────────────────────────────────── */}
+      {/* ── Main content - layout-aware ──────────────────────────────────────
+          PianoKeyboardView is a SINGLETON at the bottom of the main column so
+          PixiJS doesn't re-init on every view switch (skeleton flash + perf cost).
+          Same for FallingNotesView in falling/all modes - kept mounted across
+          the two modes that show it. Only the conditional top-content elements
+          (PianoModeBackground / SheetMusicView / sheet sidebar) mount and
+          unmount on layoutMode change. */}
       <main className="flex flex-1 overflow-hidden min-h-0 relative">
-        {layoutMode === "piano" && (
+        {/* Optional sheet sidebar (only in "all" mode) */}
-          <div className="flex flex-col flex-1 min-h-0 overflow-hidden">
-            <PianoModeBackground />
-            <PianoKeyboardView />
-          </div>
-        )}
-        {layoutMode === "falling" && (
-          <div className="flex flex-col flex-1 min-h-0 overflow-hidden">
-            <FallingNotesView />
-            <PianoKeyboardView />
-          </div>
-        )}
-        {layoutMode === "sheet" && (
-          <div className="flex flex-col flex-1 min-h-0 overflow-hidden">
-            <SheetMusicView />
-            <PianoKeyboardView />
-          </div>
-        )}
         {layoutMode === "all" && (
-          <>
           <div
             className="flex flex-col flex-shrink-0 min-h-0 overflow-hidden relative"
             style={{ flex: "0 0 42%", minWidth: 260, maxWidth: 440 }}
           >
             <SheetMusicView />
           </div>
+        )}
+
+        {/* Main column: top content varies, piano persists across all modes */}
         <div className="flex flex-col flex-1 min-h-0 overflow-hidden">
-              <FallingNotesView />
+          {layoutMode === "piano" && <PianoModeBackground />}
+          {(layoutMode === "falling" || layoutMode === "all") && <FallingNotesView />}
+          {layoutMode === "sheet" && <SheetMusicView />}
           <PianoKeyboardView />
         </div>
-          </>
-        )}
       </main>
 
       {/* ── Empty state overlay ────────────────────────────────────────────── */}
@@ -362,7 +355,7 @@ function EmptyState({ onImport }: { onImport: (file: File) => void }) {
           No file loaded
         </div>
         <div style={{ fontSize: 13, color: "var(--color-text-muted)", lineHeight: 1.6 }}>
-          Import a MIDI, MusicXML, or PDF sheet music file to start practicing
+          Import a MIDI, MusicXML, PDF sheet music, or audio recording to start practicing
         </div>
       </div>
 
@@ -406,7 +399,7 @@ function EmptyState({ onImport }: { onImport: (file: File) => void }) {
       <input
         ref={fileInputRef}
         type="file"
-        accept=".mid,.midi,.xml,.mxl,.pdf"
+        accept=".mid,.midi,.xml,.mxl,.pdf,.mp3,.wav,.m4a,.ogg,.flac,.aac"
         className="hidden"
         onChange={(e) => {
           const f = e.target.files?.[0];