diff --git a/artifacts/2026-02-22/scanner_missing_invalid_fix.md b/artifacts/2026-02-22/scanner_missing_invalid_fix.md new file mode 100644 index 000000000..0cbcadcaf --- /dev/null +++ b/artifacts/2026-02-22/scanner_missing_invalid_fix.md @@ -0,0 +1,40 @@ +# Artifact Specification: Scanner Missing and Invalid Flag Reset Fix + +## 1. Overview +This specification details the fixes implemented to resolve two major bugs regarding how the application's library scanner processes files and updates item statuses: +1. **Stuck "Issues" Status:** Books and podcasts migrated from v1 databases could have a legacy `isInvalid: true` flag. Normal scans and rescans successfully cleared `isMissing` when files were found, but `isInvalid` was never cleared. This forced items to permanently show up as having "issues" unless the user performed a heavy "Consolidate" action. +2. **Scanner Path Duplication (Symlink Bug):** When libraries were sourced from symlinked directories, `fileUtils.recurseFiles` would inadvertently duplicate paths because the underlying `fs.realpath` resolution of the symlink caused a string mismatch when stripping the root relative path. + +## 2. API & Data Contracts +- **Data Contracts**: No changes to the database schemas. The existing `isMissing` and `isInvalid` schema flags are preserved. +- **Scanner Output**: `LibraryItemScanData`, `BookScanner`, and `PodcastScanner` will actively push `isInvalid: false` to the DB when files are detected. + +## 3. Traceability (Files Modified) + +| Category | File | Change Description | +| :--- | :--- | :--- | +| **Backend** | `server/scanner/LibraryItemScanData.js` | Updated `checkLibraryItemData` to assert `isMissing: false` and `isInvalid: false` if either was `true` | +| **Backend** | `server/scanner/BookScanner.js` | Updated `rescanExistingBookLibraryItem` to assert `isMissing: false` and `isInvalid: false` if either was `true` | +| **Backend** | `server/scanner/PodcastScanner.js` | Updated `rescanExistingPodcastLibraryItem` to assert `isMissing: false` and `isInvalid: false` if either was `true` | +| **Backend** | `server/utils/fileUtils.js` | Refactored `recurseFiles` to use `fs.realpath()` for `relPathToReplace` before string manipulation to ensure correct substring replacement inside symlinks | + +## 4. Architectural Decisions +### Scanner Flags +- **Decision:** Explicitly check `|| existingLibraryItem.isInvalid` before attempting to unset the flags. +- **Reasoning:** Rather than wiping `isInvalid` indiscriminately, we check if the flag is actively `true` and the filesystem says the files exist and load correctly. If so, the item is restored to a fully healthy state. This aligns behavior seamlessly with the `isMissing` logic. + +### Symlink Substring Removal +- **Decision:** Use `fs.realpath(relPathToReplace)` before stripping prefixes. +- **Reasoning:** The `recursiveReaddirAsync` utility was configured with `realPath: true`. It inherently resolves all symlinks for deep-scanned objects. When `fileUtils` tried to truncate the root path `relPathToReplace` using string replacement, it failed on symlinked paths because `item.fullname` was resolved but `relPathToReplace` was not. Ensuring `relPathToReplace` also runs through `fs.realpath` fixes the root mismatch. + +## 5. Verification Plan +1. **Invalid Items Recovery:** + - Restart server. + - Re-scan an item that was historically tagged "invalid" via single-file root folders. + - Verify the UI "issues" count actively clears. +2. **Symlink Scans:** + - Trigger a scan on a directory mapped via symlinks (e.g., `/home/user/...` -> `/mnt/docker/...`). + - Validate that item metadata and paths register with the scanner correctly without dumping duplicate paths (`/home/user/mnt/docker/user...`) in logs or the DB. + +## 6. Limitations & Future Work +- Legacy v1 `isInvalid` items are still expected to exist in long-lived databases until they are actively rescanned or a full-library scan captures them under the new rules. No background database migration was run to mass-clear them, to ensure the file scanner correctly assesses integrity. diff --git a/artifacts/index.md b/artifacts/index.md index 94180cb9e..f2a8fc7f7 100644 --- a/artifacts/index.md +++ b/artifacts/index.md @@ -27,6 +27,7 @@ This index provides a quick reference for specification and documentation files | **2026-02-22** | [match_default_behavior.md](2026-02-22/match_default_behavior.md) | Specification for the new default "Direct Apply" match behavior and Review & Edit button. | | **2026-02-22** | [player_keyboard_shortcuts.md](2026-02-22/player_keyboard_shortcuts.md) | Specification for new player keyboard shortcuts including major skip and chapter jumps. | | **2026-02-22** | [normalized_title_filter.md](2026-02-22/normalized_title_filter.md) | Specification for the Normalized Title / Duplicate Title filter implementations and database columns. | +| **2026-02-22** | [scanner_missing_invalid_fix.md](2026-02-22/scanner_missing_invalid_fix.md) | Specification for fixing `isMissing` and legacy `isInvalid` flag reset logic during scans, and fixing symlink path duplication in `fileUtils`. | | **General** | [docs/consolidate_feature.md](docs/consolidate_feature.md) | Comprehensive documentation for the "Consolidate" feature, including conflict resolution and technical details. | | **General** | [docs/item_restructuring_guide.md](docs/item_restructuring_guide.md) | Guide for Moving, Merging, and Splitting (Promoting) library items. | | **General** | [docs/metadata_management_tools.md](docs/metadata_management_tools.md) | Documentation for Reset Metadata and Batch Reset operations. | diff --git a/server/scanner/BookScanner.js b/server/scanner/BookScanner.js index 63350e0f3..8f4d2a43a 100644 --- a/server/scanner/BookScanner.js +++ b/server/scanner/BookScanner.js @@ -404,9 +404,15 @@ class BookScanner { existingLibraryItem.isMissing = true libraryItemUpdated = true } - } else if (existingLibraryItem.isMissing) { - libraryScan.addLog(LogLevel.INFO, `Book "${bookMetadata.title}" was missing but now has media files. Setting library item as NOT missing`) - existingLibraryItem.isMissing = false + } else if (existingLibraryItem.isMissing || existingLibraryItem.isInvalid) { + if (existingLibraryItem.isMissing) { + libraryScan.addLog(LogLevel.INFO, `Book "${bookMetadata.title}" was missing but now has media files. Setting library item as NOT missing`) + existingLibraryItem.isMissing = false + } + if (existingLibraryItem.isInvalid) { + libraryScan.addLog(LogLevel.INFO, `Book "${bookMetadata.title}" was invalid but now has media files. Setting library item as NOT invalid`) + existingLibraryItem.isInvalid = false + } libraryItemUpdated = true } diff --git a/server/scanner/LibraryItemScanData.js b/server/scanner/LibraryItemScanData.js index d5a4a7a29..fc69ed1be 100644 --- a/server/scanner/LibraryItemScanData.js +++ b/server/scanner/LibraryItemScanData.js @@ -211,9 +211,10 @@ class LibraryItemScanData { existingLibraryItem.ctime = this.ctimeMs this.hasChanges = true } - if (existingLibraryItem.isMissing) { - libraryScan.addLog(LogLevel.DEBUG, `Library item "${existingLibraryItem.relPath}" was missing but now found`) + if (existingLibraryItem.isMissing || existingLibraryItem.isInvalid) { + libraryScan.addLog(LogLevel.DEBUG, `Library item "${existingLibraryItem.relPath}" was missing or invalid but now found`) existingLibraryItem.isMissing = false + existingLibraryItem.isInvalid = false this.hasChanges = true } diff --git a/server/scanner/PodcastScanner.js b/server/scanner/PodcastScanner.js index 6249cf9bc..a130547a2 100644 --- a/server/scanner/PodcastScanner.js +++ b/server/scanner/PodcastScanner.js @@ -239,6 +239,18 @@ class PodcastScanner { libraryItemUpdated = true } + if (existingLibraryItem.isMissing || existingLibraryItem.isInvalid) { + if (existingLibraryItem.isMissing) { + libraryScan.addLog(LogLevel.INFO, `Podcast "${podcastMetadata.title}" was missing but is now found. Setting library item as NOT missing`) + existingLibraryItem.isMissing = false + } + if (existingLibraryItem.isInvalid) { + libraryScan.addLog(LogLevel.INFO, `Podcast "${podcastMetadata.title}" was invalid but is now found. Setting library item as NOT invalid`) + existingLibraryItem.isInvalid = false + } + libraryItemUpdated = true + } + // Save Podcast changes to db if (hasMediaChanges) { await media.save() diff --git a/server/utils/fileUtils.js b/server/utils/fileUtils.js index 9a349bd54..cbc2163ae 100644 --- a/server/utils/fileUtils.js +++ b/server/utils/fileUtils.js @@ -192,6 +192,15 @@ module.exports.recurseFiles = async (path, relPathToReplace = null) => { relPathToReplace = path } + // Handle symlinked directories by using the real path for string replacements + let realPathToReplace = relPathToReplace + try { + realPathToReplace = await fs.realpath(relPathToReplace) + realPathToReplace = filePathToPOSIX(realPathToReplace) + if (!realPathToReplace.endsWith('/')) realPathToReplace += '/' + } catch (err) {} + + const options = { mode: rra.LIST, recursive: true, @@ -219,7 +228,7 @@ module.exports.recurseFiles = async (path, relPathToReplace = null) => { item.fullname = filePathToPOSIX(item.fullname) item.path = filePathToPOSIX(item.path) - const relpath = item.fullname.replace(relPathToReplace, '') + const relpath = item.fullname.replace(realPathToReplace, '') let reldirname = Path.dirname(relpath) if (reldirname === '.') reldirname = '' const dirname = Path.dirname(item.fullname) @@ -249,11 +258,11 @@ module.exports.recurseFiles = async (path, relPathToReplace = null) => { return true }) .map((item) => { - var isInRoot = item.path + '/' === relPathToReplace + var isInRoot = item.path + '/' === realPathToReplace return { name: item.name, - path: item.fullname.replace(relPathToReplace, ''), - reldirpath: isInRoot ? '' : item.path.replace(relPathToReplace, ''), + path: item.fullname.replace(realPathToReplace, ''), + reldirpath: isInRoot ? '' : item.path.replace(realPathToReplace, ''), fullpath: item.fullname, extension: item.extension, deep: item.deep