diff --git a/AGENTS.md b/AGENTS.md index a00d3e36e..4277c7e35 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -138,6 +138,7 @@ OpenAPI documentation available at `docs/openapi.json` - Vue.js components with Composition API where applicable - Sequelize models for database operations - No comments in code unless explicitly requested +- **Title Normalization**: When comparing or grouping by titles, always use `titleNormalized` or the `getNormalizedTitle` utility from `server/utils` to ignore prefixes, punctuation, and non-alphabetic characters. Do not use plain lowercase comparison. ### UI Components & Modals diff --git a/artifacts/2026-02-22/normalized_title_filter.md b/artifacts/2026-02-22/normalized_title_filter.md new file mode 100644 index 000000000..3d59c60a3 --- /dev/null +++ b/artifacts/2026-02-22/normalized_title_filter.md @@ -0,0 +1,28 @@ +# Normalized Title Filter Specification + +## Overview +Added a "Duplicate Title" filter option to the frontend library controls. To support performant querying of duplicate titles across potentially massive libraries, a new `titleNormalized` concept was introduced to the backend models and database schema, allowing for fast `$Op.in` and `COUNT() > 1` matching via SQL. + +## Verification Plan +1. Start server and ensure no migration errors occur +2. Verify existing books/podcasts have their `titleNormalized` fields backfilled +3. Select "Duplicate Title" in the library filter +4. Verify results contain books/podcasts that are identical or differ only by punctuation/casing + +## Architectural Decisions +- **Pre-computed database fields**: Chosen over in-memory string similarity calculations to maintain $O(1)$ query complexity at filter time instead of $O(N^2)$ Node.js looping. +- **`getNormalizedTitle` Utility**: Centralized normalization to strip all non-unicode letter characters (`/[^\p{L}]/gu`), numbers, spaces, and ignore common sorting prefixes. +- **SQLite Triggers**: Implemented via migrations to automatically keep `libraryItems.titleNormalized` in sync with updates to `books.title` and `podcasts.title`. + +## Traceability +| File | Changes | +| :--- | :--- | +| `server/utils/index.js` | Added `getNormalizedTitle` utility function. | +| `server/models/Book.js` | Added `titleNormalized` property and updated saving logic. | +| `server/models/Podcast.js` | Added `titleNormalized` property and updated saving logic. | +| `server/models/LibraryItem.js` | Added `titleNormalized` property, index, and hook copying from media. | +| `server/migrations/v2.32.9-*-columns.js` | Added migration to alter tables, backfill data, add hooks & triggers. | +| `server/utils/queries/libraryItemsBookFilters.js` | Added 'duplicates' filter handler to SQL mapping. | +| `server/utils/queries/libraryItemsPodcastFilters.js` | Added 'duplicates' filter handler to SQL mapping. | +| `client/components/controls/LibraryFilterSelect.vue` | Added "Duplicate Title" options to dropdown items. | +| `client/strings/en-us.json` | Added `LabelDuplicateTitle` translation string. | diff --git a/artifacts/2026-02-22/player_keyboard_shortcuts.md b/artifacts/2026-02-22/player_keyboard_shortcuts.md new file mode 100644 index 000000000..7b55789c1 --- /dev/null +++ b/artifacts/2026-02-22/player_keyboard_shortcuts.md @@ -0,0 +1,42 @@ +# Player Keyboard Shortcuts Specification + +## Overview +This feature introduces new keyboard shortcuts to the audio player, enabling improved playback control for power users. This expands on the existing standard keyboard interactions to include dedicated major jumps (60 seconds) and chapter navigation mapping. + +## New Keyboard Shortcuts + +The following hotkey definitions have been added or updated in the `AudioPlayer` context: + +| Context | Action | Shortcut | Description | +| :--- | :--- | :--- | :--- | +| AudioPlayer | `JUMP_FORWARD_ALT` | `Shift + Right Arrow` | Alternative shortcut for the user's standard jump forward amount. | +| AudioPlayer | `JUMP_BACKWARD_ALT` | `Shift + Left Arrow` | Alternative shortcut for the user's standard jump backward amount. | +| AudioPlayer | `JUMP_FORWARD_MAJOR` | `Ctrl + Shift + Right Arrow` | Hardcoded 60-second jump forward. | +| AudioPlayer | `JUMP_BACKWARD_MAJOR`| `Ctrl + Shift + Left Arrow` | Hardcoded 60-second jump backward. | +| AudioPlayer | `NEXT_CHAPTER` | `Shift + Up Arrow` | Skips directly to the start of the next chapter. | +| AudioPlayer | `PREV_CHAPTER` | `Shift + Down Arrow` | Skips back to the previous chapter mark (or restarts the chapter). | +| AudioPlayer | `INCREASE_PLAYBACK_RATE` | `]` (BracketRight) | Previously `Shift + Up Arrow`. Now moved to accommodate chapter navigation. | +| AudioPlayer | `DECREASE_PLAYBACK_RATE` | `[` (BracketLeft) | Previously `Shift + Down Arrow`. Now moved to accommodate chapter navigation. | + +## Implementation Details + +### Modifier Parsing +- Upgraded the hotkey parsing logic within `getHotkeyName(e)` in `client/layouts/default.vue` and `client/pages/share/_slug.vue` to actively capture `ctrlKey` and `altKey` modifier combinations, aligning with the pattern used elsewhere in the UI. + +### Player Core +- `jumpForwardMajor` and `jumpBackwardMajor` logic was integrated directly into `client/components/player/PlayerUi.vue` using its `seek` primitive capability, computing relative to the local `currentTime` and `duration`. +- Mapped the chapter navigation constants (`NEXT_CHAPTER` and `PREV_CHAPTER`) directly to existing handler functions (`goToNext()` and `prevChapter()`), keeping modifications self-contained. + +## Files Modified +| File Location | Category | Reason for Change | +| :--- | :--- | :--- | +| `AGENTS.md` | **Documentation** | Document how AI agents can lookup previous artifact specifications in `artifacts/index.md`. | +| `artifacts/docs/ux_power_user_shortcuts.md` | **Documentation** | Added Audio Player Shortcuts section. | +| `client/plugins/constants.js` | **Frontend** | Add `BracketLeft` and `BracketRight` to KeyNames registry; redefine `Hotkeys.AudioPlayer` with the new combinations. | +| `client/components/player/PlayerUi.vue` | **Frontend** | Implement the jump magnitude logic and intercept the actual shortcut emissions. | +| `client/layouts/default.vue` | **Frontend** | Add robust modifier flag interpretation in the main view wrapper. | +| `client/pages/share/_slug.vue` | **Frontend** | Replicate the exact same modifier key evaluation logic for the standalone unauthenticated player widget. | + +## Edge Cases Addressed +- **Duration Boundary checks**: Hardcoded jumps (60 sec) use bounds checking to prevent seeking below position 0, or past the overall duration of the target media. +- **Playback Rate Shortcut conflict**: Addressed by cleanly replacing the default array to brackets `[` and `]` making intuitive sense for increasing and decreasing numeric magnitudes. diff --git a/artifacts/index.md b/artifacts/index.md index 261599412..94180cb9e 100644 --- a/artifacts/index.md +++ b/artifacts/index.md @@ -26,6 +26,7 @@ This index provides a quick reference for specification and documentation files | **2026-02-22** | [centralized_keyboard_shortcuts.md](2026-02-22/centralized_keyboard_shortcuts.md) | Specification for centralizing keyboard shortcut definitions into a single configuration file. | | **2026-02-22** | [match_default_behavior.md](2026-02-22/match_default_behavior.md) | Specification for the new default "Direct Apply" match behavior and Review & Edit button. | | **2026-02-22** | [player_keyboard_shortcuts.md](2026-02-22/player_keyboard_shortcuts.md) | Specification for new player keyboard shortcuts including major skip and chapter jumps. | +| **2026-02-22** | [normalized_title_filter.md](2026-02-22/normalized_title_filter.md) | Specification for the Normalized Title / Duplicate Title filter implementations and database columns. | | **General** | [docs/consolidate_feature.md](docs/consolidate_feature.md) | Comprehensive documentation for the "Consolidate" feature, including conflict resolution and technical details. | | **General** | [docs/item_restructuring_guide.md](docs/item_restructuring_guide.md) | Guide for Moving, Merging, and Splitting (Promoting) library items. | | **General** | [docs/metadata_management_tools.md](docs/metadata_management_tools.md) | Documentation for Reset Metadata and Batch Reset operations. | diff --git a/client/components/controls/LibraryFilterSelect.vue b/client/components/controls/LibraryFilterSelect.vue index fbcf0089f..f6e59d5d3 100644 --- a/client/components/controls/LibraryFilterSelect.vue +++ b/client/components/controls/LibraryFilterSelect.vue @@ -236,6 +236,11 @@ export default { value: 'issues', sublist: false }, + { + text: this.$strings.LabelDuplicateTitle || 'Duplicate Title', + value: 'duplicates', + sublist: false + }, { text: this.$strings.LabelRSSFeedOpen, value: 'feed-open', @@ -294,6 +299,11 @@ export default { value: 'issues', sublist: false }, + { + text: this.$strings.LabelDuplicateTitle || 'Duplicate Title', + value: 'duplicates', + sublist: false + }, { text: this.$strings.LabelRSSFeedOpen, value: 'feed-open', diff --git a/client/strings/en-us.json b/client/strings/en-us.json index 98faff97e..2bd25ea5b 100644 --- a/client/strings/en-us.json +++ b/client/strings/en-us.json @@ -475,6 +475,7 @@ "LabelMinute": "Minute", "LabelMinutes": "Minutes", "LabelMissing": "Missing", + "LabelDuplicateTitle": "Duplicate Title", "LabelMissingEbook": "Has no ebook", "LabelMissingSupplementaryEbook": "Has no supplementary ebook", "LabelMoveToLibrary": "Move to Library", diff --git a/server/migrations/v2.32.9-add-title-normalized-columns.js b/server/migrations/v2.32.9-add-title-normalized-columns.js new file mode 100644 index 000000000..72b9c04e6 --- /dev/null +++ b/server/migrations/v2.32.9-add-title-normalized-columns.js @@ -0,0 +1,159 @@ +const util = require('util') +const { getNormalizedTitle } = require('../utils') + +/** + * @typedef MigrationContext + * @property {import('sequelize').QueryInterface} queryInterface - a suquelize QueryInterface object. + * @property {import('../Logger')} logger - a Logger object. + * + * @typedef MigrationOptions + * @property {MigrationContext} context - an object containing the migration context. + */ + +const migrationVersion = '2.32.9' +const migrationName = `${migrationVersion}-add-title-normalized-columns` +const loggerPrefix = `[${migrationVersion} migration]` + +async function up({ context: { queryInterface, logger } }) { + logger.info(`${loggerPrefix} UPGRADE BEGIN: ${migrationName}`) + + // 1. Add columns + await addColumn(queryInterface, logger, 'libraryItems', 'titleNormalized', { type: queryInterface.sequelize.Sequelize.STRING, allowNull: true }) + await addColumn(queryInterface, logger, 'books', 'titleNormalized', { type: queryInterface.sequelize.Sequelize.STRING, allowNull: true }) + await addColumn(queryInterface, logger, 'podcasts', 'titleNormalized', { type: queryInterface.sequelize.Sequelize.STRING, allowNull: true }) + + // 2. Backfill data for books synchronously + logger.info(`${loggerPrefix} Backfilling titleNormalized for books`) + const books = await queryInterface.sequelize.query('SELECT id, title FROM books', { type: queryInterface.sequelize.QueryTypes.SELECT }) + for (const book of books) { + if (book.title) { + const titleNormalized = getNormalizedTitle(book.title) + await queryInterface.sequelize.query('UPDATE books SET titleNormalized = :titleNormalized WHERE id = :id', { + replacements: { titleNormalized, id: book.id } + }) + } + } + + // Backfill data for podcasts + logger.info(`${loggerPrefix} Backfilling titleNormalized for podcasts`) + const podcasts = await queryInterface.sequelize.query('SELECT id, title FROM podcasts', { type: queryInterface.sequelize.QueryTypes.SELECT }) + for (const podcast of podcasts) { + if (podcast.title) { + const titleNormalized = getNormalizedTitle(podcast.title) + await queryInterface.sequelize.query('UPDATE podcasts SET titleNormalized = :titleNormalized WHERE id = :id', { + replacements: { titleNormalized, id: podcast.id } + }) + } + } + + // 3. Copy from books/podcasts to libraryItems + await copyColumn(queryInterface, logger, 'books', 'titleNormalized', 'id', 'libraryItems', 'titleNormalized', 'mediaId') + await copyColumn(queryInterface, logger, 'podcasts', 'titleNormalized', 'id', 'libraryItems', 'titleNormalized', 'mediaId') + + // 4. Add triggers + await addTrigger(queryInterface, logger, 'books', 'titleNormalized', 'id', 'libraryItems', 'titleNormalized', 'mediaId') + await addTrigger(queryInterface, logger, 'podcasts', 'titleNormalized', 'id', 'libraryItems', 'titleNormalized', 'mediaId') + + // 5. Add index on libraryItems + await addIndex(queryInterface, logger, 'libraryItems', ['libraryId', 'mediaType', { name: 'titleNormalized', collate: 'NOCASE' }]) + + logger.info(`${loggerPrefix} UPGRADE END: ${migrationName}`) +} + +async function down({ context: { queryInterface, logger } }) { + logger.info(`${loggerPrefix} DOWNGRADE BEGIN: ${migrationName}`) + + await removeIndex(queryInterface, logger, 'libraryItems', ['libraryId', 'mediaType', 'titleNormalized']) + + await removeTrigger(queryInterface, logger, 'libraryItems', 'titleNormalized', 'books') + await removeTrigger(queryInterface, logger, 'libraryItems', 'titleNormalized', 'podcasts') + + await removeColumn(queryInterface, logger, 'libraryItems', 'titleNormalized') + await removeColumn(queryInterface, logger, 'books', 'titleNormalized') + await removeColumn(queryInterface, logger, 'podcasts', 'titleNormalized') + + logger.info(`${loggerPrefix} DOWNGRADE END: ${migrationName}`) +} + +async function addIndex(queryInterface, logger, tableName, columns) { + const columnString = columns.map((column) => util.inspect(column)).join(', ') + const indexName = convertToSnakeCase(`${tableName}_${columns.map((column) => (typeof column === 'string' ? column : column.name)).join('_')}`) + try { + logger.info(`${loggerPrefix} adding index on [${columnString}] to table ${tableName}. index name: ${indexName}"`) + await queryInterface.addIndex(tableName, columns) + logger.info(`${loggerPrefix} added index on [${columnString}] to table ${tableName}. index name: ${indexName}"`) + } catch (error) { + if (error.name === 'SequelizeDatabaseError' && error.message.includes('already exists')) { + logger.info(`${loggerPrefix} index [${columnString}] for table "${tableName}" already exists`) + } else { + throw error + } + } +} + +async function removeIndex(queryInterface, logger, tableName, columns) { + logger.info(`${loggerPrefix} removing index [${columns.join(', ')}] from table "${tableName}"`) + try { + await queryInterface.removeIndex(tableName, columns) + logger.info(`${loggerPrefix} removed index [${columns.join(', ')}] from table "${tableName}"`) + } catch (error) {} +} + +async function addColumn(queryInterface, logger, table, column, options) { + logger.info(`${loggerPrefix} adding column "${column}" to table "${table}"`) + const tableDescription = await queryInterface.describeTable(table) + if (!tableDescription[column]) { + await queryInterface.addColumn(table, column, options) + logger.info(`${loggerPrefix} added column "${column}" to table "${table}"`) + } else { + logger.info(`${loggerPrefix} column "${column}" already exists in table "${table}"`) + } +} + +async function removeColumn(queryInterface, logger, table, column) { + logger.info(`${loggerPrefix} removing column "${column}" from table "${table}"`) + await queryInterface.removeColumn(table, column) + logger.info(`${loggerPrefix} removed column "${column}" from table "${table}"`) +} + +async function copyColumn(queryInterface, logger, sourceTable, sourceColumn, sourceIdColumn, targetTable, targetColumn, targetIdColumn) { + logger.info(`${loggerPrefix} copying column "${sourceColumn}" from table "${sourceTable}" to table "${targetTable}"`) + await queryInterface.sequelize.query(` + UPDATE ${targetTable} + SET ${targetColumn} = ${sourceTable}.${sourceColumn} + FROM ${sourceTable} + WHERE ${targetTable}.${targetIdColumn} = ${sourceTable}.${sourceIdColumn} + `) + logger.info(`${loggerPrefix} copied column "${sourceColumn}" from table "${sourceTable}" to table "${targetTable}"`) +} + +async function addTrigger(queryInterface, logger, sourceTable, sourceColumn, sourceIdColumn, targetTable, targetColumn, targetIdColumn) { + logger.info(`${loggerPrefix} adding trigger to update ${targetTable}.${targetColumn} when ${sourceTable}.${sourceColumn} is updated`) + const triggerName = convertToSnakeCase(`update_${targetTable}_${targetColumn}_from_${sourceTable}`) + + await queryInterface.sequelize.query(`DROP TRIGGER IF EXISTS ${triggerName}`) + + await queryInterface.sequelize.query(` + CREATE TRIGGER ${triggerName} + AFTER UPDATE OF ${sourceColumn} ON ${sourceTable} + FOR EACH ROW + BEGIN + UPDATE ${targetTable} + SET ${targetColumn} = NEW.${sourceColumn} + WHERE ${targetTable}.${targetIdColumn} = NEW.${sourceIdColumn}; + END; + `) + logger.info(`${loggerPrefix} added trigger.`) +} + +async function removeTrigger(queryInterface, logger, targetTable, targetColumn, sourceTable) { + logger.info(`${loggerPrefix} removing trigger`) + const triggerName = convertToSnakeCase(`update_${targetTable}_${targetColumn}_from_${sourceTable}`) + await queryInterface.sequelize.query(`DROP TRIGGER IF EXISTS ${triggerName}`) +} + +function convertToSnakeCase(str) { + return str.replace(/([A-Z])/g, '_$1').toLowerCase() +} + +module.exports = { up, down } diff --git a/server/models/Book.js b/server/models/Book.js index c1bbb14a5..752dd8047 100644 --- a/server/models/Book.js +++ b/server/models/Book.js @@ -146,6 +146,7 @@ class Book extends Model { }, title: DataTypes.STRING, titleIgnorePrefix: DataTypes.STRING, + titleNormalized: DataTypes.STRING, subtitle: DataTypes.STRING, publishedYear: DataTypes.STRING, publishedDate: DataTypes.STRING, @@ -407,7 +408,9 @@ class Book extends Model { this[key] = payload.metadata[key] || null if (key === 'title') { + const { getTitleIgnorePrefix, getNormalizedTitle } = require('../utils') this.titleIgnorePrefix = getTitleIgnorePrefix(this.title) + this.titleNormalized = getNormalizedTitle(this.title) } hasUpdates = true diff --git a/server/models/LibraryItem.js b/server/models/LibraryItem.js index 9b57d7eba..af1054d88 100644 --- a/server/models/LibraryItem.js +++ b/server/models/LibraryItem.js @@ -78,6 +78,8 @@ class LibraryItem extends Model { /** @type {string} */ this.titleIgnorePrefix // Only used for sorting /** @type {string} */ + this.titleNormalized // Only used for sorting + /** @type {string} */ this.authorNamesFirstLast // Only used for sorting /** @type {string} */ this.authorNamesLastFirst // Only used for sorting @@ -687,6 +689,7 @@ class LibraryItem extends Model { extraData: DataTypes.JSON, title: DataTypes.STRING, titleIgnorePrefix: DataTypes.STRING, + titleNormalized: DataTypes.STRING, authorNamesFirstLast: DataTypes.STRING, authorNamesLastFirst: DataTypes.STRING, isNotConsolidated: { @@ -719,6 +722,9 @@ class LibraryItem extends Model { { fields: ['libraryId', 'mediaType', { name: 'titleIgnorePrefix', collate: 'NOCASE' }] }, + { + fields: ['libraryId', 'mediaType', { name: 'titleNormalized', collate: 'NOCASE' }] + }, { fields: ['libraryId', 'mediaType', { name: 'authorNamesFirstLast', collate: 'NOCASE' }] }, @@ -795,6 +801,7 @@ class LibraryItem extends Model { if (instance.media) { instance.title = instance.media.title instance.titleIgnorePrefix = instance.media.titleIgnorePrefix + instance.titleNormalized = instance.media.titleNormalized if (instance.isBook) { if (instance.media.authors !== undefined) { instance.authorNamesFirstLast = instance.media.authorName diff --git a/server/models/Podcast.js b/server/models/Podcast.js index 394df0bef..1a3e029d8 100644 --- a/server/models/Podcast.js +++ b/server/models/Podcast.js @@ -1,5 +1,5 @@ const { DataTypes, Model } = require('sequelize') -const { getTitlePrefixAtEnd, getTitleIgnorePrefix } = require('../utils') +const { getTitlePrefixAtEnd, getTitleIgnorePrefix, getNormalizedTitle } = require('../utils') const Logger = require('../Logger') const libraryItemsPodcastFilters = require('../utils/queries/libraryItemsPodcastFilters') const htmlSanitizer = require('../utils/htmlSanitizer') @@ -93,6 +93,7 @@ class Podcast extends Model { { title, titleIgnorePrefix: getTitleIgnorePrefix(title), + titleNormalized: getNormalizedTitle(title), author: typeof payload.metadata.author === 'string' ? payload.metadata.author : null, releaseDate: typeof payload.metadata.releaseDate === 'string' ? payload.metadata.releaseDate : null, feedURL: typeof payload.metadata.feedUrl === 'string' ? payload.metadata.feedUrl : null, @@ -130,6 +131,7 @@ class Podcast extends Model { }, title: DataTypes.STRING, titleIgnorePrefix: DataTypes.STRING, + titleNormalized: DataTypes.STRING, author: DataTypes.STRING, releaseDate: DataTypes.STRING, feedURL: DataTypes.STRING, @@ -257,6 +259,7 @@ class Podcast extends Model { if (key === 'title') { this.titleIgnorePrefix = getTitleIgnorePrefix(this.title) + this.titleNormalized = getNormalizedTitle(this.title) } hasUpdates = true diff --git a/server/utils/index.js b/server/utils/index.js index c7700a783..d38944c6c 100644 --- a/server/utils/index.js +++ b/server/utils/index.js @@ -191,6 +191,18 @@ module.exports.getTitleIgnorePrefix = (title) => { return getTitleParts(title)[0] } +/** + * Get normalized title to use for grouping duplicates + * Removes non-alphabetic characters (numbers, punctuation, spaces) + * @param {string} title + * @returns {string} + */ +module.exports.getNormalizedTitle = (title) => { + if (!title) return '' + const sortTitle = getTitleParts(title)[0] || title + return sortTitle.toLowerCase().replace(/[^\p{L}]/gu, '') +} + /** * Put sorting prefix at the end of title * @example "The Good Book" => "Good Book, The" diff --git a/server/utils/queries/libraryItemsBookFilters.js b/server/utils/queries/libraryItemsBookFilters.js index f128b2d0c..bc3588300 100644 --- a/server/utils/queries/libraryItemsBookFilters.js +++ b/server/utils/queries/libraryItemsBookFilters.js @@ -515,6 +515,10 @@ module.exports = { isInvalid: true } ] + } else if (filterGroup === 'duplicates') { + libraryItemWhere['titleNormalized'] = { + [Sequelize.Op.in]: Sequelize.literal(`(SELECT titleNormalized FROM libraryItems WHERE libraryId = '${libraryId}' AND titleNormalized IS NOT NULL AND titleNormalized != '' GROUP BY titleNormalized HAVING COUNT(titleNormalized) > 1)`) + } } else if (filterGroup === 'progress' && user) { const mediaProgressWhere = { userId: user.id diff --git a/server/utils/queries/libraryItemsPodcastFilters.js b/server/utils/queries/libraryItemsPodcastFilters.js index 6bcddabf2..937b3194d 100644 --- a/server/utils/queries/libraryItemsPodcastFilters.js +++ b/server/utils/queries/libraryItemsPodcastFilters.js @@ -168,6 +168,10 @@ module.exports = { isInvalid: true } ] + } else if (filterGroup === 'duplicates') { + libraryItemWhere['titleNormalized'] = { + [Sequelize.Op.in]: Sequelize.literal(`(SELECT titleNormalized FROM libraryItems WHERE libraryId = '${libraryId}' AND titleNormalized IS NOT NULL AND titleNormalized != '' GROUP BY titleNormalized HAVING COUNT(titleNormalized) > 1)`) + } } else if (filterGroup === 'recent') { libraryItemWhere['createdAt'] = { [Sequelize.Op.gte]: new Date(new Date() - 60 * 24 * 60 * 60 * 1000) // 60 days ago