feat: implement duplicate title normalized filter

This commit is contained in:
Tiberiu Ichim 2026-02-22 16:46:14 +02:00
parent aa85106681
commit ead215e777
13 changed files with 276 additions and 1 deletions

View file

@ -138,6 +138,7 @@ OpenAPI documentation available at `docs/openapi.json`
- Vue.js components with Composition API where applicable
- Sequelize models for database operations
- No comments in code unless explicitly requested
- **Title Normalization**: When comparing or grouping by titles, always use `titleNormalized` or the `getNormalizedTitle` utility from `server/utils` to ignore prefixes, punctuation, and non-alphabetic characters. Do not use plain lowercase comparison.
### UI Components & Modals

View file

@ -0,0 +1,28 @@
# Normalized Title Filter Specification
## Overview
Added a "Duplicate Title" filter option to the frontend library controls. To support performant querying of duplicate titles across potentially massive libraries, a new `titleNormalized` concept was introduced to the backend models and database schema, allowing for fast `$Op.in` and `COUNT() > 1` matching via SQL.
## Verification Plan
1. Start server and ensure no migration errors occur
2. Verify existing books/podcasts have their `titleNormalized` fields backfilled
3. Select "Duplicate Title" in the library filter
4. Verify results contain books/podcasts that are identical or differ only by punctuation/casing
## Architectural Decisions
- **Pre-computed database fields**: Chosen over in-memory string similarity calculations to maintain $O(1)$ query complexity at filter time instead of $O(N^2)$ Node.js looping.
- **`getNormalizedTitle` Utility**: Centralized normalization to strip all non-unicode letter characters (`/[^\p{L}]/gu`), numbers, spaces, and ignore common sorting prefixes.
- **SQLite Triggers**: Implemented via migrations to automatically keep `libraryItems.titleNormalized` in sync with updates to `books.title` and `podcasts.title`.
## Traceability
| File | Changes |
| :--- | :--- |
| `server/utils/index.js` | Added `getNormalizedTitle` utility function. |
| `server/models/Book.js` | Added `titleNormalized` property and updated saving logic. |
| `server/models/Podcast.js` | Added `titleNormalized` property and updated saving logic. |
| `server/models/LibraryItem.js` | Added `titleNormalized` property, index, and hook copying from media. |
| `server/migrations/v2.32.9-*-columns.js` | Added migration to alter tables, backfill data, add hooks & triggers. |
| `server/utils/queries/libraryItemsBookFilters.js` | Added 'duplicates' filter handler to SQL mapping. |
| `server/utils/queries/libraryItemsPodcastFilters.js` | Added 'duplicates' filter handler to SQL mapping. |
| `client/components/controls/LibraryFilterSelect.vue` | Added "Duplicate Title" options to dropdown items. |
| `client/strings/en-us.json` | Added `LabelDuplicateTitle` translation string. |

View file

@ -0,0 +1,42 @@
# Player Keyboard Shortcuts Specification
## Overview
This feature introduces new keyboard shortcuts to the audio player, enabling improved playback control for power users. This expands on the existing standard keyboard interactions to include dedicated major jumps (60 seconds) and chapter navigation mapping.
## New Keyboard Shortcuts
The following hotkey definitions have been added or updated in the `AudioPlayer` context:
| Context | Action | Shortcut | Description |
| :--- | :--- | :--- | :--- |
| AudioPlayer | `JUMP_FORWARD_ALT` | `Shift + Right Arrow` | Alternative shortcut for the user's standard jump forward amount. |
| AudioPlayer | `JUMP_BACKWARD_ALT` | `Shift + Left Arrow` | Alternative shortcut for the user's standard jump backward amount. |
| AudioPlayer | `JUMP_FORWARD_MAJOR` | `Ctrl + Shift + Right Arrow` | Hardcoded 60-second jump forward. |
| AudioPlayer | `JUMP_BACKWARD_MAJOR`| `Ctrl + Shift + Left Arrow` | Hardcoded 60-second jump backward. |
| AudioPlayer | `NEXT_CHAPTER` | `Shift + Up Arrow` | Skips directly to the start of the next chapter. |
| AudioPlayer | `PREV_CHAPTER` | `Shift + Down Arrow` | Skips back to the previous chapter mark (or restarts the chapter). |
| AudioPlayer | `INCREASE_PLAYBACK_RATE` | `]` (BracketRight) | Previously `Shift + Up Arrow`. Now moved to accommodate chapter navigation. |
| AudioPlayer | `DECREASE_PLAYBACK_RATE` | `[` (BracketLeft) | Previously `Shift + Down Arrow`. Now moved to accommodate chapter navigation. |
## Implementation Details
### Modifier Parsing
- Upgraded the hotkey parsing logic within `getHotkeyName(e)` in `client/layouts/default.vue` and `client/pages/share/_slug.vue` to actively capture `ctrlKey` and `altKey` modifier combinations, aligning with the pattern used elsewhere in the UI.
### Player Core
- `jumpForwardMajor` and `jumpBackwardMajor` logic was integrated directly into `client/components/player/PlayerUi.vue` using its `seek` primitive capability, computing relative to the local `currentTime` and `duration`.
- Mapped the chapter navigation constants (`NEXT_CHAPTER` and `PREV_CHAPTER`) directly to existing handler functions (`goToNext()` and `prevChapter()`), keeping modifications self-contained.
## Files Modified
| File Location | Category | Reason for Change |
| :--- | :--- | :--- |
| `AGENTS.md` | **Documentation** | Document how AI agents can lookup previous artifact specifications in `artifacts/index.md`. |
| `artifacts/docs/ux_power_user_shortcuts.md` | **Documentation** | Added Audio Player Shortcuts section. |
| `client/plugins/constants.js` | **Frontend** | Add `BracketLeft` and `BracketRight` to KeyNames registry; redefine `Hotkeys.AudioPlayer` with the new combinations. |
| `client/components/player/PlayerUi.vue` | **Frontend** | Implement the jump magnitude logic and intercept the actual shortcut emissions. |
| `client/layouts/default.vue` | **Frontend** | Add robust modifier flag interpretation in the main view wrapper. |
| `client/pages/share/_slug.vue` | **Frontend** | Replicate the exact same modifier key evaluation logic for the standalone unauthenticated player widget. |
## Edge Cases Addressed
- **Duration Boundary checks**: Hardcoded jumps (60 sec) use bounds checking to prevent seeking below position 0, or past the overall duration of the target media.
- **Playback Rate Shortcut conflict**: Addressed by cleanly replacing the default array to brackets `[` and `]` making intuitive sense for increasing and decreasing numeric magnitudes.

View file

@ -26,6 +26,7 @@ This index provides a quick reference for specification and documentation files
| **2026-02-22** | [centralized_keyboard_shortcuts.md](2026-02-22/centralized_keyboard_shortcuts.md) | Specification for centralizing keyboard shortcut definitions into a single configuration file. |
| **2026-02-22** | [match_default_behavior.md](2026-02-22/match_default_behavior.md) | Specification for the new default "Direct Apply" match behavior and Review & Edit button. |
| **2026-02-22** | [player_keyboard_shortcuts.md](2026-02-22/player_keyboard_shortcuts.md) | Specification for new player keyboard shortcuts including major skip and chapter jumps. |
| **2026-02-22** | [normalized_title_filter.md](2026-02-22/normalized_title_filter.md) | Specification for the Normalized Title / Duplicate Title filter implementations and database columns. |
| **General** | [docs/consolidate_feature.md](docs/consolidate_feature.md) | Comprehensive documentation for the "Consolidate" feature, including conflict resolution and technical details. |
| **General** | [docs/item_restructuring_guide.md](docs/item_restructuring_guide.md) | Guide for Moving, Merging, and Splitting (Promoting) library items. |
| **General** | [docs/metadata_management_tools.md](docs/metadata_management_tools.md) | Documentation for Reset Metadata and Batch Reset operations. |

View file

@ -236,6 +236,11 @@ export default {
value: 'issues',
sublist: false
},
{
text: this.$strings.LabelDuplicateTitle || 'Duplicate Title',
value: 'duplicates',
sublist: false
},
{
text: this.$strings.LabelRSSFeedOpen,
value: 'feed-open',
@ -294,6 +299,11 @@ export default {
value: 'issues',
sublist: false
},
{
text: this.$strings.LabelDuplicateTitle || 'Duplicate Title',
value: 'duplicates',
sublist: false
},
{
text: this.$strings.LabelRSSFeedOpen,
value: 'feed-open',

View file

@ -475,6 +475,7 @@
"LabelMinute": "Minute",
"LabelMinutes": "Minutes",
"LabelMissing": "Missing",
"LabelDuplicateTitle": "Duplicate Title",
"LabelMissingEbook": "Has no ebook",
"LabelMissingSupplementaryEbook": "Has no supplementary ebook",
"LabelMoveToLibrary": "Move to Library",

View file

@ -0,0 +1,159 @@
const util = require('util')
const { getNormalizedTitle } = require('../utils')
/**
* @typedef MigrationContext
* @property {import('sequelize').QueryInterface} queryInterface - a suquelize QueryInterface object.
* @property {import('../Logger')} logger - a Logger object.
*
* @typedef MigrationOptions
* @property {MigrationContext} context - an object containing the migration context.
*/
const migrationVersion = '2.32.9'
const migrationName = `${migrationVersion}-add-title-normalized-columns`
const loggerPrefix = `[${migrationVersion} migration]`
async function up({ context: { queryInterface, logger } }) {
logger.info(`${loggerPrefix} UPGRADE BEGIN: ${migrationName}`)
// 1. Add columns
await addColumn(queryInterface, logger, 'libraryItems', 'titleNormalized', { type: queryInterface.sequelize.Sequelize.STRING, allowNull: true })
await addColumn(queryInterface, logger, 'books', 'titleNormalized', { type: queryInterface.sequelize.Sequelize.STRING, allowNull: true })
await addColumn(queryInterface, logger, 'podcasts', 'titleNormalized', { type: queryInterface.sequelize.Sequelize.STRING, allowNull: true })
// 2. Backfill data for books synchronously
logger.info(`${loggerPrefix} Backfilling titleNormalized for books`)
const books = await queryInterface.sequelize.query('SELECT id, title FROM books', { type: queryInterface.sequelize.QueryTypes.SELECT })
for (const book of books) {
if (book.title) {
const titleNormalized = getNormalizedTitle(book.title)
await queryInterface.sequelize.query('UPDATE books SET titleNormalized = :titleNormalized WHERE id = :id', {
replacements: { titleNormalized, id: book.id }
})
}
}
// Backfill data for podcasts
logger.info(`${loggerPrefix} Backfilling titleNormalized for podcasts`)
const podcasts = await queryInterface.sequelize.query('SELECT id, title FROM podcasts', { type: queryInterface.sequelize.QueryTypes.SELECT })
for (const podcast of podcasts) {
if (podcast.title) {
const titleNormalized = getNormalizedTitle(podcast.title)
await queryInterface.sequelize.query('UPDATE podcasts SET titleNormalized = :titleNormalized WHERE id = :id', {
replacements: { titleNormalized, id: podcast.id }
})
}
}
// 3. Copy from books/podcasts to libraryItems
await copyColumn(queryInterface, logger, 'books', 'titleNormalized', 'id', 'libraryItems', 'titleNormalized', 'mediaId')
await copyColumn(queryInterface, logger, 'podcasts', 'titleNormalized', 'id', 'libraryItems', 'titleNormalized', 'mediaId')
// 4. Add triggers
await addTrigger(queryInterface, logger, 'books', 'titleNormalized', 'id', 'libraryItems', 'titleNormalized', 'mediaId')
await addTrigger(queryInterface, logger, 'podcasts', 'titleNormalized', 'id', 'libraryItems', 'titleNormalized', 'mediaId')
// 5. Add index on libraryItems
await addIndex(queryInterface, logger, 'libraryItems', ['libraryId', 'mediaType', { name: 'titleNormalized', collate: 'NOCASE' }])
logger.info(`${loggerPrefix} UPGRADE END: ${migrationName}`)
}
async function down({ context: { queryInterface, logger } }) {
logger.info(`${loggerPrefix} DOWNGRADE BEGIN: ${migrationName}`)
await removeIndex(queryInterface, logger, 'libraryItems', ['libraryId', 'mediaType', 'titleNormalized'])
await removeTrigger(queryInterface, logger, 'libraryItems', 'titleNormalized', 'books')
await removeTrigger(queryInterface, logger, 'libraryItems', 'titleNormalized', 'podcasts')
await removeColumn(queryInterface, logger, 'libraryItems', 'titleNormalized')
await removeColumn(queryInterface, logger, 'books', 'titleNormalized')
await removeColumn(queryInterface, logger, 'podcasts', 'titleNormalized')
logger.info(`${loggerPrefix} DOWNGRADE END: ${migrationName}`)
}
async function addIndex(queryInterface, logger, tableName, columns) {
const columnString = columns.map((column) => util.inspect(column)).join(', ')
const indexName = convertToSnakeCase(`${tableName}_${columns.map((column) => (typeof column === 'string' ? column : column.name)).join('_')}`)
try {
logger.info(`${loggerPrefix} adding index on [${columnString}] to table ${tableName}. index name: ${indexName}"`)
await queryInterface.addIndex(tableName, columns)
logger.info(`${loggerPrefix} added index on [${columnString}] to table ${tableName}. index name: ${indexName}"`)
} catch (error) {
if (error.name === 'SequelizeDatabaseError' && error.message.includes('already exists')) {
logger.info(`${loggerPrefix} index [${columnString}] for table "${tableName}" already exists`)
} else {
throw error
}
}
}
async function removeIndex(queryInterface, logger, tableName, columns) {
logger.info(`${loggerPrefix} removing index [${columns.join(', ')}] from table "${tableName}"`)
try {
await queryInterface.removeIndex(tableName, columns)
logger.info(`${loggerPrefix} removed index [${columns.join(', ')}] from table "${tableName}"`)
} catch (error) {}
}
async function addColumn(queryInterface, logger, table, column, options) {
logger.info(`${loggerPrefix} adding column "${column}" to table "${table}"`)
const tableDescription = await queryInterface.describeTable(table)
if (!tableDescription[column]) {
await queryInterface.addColumn(table, column, options)
logger.info(`${loggerPrefix} added column "${column}" to table "${table}"`)
} else {
logger.info(`${loggerPrefix} column "${column}" already exists in table "${table}"`)
}
}
async function removeColumn(queryInterface, logger, table, column) {
logger.info(`${loggerPrefix} removing column "${column}" from table "${table}"`)
await queryInterface.removeColumn(table, column)
logger.info(`${loggerPrefix} removed column "${column}" from table "${table}"`)
}
async function copyColumn(queryInterface, logger, sourceTable, sourceColumn, sourceIdColumn, targetTable, targetColumn, targetIdColumn) {
logger.info(`${loggerPrefix} copying column "${sourceColumn}" from table "${sourceTable}" to table "${targetTable}"`)
await queryInterface.sequelize.query(`
UPDATE ${targetTable}
SET ${targetColumn} = ${sourceTable}.${sourceColumn}
FROM ${sourceTable}
WHERE ${targetTable}.${targetIdColumn} = ${sourceTable}.${sourceIdColumn}
`)
logger.info(`${loggerPrefix} copied column "${sourceColumn}" from table "${sourceTable}" to table "${targetTable}"`)
}
async function addTrigger(queryInterface, logger, sourceTable, sourceColumn, sourceIdColumn, targetTable, targetColumn, targetIdColumn) {
logger.info(`${loggerPrefix} adding trigger to update ${targetTable}.${targetColumn} when ${sourceTable}.${sourceColumn} is updated`)
const triggerName = convertToSnakeCase(`update_${targetTable}_${targetColumn}_from_${sourceTable}`)
await queryInterface.sequelize.query(`DROP TRIGGER IF EXISTS ${triggerName}`)
await queryInterface.sequelize.query(`
CREATE TRIGGER ${triggerName}
AFTER UPDATE OF ${sourceColumn} ON ${sourceTable}
FOR EACH ROW
BEGIN
UPDATE ${targetTable}
SET ${targetColumn} = NEW.${sourceColumn}
WHERE ${targetTable}.${targetIdColumn} = NEW.${sourceIdColumn};
END;
`)
logger.info(`${loggerPrefix} added trigger.`)
}
async function removeTrigger(queryInterface, logger, targetTable, targetColumn, sourceTable) {
logger.info(`${loggerPrefix} removing trigger`)
const triggerName = convertToSnakeCase(`update_${targetTable}_${targetColumn}_from_${sourceTable}`)
await queryInterface.sequelize.query(`DROP TRIGGER IF EXISTS ${triggerName}`)
}
function convertToSnakeCase(str) {
return str.replace(/([A-Z])/g, '_$1').toLowerCase()
}
module.exports = { up, down }

View file

@ -146,6 +146,7 @@ class Book extends Model {
},
title: DataTypes.STRING,
titleIgnorePrefix: DataTypes.STRING,
titleNormalized: DataTypes.STRING,
subtitle: DataTypes.STRING,
publishedYear: DataTypes.STRING,
publishedDate: DataTypes.STRING,
@ -407,7 +408,9 @@ class Book extends Model {
this[key] = payload.metadata[key] || null
if (key === 'title') {
const { getTitleIgnorePrefix, getNormalizedTitle } = require('../utils')
this.titleIgnorePrefix = getTitleIgnorePrefix(this.title)
this.titleNormalized = getNormalizedTitle(this.title)
}
hasUpdates = true

View file

@ -78,6 +78,8 @@ class LibraryItem extends Model {
/** @type {string} */
this.titleIgnorePrefix // Only used for sorting
/** @type {string} */
this.titleNormalized // Only used for sorting
/** @type {string} */
this.authorNamesFirstLast // Only used for sorting
/** @type {string} */
this.authorNamesLastFirst // Only used for sorting
@ -687,6 +689,7 @@ class LibraryItem extends Model {
extraData: DataTypes.JSON,
title: DataTypes.STRING,
titleIgnorePrefix: DataTypes.STRING,
titleNormalized: DataTypes.STRING,
authorNamesFirstLast: DataTypes.STRING,
authorNamesLastFirst: DataTypes.STRING,
isNotConsolidated: {
@ -719,6 +722,9 @@ class LibraryItem extends Model {
{
fields: ['libraryId', 'mediaType', { name: 'titleIgnorePrefix', collate: 'NOCASE' }]
},
{
fields: ['libraryId', 'mediaType', { name: 'titleNormalized', collate: 'NOCASE' }]
},
{
fields: ['libraryId', 'mediaType', { name: 'authorNamesFirstLast', collate: 'NOCASE' }]
},
@ -795,6 +801,7 @@ class LibraryItem extends Model {
if (instance.media) {
instance.title = instance.media.title
instance.titleIgnorePrefix = instance.media.titleIgnorePrefix
instance.titleNormalized = instance.media.titleNormalized
if (instance.isBook) {
if (instance.media.authors !== undefined) {
instance.authorNamesFirstLast = instance.media.authorName

View file

@ -1,5 +1,5 @@
const { DataTypes, Model } = require('sequelize')
const { getTitlePrefixAtEnd, getTitleIgnorePrefix } = require('../utils')
const { getTitlePrefixAtEnd, getTitleIgnorePrefix, getNormalizedTitle } = require('../utils')
const Logger = require('../Logger')
const libraryItemsPodcastFilters = require('../utils/queries/libraryItemsPodcastFilters')
const htmlSanitizer = require('../utils/htmlSanitizer')
@ -93,6 +93,7 @@ class Podcast extends Model {
{
title,
titleIgnorePrefix: getTitleIgnorePrefix(title),
titleNormalized: getNormalizedTitle(title),
author: typeof payload.metadata.author === 'string' ? payload.metadata.author : null,
releaseDate: typeof payload.metadata.releaseDate === 'string' ? payload.metadata.releaseDate : null,
feedURL: typeof payload.metadata.feedUrl === 'string' ? payload.metadata.feedUrl : null,
@ -130,6 +131,7 @@ class Podcast extends Model {
},
title: DataTypes.STRING,
titleIgnorePrefix: DataTypes.STRING,
titleNormalized: DataTypes.STRING,
author: DataTypes.STRING,
releaseDate: DataTypes.STRING,
feedURL: DataTypes.STRING,
@ -257,6 +259,7 @@ class Podcast extends Model {
if (key === 'title') {
this.titleIgnorePrefix = getTitleIgnorePrefix(this.title)
this.titleNormalized = getNormalizedTitle(this.title)
}
hasUpdates = true

View file

@ -191,6 +191,18 @@ module.exports.getTitleIgnorePrefix = (title) => {
return getTitleParts(title)[0]
}
/**
* Get normalized title to use for grouping duplicates
* Removes non-alphabetic characters (numbers, punctuation, spaces)
* @param {string} title
* @returns {string}
*/
module.exports.getNormalizedTitle = (title) => {
if (!title) return ''
const sortTitle = getTitleParts(title)[0] || title
return sortTitle.toLowerCase().replace(/[^\p{L}]/gu, '')
}
/**
* Put sorting prefix at the end of title
* @example "The Good Book" => "Good Book, The"

View file

@ -515,6 +515,10 @@ module.exports = {
isInvalid: true
}
]
} else if (filterGroup === 'duplicates') {
libraryItemWhere['titleNormalized'] = {
[Sequelize.Op.in]: Sequelize.literal(`(SELECT titleNormalized FROM libraryItems WHERE libraryId = '${libraryId}' AND titleNormalized IS NOT NULL AND titleNormalized != '' GROUP BY titleNormalized HAVING COUNT(titleNormalized) > 1)`)
}
} else if (filterGroup === 'progress' && user) {
const mediaProgressWhere = {
userId: user.id

View file

@ -168,6 +168,10 @@ module.exports = {
isInvalid: true
}
]
} else if (filterGroup === 'duplicates') {
libraryItemWhere['titleNormalized'] = {
[Sequelize.Op.in]: Sequelize.literal(`(SELECT titleNormalized FROM libraryItems WHERE libraryId = '${libraryId}' AND titleNormalized IS NOT NULL AND titleNormalized != '' GROUP BY titleNormalized HAVING COUNT(titleNormalized) > 1)`)
}
} else if (filterGroup === 'recent') {
libraryItemWhere['createdAt'] = {
[Sequelize.Op.gte]: new Date(new Date() - 60 * 24 * 60 * 60 * 1000) // 60 days ago