initial comment

2026-01-24 17:43:28 -05:00
commit fe40adfd38
72 changed files with 19614 additions and 0 deletions
--- a/QUALITY-CHECKING.md
+++ b/QUALITY-CHECKING.md
@@ -0,0 +1,602 @@
+# Quality Checking Feature
+
+## Overview
+
+encoderPro now includes intelligent quality analysis that detects source video quality and warns you if encoding will degrade quality to a noticeable degree. This prevents accidental quality loss when re-encoding high-quality source material.
+
+---
+
+## Key Features
+
+### 1. **Automatic Quality Detection**
+- Analyzes source video bitrate, resolution, codec, and FPS
+- Calculates quality score (0-100) based on bitrate per pixel
+- Detects HDR content
+- Accounts for codec efficiency (H.265 vs H.264 vs AV1)
+
+### 2. **Pre-Encoding Quality Comparison**
+- Estimates target encoding bitrate based on your CRF settings
+- Compares source quality to estimated target quality
+- Warns if quality drop exceeds configurable thresholds
+- Provides detailed quality metrics for decision making
+
+### 3. **Processed File Tracking**
+- Database tracks all processed files
+- Automatically skips already-encoded files on re-runs
+- Maintains processing history with quality metrics
+- Supports manual reset for re-processing
+
+### 4. **Configurable Behavior**
+- Enable/disable quality checking
+- Adjustable warning and error thresholds
+- Option to skip degraded files automatically
+- User prompts for manual confirmation
+
+---
+
+## Configuration
+
+Add this section to your `config.yaml`:
+
+```yaml
+quality_check:
+  # Enable pre-encoding quality analysis
+  enabled: true
+
+  # Warning threshold - warn if quality will drop by this many points (0-100 scale)
+  warn_threshold: 10.0
+
+  # Error threshold - fail/skip if quality will drop by this many points
+  error_threshold: 20.0
+
+  # Automatically skip files where encoding would degrade quality
+  skip_on_degradation: false
+
+  # Prompt user for confirmation when warnings detected (CLI only)
+  prompt_on_warning: true
+```
+
+### Configuration Options Explained
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `enabled` | boolean | `true` | Enable/disable quality checking entirely |
+| `warn_threshold` | float | `10.0` | Quality score drop that triggers a warning |
+| `error_threshold` | float | `20.0` | Quality score drop that triggers an error |
+| `skip_on_degradation` | boolean | `false` | Auto-skip files with quality degradation |
+| `prompt_on_warning` | boolean | `true` | Ask user for confirmation on warnings |
+
+---
+
+## How Quality Scores Work
+
+### Quality Score (0-100)
+
+The quality checker calculates a score based on **bits per pixel per frame**:
+
+```
+Quality Score = f(bitrate, resolution, fps, codec_efficiency)
+```
+
+**Score Ranges:**
+- **95-100:** Near-lossless / Exceptional quality
+- **85-95:** Excellent / Archival quality
+- **70-85:** Good / Visually transparent
+- **50-70:** Acceptable / Minor compression visible
+- **0-50:** Poor / Heavy compression artifacts
+
+### Codec Efficiency Multipliers
+
+The quality checker accounts for codec efficiency:
+
+| Codec | Multiplier | Notes |
+|-------|------------|-------|
+| AV1 | 1.8x | Most efficient codec |
+| H.265/HEVC | 1.5x | ~50% better than H.264 |
+| H.264/AVC | 1.0x | Baseline reference |
+| MPEG-2 | 0.5x | Older, less efficient |
+| MPEG-4 | 0.7x | Older codec |
+
+**Example:** A video at 5 Mbps H.265 has similar quality to 7.5 Mbps H.264.
+
+### Bits Per Pixel Ranges
+
+| BPP Range | Quality | Typical Use Case |
+|-----------|---------|------------------|
+| > 0.5 | Near-lossless (95-100) | Professional archival |
+| 0.3 - 0.5 | Excellent (85-95) | High-quality archival |
+| 0.2 - 0.3 | Good (70-85) | **Sweet spot for home media** |
+| 0.1 - 0.2 | Acceptable (50-70) | Streaming services |
+| < 0.1 | Poor (0-50) | Heavy compression |
+
+---
+
+## Usage Examples
+
+### Example 1: Analyze Video Quality
+
+```bash
+python3 quality_checker.py /movies/example.mkv
+```
+
+**Output:**
+```
+Video Quality Analysis:
+  Resolution: 1920x1080
+  Bitrate: 8.5 Mbps
+  Codec: h264
+  FPS: 23.98
+  HDR: No
+  Quality Score: 72.3/100
+```
+
+### Example 2: Check Before Encoding
+
+```python
+from quality_checker import QualityChecker
+from pathlib import Path
+
+checker = QualityChecker()
+
+# Define your encoding profile
+profile = {
+    'encoder': 'nvidia_nvenc_h265',
+    'quality': 21  # CRF value
+}
+
+# Check quality before encoding
+result = checker.check_before_encode(
+    Path('/movies/example.mkv'),
+    profile,
+    warn_threshold=10.0,
+    error_threshold=20.0
+)
+
+if result['ok']:
+    print("✅ Safe to encode")
+elif result['warning']:
+    print(f"⚠️ Warning: {result['message']}")
+elif result['error']:
+    print(f"❌ Error: {result['message']}")
+```
+
+### Example 3: Batch Analysis
+
+See `example_quality_check.py` for comprehensive examples including:
+- Single file analysis
+- Quality degradation checks
+- Comprehensive pre-encode checks
+- Batch directory analysis
+
+---
+
+## Understanding Quality Degradation
+
+### What Causes Quality Degradation?
+
+Encoding can degrade quality when:
+
+1. **Source is already high quality**
+   - Source: 95/100 (near-lossless BluRay rip at 25 Mbps)
+   - Target: 75/100 (H.265 CRF 21 at 8 Mbps)
+   - **Drop:** 20 points ⚠️
+
+2. **Target bitrate is too low**
+   - Source: 4K movie at 40 Mbps
+   - Target: 4K at 10 Mbps (too aggressive)
+   - **Result:** Visible compression artifacts
+
+3. **Re-encoding already compressed content**
+   - Source: Web-DL already at CRF 23
+   - Target: Re-encode at CRF 21
+   - **Result:** Quality loss from generation loss
+
+### When to Proceed Despite Warnings
+
+**Safe to proceed:**
+- Source quality 50-70 (already compressed)
+- Quality drop < 10 points
+- Testing settings on sample files
+- Source is a screen recording or low-quality capture
+
+**Consider skipping:**
+- Source quality > 90 (near-lossless)
+- Quality drop > 20 points
+- Archival content you want to preserve
+- BluRay remuxes or untouched sources
+
+---
+
+## Real-World Examples
+
+### Example 1: BluRay Remux (Skip Recommended)
+
+```
+Source Quality Analysis:
+  Resolution: 1920x1080
+  Bitrate: 28.5 Mbps (H.264)
+  Quality Score: 96/100 (Near-lossless)
+
+Target Settings (CRF 21 H.265):
+  Estimated Bitrate: 7.2 Mbps
+  Target Quality: 73/100
+
+⚠️ WARNING: Quality will drop by 23 points
+❌ Recommendation: SKIP - Source is too high quality
+```
+
+**Action:** Skip encoding or use CRF 18-19 for archival quality.
+
+---
+
+### Example 2: Web-DL (Safe to Encode)
+
+```
+Source Quality Analysis:
+  Resolution: 1920x1080
+  Bitrate: 6.2 Mbps (H.264)
+  Quality Score: 68/100 (Good)
+
+Target Settings (CRF 21 H.265):
+  Estimated Bitrate: 5.8 Mbps
+  Target Quality: 71/100
+
+✅ OK: Quality will improve by 3 points
+✅ Recommendation: PROCEED - Safe to encode
+```
+
+**Action:** Proceed with encoding. Will remove subtitles without quality loss.
+
+---
+
+### Example 3: Already Encoded (Warning)
+
+```
+Source Quality Analysis:
+  Resolution: 1920x1080
+  Bitrate: 4.8 Mbps (H.265)
+  Quality Score: 65/100 (Acceptable)
+
+Target Settings (CRF 21 H.265):
+  Estimated Bitrate: 5.2 Mbps
+  Target Quality: 68/100
+
+⚠️ WARNING: Re-encoding already compressed H.265
+⚠️ Recommendation: Consider CRF 19-20 to avoid generation loss
+```
+
+**Action:** Consider lower CRF or skip. Source is already H.265.
+
+---
+
+### Example 4: Low Quality Source (Safe)
+
+```
+Source Quality Analysis:
+  Resolution: 1920x1080
+  Bitrate: 2.8 Mbps (H.264)
+  Quality Score: 52/100 (Acceptable)
+
+Target Settings (CRF 21 H.265):
+  Estimated Bitrate: 5.8 Mbps
+  Target Quality: 71/100
+
+✅ OK: Quality will improve by 19 points
+✅ Recommendation: PROCEED - Encoding will improve quality
+```
+
+**Action:** Proceed. H.265 encoding will improve perceived quality.
+
+---
+
+## HDR Content Detection
+
+The quality checker automatically detects HDR content:
+
+### HDR Detection Criteria
+
+1. **Transfer characteristics:**
+   - SMPTE 2084 (HDR10)
+   - ARIB STD-B67 (HLG)
+
+2. **Color primaries:**
+   - BT.2020 color space
+
+3. **Metadata tags:**
+   - Any tag containing "HDR"
+
+### HDR Warning
+
+If HDR content is detected and `hdr_handling` is not set to `preserve`:
+
+```
+⚠️ HDR content detected but HDR handling is not set to 'preserve'
+```
+
+**Action:** Update your profile:
+```yaml
+profiles:
+  your_profile:
+    hdr_handling: preserve
+```
+
+---
+
+## Processed File Tracking
+
+### How It Works
+
+1. **First run:** All files scanned and added to database as `pending`
+2. **Encoding:** Files marked as `processing` → `completed` or `failed`
+3. **Second run:** Completed files automatically skipped
+4. **Re-processing:** Reset state to `pending` if needed
+
+### Database Schema
+
+```sql
+CREATE TABLE files (
+    id INTEGER PRIMARY KEY,
+    filepath TEXT UNIQUE NOT NULL,
+    state TEXT NOT NULL,  -- pending/processing/completed/failed/skipped
+    profile_name TEXT,
+    encoder_used TEXT,
+    encode_time_seconds REAL,
+    fps REAL,
+    original_size INTEGER,
+    encoded_size INTEGER,
+    source_quality_score REAL,  -- NEW
+    target_quality_score REAL,  -- NEW
+    created_at TIMESTAMP,
+    updated_at TIMESTAMP
+);
+```
+
+### Manually Reset Files
+
+```bash
+# Reset specific file
+python3 dbmanage.py --reset-file /movies/example.mkv
+
+# Reset all files
+python3 dbmanage.py --reset-all
+
+# Reset only failed files
+python3 dbmanage.py --reset-failed
+```
+
+---
+
+## Command-Line Options
+
+### Check Quality Only (No Encoding)
+
+```bash
+# Analyze single file
+python3 quality_checker.py /movies/example.mkv
+
+# Batch analyze directory
+python3 example_quality_check.py /movies/
+```
+
+### Override Quality Check
+
+```bash
+# Disable quality check for this run
+python3 reencode.py -c config.yaml --no-quality-check
+
+# Force encode despite warnings
+python3 reencode.py -c config.yaml --force
+```
+
+### Skip Processed Files
+
+```bash
+# Skip already completed files (default behavior)
+python3 reencode.py -c config.yaml
+
+# Re-process all files (ignore database)
+python3 reencode.py -c config.yaml --reprocess-all
+```
+
+---
+
+## Best Practices
+
+### 1. **Start with Quality Analysis**
+
+Before encoding your entire library:
+
+```bash
+# Analyze quality distribution
+python3 example_quality_check.py /movies/ > quality_report.txt
+```
+
+Review the report to understand your source quality distribution.
+
+---
+
+### 2. **Use Appropriate Thresholds**
+
+**Conservative (Preserve Quality):**
+```yaml
+quality_check:
+  warn_threshold: 5.0    # Warn on small drops
+  error_threshold: 10.0  # Error on moderate drops
+  skip_on_degradation: true
+```
+
+**Balanced (Recommended):**
+```yaml
+quality_check:
+  warn_threshold: 10.0
+  error_threshold: 20.0
+  skip_on_degradation: false
+```
+
+**Aggressive (Maximum Compression):**
+```yaml
+quality_check:
+  warn_threshold: 20.0
+  error_threshold: 30.0
+  skip_on_degradation: false
+```
+
+---
+
+### 3. **Quality-Based Profiles**
+
+Use resolution rules to apply different profiles based on source quality:
+
+```yaml
+advanced:
+  resolution_rules:
+    enabled: true
+    quality_rules:
+      - min_quality: 90    # Near-lossless sources
+        profile: quality_gpu  # Use high-quality preset
+      - min_quality: 70
+        profile: sweetspot_gpu  # Balanced
+      - min_quality: 50
+        profile: balanced_gpu   # Already compressed
+      - min_quality: 0
+        profile: fast_gpu       # Low quality sources
+```
+
+---
+
+### 4. **Monitor Processing Logs**
+
+Quality checks are logged:
+
+```
+2025-01-15 10:30:22 - INFO - Analyzing quality: example.mkv
+2025-01-15 10:30:23 - INFO - Source quality: 72.3/100
+2025-01-15 10:30:23 - INFO - Target quality: 75.1/100
+2025-01-15 10:30:23 - INFO - ✅ Quality check passed
+```
+
+Review logs to identify patterns and adjust settings.
+
+---
+
+## Troubleshooting
+
+### Quality Check Fails
+
+**Problem:** `Failed to analyze source video quality`
+
+**Solutions:**
+1. Check ffprobe is installed: `ffprobe -version`
+2. Verify file is readable: `ls -la /path/to/file.mkv`
+3. Check file is valid video: `ffprobe /path/to/file.mkv`
+4. Disable quality check temporarily:
+   ```yaml
+   quality_check:
+     enabled: false
+   ```
+
+---
+
+### Inaccurate Quality Scores
+
+**Problem:** Quality scores don't match expectations
+
+**Explanation:**
+- Quality scores are **estimates** based on bitrate per pixel
+- Different content has different complexity
+  - Simple animation: Lower bitrate, high quality
+  - Grain-heavy film: Higher bitrate needed
+- Scores are relative, not absolute
+
+**Solution:** Use thresholds as guidelines, not absolute rules.
+
+---
+
+### Too Many Warnings
+
+**Problem:** Getting warnings on files you want to encode
+
+**Solutions:**
+
+1. **Increase thresholds:**
+   ```yaml
+   quality_check:
+     warn_threshold: 15.0  # From 10.0
+   ```
+
+2. **Disable warnings:**
+   ```yaml
+   quality_check:
+     prompt_on_warning: false
+   ```
+
+3. **Disable quality check:**
+   ```yaml
+   quality_check:
+     enabled: false
+   ```
+
+---
+
+## API Reference
+
+See `quality_checker.py` for full API documentation.
+
+### QualityChecker Class
+
+```python
+class QualityChecker:
+    def analyze_quality(self, filepath: Path) -> Optional[VideoQuality]:
+        """Analyze video quality metrics"""
+
+    def will_degrade_quality(
+        self,
+        source_quality: VideoQuality,
+        target_bitrate: int,
+        target_codec: str,
+        threshold: float = 10.0
+    ) -> Tuple[bool, str]:
+        """Check if encoding will significantly degrade quality"""
+
+    def check_before_encode(
+        self,
+        filepath: Path,
+        profile: Dict,
+        warn_threshold: float = 10.0,
+        error_threshold: float = 20.0
+    ) -> Dict:
+        """Comprehensive quality check before encoding"""
+
+    def estimate_target_bitrate(
+        self,
+        profile: Dict,
+        resolution: Tuple[int, int],
+        fps: float
+    ) -> int:
+        """Estimate target bitrate based on profile settings"""
+```
+
+---
+
+## Summary
+
+The quality checking feature provides:
+
+✅ **Intelligent quality analysis** - Understand your source material
+✅ **Degradation detection** - Avoid accidental quality loss
+✅ **Processed file tracking** - Skip already-encoded files
+✅ **Configurable behavior** - Customize to your needs
+✅ **HDR detection** - Preserve HDR metadata
+✅ **Detailed logging** - Monitor quality decisions
+
+**Best Use Cases:**
+- Large mixed-quality libraries
+- Preserving high-quality sources
+- Avoiding re-encoding already compressed files
+- Understanding quality impact before encoding
+
+**When to Disable:**
+- All sources are known low quality
+- Testing encoding settings
+- Processing screen recordings
+- Time-sensitive batch jobs