Merge branch #buildborderless/OpenSight-Deepfake-Detection-Models-Playground' into 'LPX55/Deepfake-Detection-Playground-Dev'

.github/instructions/copilot-instructions.md

@@ -1,34 +1,50 @@
 
-# Copilot Instructions for OpenSight-Deepfake-Detection-Models-Playground
+
+# Copilot Instructions for OpenSight-Deepfake-Detection-Models-Playground (2025)
 
 ## Project Overview
-This project is a modular, agent-driven toolkit for deepfake detection and digital forensics. It uses an ensemble of models, advanced forensic tools, and smart agents to provide explainable, extensible, and robust detection—optimized for integration with vision LLMs and multimodal AI agents.
+OpenSight is a modular, agent-driven toolkit for deepfake detection and digital forensics. It leverages an ensemble of models, advanced forensic tools, and smart agents for explainable, extensible, and robust detection. The system is optimized for integration with vision LLMs and multimodal AI agents, and supports logging to Hugging Face datasets.
 
-## Architecture & Key Components
-- **Entrypoint:** `app.py` (Gradio app + MCP server).
-- **Forensics:** `forensics/` (e.g., `ela.py`, `gradient.py`, `minmax.py`, `bitplane.py`, `wavelet.py`). Each file implements a forensic technique, callable via LLM/MCP.
-- **Agents:** `agents/` (e.g., `EnsembleMonitorAgent`, `ModelWeightManager`, `ContextualIntelligenceAgent`, `ForensicAnomalyDetectionAgent`). Agents coordinate model weighting, context inference, and anomaly detection.
-- **Model Management:** Models are registered in `utils/registry.py` and managed as an ensemble with dynamic, context-aware weighting.
-- **Utilities:** `utils/` (logging, augmentation, registry, health checks).
+## Key Technologies
+- **Gradio**: Main UI and API server (`app.py`).
+- **Hugging Face Hub**: Model and dataset management, logging, and deployment.
+- **Git LFS**: Required for storing binary files (e.g., PNGs) in the repo. See `.gitattributes` for tracked types.
+- **Agents**: Smart agents for ensemble monitoring, weight optimization, system health, context intelligence, and anomaly detection (`agents/`).
+- **Forensic Tools**: Modular forensic techniques in `forensics/` (ELA, gradient, minmax, bitplane, wavelet, exif, etc.).
+- **Model Management**: Models are registered and managed in `utils/registry.py`, loaded via ONNX/Hugging Face/Gradio API.
+- **Utilities**: Logging, augmentation, health checks, and more in `utils/`.
 
 ## Data Flow & Prediction Pipeline
-1. **Image Preprocessing:** Normalize to PIL RGB; optionally augment (rotate, noise, sharpen).
-2. **Agent Initialization:** Monitoring, optimization, and context agents are set up.
-3. **Model Inference:** Each model predicts independently; results tracked by agents.
-4. **Consensus:** Model weights are dynamically adjusted based on context and agent feedback.
-5. **Forensic Analysis:** Multiple forensic tools run in parallel; outputs analyzed for anomalies.
-6. **Logging:** All results (images, predictions, agent data) are logged to Hugging Face datasets.
-
-## Developer Workflows
-- **Run the App:**  
+1. **Image Preprocessing**: Normalize to PIL RGB, optional augmentation (rotate, noise, sharpen).
+2. **Agent Initialization**: Monitoring, optimization, and context agents setup.
+3. **Model Inference**: Each model predicts independently; results tracked by agents.
+4. **Consensus**: Model weights dynamically adjusted based on context and agent feedback.
+5. **Forensic Analysis**: Multiple forensic tools run in parallel; outputs analyzed for anomalies.
+6. **Logging**: All results (images, predictions, agent data) are logged to Hugging Face datasets (`hf_logger.py`).
+
+## Developer Workflow
+- **Run the App:**
+  ```bash
+  python app.py
+  ```
+- **Dependencies:** See `requirements.txt` (gradio, PIL, numpy, torch, huggingface_hub, etc.).
+- **Binary Files:** All PNGs and other binaries must be tracked with Git LFS. If you see a push error, run:
+  ```bash
+  git lfs track "*.png"
+  git add .gitattributes
+  git add <yourfile.png>
+  git commit -m "Track PNG files with Git LFS"
+  git push origin main
+  ```
+  If the file is already in history, use:
   ```bash
-  python app_optimized.py
+  git lfs migrate import --include="*.png"
+  git push origin main
   ```
-- **Dependencies:** See `requirements.txt` (notably: gradio, PIL, numpy, torch, smolagents, etc.).
 - **Extending Forensics/Agents:** Add new tools in `forensics/`, new agents in `agents/`, and register in the main app.
 - **Testing:** Unit tests for agents/models are planned (see roadmap in `README.md`).
 
-## Project-Specific Patterns & Conventions
+## Project Patterns & Conventions
 - **Forensic Tool Naming:** Use `tool_*` or descriptive names (e.g., `tool_ela`, `tool_waveletnoise`).
 - **Agent Classes:** Use `*Agent` suffix (e.g., `EnsembleMonitorAgent`).
 - **API Exposure:** Functions are exposed for LLM/MCP calls with clear parameter/return docs (see `README.md`).
@@ -38,7 +54,7 @@ This project is a modular, agent-driven toolkit for deepfake detection and digit
 ## Integration & Extension
 - **Add Models:** Update ensemble logic and register in agent system.
 - **Add Forensic Tools:** Implement in `forensics/`, expose via main app, and document parameters/returns.
-- **LLM/Multimodal Integration:** Hybrid input strategies (e.g., ELA+RGB, metadata+image) are encouraged; see `README.md` for detailed tables and guidance.
+- **LLM/Multimodal Integration:** Hybrid input strategies (e.g., ELA+RGB, metadata+image) are encouraged; see `README.md` for details.
 
 ## References
 - **Forensic Techniques:** See `forensics/` for implementation details.
@@ -47,4 +63,4 @@ This project is a modular, agent-driven toolkit for deepfake detection and digit
 - **Roadmap:** Ongoing and planned features are tracked in `README.md`.
 
 ---
-**Tip:** When extending or debugging, always check agent logic and consensus weighting, as these are central to system behavior.
+**Tip:** Always check agent logic and consensus weighting when extending or debugging, as these are central to system behavior. For binary file push errors, ensure Git LFS is set up and files are tracked correctly.

.python-version

@@ -1 +0,0 @@
-3.11

README.md

@@ -1,25 +1,26 @@
 ---
 title: MCP Toolkit - Re-Thinking Deepfake Detection & Forensics
-description: MCP Server for Project OpenSight's Deepfake Detection & Digital Forensics Tools
+description: >-
+  MCP Server for Project OpenSight's Deepfake Detection & Digital Forensics
+  Tools
 emoji: 🚑
 colorFrom: yellow
 colorTo: yellow
 sdk: gradio
-sdk_version: 5.33.0
+sdk_version: 5.49.1
 app_file: app.py
 pinned: true
 models:
 - aiwithoutborders-xyz/OpenSight-CommunityForensics-Deepfake-ViT
 license: mit
 tags:
-  - deepfake-detection
-  - ai-agents
-  - leaderboards
-  - deepfake
-  - detection
-  - ensemble
-  - forensics
-
+- deepfake-detection
+- ai-agents
+- leaderboards
+- deepfake
+- detection
+- ensemble
+- forensics
 ---
 
 # The Detection Dilemma: The Degentic Games

agents/ensemble_weights.py

@@ -38,8 +38,6 @@ class ContextualWeightOverrideAgent:
         agent_logger.log("weight_optimization", "info", f"Combined context overrides: {combined_overrides}")
         return combined_overrides
 
-
-
 class ModelWeightManager:
     def __init__(self, strongest_model_id: str = None):
         agent_logger = AgentLogger()
@@ -49,8 +47,8 @@ class ModelWeightManager:
         if num_models > 0:
             if strongest_model_id and strongest_model_id in MODEL_REGISTRY:
                 agent_logger.log("weight_optimization", "info", f"Designating '{strongest_model_id}' as the strongest model.")
-                # Assign a high weight to the strongest model (e.g., 50%)
-                strongest_weight_share = 0.5
+                # Assign a high weight to the strongest model (e.g., 40%)
+                strongest_weight_share = 0.4
                 self.base_weights = {strongest_model_id: strongest_weight_share}
                 remaining_models = [mid for mid in MODEL_REGISTRY.keys() if mid != strongest_model_id]
                 if remaining_models:
@@ -126,7 +124,7 @@ class ModelWeightManager:
         """Check if models agree on prediction"""
         agent_logger.log("weight_optimization", "info", "Checking for consensus among model predictions.")
         non_none_predictions = [p.get("Label") for p in predictions.values() if p is not None and isinstance(p, dict) and p.get("Label") is not None and p.get("Label") != "Error"]
-        agent_logger.log("weight_optimization", "debug", f"Non-none predictions for consensus check: {non_none_predictions}")
+        agent_logger.log("weight_optimization", "info", f"Non-none predictions for consensus check: {non_none_predictions}")
         result = len(non_none_predictions) > 0 and len(set(non_none_predictions)) == 1
         agent_logger.log("weight_optimization", "info", f"Consensus detected: {result}")
         return result
@@ -135,7 +133,7 @@ class ModelWeightManager:
         """Check if models have conflicting predictions"""
         agent_logger.log("weight_optimization", "info", "Checking for conflicts among model predictions.")
         non_none_predictions = [p.get("Label") for p in predictions.values() if p is not None and isinstance(p, dict) and p.get("Label") is not None and p.get("Label") != "Error"]
-        agent_logger.log("weight_optimization", "debug", f"Non-none predictions for conflict check: {non_none_predictions}")
+        agent_logger.log("weight_optimization", "info", f"Non-none predictions for conflict check: {non_none_predictions}")
         result = len(non_none_predictions) > 1 and len(set(non_none_predictions)) > 1
         agent_logger.log("weight_optimization", "info", f"Conflicts detected: {result}")
         return result
@@ -154,4 +152,4 @@ class ModelWeightManager:
                 return {} # No models registered
         normalized = {k: v/total for k, v in weights.items()}
         agent_logger.log("weight_optimization", "info", f"Weights normalized. Total sum: {sum(normalized.values()):.2f}")
-        return normalized
+        return normalized 

forensics/__init__.py

@@ -4,6 +4,7 @@ from .ela import ELA
 from .gradient import gradient_processing
 from .minmax import minmax_process
 from .wavelet import noise_estimation
+from .jpeg_compression import estimate_qf
 
 __all__ = [
     'bit_plane_extractor',
@@ -11,5 +12,6 @@ __all__ = [
      # 'exif_full_dump',
     'gradient_processing',
     'minmax_process',
-    'noise_estimation'
-] 
+    'noise_estimation',
+    'estimate_qf'
+]

forensics/jpeg_compression.py

@@ -0,0 +1,157 @@
+import cv2 as cv
+import numpy as np
+from PIL import Image
+
+DCT_SIZE = 8
+TABLE_SIZE = DCT_SIZE ** 2
+ZIG_ZAG = [
+    [0, 0],
+    [0, 1],
+    [1, 0],
+    [2, 0],
+    [1, 1],
+    [0, 2],
+    [0, 3],
+    [1, 2],
+    [2, 1],
+    [3, 0],
+    [4, 0],
+    [3, 1],
+    [2, 2],
+    [1, 3],
+    [0, 4],
+    [0, 5],
+    [1, 4],
+    [2, 3],
+    [3, 2],
+    [4, 1],
+    [5, 0],
+    [6, 0],
+    [5, 1],
+    [4, 2],
+    [3, 3],
+    [2, 4],
+    [1, 5],
+    [0, 6],
+    [0, 7],
+    [1, 6],
+    [2, 5],
+    [3, 4],
+    [4, 4],
+    [5, 3],
+    [6, 2],
+    [7, 1],
+    [7, 2],
+    [6, 3],
+    [5, 4],
+    [4, 5],
+    [3, 5],
+    [2, 6],
+    [1, 7],
+    [2, 7],
+    [3, 6],
+    [4, 5],
+    [5, 4],
+    [6, 3],
+    [7, 2],
+    [7, 3],
+    [6, 4],
+    [5, 5],
+    [4, 6],
+    [3, 7],
+    [4, 7],
+    [5, 6],
+    [6, 5],
+    [7, 4],
+    [7, 5],
+    [6, 6],
+    [5, 7],
+    [6, 7],
+    [7, 6],
+    [7, 7],
+]
+
+
+def compress_jpg(image: Image.Image, quality, color=True):
+    """Compress a PIL image to JPEG format with specified quality.
+    
+    Args:
+        image: Input PIL image (RGB format)
+        quality: JPEG compression quality (1-100)
+        color: Whether to preserve color (BGR format)
+    
+    Returns:
+        np.ndarray: Decompressed image in BGR or grayscale format
+    """
+    # Convert PIL image to OpenCV BGR format
+    img_np = np.array(image)
+    if color:
+        img_np = cv.cvtColor(img_np, cv.COLOR_RGB2BGR)
+    
+    _, buffer = cv.imencode(".jpg", img_np, [cv.IMWRITE_JPEG_QUALITY, quality])
+    return cv.imdecode(buffer, cv.IMREAD_COLOR if color else cv.IMREAD_GRAYSCALE)
+
+
+def loss_curve(image: Image.Image, qualities=tuple(range(1, 101)), normalize=True):
+    """Calculate JPEG compression loss curve for quality estimation.
+    
+    Args:
+        image: Input PIL image (RGB format)
+        qualities: Quality values to test (1-100)
+        normalize: Whether to normalize the output curve
+    
+    Returns:
+        np.ndarray: Mean absolute difference values across quality levels
+    """
+    # Convert input image to grayscale BGR for compression testing
+    img_np = np.array(image)
+    if len(img_np.shape) == 3:
+        x = cv.cvtColor(img_np, cv.COLOR_RGB2GRAY)
+    else:
+        x = img_np
+    
+    c = np.array(
+        [cv.mean(cv.absdiff(compress_jpg(x, q, False), x))[0] for q in qualities]
+    )
+    if normalize:
+        c = cv.normalize(c, None, 0, 1, cv.NORM_MINMAX).flatten()
+    return c
+
+
+def estimate_qf(image):
+    return np.argmin(loss_curve(image))
+
+
+def get_tables(quality):
+    luma = np.array(
+        [
+            [16, 11, 10, 16, 24, 40, 51, 61],
+            [12, 12, 14, 19, 26, 58, 60, 55],
+            [14, 13, 16, 24, 40, 57, 69, 56],
+            [14, 17, 22, 29, 51, 87, 80, 62],
+            [18, 22, 37, 56, 68, 109, 103, 77],
+            [24, 35, 55, 64, 81, 104, 113, 92],
+            [49, 64, 78, 87, 103, 121, 120, 101],
+            [72, 92, 95, 98, 112, 100, 103, 99],
+        ]
+    )
+    chroma = np.array(
+        [
+            [17, 18, 24, 47, 99, 99, 99, 99],
+            [18, 21, 26, 66, 99, 99, 99, 99],
+            [24, 26, 56, 99, 99, 99, 99, 99],
+            [47, 66, 99, 99, 99, 99, 99, 99],
+            [99, 99, 99, 99, 99, 99, 99, 99],
+            [99, 99, 99, 99, 99, 99, 99, 99],
+            [99, 99, 99, 99, 99, 99, 99, 99],
+            [99, 99, 99, 99, 99, 99, 99, 99],
+        ]
+    )
+    quality = np.clip(quality, 1, 100)
+    if quality < 50:
+        quality = 5000 / quality
+    else:
+        quality = 200 - quality * 2
+    tables = np.concatenate((luma[:, :, np.newaxis], chroma[:, :, np.newaxis]), axis=2)
+    tables = (tables * quality + 50) / 100
+    return np.clip(tables, 1, 255).astype(int)

utils/model_loader.py

@@ -86,11 +86,11 @@ def register_all_models(MODEL_PATHS, CLASS_NAMES, device, infer_onnx_model, prep
             if model_key == "model_1":
                 contributor = "haywoodsloan"
                 architecture = "SwinV2"
-                dataset = "DeepFakeDetection"
+                dataset = "Mixed"
             elif model_key == "model_2":
                 contributor = "Heem2"
                 architecture = "ViT"
-                dataset = "DeepFakeDetection"
+                dataset = "Mixed"
             elif model_key == "model_3":
                 contributor = "Organika"
                 architecture = "VIT"
@@ -126,7 +126,7 @@ def register_all_models(MODEL_PATHS, CLASS_NAMES, device, infer_onnx_model, prep
         elif model_key == "model_8":
             contributor = "aiwithoutborders-xyz"
             architecture = "ViT"
-            dataset = "DeepfakeDetection"
+            dataset = "Massive"
             display_name_parts = [model_num]
             if architecture and architecture not in ["Unknown"]:
                 display_name_parts.append(architecture)