battlewords/modules/storage.md

# Storage Module (modules/storage.py) Usage Guide

The storage.py module provides helper functions for:

• Generating permalinks for 3D viewer projects.
• Uploading files in batches to a Hugging Face repository.
• Managing URL shortening by storing (short URL, full URL) pairs in a JSON file on the repository.
• Retrieving full URLs from short URL IDs and vice versa.
• Handle specific file types for 3D models, images, video and audio.
• 🔑 Cryptographic key management for Open Badge 3.0 issuers.

Key Functions

1. generate_permalink(valid_files, base_url_external, permalink_viewer_url="surn-3d-viewer.hf.space")

• Purpose:
  Given a list of file paths, it looks for exactly one model file (with an extension defined in model_extensions) and exactly two image files (extensions defined in image_extensions). If the criteria are met, it returns a permalink URL built from the base URL and query parameters.
• **Usage Example:**from modules.storage import generate_permalink

valid_files = [ "models/3d_model.glb", "images/model_texture.png", "images/model_depth.png" ] base_url_external = "https://huggingface.co/datasets/Surn/Storage/resolve/main/saved_models/my_model" permalink = generate_permalink(valid_files, base_url_external) if permalink: print("Permalink:", permalink)

2. generate_permalink_from_urls(model_url, hm_url, img_url, permalink_viewer_url="surn-3d-viewer.hf.space")

• Purpose:
  Constructs a permalink URL by combining individual URLs for a 3D model (model_url), height map (hm_url), and image (img_url) into a single URL with corresponding query parameters.
• **Usage Example:**from modules.storage import generate_permalink_from_urls

model_url = "https://example.com/model.glb" hm_url = "https://example.com/heightmap.png" img_url = "https://example.com/source.png"

permalink = generate_permalink_from_urls(model_url, hm_url, img_url) print("Generated Permalink:", permalink)

3. upload_files_to_repo(files, repo_id, folder_name, create_permalink=False, repo_type="dataset", permalink_viewer_url="surn-3d-viewer.hf.space")

• Purpose:
  Uploads a batch of files (each file represented as a path string) to a specified Hugging Face repository (e.g. "Surn/Storage") under a given folder. The function's return type is Union[Dict[str, Any], List[Tuple[Any, str]]].

  • When create_permalink is True and exactly three valid files (one model and two images) are provided, the function returns a dictionary:{ "response": , "permalink": "", "short_permalink": "" } - Otherwise (or if create_permalink is False or conditions for permalink creation are not met), it returns a list of tuples, where each tuple is (upload_folder_response, individual_file_link).
  • If no valid files are provided, it returns an empty list [] (this case should ideally also return the dictionary with empty/None values for consistency, but currently returns [] as per the code).

• Usage Example:

  **a. Uploading with permalink creation:**from modules.storage import upload_files_to_repo

files_for_permalink = [ "local/path/to/model.glb", "local/path/to/heightmap.png", "local/path/to/image.png" ] repo_id = "Surn/Storage" # Make sure this is defined, e.g., from constants or environment variables folder_name = "my_new_model_with_permalink"

upload_result = upload_files_to_repo( files_for_permalink, repo_id, folder_name, create_permalink=True )

if isinstance(upload_result, dict): print("Upload Response:", upload_result.get("response")) print("Full Permalink:", upload_result.get("permalink")) print("Short Permalink:", upload_result.get("short_permalink")) elif upload_result: # Check if list is not empty print("Upload Response for individual files:") for res, link in upload_result: print(f" Response: {res}, Link: {link}") else: print("No files uploaded or error occurred.") **b. Uploading without permalink creation (or if conditions for permalink are not met):**from modules.storage import upload_files_to_repo

files_individual = [ "local/path/to/another_model.obj", "local/path/to/texture.jpg" ] repo_id = "Surn/Storage" folder_name = "my_other_uploads"

upload_results_list = upload_files_to_repo( files_individual, repo_id, folder_name, create_permalink=False # Or if create_permalink=True but not 1 model & 2 images )

if upload_results_list: # Will be a list of tuples print("Upload results for individual files:") for res, link in upload_results_list: print(f" Upload Response: {res}, File Link: {link}") else: print("No files uploaded or error occurred.")

4. URL Shortening Functions: gen_full_url(...) and Helpers

The module also enables URL shortening by managing a JSON file (e.g. shortener.json) in a Hugging Face repository. It supports CRUD-like operations:

• Read: Look up the full URL using a provided short URL ID.
• Create: Generate a new short URL ID for a full URL if no existing mapping exists.
• Update/Conflict Handling:
  If both short URL ID and full URL are provided, it checks consistency and either confirms or reports a conflict.

gen_full_url(short_url=None, full_url=None, repo_id=None, repo_type="dataset", permalink_viewer_url="surn-3d-viewer.hf.space", json_file="shortener.json")

• Purpose:
  Based on which parameter is provided, it retrieves or creates a mapping between a short URL ID and a full URL.

  • If only short_url (the ID) is given, it returns the corresponding full_url.
  • If only full_url is given, it looks up an existing short_url ID or generates and stores a new one.
  • If both are given, it validates and returns the mapping or an error status.

• Returns: A tuple (status_message, result_url), where status_message indicates the outcome (e.g., "success_retrieved_full", "created_short") and result_url is the relevant URL (full or short ID).

• Usage Examples:

  **a. Convert a full URL into a short URL ID:**from modules.storage import gen_full_url

from modules.constants import HF_REPO_ID, SHORTENER_JSON_FILE # Assuming these are defined

full_permalink = "https://surn-3d-viewer.hf.space/?3d=https%3A%2F%2Fexample.com%2Fmodel.glb&hm=https%3A%2F%2Fexample.com%2Fheightmap.png&image=https%3A%2F%2Fexample.com%2Fsource.png"

status, short_id = gen_full_url( full_url=full_permalink, repo_id=HF_REPO_ID, json_file=SHORTENER_JSON_FILE ) print("Status:", status) if status == "created_short" or status == "success_retrieved_short": print("Shortened URL ID:", short_id) # Construct the full short URL for sharing: # permalink_viewer_url = "surn-3d-viewer.hf.space" # Or from constants # shareable_short_url = f"https://{permalink_viewer_url}/?sid={short_id}" # print("Shareable Short URL:", shareable_short_url) **b. Retrieve the full URL from a short URL ID:**from modules.storage import gen_full_url from modules.constants import HF_REPO_ID, SHORTENER_JSON_FILE # Assuming these are defined

short_id_to_lookup = "aBcDeFg1" # Example short URL ID

status, retrieved_full_url = gen_full_url( short_url=short_id_to_lookup, repo_id=HF_REPO_ID, json_file=SHORTENER_JSON_FILE ) print("Status:", status) if status == "success_retrieved_full": print("Retrieved Full URL:", retrieved_full_url)

🔑 Cryptographic Key Management Functions

5. store_issuer_keypair(issuer_id, public_key, private_key, repo_id=None)

• Purpose:
  Securely store cryptographic keys for an issuer in a private Hugging Face repository. Private keys are encrypted before storage.
• ⚠️ IMPORTANT: This function requires a PRIVATE Hugging Face repository to ensure the security of stored private keys. Never use this with public repositories.
• **Storage Structure:**keys/issuers/{issuer_id}/ ├── private_key.json (encrypted) └── public_key.json- Returns: bool - True if keys were stored successfully, False otherwise.
• **Usage Example:**from modules.storage import store_issuer_keypair

Example Ed25519 keys (multibase encoded)

issuer_id = "https://example.edu/issuers/565049" public_key = "z6MkhaXgBZDvotDkL5257faiztiGiC2QtKLGpbnnEGta2doK" private_key = "z3u2MQhLnQw7nvJRGJCdKdqfXHV4N7BLKuEGFWnJqsVSdgYv"

success = store_issuer_keypair(issuer_id, public_key, private_key) if success: print("Keys stored successfully") else: print("Failed to store keys")

6. get_issuer_keypair(issuer_id, repo_id=None)

• Purpose:
  Retrieve and decrypt stored cryptographic keys for an issuer from the private Hugging Face repository.
• ⚠️ IMPORTANT: This function accesses a PRIVATE Hugging Face repository containing encrypted private keys. Ensure proper access control and security measures.
• Returns: Tuple[Optional[str], Optional[str]] - (public_key, private_key) or (None, None) if not found.
• **Usage Example:**from modules.storage import get_issuer_keypair

issuer_id = "https://example.edu/issuers/565049" public_key, private_key = get_issuer_keypair(issuer_id)

if public_key and private_key: print("Keys retrieved successfully") print(f"Public key: {public_key}") # Use private_key for signing operations else: print("Keys not found or error occurred")

7. get_verification_methods_registry(repo_id=None)

• Purpose:
  Retrieve the global verification methods registry containing all registered issuer public keys.
• Returns: Dict[str, Any] - Registry data containing all verification methods.
• **Usage Example:**from modules.storage import get_verification_methods_registry

registry = get_verification_methods_registry() methods = registry.get("verification_methods", [])

for method in methods: print(f"Issuer: {method['issuer_id']}") print(f"Public Key: {method['public_key']}") print(f"Key Type: {method['key_type']}") print("---")

8. list_issuer_ids(repo_id=None)

• Purpose:
  List all issuer IDs that have stored keys in the repository.
• Returns: List[str] - List of issuer IDs.
• **Usage Example:**from modules.storage import list_issuer_ids

issuer_ids = list_issuer_ids() print("Registered issuers:") for issuer_id in issuer_ids: print(f" - {issuer_id}")

Notes

• Authentication: All functions that interact with Hugging Face Hub use the HF API token defined as HF_API_TOKEN in modules/constants.py. Ensure this environment variable is correctly set.
• Constants: Functions like gen_full_url and upload_files_to_repo (when creating short links) rely on HF_REPO_ID and SHORTENER_JSON_FILE from modules/constants.py for the URL shortening feature.
• 🔐 Private Repository Requirement: Key management functions require a PRIVATE Hugging Face repository to ensure the security of stored encrypted private keys. Never use these functions with public repositories.
• File Types: Only files with extensions included in upload_file_types (a combination of model_extensions and image_extensions from modules/constants.py) are processed by upload_files_to_repo.
• Repository Configuration: When using URL shortening, file uploads, and key management, ensure that the specified Hugging Face repository (e.g., defined by HF_REPO_ID) exists and that you have write permissions.
• Temporary Directory: upload_files_to_repo temporarily copies files to a local directory (configured by TMPDIR in modules/constants.py) before uploading.
• Key Encryption: Private keys are encrypted using basic XOR encryption (demo implementation). In production environments, upgrade to proper encryption like Fernet from the cryptography library.
• Error Handling: Functions include basic error handling (e.g., catching RepositoryNotFoundError, EntryNotFoundError, JSON decoding errors, or upload issues) and print messages to the console for debugging. Review function return values to handle these cases appropriately in your application.

🔒 Security Considerations for Key Management

1. Private Repository Only: Always use private repositories for key storage to protect cryptographic material.
2. Key Sanitization: Issuer IDs are sanitized for file system compatibility (replacing special characters with underscores).
3. Encryption: Private keys are encrypted before storage. Upgrade to Fernet encryption in production.
4. Access Control: Implement proper authentication and authorization for key access.
5. Key Rotation: Consider implementing key rotation mechanisms for enhanced security.
6. Audit Logging: Monitor key access and usage patterns for security auditing.

---

This guide provides the essential usage examples for interacting with the storage, URL-shortening, and cryptographic key management functionality. You can integrate these examples into your application or use them as a reference when extending functionality.