README_CSV_DEPLOYMENT.md

# CSV-First Deployment for Hugging Face Spaces

This document explains how to deploy your Optimus Agent Performance dashboard to Hugging Face Spaces using a CSV-first approach to avoid API rate limiting issues.

## Overview

The CSV-first approach works by:
1. **Local Data Generation**: Run data fetching and preprocessing locally where you have full API access
2. **CSV Export**: Save preprocessed data to CSV files
3. **Space Deployment**: Upload CSV files to your Hugging Face Space
4. **CSV-First Loading**: The Space app prioritizes loading from CSV files over making API calls

## Files Added for CSV-First Deployment

### 1. `load_from_csv.py`
Contains functions to load data from CSV files:
- `load_apr_data_from_csv()` - Loads APR data from CSV
- `load_roi_data_from_csv()` - Loads ROI data from CSV
- `load_statistics_from_csv()` - Loads statistics from CSV
- `check_csv_data_availability()` - Checks which CSV files are available
- `get_data_freshness_info()` - Analyzes how fresh the CSV data is

### 2. `generate_csv_for_space.py`
Interactive script to generate CSV files locally:
- Fetches fresh data from the API
- Applies all preprocessing (including `initial_value_fixer.py`)
- Saves to CSV files ready for Space deployment
- Provides deployment guidance

### 3. Modified `app.py`
Updated visualization functions with CSV-first logic:
- `generate_apr_visualizations()` - Now tries CSV first, falls back to API
- `generate_roi_visualizations()` - Now tries CSV first, falls back to API
- Imports CSV loading functions
- Maintains backward compatibility

## Deployment Workflow

### Step 1: Generate CSV Files Locally

Run the CSV generation script on your local machine:

```bash
python generate_csv_for_space.py
```

This will:
- Check existing CSV files and their freshness
- Fetch fresh data from the API (no rate limits locally)
- Apply preprocessing using `initial_value_fixer.py`
- Save CSV files: `optimus_apr_values.csv`, `optimus_apr_statistics.csv`, `optimus_roi_values.csv`

### Step 2: Upload Files to Hugging Face Space

Upload these files to your Hugging Face Space repository:
- `app.py` (modified with CSV-first logic)
- `load_from_csv.py` (new CSV loading functions)
- `initial_value_fixer.py` (existing preprocessing)
- `fetch_and_preprocess_data.py` (existing data functions)
- `optimus_apr_values.csv` (generated data)
- `optimus_apr_statistics.csv` (generated statistics)
- `optimus_roi_values.csv` (generated data)

### Step 3: Deploy Space

Your Space will now:
1. **Try CSV first**: Load data from uploaded CSV files
2. **Fast loading**: No API calls needed, instant visualization
3. **Fallback**: If CSV files are missing, fall back to API (with rate limits)
4. **Error handling**: Show appropriate messages if both CSV and API fail

## CSV File Structure

### APR Data (`optimus_apr_values.csv`)
```csv
apr,adjusted_apr,timestamp,agent_id,agent_name,is_dummy,metric_type,roi,address
15.5,12.3,2025-06-14 10:30:00,123,Agent_1,False,APR,0.85,0x123...
```

### ROI Data (`optimus_roi_values.csv`)
```csv
roi,timestamp,agent_id,agent_name,is_dummy,metric_type
0.85,2025-06-14 10:30:00,123,Agent_1,False,ROI
```

### Statistics (`optimus_apr_statistics.csv`)
```csv
agent_id,agent_name,total_points,apr_points,avg_apr,avg_adjusted_apr,latest_timestamp
123,Agent_1,100,50,15.5,12.3,2025-06-14 10:30:00
```

## Benefits

### ✅ Advantages
- **No Rate Limits**: Space doesn't make API calls
- **Fast Loading**: Instant visualization from CSV
- **Reliable**: No dependency on external API availability
- **Preprocessed**: Data includes all transformations from `initial_value_fixer.py`
- **Fresh Data**: Update CSV files as needed locally

### ⚠️ Considerations
- **Manual Updates**: Need to regenerate CSV files for fresh data
- **File Size**: CSV files add to Space storage
- **Data Staleness**: CSV data becomes stale over time

## Updating Data

To update your Space with fresh data:

1. **Run locally**: `python generate_csv_for_space.py`
2. **Upload new CSV files** to your Space repository
3. **Space auto-updates** with new data

## Monitoring

### Data Freshness
The app logs data freshness information:
```python
from load_from_csv import get_data_freshness_info
freshness = get_data_freshness_info()
# Shows how old the CSV data is
```

### CSV Availability
Check which CSV files are available:
```python
from load_from_csv import check_csv_data_availability
availability = check_csv_data_availability()
# Shows file status, size, modification date
```

## Troubleshooting

### CSV Files Not Loading
1. Check file names match exactly: `optimus_apr_values.csv`, `optimus_roi_values.csv`
2. Verify CSV files are in the Space root directory
3. Check Space logs for CSV loading errors

### Data Issues
1. Regenerate CSV files locally with fresh API data
2. Verify preprocessing ran correctly in `generate_csv_for_space.py`
3. Check CSV file structure matches expected format

### Fallback to API
If CSV loading fails, the app falls back to API calls:
1. This may trigger rate limits in Hugging Face Spaces
2. Check Space logs for API errors
3. Ensure CSV files are properly uploaded

## Best Practices

1. **Regular Updates**: Run `generate_csv_for_space.py` daily or weekly
2. **Monitor Logs**: Check Space logs for CSV loading status
3. **Backup CSV**: Keep local copies of generated CSV files
4. **Test Locally**: Test CSV loading locally before deploying
5. **Version Control**: Track CSV generation timestamps

## Example Space Configuration

Your Space should include these files:
```
├── app.py                          # Modified with CSV-first logic
├── load_from_csv.py               # CSV loading functions
├── initial_value_fixer.py         # Preprocessing logic
├── fetch_and_preprocess_data.py   # Data functions
├── optimus_apr_values.csv         # Generated APR data
├── optimus_apr_statistics.csv     # Generated statistics
├── optimus_roi_values.csv         # Generated ROI data
├── requirements.txt               # Dependencies
└── README.md                      # Space documentation
```

This approach ensures your Hugging Face Space runs reliably without hitting API rate limits while maintaining all the preprocessing and visualization features of your application.