|
|
""" |
|
|
Advanced 3D Reconstruction from Single or Multiple Images |
|
|
Academic-grade pipeline with responsible AI considerations, multi-image support, |
|
|
quality metrics, multiple export formats, and interactive visualization |
|
|
""" |
|
|
|
|
|
import gradio as gr |
|
|
import numpy as np |
|
|
import torch |
|
|
from PIL import Image |
|
|
from transformers import GLPNForDepthEstimation, GLPNImageProcessor, DPTForDepthEstimation, DPTImageProcessor |
|
|
import open3d as o3d |
|
|
import plotly.graph_objects as go |
|
|
import matplotlib.pyplot as plt |
|
|
import io |
|
|
import json |
|
|
import time |
|
|
from pathlib import Path |
|
|
import tempfile |
|
|
import zipfile |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
THEORY_TEXT = """ |
|
|
## Theoretical Background |
|
|
|
|
|
## About This Tool |
|
|
|
|
|
This application demonstrates how artificial intelligence can convert 2D photographs into interactive 3D models automatically, with a focus on responsible AI practices. |
|
|
|
|
|
### What Makes This Special |
|
|
|
|
|
**Traditional Approach:** |
|
|
- Need special equipment (3D scanner, multiple cameras) |
|
|
- Requires technical expertise |
|
|
- Time-consuming process |
|
|
- Expensive |
|
|
--- |
|
|
|
|
|
## The Technology |
|
|
|
|
|
### AI Models Used |
|
|
|
|
|
This tool uses state-of-the-art artificial intelligence models: |
|
|
|
|
|
|
|
|
### Depth Estimation Technology |
|
|
|
|
|
**GLPN (Global-Local Path Networks)** |
|
|
- Paper: Kim et al., CVPR 2022 |
|
|
- Optimized for: Indoor/outdoor architectural scenes |
|
|
- Training: NYU Depth V2 (urban indoor environments) |
|
|
- Best for: Building interiors, street-level views, architectural details |
|
|
- Geographic advantage: Fast processing for field documentation |
|
|
|
|
|
**DPT (Dense Prediction Transformer)** |
|
|
- Paper: Ranftl et al., ICCV 2021 |
|
|
- Optimized for: Complex urban scenes |
|
|
- Training: Multiple datasets (urban and natural environments) |
|
|
- Best for: Wide-area urban landscapes, complex built environments |
|
|
- Geographic advantage: Superior accuracy for planning-grade documentation |
|
|
|
|
|
### Multi-Image Reconstruction |
|
|
|
|
|
**Single Image Mode:** |
|
|
- Fast processing |
|
|
- Works with limited data |
|
|
- Best for quick assessments |
|
|
- Limitations: Single viewpoint, scale ambiguity |
|
|
|
|
|
**Multiple Image Mode (NEW):** |
|
|
- Improved coverage and accuracy |
|
|
- Combines depth maps from different viewpoints |
|
|
- Reduces occlusion issues |
|
|
- Better overall 3D representation |
|
|
- Note: Images should be of the same object/scene from different angles |
|
|
|
|
|
### How It Works (Simple) |
|
|
1. **AI looks at photo(s)** → Recognizes objects, patterns, perspective |
|
|
2. **Estimates distance** → Figures out what's close, what's far |
|
|
3. **Creates 3D points** → Places colored dots in 3D space |
|
|
4. **Builds surface** → Connects dots into smooth shape |
|
|
5. **Multi-view fusion** (if multiple images) → Combines information for better accuracy |
|
|
|
|
|
### Responsible AI Considerations |
|
|
|
|
|
This tool is designed with responsible AI principles in mind: |
|
|
|
|
|
**1. Privacy Protection:** |
|
|
- All processing happens locally - no data sent to external servers |
|
|
- No image storage or retention after processing |
|
|
- No facial recognition or identity tracking |
|
|
- Users maintain full control over their data |
|
|
- Recommendation: Avoid uploading images with identifiable individuals |
|
|
|
|
|
**2. Explainability & Transparency:** |
|
|
- Depth map visualization shows how AI "sees" the scene |
|
|
- Quality metrics provide confidence indicators |
|
|
- Processing steps are clearly documented |
|
|
- Model limitations are explicitly stated |
|
|
- Users can verify reconstruction quality |
|
|
|
|
|
**3. Fairness & Bias Awareness:** |
|
|
- Models trained primarily on indoor/urban scenes |
|
|
- May perform differently on underrepresented scene types |
|
|
- Quality metrics help identify potential biases |
|
|
- Users should validate results for critical applications |
|
|
|
|
|
**4. Intended Use & Limitations:** |
|
|
- Designed for educational and research purposes |
|
|
- Not suitable for: safety-critical applications, surveillance, or precise measurements |
|
|
- Best for: visualization, preliminary analysis, teaching |
|
|
- Scale ambiguity: requires ground control for absolute measurements |
|
|
|
|
|
**5. Data Governance:** |
|
|
- Open-source models with documented training data |
|
|
- No proprietary algorithms or black boxes |
|
|
- Full transparency in reconstruction pipeline |
|
|
- Users can audit and validate the process |
|
|
|
|
|
### Spatial Data Pipeline |
|
|
|
|
|
Our reconstruction pipeline generates geospatially-relevant data: |
|
|
|
|
|
**1. Monocular Depth Estimation** |
|
|
- Challenge: Extracting 3D spatial information from 2D photographs |
|
|
- Application: Similar to photogrammetry but from single images |
|
|
- Output: Relative depth maps for spatial analysis |
|
|
- Use case: Quick field assessment without specialized equipment |
|
|
|
|
|
**2. Point Cloud Generation (Spatial Coordinates)** |
|
|
- Creates 3D coordinate system (X, Y, Z) from pixels |
|
|
- Each point: Geographic location + RGB color information |
|
|
- Compatible with: GIS software, CAD tools, spatial databases |
|
|
- Use case: Integration with existing urban datasets |
|
|
|
|
|
**3. 3D Mesh Generation (Surface Models)** |
|
|
- Creates continuous surface from discrete points |
|
|
- Similar to: Digital terrain models (DTMs) for buildings |
|
|
- Output formats: Compatible with ArcGIS, QGIS, SketchUp |
|
|
- Use case: 3D city models, urban visualization |
|
|
|
|
|
### Spatial Quality Metrics |
|
|
|
|
|
**For Urban Planning Applications:** |
|
|
|
|
|
- **Point Cloud Density**: 290K+ points = high spatial resolution |
|
|
- **Geometric Accuracy**: Manifold checks ensure valid topology |
|
|
- **Surface Continuity**: Watertight meshes = complete volume calculations |
|
|
- **Data Fidelity**: Triangle count indicates level of detail |
|
|
|
|
|
**Limitations for Geographic Applications:** |
|
|
|
|
|
1. **Scale Ambiguity**: Requires ground control points for absolute measurements |
|
|
2. **Single Viewpoint**: Cannot capture occluded facades or hidden spaces (reduced with multi-image mode) |
|
|
3. **No Georeferencing**: Outputs in local coordinates, not global (lat/lon) |
|
|
4. **Weather Dependent**: Best results with clear, well-lit conditions |
|
|
|
|
|
""" |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def check_image_privacy(image): |
|
|
""" |
|
|
Check if image might contain sensitive information. |
|
|
Returns warnings if potential privacy concerns detected. |
|
|
""" |
|
|
warnings = [] |
|
|
|
|
|
|
|
|
width, height = image.size |
|
|
if width * height > 4000 * 3000: |
|
|
warnings.append("⚠️ High-resolution image detected. Ensure it doesn't contain identifiable individuals.") |
|
|
|
|
|
|
|
|
aspect_ratio = width / height |
|
|
if aspect_ratio > 2.5 or aspect_ratio < 0.4: |
|
|
warnings.append("ℹ️ Unusual aspect ratio detected. Common in security camera footage.") |
|
|
|
|
|
return warnings |
|
|
|
|
|
def generate_explainability_report(metrics, depth_stats): |
|
|
""" |
|
|
Generate an explainability report for the reconstruction. |
|
|
Helps users understand how the AI made decisions. |
|
|
""" |
|
|
report = "### 🔍 AI Decision Explainability\n\n" |
|
|
|
|
|
|
|
|
depth_range = depth_stats['max'] - depth_stats['min'] |
|
|
depth_variation = depth_stats['std'] / depth_stats['mean'] |
|
|
|
|
|
if depth_variation > 0.5: |
|
|
report += "- **High depth variation detected**: Scene has significant depth differences (good for reconstruction)\n" |
|
|
else: |
|
|
report += "- **Low depth variation**: Scene is relatively flat (may limit 3D detail)\n" |
|
|
|
|
|
|
|
|
outlier_ratio = metrics['outliers_removed'] / metrics['initial_points'] |
|
|
if outlier_ratio < 0.05: |
|
|
report += "- **Clean depth estimation**: AI is confident about depth predictions (< 5% outliers)\n" |
|
|
elif outlier_ratio < 0.15: |
|
|
report += "- **Moderate noise**: Some uncertainty in depth predictions (normal for complex scenes)\n" |
|
|
else: |
|
|
report += "- **High uncertainty**: AI struggled with this scene (> 15% outliers removed)\n" |
|
|
|
|
|
|
|
|
if metrics['is_watertight']: |
|
|
report += "- **Complete surface reconstruction**: AI successfully closed all gaps\n" |
|
|
else: |
|
|
report += "- **Incomplete surface**: Some areas couldn't be reconstructed (occluded or ambiguous)\n" |
|
|
|
|
|
|
|
|
if metrics['is_edge_manifold'] and outlier_ratio < 0.1: |
|
|
report += "\n**Overall Confidence**: ✅ High - Results are reliable\n" |
|
|
elif metrics['is_vertex_manifold']: |
|
|
report += "\n**Overall Confidence**: ⚠️ Medium - Results are usable but verify quality\n" |
|
|
else: |
|
|
report += "\n**Overall Confidence**: ❌ Low - Results may need manual correction\n" |
|
|
|
|
|
return report |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
print("Loading GLPN model...") |
|
|
glpn_processor = GLPNImageProcessor.from_pretrained("vinvino02/glpn-nyu") |
|
|
glpn_model = GLPNForDepthEstimation.from_pretrained("vinvino02/glpn-nyu") |
|
|
print("GLPN model loaded successfully!") |
|
|
|
|
|
|
|
|
dpt_model = None |
|
|
dpt_processor = None |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def estimate_depth_for_image(image, model_choice): |
|
|
"""Estimate depth for a single image""" |
|
|
if model_choice == "GLPN (Recommended)": |
|
|
processor = glpn_processor |
|
|
model = glpn_model |
|
|
else: |
|
|
global dpt_model, dpt_processor |
|
|
if dpt_model is None: |
|
|
print("Loading DPT model (first time only)...") |
|
|
dpt_processor = DPTImageProcessor.from_pretrained("Intel/dpt-large") |
|
|
dpt_model = DPTForDepthEstimation.from_pretrained("Intel/dpt-large") |
|
|
processor = dpt_processor |
|
|
model = dpt_model |
|
|
|
|
|
inputs = processor(images=image, return_tensors="pt") |
|
|
|
|
|
with torch.no_grad(): |
|
|
outputs = model(**inputs) |
|
|
predicted_depth = outputs.predicted_depth |
|
|
|
|
|
return predicted_depth |
|
|
|
|
|
def merge_point_clouds(point_clouds, colors_list): |
|
|
""" |
|
|
Merge multiple point clouds with basic alignment. |
|
|
Note: This is a simple merging strategy. For better results, |
|
|
consider using registration algorithms (ICP, etc.) |
|
|
""" |
|
|
all_points = [] |
|
|
all_colors = [] |
|
|
|
|
|
for i, (points, colors) in enumerate(zip(point_clouds, colors_list)): |
|
|
|
|
|
offset = np.array([i * 0.5, 0, 0]) |
|
|
all_points.append(points + offset) |
|
|
all_colors.append(colors) |
|
|
|
|
|
merged_points = np.vstack(all_points) |
|
|
merged_colors = np.vstack(all_colors) |
|
|
|
|
|
return merged_points, merged_colors |
|
|
|
|
|
def process_image(images, model_choice="GLPN (Recommended)", visualization_type="mesh", enable_privacy_check=True): |
|
|
"""Main processing pipeline - supports single or multiple images""" |
|
|
|
|
|
def _generate_quality_assessment(metrics): |
|
|
"""Generate quality assessment based on metrics""" |
|
|
assessment = [] |
|
|
|
|
|
|
|
|
outlier_pct = (metrics['outliers_removed'] / metrics['initial_points']) * 100 |
|
|
if outlier_pct < 5: |
|
|
assessment.append("Very clean depth estimation (low noise)") |
|
|
elif outlier_pct < 15: |
|
|
assessment.append("Good depth quality (normal noise level)") |
|
|
else: |
|
|
assessment.append("High noise in depth estimation") |
|
|
|
|
|
|
|
|
if metrics['is_edge_manifold'] and metrics['is_vertex_manifold']: |
|
|
assessment.append("Excellent topology - mesh is well-formed") |
|
|
elif metrics['is_vertex_manifold']: |
|
|
assessment.append("Good local topology but has some edge issues") |
|
|
else: |
|
|
assessment.append("Topology issues present - may need cleanup") |
|
|
|
|
|
|
|
|
if metrics['is_watertight']: |
|
|
assessment.append("Watertight mesh - ready for 3D printing!") |
|
|
else: |
|
|
assessment.append("Not watertight - use MeshLab's 'Close Holes' for 3D printing") |
|
|
|
|
|
|
|
|
if metrics['triangles'] > 1000000: |
|
|
assessment.append("Very detailed mesh - may be slow in some software") |
|
|
elif metrics['triangles'] > 500000: |
|
|
assessment.append("High detail mesh - good quality") |
|
|
else: |
|
|
assessment.append("Moderate detail - good balance of quality and performance") |
|
|
|
|
|
return "\n".join(f"- {item}" for item in assessment) |
|
|
|
|
|
if images is None or len(images) == 0: |
|
|
return None, None, None, "Please upload at least one image.", None, None |
|
|
|
|
|
|
|
|
if not isinstance(images, list): |
|
|
images = [images] |
|
|
|
|
|
try: |
|
|
num_images = len(images) |
|
|
print(f"Starting reconstruction with {num_images} image(s) using {model_choice}...") |
|
|
|
|
|
|
|
|
privacy_warnings = [] |
|
|
if enable_privacy_check: |
|
|
for idx, img in enumerate(images): |
|
|
warnings = check_image_privacy(img) |
|
|
if warnings: |
|
|
privacy_warnings.extend([f"Image {idx+1}: {w}" for w in warnings]) |
|
|
|
|
|
privacy_report = "" |
|
|
if privacy_warnings: |
|
|
privacy_report = "### 🔒 Privacy Considerations\n\n" + "\n".join(privacy_warnings) + "\n\n" |
|
|
|
|
|
|
|
|
all_point_clouds = [] |
|
|
all_colors = [] |
|
|
depth_visualizations = [] |
|
|
depth_stats_list = [] |
|
|
total_depth_time = 0 |
|
|
|
|
|
for idx, image in enumerate(images): |
|
|
print(f"\n=== Processing Image {idx+1}/{num_images} ===") |
|
|
|
|
|
|
|
|
print(f"Image {idx+1}: Preprocessing...") |
|
|
new_height = 480 if image.height > 480 else image.height |
|
|
new_height -= (new_height % 32) |
|
|
new_width = int(new_height * image.width / image.height) |
|
|
diff = new_width % 32 |
|
|
new_width = new_width - diff if diff < 16 else new_width + (32 - diff) |
|
|
new_size = (new_width, new_height) |
|
|
image = image.resize(new_size, Image.LANCZOS) |
|
|
print(f"Image {idx+1} resized to: {new_size}") |
|
|
|
|
|
|
|
|
print(f"Image {idx+1}: Estimating depth...") |
|
|
start_time = time.time() |
|
|
predicted_depth = estimate_depth_for_image(image, model_choice) |
|
|
depth_time = time.time() - start_time |
|
|
total_depth_time += depth_time |
|
|
print(f"Image {idx+1}: Depth estimation completed in {depth_time:.2f}s") |
|
|
|
|
|
|
|
|
pad = 16 |
|
|
output = predicted_depth.squeeze().cpu().numpy() * 1000.0 |
|
|
output = output[pad:-pad, pad:-pad] |
|
|
image_cropped = image.crop((pad, pad, image.width - pad, image.height - pad)) |
|
|
|
|
|
|
|
|
depth_height, depth_width = output.shape |
|
|
img_width, img_height = image_cropped.size |
|
|
|
|
|
if depth_height != img_height or depth_width != img_width: |
|
|
from scipy import ndimage |
|
|
zoom_factors = (img_height / depth_height, img_width / depth_width) |
|
|
output = ndimage.zoom(output, zoom_factors, order=1) |
|
|
|
|
|
image = image_cropped |
|
|
|
|
|
|
|
|
depth_stats = { |
|
|
'min': float(np.min(output)), |
|
|
'max': float(np.max(output)), |
|
|
'mean': float(np.mean(output)), |
|
|
'std': float(np.std(output)) |
|
|
} |
|
|
depth_stats_list.append(depth_stats) |
|
|
|
|
|
|
|
|
fig, ax = plt.subplots(1, 2, figsize=(14, 7)) |
|
|
ax[0].imshow(image) |
|
|
ax[0].set_title(f'Image {idx+1}: Original', fontsize=14, fontweight='bold') |
|
|
ax[0].axis('off') |
|
|
|
|
|
im = ax[1].imshow(output, cmap='plasma') |
|
|
ax[1].set_title(f'Image {idx+1}: Depth Map', fontsize=14, fontweight='bold') |
|
|
ax[1].axis('off') |
|
|
plt.colorbar(im, ax=ax[1], fraction=0.046, pad=0.04) |
|
|
plt.tight_layout() |
|
|
|
|
|
buf = io.BytesIO() |
|
|
plt.savefig(buf, format='png', dpi=150, bbox_inches='tight') |
|
|
buf.seek(0) |
|
|
depth_viz = Image.open(buf) |
|
|
depth_visualizations.append(depth_viz) |
|
|
plt.close() |
|
|
|
|
|
|
|
|
print(f"Image {idx+1}: Generating point cloud...") |
|
|
width, height = image.size |
|
|
|
|
|
if output.shape != (height, width): |
|
|
from scipy import ndimage |
|
|
zoom_factors = (height / output.shape[0], width / output.shape[1]) |
|
|
output = ndimage.zoom(output, zoom_factors, order=1) |
|
|
|
|
|
depth_image = (output * 255 / np.max(output)).astype(np.uint8) |
|
|
image_array = np.array(image) |
|
|
|
|
|
depth_o3d = o3d.geometry.Image(depth_image) |
|
|
image_o3d = o3d.geometry.Image(image_array) |
|
|
rgbd_image = o3d.geometry.RGBDImage.create_from_color_and_depth( |
|
|
image_o3d, depth_o3d, convert_rgb_to_intensity=False |
|
|
) |
|
|
|
|
|
camera_intrinsic = o3d.camera.PinholeCameraIntrinsic() |
|
|
camera_intrinsic.set_intrinsics(width, height, 500, 500, width/2, height/2) |
|
|
|
|
|
pcd_temp = o3d.geometry.PointCloud.create_from_rgbd_image(rgbd_image, camera_intrinsic) |
|
|
|
|
|
|
|
|
all_point_clouds.append(np.asarray(pcd_temp.points)) |
|
|
all_colors.append(np.asarray(pcd_temp.colors)) |
|
|
|
|
|
print(f"Image {idx+1}: Generated {len(pcd_temp.points)} points") |
|
|
|
|
|
|
|
|
if len(depth_visualizations) == 1: |
|
|
combined_depth_viz = depth_visualizations[0] |
|
|
else: |
|
|
|
|
|
cols = min(2, len(depth_visualizations)) |
|
|
rows = (len(depth_visualizations) + cols - 1) // cols |
|
|
|
|
|
fig, axes = plt.subplots(rows, cols, figsize=(14 * cols, 7 * rows)) |
|
|
if rows == 1: |
|
|
axes = [axes] if cols == 1 else axes |
|
|
else: |
|
|
axes = axes.flatten() |
|
|
|
|
|
for idx, depth_viz in enumerate(depth_visualizations): |
|
|
axes[idx].imshow(depth_viz) |
|
|
axes[idx].axis('off') |
|
|
axes[idx].set_title(f'Image {idx+1}', fontsize=16, fontweight='bold') |
|
|
|
|
|
|
|
|
for idx in range(len(depth_visualizations), len(axes)): |
|
|
axes[idx].axis('off') |
|
|
|
|
|
plt.tight_layout() |
|
|
buf = io.BytesIO() |
|
|
plt.savefig(buf, format='png', dpi=150, bbox_inches='tight') |
|
|
buf.seek(0) |
|
|
combined_depth_viz = Image.open(buf) |
|
|
plt.close() |
|
|
|
|
|
|
|
|
print(f"\nMerging {num_images} point cloud(s)...") |
|
|
if num_images > 1: |
|
|
merged_points, merged_colors = merge_point_clouds(all_point_clouds, all_colors) |
|
|
else: |
|
|
merged_points = all_point_clouds[0] |
|
|
merged_colors = all_colors[0] |
|
|
|
|
|
|
|
|
pcd = o3d.geometry.PointCloud() |
|
|
pcd.points = o3d.utility.Vector3dVector(merged_points) |
|
|
pcd.colors = o3d.utility.Vector3dVector(merged_colors) |
|
|
|
|
|
initial_points = len(pcd.points) |
|
|
print(f"Combined point cloud: {initial_points} points") |
|
|
|
|
|
|
|
|
print("Cleaning combined point cloud...") |
|
|
cl, ind = pcd.remove_statistical_outlier(nb_neighbors=20, std_ratio=2.0) |
|
|
pcd = pcd.select_by_index(ind) |
|
|
outliers_removed = initial_points - len(pcd.points) |
|
|
print(f"Removed {outliers_removed} outliers") |
|
|
|
|
|
|
|
|
print("Estimating normals...") |
|
|
pcd.estimate_normals() |
|
|
pcd.orient_normals_to_align_with_direction() |
|
|
|
|
|
|
|
|
print("Creating mesh...") |
|
|
mesh_start = time.time() |
|
|
mesh = o3d.geometry.TriangleMesh.create_from_point_cloud_poisson( |
|
|
pcd, depth=10, n_threads=1 |
|
|
)[0] |
|
|
|
|
|
|
|
|
print("Transferring colors to mesh...") |
|
|
pcd_tree = o3d.geometry.KDTreeFlann(pcd) |
|
|
mesh_colors = [] |
|
|
for vertex in mesh.vertices: |
|
|
[_, idx, _] = pcd_tree.search_knn_vector_3d(vertex, 1) |
|
|
mesh_colors.append(pcd.colors[idx[0]]) |
|
|
mesh.vertex_colors = o3d.utility.Vector3dVector(np.array(mesh_colors)) |
|
|
|
|
|
|
|
|
rotation = mesh.get_rotation_matrix_from_xyz((np.pi, 0, 0)) |
|
|
mesh.rotate(rotation, center=(0, 0, 0)) |
|
|
mesh_time = time.time() - mesh_start |
|
|
print(f"Mesh created in {mesh_time:.2f}s") |
|
|
|
|
|
|
|
|
print("Computing metrics...") |
|
|
mesh.compute_vertex_normals() |
|
|
|
|
|
metrics = { |
|
|
'model_used': model_choice, |
|
|
'num_images': num_images, |
|
|
'depth_estimation_time': f"{total_depth_time:.2f}s", |
|
|
'mesh_reconstruction_time': f"{mesh_time:.2f}s", |
|
|
'total_time': f"{total_depth_time + mesh_time:.2f}s", |
|
|
'initial_points': initial_points, |
|
|
'outliers_removed': outliers_removed, |
|
|
'final_points': len(pcd.points), |
|
|
'vertices': len(mesh.vertices), |
|
|
'triangles': len(mesh.triangles), |
|
|
'is_edge_manifold': mesh.is_edge_manifold(), |
|
|
'is_vertex_manifold': mesh.is_vertex_manifold(), |
|
|
'is_watertight': mesh.is_watertight(), |
|
|
} |
|
|
|
|
|
|
|
|
surface_area_computed = False |
|
|
try: |
|
|
surface_area = mesh.get_surface_area() |
|
|
if surface_area > 0: |
|
|
metrics['surface_area'] = float(surface_area) |
|
|
surface_area_computed = True |
|
|
except: |
|
|
pass |
|
|
|
|
|
if not surface_area_computed: |
|
|
try: |
|
|
vertices = np.asarray(mesh.vertices) |
|
|
triangles = np.asarray(mesh.triangles) |
|
|
v0 = vertices[triangles[:, 0]] |
|
|
v1 = vertices[triangles[:, 1]] |
|
|
v2 = vertices[triangles[:, 2]] |
|
|
cross = np.cross(v1 - v0, v2 - v0) |
|
|
areas = 0.5 * np.linalg.norm(cross, axis=1) |
|
|
total_area = np.sum(areas) |
|
|
metrics['surface_area'] = float(total_area) |
|
|
surface_area_computed = True |
|
|
except: |
|
|
metrics['surface_area'] = "Unable to compute" |
|
|
|
|
|
|
|
|
try: |
|
|
if mesh.is_watertight(): |
|
|
volume = mesh.get_volume() |
|
|
metrics['volume'] = float(volume) |
|
|
else: |
|
|
metrics['volume'] = None |
|
|
except: |
|
|
metrics['volume'] = None |
|
|
|
|
|
print("Metrics computed!") |
|
|
|
|
|
|
|
|
print("Creating 3D visualization...") |
|
|
points = np.asarray(pcd.points) |
|
|
colors = np.asarray(pcd.colors) |
|
|
|
|
|
if visualization_type == "point_cloud": |
|
|
scatter = go.Scatter3d( |
|
|
x=points[:, 0], y=points[:, 1], z=points[:, 2], |
|
|
mode='markers', |
|
|
marker=dict( |
|
|
size=2, |
|
|
color=['rgb({},{},{})'.format(int(r*255), int(g*255), int(b*255)) |
|
|
for r, g, b in colors], |
|
|
), |
|
|
name='Point Cloud' |
|
|
) |
|
|
|
|
|
layout = go.Layout( |
|
|
scene=dict( |
|
|
xaxis=dict(visible=False), |
|
|
yaxis=dict(visible=False), |
|
|
zaxis=dict(visible=False), |
|
|
aspectmode='data', |
|
|
camera=dict(eye=dict(x=1.5, y=1.5, z=1.5)) |
|
|
), |
|
|
margin=dict(l=0, r=0, t=30, b=0), |
|
|
height=700, |
|
|
title="Point Cloud" |
|
|
) |
|
|
|
|
|
plotly_fig = go.Figure(data=[scatter], layout=layout) |
|
|
|
|
|
elif visualization_type == "mesh": |
|
|
vertices = np.asarray(mesh.vertices) |
|
|
triangles = np.asarray(mesh.triangles) |
|
|
|
|
|
if mesh.has_vertex_colors(): |
|
|
vertex_colors = np.asarray(mesh.vertex_colors) |
|
|
colors_rgb = ['rgb({},{},{})'.format(int(r*255), int(g*255), int(b*255)) |
|
|
for r, g, b in vertex_colors] |
|
|
|
|
|
mesh_trace = go.Mesh3d( |
|
|
x=vertices[:, 0], y=vertices[:, 1], z=vertices[:, 2], |
|
|
i=triangles[:, 0], j=triangles[:, 1], k=triangles[:, 2], |
|
|
vertexcolor=colors_rgb, |
|
|
opacity=0.95, |
|
|
name='Mesh', |
|
|
lighting=dict(ambient=0.5, diffuse=0.8, specular=0.2), |
|
|
lightposition=dict(x=100, y=100, z=100) |
|
|
) |
|
|
else: |
|
|
mesh_trace = go.Mesh3d( |
|
|
x=vertices[:, 0], y=vertices[:, 1], z=vertices[:, 2], |
|
|
i=triangles[:, 0], j=triangles[:, 1], k=triangles[:, 2], |
|
|
color='lightblue', |
|
|
opacity=0.9, |
|
|
name='Mesh' |
|
|
) |
|
|
|
|
|
layout = go.Layout( |
|
|
scene=dict( |
|
|
xaxis=dict(visible=False), |
|
|
yaxis=dict(visible=False), |
|
|
zaxis=dict(visible=False), |
|
|
aspectmode='data', |
|
|
camera=dict(eye=dict(x=1.5, y=1.5, z=1.5)) |
|
|
), |
|
|
margin=dict(l=0, r=0, t=30, b=0), |
|
|
height=700, |
|
|
title="3D Mesh" |
|
|
) |
|
|
|
|
|
plotly_fig = go.Figure(data=[mesh_trace], layout=layout) |
|
|
|
|
|
else: |
|
|
from plotly.subplots import make_subplots |
|
|
|
|
|
vertices = np.asarray(mesh.vertices) |
|
|
triangles = np.asarray(mesh.triangles) |
|
|
|
|
|
scatter = go.Scatter3d( |
|
|
x=points[:, 0], y=points[:, 1], z=points[:, 2], |
|
|
mode='markers', |
|
|
marker=dict( |
|
|
size=2, |
|
|
color=['rgb({},{},{})'.format(int(r*255), int(g*255), int(b*255)) |
|
|
for r, g, b in colors], |
|
|
), |
|
|
name='Point Cloud' |
|
|
) |
|
|
|
|
|
if mesh.has_vertex_colors(): |
|
|
vertex_colors = np.asarray(mesh.vertex_colors) |
|
|
colors_rgb = ['rgb({},{},{})'.format(int(r*255), int(g*255), int(b*255)) |
|
|
for r, g, b in vertex_colors] |
|
|
|
|
|
mesh_trace = go.Mesh3d( |
|
|
x=vertices[:, 0], y=vertices[:, 1], z=vertices[:, 2], |
|
|
i=triangles[:, 0], j=triangles[:, 1], k=triangles[:, 2], |
|
|
vertexcolor=colors_rgb, |
|
|
opacity=0.95, |
|
|
name='Mesh', |
|
|
lighting=dict(ambient=0.5, diffuse=0.8, specular=0.2), |
|
|
lightposition=dict(x=100, y=100, z=100) |
|
|
) |
|
|
else: |
|
|
mesh_trace = go.Mesh3d( |
|
|
x=vertices[:, 0], y=vertices[:, 1], z=vertices[:, 2], |
|
|
i=triangles[:, 0], j=triangles[:, 1], k=triangles[:, 2], |
|
|
color='lightblue', |
|
|
opacity=0.9, |
|
|
name='Mesh' |
|
|
) |
|
|
|
|
|
plotly_fig = make_subplots( |
|
|
rows=1, cols=2, |
|
|
specs=[[{'type': 'scatter3d'}, {'type': 'scatter3d'}]], |
|
|
subplot_titles=('Point Cloud', '3D Mesh'), |
|
|
horizontal_spacing=0.05 |
|
|
) |
|
|
|
|
|
plotly_fig.add_trace(scatter, row=1, col=1) |
|
|
plotly_fig.add_trace(mesh_trace, row=1, col=2) |
|
|
|
|
|
plotly_fig.update_layout( |
|
|
scene=dict( |
|
|
xaxis=dict(visible=False), |
|
|
yaxis=dict(visible=False), |
|
|
zaxis=dict(visible=False), |
|
|
aspectmode='data', |
|
|
camera=dict(eye=dict(x=1.5, y=1.5, z=1.5)) |
|
|
), |
|
|
scene2=dict( |
|
|
xaxis=dict(visible=False), |
|
|
yaxis=dict(visible=False), |
|
|
zaxis=dict(visible=False), |
|
|
aspectmode='data', |
|
|
camera=dict(eye=dict(x=1.5, y=1.5, z=1.5)) |
|
|
), |
|
|
height=600, |
|
|
showlegend=False, |
|
|
margin=dict(l=0, r=0, t=50, b=0) |
|
|
) |
|
|
|
|
|
print("3D visualization created!") |
|
|
|
|
|
|
|
|
print("Exporting files...") |
|
|
temp_dir = tempfile.mkdtemp() |
|
|
|
|
|
|
|
|
pcd_path = Path(temp_dir) / "point_cloud.ply" |
|
|
o3d.io.write_point_cloud(str(pcd_path), pcd) |
|
|
|
|
|
|
|
|
mesh_path = Path(temp_dir) / "mesh.ply" |
|
|
o3d.io.write_triangle_mesh(str(mesh_path), mesh) |
|
|
|
|
|
|
|
|
mesh_obj_path = Path(temp_dir) / "mesh.obj" |
|
|
o3d.io.write_triangle_mesh(str(mesh_obj_path), mesh) |
|
|
|
|
|
|
|
|
mesh_stl_path = Path(temp_dir) / "mesh.stl" |
|
|
o3d.io.write_triangle_mesh(str(mesh_stl_path), mesh) |
|
|
|
|
|
|
|
|
metrics_path = Path(temp_dir) / "metrics.json" |
|
|
with open(metrics_path, 'w') as f: |
|
|
json.dump(metrics, f, indent=2, default=str) |
|
|
|
|
|
|
|
|
zip_path = Path(temp_dir) / "reconstruction_complete.zip" |
|
|
with zipfile.ZipFile(zip_path, 'w', zipfile.ZIP_DEFLATED) as zipf: |
|
|
zipf.write(pcd_path, pcd_path.name) |
|
|
zipf.write(mesh_path, mesh_path.name) |
|
|
zipf.write(mesh_obj_path, mesh_obj_path.name) |
|
|
zipf.write(mesh_stl_path, mesh_stl_path.name) |
|
|
zipf.write(metrics_path, metrics_path.name) |
|
|
|
|
|
print("Files exported!") |
|
|
|
|
|
|
|
|
assessment = _generate_quality_assessment(metrics) |
|
|
|
|
|
|
|
|
avg_depth_stats = { |
|
|
'min': np.mean([d['min'] for d in depth_stats_list]), |
|
|
'max': np.mean([d['max'] for d in depth_stats_list]), |
|
|
'mean': np.mean([d['mean'] for d in depth_stats_list]), |
|
|
'std': np.mean([d['std'] for d in depth_stats_list]) |
|
|
} |
|
|
explainability = generate_explainability_report(metrics, avg_depth_stats) |
|
|
|
|
|
multi_image_note = "" |
|
|
if num_images > 1: |
|
|
multi_image_note = f""" |
|
|
### 📸 Multi-Image Reconstruction |
|
|
- **Number of Images**: {num_images} |
|
|
- **Combined Points**: {initial_points:,} (before cleaning) |
|
|
- **Advantage**: Better coverage and reduced occlusion compared to single image |
|
|
- **Note**: Images were combined using simple spatial offset. For production use, consider advanced registration algorithms (ICP, feature matching). |
|
|
""" |
|
|
|
|
|
report = f""" |
|
|
## Reconstruction Complete! |
|
|
|
|
|
{privacy_report} |
|
|
|
|
|
{multi_image_note} |
|
|
|
|
|
### Performance Metrics |
|
|
- **Model Used**: {metrics['model_used']} |
|
|
- **Number of Images**: {metrics['num_images']} |
|
|
- **Depth Estimation Time**: {metrics['depth_estimation_time']} |
|
|
- **Mesh Reconstruction Time**: {metrics['mesh_reconstruction_time']} |
|
|
- **Total Processing Time**: {metrics['total_time']} |
|
|
|
|
|
### Point Cloud Statistics |
|
|
- **Initial Points**: {metrics['initial_points']:,} |
|
|
- **Outliers Removed**: {metrics['outliers_removed']:,} ({(metrics['outliers_removed']/metrics['initial_points']*100):.1f}%) |
|
|
- **Final Points**: {metrics['final_points']:,} |
|
|
|
|
|
### Mesh Quality |
|
|
- **Vertices**: {metrics['vertices']:,} |
|
|
- **Triangles**: {metrics['triangles']:,} |
|
|
- **Edge Manifold**: {'✓ Good topology' if metrics['is_edge_manifold'] else '✗ Has non-manifold edges'} |
|
|
- **Vertex Manifold**: {'✓ Clean vertices' if metrics['is_vertex_manifold'] else '✗ Has non-manifold vertices'} |
|
|
- **Watertight**: {'✓ Closed surface (3D printable)' if metrics['is_watertight'] else '✗ Has boundaries (needs repair for 3D printing)'} |
|
|
- **Surface Area**: {metrics['surface_area'] if isinstance(metrics['surface_area'], str) else f"{metrics['surface_area']:.2f}"} |
|
|
- **Volume**: {f"{metrics['volume']:.2f}" if metrics.get('volume') else 'N/A (not watertight)'} |
|
|
|
|
|
### Quality Assessment |
|
|
{assessment} |
|
|
|
|
|
{explainability} |
|
|
|
|
|
### Files Exported |
|
|
- Point Cloud: PLY format |
|
|
- Mesh: PLY, OBJ, STL formats |
|
|
- Quality Metrics: JSON |
|
|
|
|
|
**Download the complete package below!** |
|
|
""" |
|
|
|
|
|
print("SUCCESS! Returning results...") |
|
|
return combined_depth_viz, plotly_fig, str(zip_path), report, json.dumps(metrics, indent=2, default=str), privacy_report |
|
|
|
|
|
except Exception as e: |
|
|
import traceback |
|
|
error_msg = f"Error during reconstruction:\n{str(e)}\n\nTraceback:\n{traceback.format_exc()}" |
|
|
print(error_msg) |
|
|
return None, None, None, error_msg, None, None |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
with gr.Blocks(title="Advanced 3D Reconstruction", theme=gr.themes.Soft()) as demo: |
|
|
|
|
|
gr.Markdown(""" |
|
|
# 🗿️ 3D Urban Reconstruction from Single or Multiple Images |
|
|
|
|
|
Transform 2D photographs into 3D spatial models with **Responsible AI** practices |
|
|
|
|
|
Upload one or multiple photographs to generate interactive 3D models with exportable spatial data. |
|
|
|
|
|
**New Features:** |
|
|
- ✨ **Multi-image support** for better coverage and accuracy |
|
|
- 🔒 **Privacy protection** with local processing |
|
|
- 🔍 **AI explainability** to understand reconstruction decisions |
|
|
""") |
|
|
|
|
|
with gr.Tabs(): |
|
|
|
|
|
|
|
|
with gr.Tab("🔧 Reconstruction"): |
|
|
with gr.Row(): |
|
|
with gr.Column(scale=1): |
|
|
gr.Markdown(""" |
|
|
### Upload Images |
|
|
Upload **1-5 images** of the same object/scene from different angles for best results. |
|
|
- Single image: Fast processing |
|
|
- Multiple images: Better coverage, improved quality |
|
|
""") |
|
|
|
|
|
input_images = gr.File( |
|
|
file_count="multiple", |
|
|
file_types=["image"], |
|
|
label="Upload Image(s) - Supports: JPG, PNG, BMP", |
|
|
type="filepath" |
|
|
) |
|
|
|
|
|
gr.Markdown("### Model Settings") |
|
|
model_choice = gr.Radio( |
|
|
choices=["GLPN (Recommended)", "DPT (High Quality)"], |
|
|
value="GLPN (Recommended)", |
|
|
label="Depth Estimation Model" |
|
|
) |
|
|
|
|
|
visualization_type = gr.Radio( |
|
|
choices=["mesh", "point_cloud", "both"], |
|
|
value="mesh", |
|
|
label="3D Visualization Type" |
|
|
) |
|
|
|
|
|
gr.Markdown("### Responsible AI Settings") |
|
|
privacy_check = gr.Checkbox( |
|
|
value=True, |
|
|
label="Enable privacy checks (recommended)", |
|
|
info="Warns if images might contain sensitive information" |
|
|
) |
|
|
|
|
|
reconstruct_btn = gr.Button("🚀 Start Reconstruction", variant="primary", size="lg") |
|
|
|
|
|
with gr.Column(scale=2): |
|
|
depth_output = gr.Image(label="Depth Map Visualization") |
|
|
viewer_3d = gr.Plot(label="Interactive 3D Viewer (Rotate, Zoom, Pan)") |
|
|
|
|
|
with gr.Row(): |
|
|
with gr.Column(): |
|
|
metrics_output = gr.Markdown(label="Reconstruction Report") |
|
|
with gr.Column(): |
|
|
json_output = gr.Textbox(label="Raw Metrics (JSON)", lines=10) |
|
|
|
|
|
with gr.Row(): |
|
|
download_output = gr.File(label="📦 Download Complete Package (ZIP)") |
|
|
|
|
|
|
|
|
def process_uploaded_files(files, model, viz_type, privacy): |
|
|
if files is None or len(files) == 0: |
|
|
return None, None, None, "Please upload at least one image.", None, None |
|
|
|
|
|
|
|
|
images = [] |
|
|
for file_path in files: |
|
|
img = Image.open(file_path) |
|
|
images.append(img) |
|
|
|
|
|
return process_image(images, model, viz_type, privacy) |
|
|
|
|
|
reconstruct_btn.click( |
|
|
fn=process_uploaded_files, |
|
|
inputs=[input_images, model_choice, visualization_type, privacy_check], |
|
|
outputs=[depth_output, viewer_3d, download_output, metrics_output, json_output, gr.Textbox(visible=False)] |
|
|
) |
|
|
|
|
|
|
|
|
with gr.Tab("🛡️ Responsible AI"): |
|
|
gr.Markdown(""" |
|
|
## Responsible AI Framework |
|
|
|
|
|
This application implements responsible AI principles to ensure ethical and safe use of AI technology. |
|
|
|
|
|
### 1. Privacy Protection 🔒 |
|
|
|
|
|
**What we do:** |
|
|
- **Local Processing Only**: All computation happens in your browser/server - no data sent to external APIs |
|
|
- **No Data Retention**: Images are processed in memory and deleted immediately after reconstruction |
|
|
- **No Tracking**: We don't collect, store, or analyze user data |
|
|
- **Privacy Warnings**: System alerts you if uploaded images might contain sensitive information |
|
|
|
|
|
**User Responsibilities:** |
|
|
- Avoid uploading images with identifiable individuals without consent |
|
|
- Don't use for surveillance or unauthorized monitoring |
|
|
- Be mindful of private/sensitive locations |
|
|
- Follow local privacy laws and regulations |
|
|
|
|
|
**Technical Safeguards:** |
|
|
- No facial recognition algorithms |
|
|
- No identity tracking features |
|
|
- No cloud storage or external data transmission |
|
|
- User maintains full data ownership |
|
|
|
|
|
--- |
|
|
|
|
|
### 2. Explainability & Transparency 🔍 |
|
|
|
|
|
**Understanding AI Decisions:** |
|
|
|
|
|
The system provides multiple layers of explainability: |
|
|
|
|
|
**Depth Map Visualization:** |
|
|
- Shows exactly how AI interprets scene depth |
|
|
- Color coding reveals AI's confidence (yellow/red = far, purple/blue = near) |
|
|
- Allows manual verification of depth estimates |
|
|
|
|
|
**Quality Metrics:** |
|
|
- **Outlier Percentage**: Shows AI uncertainty (< 5% = high confidence) |
|
|
- **Manifold Properties**: Indicates reconstruction reliability |
|
|
- **Watertight Status**: Reveals completeness of 3D model |
|
|
|
|
|
**Explainability Report:** |
|
|
- Plain-language explanation of AI decisions |
|
|
- Confidence levels for reconstruction quality |
|
|
- Warnings about potential issues |
|
|
|
|
|
**Model Transparency:** |
|
|
- Open-source models (GLPN, DPT) with published papers |
|
|
- Documented training data (NYU Depth V2, etc.) |
|
|
- Known limitations explicitly stated |
|
|
|
|
|
--- |
|
|
|
|
|
### 3. Fairness & Bias Awareness ⚖️ |
|
|
|
|
|
**Known Biases:** |
|
|
|
|
|
Our AI models have inherent biases based on their training data: |
|
|
|
|
|
**Geographic Bias:** |
|
|
- Trained primarily on urban/indoor scenes from developed countries |
|
|
- May underperform on architectural styles from underrepresented regions |
|
|
- Less accurate for non-Western building structures |
|
|
|
|
|
**Scene Type Bias:** |
|
|
- Optimized for indoor environments |
|
|
- Better performance on structured scenes (rooms, buildings) |
|
|
- May struggle with natural landscapes, outdoor scenes |
|
|
|
|
|
**Lighting Bias:** |
|
|
- Trained on well-lit images |
|
|
- Reduced accuracy in low-light conditions |
|
|
- May fail on images with extreme shadows |
|
|
|
|
|
**Mitigation Strategies:** |
|
|
- Quality metrics help identify poor reconstructions |
|
|
- Multiple model options (GLPN vs DPT) for different scenarios |
|
|
- User can validate results visually |
|
|
- Clear documentation of limitations |
|
|
|
|
|
--- |
|
|
|
|
|
### 4. Intended Use & Limitations ⚠️ |
|
|
|
|
|
**Appropriate Uses:** |
|
|
- ✅ Educational demonstrations and learning |
|
|
- ✅ Research and academic projects |
|
|
- ✅ Preliminary architectural visualization |
|
|
- ✅ Art and creative projects |
|
|
- ✅ Rapid prototyping and concept exploration |
|
|
|
|
|
**Inappropriate Uses:** |
|
|
- ❌ Safety-critical applications (structural engineering, medical) |
|
|
- ❌ Surveillance or unauthorized monitoring |
|
|
- ❌ Precise measurements without ground truth validation |
|
|
- ❌ Legal evidence or forensic analysis |
|
|
- ❌ Automated decision-making affecting individuals |
|
|
|
|
|
**Key Limitations:** |
|
|
|
|
|
1. **Scale Ambiguity**: Outputs are relative, not absolute measurements |
|
|
2. **Single Viewpoint**: Cannot see occluded/hidden areas (reduced with multi-image) |
|
|
3. **No Georeferencing**: Local coordinates, not GPS/global positioning |
|
|
4. **Monocular Limitations**: Less accurate than stereo or LiDAR systems |
|
|
5. **Training Data Constraints**: Best for similar scenes to training data |
|
|
|
|
|
--- |
|
|
|
|
|
### 5. Data Governance & Transparency 📊 |
|
|
|
|
|
**Model Provenance:** |
|
|
|
|
|
All AI models used in this application are fully transparent: |
|
|
|
|
|
| Model | Source | Training Data | License | Paper | |
|
|
|-------|--------|---------------|---------|-------| |
|
|
| GLPN | Hugging Face | NYU Depth V2 | Apache 2.0 | Kim et al., CVPR 2022 | |
|
|
| DPT | Intel/Hugging Face | Mixed datasets | Apache 2.0 | Ranftl et al., ICCV 2021 | |
|
|
|
|
|
**Training Data:** |
|
|
- NYU Depth V2: Indoor scenes from New York apartments |
|
|
- MIX 6: Mixed indoor/outdoor scenes |
|
|
- Primarily North American and European locations |
|
|
- Limited representation of other regions |
|
|
|
|
|
**No Proprietary Black Boxes:** |
|
|
- All models are open-source |
|
|
- Architecture and weights publicly available |
|
|
- No hidden proprietary algorithms |
|
|
- Users can audit model behavior |
|
|
|
|
|
--- |
|
|
|
|
|
### 6. Environmental Considerations 🌍 |
|
|
|
|
|
**Computational Efficiency:** |
|
|
- Optimized for CPU inference (no GPU required) |
|
|
- GLPN model: Fast processing (~0.3-2.5s per image) |
|
|
- Minimal energy consumption compared to cloud-based solutions |
|
|
- Local processing reduces data transfer energy costs |
|
|
|
|
|
--- |
|
|
|
|
|
### 7. Ethical Guidelines for Users 📖 |
|
|
|
|
|
**Before Using This Tool:** |
|
|
|
|
|
1. **Consent**: Ensure you have rights to process uploaded images |
|
|
2. **Privacy**: Verify images don't contain identifiable individuals without consent |
|
|
3. **Purpose**: Confirm your use case aligns with intended applications |
|
|
4. **Validation**: Don't rely solely on AI outputs for critical decisions |
|
|
5. **Attribution**: Credit the open-source models and datasets used |
|
|
|
|
|
**Reporting Issues:** |
|
|
|
|
|
If you discover: |
|
|
- Unexpected biases or failure modes |
|
|
- Privacy concerns or vulnerabilities |
|
|
- Misuse potential or ethical issues |
|
|
|
|
|
Please report to the development team for continuous improvement. |
|
|
|
|
|
--- |
|
|
|
|
|
### 8. Continuous Improvement 🔄 |
|
|
|
|
|
**How We're Working to Improve:** |
|
|
|
|
|
- Expanding training data diversity |
|
|
- Developing bias detection metrics |
|
|
- Improving explainability features |
|
|
- Adding more privacy safeguards |
|
|
- Documenting edge cases and limitations |
|
|
|
|
|
**User Feedback:** |
|
|
Your feedback helps us improve responsible AI practices. Please share: |
|
|
- Unexpected results or biases observed |
|
|
- Suggestions for better explainability |
|
|
- Privacy concerns or recommendations |
|
|
- Use cases we haven't considered |
|
|
|
|
|
--- |
|
|
|
|
|
## References |
|
|
|
|
|
- [Responsible AI Practices](https://ai.google/responsibilities/responsible-ai-practices/) |
|
|
- [Microsoft Responsible AI Principles](https://www.microsoft.com/en-us/ai/responsible-ai) |
|
|
- [Partnership on AI](https://partnershiponai.org/) |
|
|
- [Montreal Declaration for Responsible AI](https://www.montrealdeclaration-responsibleai.com/) |
|
|
""") |
|
|
|
|
|
|
|
|
with gr.Tab("📚 Theory & Background"): |
|
|
gr.Markdown(THEORY_TEXT) |
|
|
|
|
|
gr.Markdown(""" |
|
|
## Reconstruction Pipeline Details |
|
|
|
|
|
This application uses an **11-step automated pipeline**: |
|
|
|
|
|
1. **Image Preprocessing**: Resize to model requirements (divisible by 32) |
|
|
2. **Depth Estimation**: Neural network inference (GLPN or DPT) for each image |
|
|
3. **Depth Visualization**: Create comparison images |
|
|
4. **Point Cloud Generation**: Back-project using pinhole camera model |
|
|
5. **Multi-View Fusion**: Merge point clouds from multiple images (if applicable) |
|
|
6. **Outlier Removal**: Statistical filtering (20 neighbors, 2.0 std ratio) |
|
|
7. **Normal Estimation**: Local plane fitting for surface orientation |
|
|
8. **Mesh Reconstruction**: Poisson surface reconstruction (depth=10) |
|
|
9. **Quality Metrics**: Compute manifold properties and geometric measures |
|
|
10. **3D Visualization**: Create interactive Plotly figure |
|
|
11. **File Export**: Generate PLY, OBJ, STL formats |
|
|
|
|
|
### Multi-Image Processing |
|
|
|
|
|
When multiple images are provided: |
|
|
- Each image is processed independently for depth estimation |
|
|
- Point clouds are generated from each image |
|
|
- Simple spatial offset applied to prevent overlap |
|
|
- Combined point cloud undergoes unified cleaning and meshing |
|
|
|
|
|
**Note**: Current implementation uses basic merging. Production systems would use: |
|
|
- Feature matching (SIFT, ORB) for correspondence |
|
|
- Structure-from-Motion (SfM) for camera pose estimation |
|
|
- Iterative Closest Point (ICP) for fine alignment |
|
|
- Bundle adjustment for global optimization |
|
|
|
|
|
### Default Parameters Used |
|
|
|
|
|
- **Poisson Depth**: 10 (balanced detail vs speed) |
|
|
- **Outlier Neighbors**: 20 points |
|
|
- **Outlier Std Ratio**: 2.0 |
|
|
- **Focal Length**: 500 (pixels) |
|
|
- **Normal Radius**: 0.1 (search radius) |
|
|
|
|
|
These parameters are optimized for general use cases and provide good results for most indoor scenes. |
|
|
|
|
|
## Key References |
|
|
|
|
|
1. **Kim, D., et al. (2022)**. "Global-Local Path Networks for Monocular Depth Estimation with Vertical CutDepth." *CVPR 2022* |
|
|
2. **Ranftl, R., et al. (2021)**. "Vision Transformers for Dense Prediction." *ICCV 2021* |
|
|
3. **Kazhdan, M., et al. (2006)**. "Poisson Surface Reconstruction." *Eurographics Symposium on Geometry Processing* |
|
|
|
|
|
## Model Comparison |
|
|
|
|
|
| Feature | GLPN (Recommended) | DPT (High Quality) | |
|
|
|---------|-------------------|-------------------| |
|
|
| **Speed** | Fast (~0.3-2.5s) | Slower (~0.8-6.5s) | |
|
|
| **Quality** | Good | Excellent | |
|
|
| **Memory** | Low (~2GB) | High (~5GB) | |
|
|
| **Best For** | Indoor scenes, Real-time | Complex scenes, Highest quality | |
|
|
| **Training** | NYU Depth V2 | Multiple datasets | |
|
|
|
|
|
### When to Use Each Model: |
|
|
|
|
|
**Choose GLPN if:** |
|
|
- Processing indoor scenes (rooms, furniture) |
|
|
- Speed is important |
|
|
- Running on limited hardware |
|
|
- Need real-time performance |
|
|
|
|
|
**Choose DPT if:** |
|
|
- Need highest quality results |
|
|
- Processing complex/outdoor scenes |
|
|
- Speed is not critical |
|
|
- Have sufficient memory/GPU |
|
|
""") |
|
|
|
|
|
|
|
|
with gr.Tab("📖 Usage Guide"): |
|
|
gr.Markdown(""" |
|
|
## How to Use This Application |
|
|
|
|
|
### Step 1: Upload Image(s) |
|
|
|
|
|
**Single Image Mode:** |
|
|
- Click on the upload area and select one image |
|
|
- Best for: Quick reconstruction, simple objects |
|
|
- Processing time: Fast |
|
|
|
|
|
**Multiple Image Mode (NEW):** |
|
|
- Select 2-5 images of the same object from different angles |
|
|
- Best for: Better coverage, complex objects, reduced occlusions |
|
|
- Processing time: Longer (scales with number of images) |
|
|
- **Tip**: Take photos from 45-90 degree intervals around the object |
|
|
|
|
|
**Image Requirements:** |
|
|
- **Format**: JPG, PNG, or BMP |
|
|
- **Resolution**: 512-1024px recommended |
|
|
- **Lighting**: Well-lit, minimal shadows |
|
|
- **Content**: Objects with texture, clear depth cues |
|
|
|
|
|
**Multi-Image Tips:** |
|
|
- Keep camera distance roughly consistent |
|
|
- Overlap between views improves reconstruction |
|
|
- Avoid motion blur between shots |
|
|
- Same lighting conditions across all images |
|
|
|
|
|
--- |
|
|
|
|
|
### Step 2: Configure Settings |
|
|
|
|
|
**Model Selection:** |
|
|
- **GLPN (Recommended)**: Fast, good for indoor scenes |
|
|
- **DPT (High Quality)**: Slower but higher quality |
|
|
|
|
|
**Visualization Type:** |
|
|
- **Mesh**: Solid 3D surface (recommended) |
|
|
- **Point Cloud**: Individual 3D points |
|
|
- **Both**: Side-by-side comparison |
|
|
|
|
|
**Privacy Settings:** |
|
|
- Keep "Enable privacy checks" ON (recommended) |
|
|
- System will warn about potential privacy concerns |
|
|
|
|
|
--- |
|
|
|
|
|
### Step 3: Start Reconstruction |
|
|
- Click "🚀 Start Reconstruction" |
|
|
- Wait for processing (10-90 seconds depending on number of images) |
|
|
- Results appear automatically |
|
|
|
|
|
--- |
|
|
|
|
|
### Step 4: Explore Results |
|
|
|
|
|
**Depth Map(s):** |
|
|
- Shows original image(s) next to depth estimates |
|
|
- Color coding: Yellow/Red = Far, Purple/Blue = Near |
|
|
- Multiple images show grid of all depth maps |
|
|
|
|
|
**Interactive 3D Viewer:** |
|
|
- **Rotate**: Click and drag |
|
|
- **Zoom**: Scroll wheel |
|
|
- **Pan**: Right-click and drag |
|
|
- **Reset**: Double-click |
|
|
|
|
|
**Reconstruction Report:** |
|
|
- Performance metrics |
|
|
- Quality assessment |
|
|
- AI explainability (confidence levels) |
|
|
- Privacy warnings (if any) |
|
|
|
|
|
--- |
|
|
|
|
|
### Step 5: Download Results |
|
|
|
|
|
ZIP package contains: |
|
|
- `point_cloud.ply` - 3D points with colors |
|
|
- `mesh.ply` - Full mesh with metadata |
|
|
- `mesh.obj` - Standard format (most compatible) |
|
|
- `mesh.stl` - For 3D printing |
|
|
- `metrics.json` - All quality metrics |
|
|
|
|
|
--- |
|
|
|
|
|
## Viewing Downloaded Files |
|
|
|
|
|
**Free Software:** |
|
|
- **MeshLab**: Best for beginners - https://www.meshlab.net/ |
|
|
- **Blender**: Advanced 3D modeling - https://www.blender.org/ |
|
|
- **CloudCompare**: Point cloud analysis - https://www.cloudcompare.org/ |
|
|
|
|
|
**Online Viewers:** |
|
|
- https://3dviewer.net/ |
|
|
- https://www.creators3d.com/online-viewer |
|
|
|
|
|
--- |
|
|
|
|
|
## Tips for Best Results |
|
|
|
|
|
### Single Image Mode: |
|
|
- Use well-lit images |
|
|
- Include depth cues (corners, edges) |
|
|
- Avoid reflective surfaces |
|
|
- Indoor scenes work best |
|
|
|
|
|
### Multiple Image Mode: |
|
|
- Take 3-5 photos from different angles |
|
|
- Maintain 45-90 degree spacing |
|
|
- Keep consistent distance from object |
|
|
- Ensure 30-50% overlap between views |
|
|
- Use same lighting for all shots |
|
|
|
|
|
### What to Avoid: |
|
|
- Motion blur |
|
|
- Extreme close-ups |
|
|
- Transparent objects |
|
|
- Mirrors or glass |
|
|
- Uniform textures |
|
|
- Very dark images |
|
|
|
|
|
--- |
|
|
|
|
|
## Troubleshooting |
|
|
|
|
|
**"Please upload at least one image"** |
|
|
- Ensure files are selected before clicking reconstruct |
|
|
- Check file format (JPG, PNG, BMP only) |
|
|
|
|
|
**Mesh has holes/artifacts** |
|
|
- Normal for single-view reconstruction |
|
|
- Try multiple images for better coverage |
|
|
- Use MeshLab's "Close Holes" tool if needed |
|
|
|
|
|
**Processing is slow** |
|
|
- Use GLPN model instead of DPT |
|
|
- Reduce number of images |
|
|
- Use smaller image resolution |
|
|
|
|
|
**"Not watertight" warning** |
|
|
- Common for complex scenes |
|
|
- Still usable for visualization |
|
|
- For 3D printing: use mesh repair tools |
|
|
|
|
|
**Privacy warnings** |
|
|
- Review uploaded images |
|
|
- Remove identifiable information if needed |
|
|
- Disable privacy checks if false positive |
|
|
""") |
|
|
|
|
|
|
|
|
with gr.Tab("📄 Citation & Credits"): |
|
|
gr.Markdown(""" |
|
|
## Academic Citation |
|
|
|
|
|
If you use this tool in your research or projects, please cite the underlying models: |
|
|
|
|
|
### For GLPN Model: |
|
|
```bibtex |
|
|
@inproceedings{kim2022global, |
|
|
title={Global-Local Path Networks for Monocular Depth Estimation with Vertical CutDepth}, |
|
|
author={Kim, Doyeon and Ga, Woonghyun and Ahn, Pyungwhan and Joo, Donggyu and Chun, Sehwan and Kim, Junmo}, |
|
|
booktitle={CVPR}, |
|
|
year={2022} |
|
|
} |
|
|
``` |
|
|
|
|
|
### For DPT Model: |
|
|
```bibtex |
|
|
@inproceedings{ranftl2021vision, |
|
|
title={Vision Transformers for Dense Prediction}, |
|
|
author={Ranftl, Ren{\'e} and Bochkovskiy, Alexey and Koltun, Vladlen}, |
|
|
booktitle={ICCV}, |
|
|
year={2021} |
|
|
} |
|
|
``` |
|
|
|
|
|
## Open Source Components |
|
|
|
|
|
This application is built with: |
|
|
- **Transformers** (Hugging Face): Model inference |
|
|
- **Open3D**: Point cloud and mesh processing |
|
|
- **PyTorch**: Deep learning framework |
|
|
- **Plotly**: Interactive 3D visualization |
|
|
- **Gradio**: Web interface |
|
|
- **NumPy & SciPy**: Scientific computing |
|
|
|
|
|
## Acknowledgments |
|
|
|
|
|
- NYU Depth V2 dataset creators |
|
|
- Open3D development team |
|
|
- Hugging Face community |
|
|
- Academic researchers advancing monocular depth estimation |
|
|
|
|
|
## License & Terms |
|
|
|
|
|
- Models: Apache 2.0 License |
|
|
- This application: Educational and research use |
|
|
- Commercial use: Verify model licenses |
|
|
- No warranty provided for accuracy or fitness for purpose |
|
|
|
|
|
## Contact & Feedback |
|
|
|
|
|
For questions, bug reports, or suggestions regarding responsible AI implementation, |
|
|
please contact the development team. |
|
|
""") |
|
|
|
|
|
|
|
|
gr.Markdown(""" |
|
|
--- |
|
|
**🔒 Privacy Notice**: All processing happens locally. No data is transmitted to external servers. |
|
|
|
|
|
**⚠️ Disclaimer**: This tool is for educational and research purposes. Not suitable for safety-critical applications or precise measurements. |
|
|
|
|
|
**📊 Responsible AI**: Built with privacy protection, explainability, and fairness considerations. |
|
|
""") |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
if __name__ == "__main__": |
|
|
demo.launch(share=True) |
