Files changed (1) hide show
  1. README.md +38 -19
README.md CHANGED
@@ -11,11 +11,12 @@ datasets:
11
 
12
  ### Description:
13
 
14
- ``mindmap`` is a 3D diffusion policy that generates robot trajectories based on a semantic 3D reconstruction of the environment,
15
- enabling robots with spatial memory.
16
 
17
  Trained models are available on Hugging Face: [PhysicalAI-Robotics-mindmap-Checkpoints](https://huggingface.co/nvidia/PhysicalAI-Robotics-mindmap-Checkpoints)
18
 
 
 
19
  ### License/Terms of Use
20
 
21
  - Model: [NVIDIA Open Model License Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/)
@@ -33,6 +34,11 @@ The trained ``mindmap`` policies allow for quick evaluation of the ``mindmap`` c
33
  - Developers: Integrate and customize AI for various robotic applications.
34
  - Startups & Companies: Accelerate robotics development and reduce training costs.
35
 
 
 
 
 
 
36
 
37
  ## References(s):
38
 
@@ -90,6 +96,9 @@ The trained ``mindmap`` policies allow for quick evaluation of the ``mindmap`` c
90
  - Gripper: `[PREDICTION_HORIZON, NUM_GRIPPERS, 8]` - consisting of end-effector translation, rotation (quaternion, wxyz) and closedness
91
  - Head Yaw: `[PREDICTION_HORIZON, 1]` - only for humanoid embodiments
92
 
 
 
 
93
 
94
  ## Software Integration:
95
  **Runtime Engine(s):** PyTorch
@@ -108,6 +117,10 @@ The trained ``mindmap`` policies allow for quick evaluation of the ``mindmap`` c
108
  **Preferred/Supported Operating System(s):**
109
  * Linux
110
 
 
 
 
 
111
  ## Model Version(s):
112
 
113
  This is the initial version of the model, version 1.0.0
@@ -120,7 +133,24 @@ Datasets:
120
  - drill_in_box_checkpoint: [GR1 Drill in Box Dataset](https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-mindmap-GR1-Drill-in-Box)
121
  - stick_in_bin_checkpoint: [GR1 Stick in Bin Dataset](https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-mindmap-GR1-Stick-in-Bin)
122
 
123
- The models were trained on 100 (GR1) and 130 (Franka) demonstrations. The evaluation set consisted of 20 distinct demonstrations. Closed loop testing was performed on 100 demonstrations mutually exclusive from the training set.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
124
 
125
  # Inference:
126
 
@@ -132,9 +162,9 @@ The models were trained on 100 (GR1) and 130 (Franka) demonstrations. The evalua
132
 
133
  This model is not tested or intended for use in mission critical applications that require functional safety. The use of the model in those applications is at the user's own risk and sole responsibility, including taking the necessary steps to add needed guardrails or safety mechanisms.
134
 
135
- - Risk: This policy is only effective on the exact simulation environment it was trained on.
136
- - Mitigation: Need to retrain the model on new simulation environments.
137
- - Risk: The policy was never tested on a physical robot and likely only works in simulation
138
  - Mitigation: Expand training, testing and validation on physical robot platforms.
139
 
140
  ## Ethical Considerations:
@@ -143,19 +173,14 @@ NVIDIA believes Trustworthy AI is a shared responsibility and we have establishe
143
 
144
  For more detailed information on ethical considerations for this model, please see the Model Card++ Explainability, Bias, Safety & Security, and Privacy Subcards.
145
 
146
- Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).
147
 
148
  # Bias
149
  Field | Response
150
  :---------------------------------------------------------------------------------------------------|:---------------
151
  Participation considerations from adversely impacted groups [protected classes](https://www.senate.ca.gov/content/protected-classes) in model design and testing: | Not Applicable
152
  Bias Metric (If Measured): | Not Applicable
153
- (For GPAI Models) Which characteristic (feature) show(s) the greatest difference in performance?: | Not Applicable
154
- (For GPAI Models): Which feature(s) have have the worst performance overall? | Not Applicable
155
  Measures taken to mitigate against unwanted bias: | Not Applicable
156
- (For GPAI Models): If using internal data, description of methods implemented in data acquisition or processing, if any, to address the prevalence of identifiable biases in the training, testing, and validation data: | Not Applicable
157
- (For GPAI Models): Tools used to assess statistical imbalances and highlight patterns that may introduce bias into AI models: | Not Applicable
158
- (For GPAI Models): Tools used to assess statistical imbalances and highlight patterns that may introduce bias into AI models: | Not Applicable
159
 
160
  # Explainability
161
  Field | Response
@@ -164,10 +189,9 @@ Intended Task/Domain:
164
  Model Type: | Denoising Diffusion Probabilistic Model
165
  Intended Users: | Roboticists and researchers in academia and industry who are interested in robot manipulation research
166
  Output: | Actions consisting of end-effector poses, gripper states and head orientation.
167
- (For GPAI Models): Tools used to evaluate datasets to identify synthetic data and ensure data authenticity. | Not Applicable
168
  Describe how the model works: | ``mindmap`` is a Denoising Diffusion Probabilistic Model that samples robot trajectories conditioned on sensor observations and a 3D reconstruction of the environment.
169
  Name the adversely impacted groups this has been tested to deliver comparable outcomes regardless of: | Not Applicable
170
- Technical Limitations & Mitigation: | - The policy is only effective on the exact simulation environment it was trained on. - The policy was never tested on a physical robot and likely only works in simulation.
171
  Verified to have met prescribed NVIDIA quality standards: | Yes
172
  Performance Metrics: | Closed loop success rate on simulated robotic manipulation tasks.
173
  Potential Known Risks: | The model might be susceptible to rendering changes on the simulation tasks it was trained on.
@@ -177,9 +201,6 @@ Licensing:
177
  Field | Response
178
  :---------------------------------------------------|:----------------------------------
179
  Model Application Field(s): | Robotics
180
- Describe the life critical impact (if present). | Not Applicable
181
- (For GPAI Models): Description of methods implemented in data acquisition or processing, if any, to address other types of potentially harmful data in the training, testing, and validation data: | Not GPAI
182
- (For GPAI Models): Description of any methods implemented in data acquisition or processing, if any, to address illegal or harmful content in the training data, including, but not limited to, child sexual abuse material (CSAM) and non-consensual intimate imagery (NCII) | Not GPAI
183
  Use Case Restrictions: | Abide by [NVIDIA Open Model License Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/)
184
  Model and dataset restrictions: | The Principle of least privilege (PoLP) is applied limiting access for dataset generation and model development. Restrictions enforce dataset access during training, and dataset license constraints adhered to.
185
 
@@ -188,8 +209,6 @@ Field
188
  :----------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------
189
  Generatable or reverse engineerable personal data? | No
190
  Personal data used to create this model? | No
191
- Was consent obtained for any personal data used? | Not Applicable
192
- (For GPAI Models): A description of any methods implemented in data acquisition or processing, if any, to address the prevalence of personal data in the training data, where relevant and applicable. | Not Applicable
193
  How often is dataset reviewed? | Before Release
194
  Is there provenance for all datasets used in training? | Yes
195
  Does data labeling (annotation, metadata) comply with privacy laws? | Yes
 
11
 
12
  ### Description:
13
 
14
+ ``mindmap`` is a 3D diffusion policy that generates robot trajectories based on a semantic 3D reconstruction of the environment, enabling robots with spatial memory.
 
15
 
16
  Trained models are available on Hugging Face: [PhysicalAI-Robotics-mindmap-Checkpoints](https://huggingface.co/nvidia/PhysicalAI-Robotics-mindmap-Checkpoints)
17
 
18
+ This model is ready for commercial/non-commercial use
19
+
20
  ### License/Terms of Use
21
 
22
  - Model: [NVIDIA Open Model License Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/)
 
34
  - Developers: Integrate and customize AI for various robotic applications.
35
  - Startups & Companies: Accelerate robotics development and reduce training costs.
36
 
37
+ ### Release Date
38
+
39
+ Github 09/26/2025 via github.com/NVlabs/nvblox_mindmap
40
+
41
+ Hugging Face 09/26/2025 via huggingface.co/nvidia/PhysicalAI-Robotics-mindmap-Checkpoints
42
 
43
  ## References(s):
44
 
 
96
  - Gripper: `[PREDICTION_HORIZON, NUM_GRIPPERS, 8]` - consisting of end-effector translation, rotation (quaternion, wxyz) and closedness
97
  - Head Yaw: `[PREDICTION_HORIZON, 1]` - only for humanoid embodiments
98
 
99
+ Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems.
100
+ By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries),
101
+ the model achieves faster training and inference times compared to CPU-only solutions.
102
 
103
  ## Software Integration:
104
  **Runtime Engine(s):** PyTorch
 
117
  **Preferred/Supported Operating System(s):**
118
  * Linux
119
 
120
+ The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment.
121
+ Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks,
122
+ meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.
123
+
124
  ## Model Version(s):
125
 
126
  This is the initial version of the model, version 1.0.0
 
133
  - drill_in_box_checkpoint: [GR1 Drill in Box Dataset](https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-mindmap-GR1-Drill-in-Box)
134
  - stick_in_bin_checkpoint: [GR1 Stick in Bin Dataset](https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-mindmap-GR1-Stick-in-Bin)
135
 
136
+
137
+ **Data Modality:** Image, 3D reconstruction, robot states
138
+
139
+ **Image Training Data Size:** Less than a Million Images
140
+
141
+ **3D reconstruction, robot state Data Size:** Less than a Million Samples
142
+
143
+ **Data Collection Method by dataset:**
144
+ * Synthetic
145
+ * Human teleoperation
146
+ * Automatic trajectory generation
147
+
148
+ **Properties:**
149
+
150
+ The models were trained on 100 (GR1) and 130 (Franka) demonstrations.
151
+ The evaluation set consisted of 20 distinct demonstrations.
152
+ Closed loop testing was performed on 100 demonstrations mutually exclusive from the training set.
153
+ The training data is synthetic only and fully generated in Isaac Lab.
154
 
155
  # Inference:
156
 
 
162
 
163
  This model is not tested or intended for use in mission critical applications that require functional safety. The use of the model in those applications is at the user's own risk and sole responsibility, including taking the necessary steps to add needed guardrails or safety mechanisms.
164
 
165
+ - Limitation: This policy is only effective in the exact simulation environment in which it was trained.
166
+ - Mitigation: Recommended to retrain the model in new simulation environments.
167
+ - Limitation: The policy was not tested on a physical robot and likely only works in simulation.
168
  - Mitigation: Expand training, testing and validation on physical robot platforms.
169
 
170
  ## Ethical Considerations:
 
173
 
174
  For more detailed information on ethical considerations for this model, please see the Model Card++ Explainability, Bias, Safety & Security, and Privacy Subcards.
175
 
176
+ Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).
177
 
178
  # Bias
179
  Field | Response
180
  :---------------------------------------------------------------------------------------------------|:---------------
181
  Participation considerations from adversely impacted groups [protected classes](https://www.senate.ca.gov/content/protected-classes) in model design and testing: | Not Applicable
182
  Bias Metric (If Measured): | Not Applicable
 
 
183
  Measures taken to mitigate against unwanted bias: | Not Applicable
 
 
 
184
 
185
  # Explainability
186
  Field | Response
 
189
  Model Type: | Denoising Diffusion Probabilistic Model
190
  Intended Users: | Roboticists and researchers in academia and industry who are interested in robot manipulation research
191
  Output: | Actions consisting of end-effector poses, gripper states and head orientation.
 
192
  Describe how the model works: | ``mindmap`` is a Denoising Diffusion Probabilistic Model that samples robot trajectories conditioned on sensor observations and a 3D reconstruction of the environment.
193
  Name the adversely impacted groups this has been tested to deliver comparable outcomes regardless of: | Not Applicable
194
+ Technical Limitations & Mitigation: | - Limitation: This policy is only effective in the exact simulation environment in which it was trained. Mitigation: Recommended to retrain the model in new simulation environments. - Limitation: The policy was not tested on a physical robot and likely only works in simulation. Mitigation: Expand training, testing and validation on physical robot platforms.
195
  Verified to have met prescribed NVIDIA quality standards: | Yes
196
  Performance Metrics: | Closed loop success rate on simulated robotic manipulation tasks.
197
  Potential Known Risks: | The model might be susceptible to rendering changes on the simulation tasks it was trained on.
 
201
  Field | Response
202
  :---------------------------------------------------|:----------------------------------
203
  Model Application Field(s): | Robotics
 
 
 
204
  Use Case Restrictions: | Abide by [NVIDIA Open Model License Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/)
205
  Model and dataset restrictions: | The Principle of least privilege (PoLP) is applied limiting access for dataset generation and model development. Restrictions enforce dataset access during training, and dataset license constraints adhered to.
206
 
 
209
  :----------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------
210
  Generatable or reverse engineerable personal data? | No
211
  Personal data used to create this model? | No
 
 
212
  How often is dataset reviewed? | Before Release
213
  Is there provenance for all datasets used in training? | Yes
214
  Does data labeling (annotation, metadata) comply with privacy laws? | Yes