update model card
#4
by
remostei
- opened
README.md
CHANGED
|
@@ -11,11 +11,12 @@ datasets:
|
|
| 11 |
|
| 12 |
### Description:
|
| 13 |
|
| 14 |
-
``mindmap`` is a 3D diffusion policy that generates robot trajectories based on a semantic 3D reconstruction of the environment,
|
| 15 |
-
enabling robots with spatial memory.
|
| 16 |
|
| 17 |
Trained models are available on Hugging Face: [PhysicalAI-Robotics-mindmap-Checkpoints](https://huggingface.co/nvidia/PhysicalAI-Robotics-mindmap-Checkpoints)
|
| 18 |
|
|
|
|
|
|
|
| 19 |
### License/Terms of Use
|
| 20 |
|
| 21 |
- Model: [NVIDIA Open Model License Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/)
|
|
@@ -33,6 +34,11 @@ The trained ``mindmap`` policies allow for quick evaluation of the ``mindmap`` c
|
|
| 33 |
- Developers: Integrate and customize AI for various robotic applications.
|
| 34 |
- Startups & Companies: Accelerate robotics development and reduce training costs.
|
| 35 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 36 |
|
| 37 |
## References(s):
|
| 38 |
|
|
@@ -90,6 +96,9 @@ The trained ``mindmap`` policies allow for quick evaluation of the ``mindmap`` c
|
|
| 90 |
- Gripper: `[PREDICTION_HORIZON, NUM_GRIPPERS, 8]` - consisting of end-effector translation, rotation (quaternion, wxyz) and closedness
|
| 91 |
- Head Yaw: `[PREDICTION_HORIZON, 1]` - only for humanoid embodiments
|
| 92 |
|
|
|
|
|
|
|
|
|
|
| 93 |
|
| 94 |
## Software Integration:
|
| 95 |
**Runtime Engine(s):** PyTorch
|
|
@@ -108,6 +117,10 @@ The trained ``mindmap`` policies allow for quick evaluation of the ``mindmap`` c
|
|
| 108 |
**Preferred/Supported Operating System(s):**
|
| 109 |
* Linux
|
| 110 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 111 |
## Model Version(s):
|
| 112 |
|
| 113 |
This is the initial version of the model, version 1.0.0
|
|
@@ -120,7 +133,24 @@ Datasets:
|
|
| 120 |
- drill_in_box_checkpoint: [GR1 Drill in Box Dataset](https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-mindmap-GR1-Drill-in-Box)
|
| 121 |
- stick_in_bin_checkpoint: [GR1 Stick in Bin Dataset](https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-mindmap-GR1-Stick-in-Bin)
|
| 122 |
|
| 123 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 124 |
|
| 125 |
# Inference:
|
| 126 |
|
|
@@ -132,9 +162,9 @@ The models were trained on 100 (GR1) and 130 (Franka) demonstrations. The evalua
|
|
| 132 |
|
| 133 |
This model is not tested or intended for use in mission critical applications that require functional safety. The use of the model in those applications is at the user's own risk and sole responsibility, including taking the necessary steps to add needed guardrails or safety mechanisms.
|
| 134 |
|
| 135 |
-
-
|
| 136 |
-
- Mitigation:
|
| 137 |
-
-
|
| 138 |
- Mitigation: Expand training, testing and validation on physical robot platforms.
|
| 139 |
|
| 140 |
## Ethical Considerations:
|
|
@@ -143,19 +173,14 @@ NVIDIA believes Trustworthy AI is a shared responsibility and we have establishe
|
|
| 143 |
|
| 144 |
For more detailed information on ethical considerations for this model, please see the Model Card++ Explainability, Bias, Safety & Security, and Privacy Subcards.
|
| 145 |
|
| 146 |
-
Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).
|
| 147 |
|
| 148 |
# Bias
|
| 149 |
Field | Response
|
| 150 |
:---------------------------------------------------------------------------------------------------|:---------------
|
| 151 |
Participation considerations from adversely impacted groups [protected classes](https://www.senate.ca.gov/content/protected-classes) in model design and testing: | Not Applicable
|
| 152 |
Bias Metric (If Measured): | Not Applicable
|
| 153 |
-
(For GPAI Models) Which characteristic (feature) show(s) the greatest difference in performance?: | Not Applicable
|
| 154 |
-
(For GPAI Models): Which feature(s) have have the worst performance overall? | Not Applicable
|
| 155 |
Measures taken to mitigate against unwanted bias: | Not Applicable
|
| 156 |
-
(For GPAI Models): If using internal data, description of methods implemented in data acquisition or processing, if any, to address the prevalence of identifiable biases in the training, testing, and validation data: | Not Applicable
|
| 157 |
-
(For GPAI Models): Tools used to assess statistical imbalances and highlight patterns that may introduce bias into AI models: | Not Applicable
|
| 158 |
-
(For GPAI Models): Tools used to assess statistical imbalances and highlight patterns that may introduce bias into AI models: | Not Applicable
|
| 159 |
|
| 160 |
# Explainability
|
| 161 |
Field | Response
|
|
@@ -164,10 +189,9 @@ Intended Task/Domain:
|
|
| 164 |
Model Type: | Denoising Diffusion Probabilistic Model
|
| 165 |
Intended Users: | Roboticists and researchers in academia and industry who are interested in robot manipulation research
|
| 166 |
Output: | Actions consisting of end-effector poses, gripper states and head orientation.
|
| 167 |
-
(For GPAI Models): Tools used to evaluate datasets to identify synthetic data and ensure data authenticity. | Not Applicable
|
| 168 |
Describe how the model works: | ``mindmap`` is a Denoising Diffusion Probabilistic Model that samples robot trajectories conditioned on sensor observations and a 3D reconstruction of the environment.
|
| 169 |
Name the adversely impacted groups this has been tested to deliver comparable outcomes regardless of: | Not Applicable
|
| 170 |
-
Technical Limitations & Mitigation: | -
|
| 171 |
Verified to have met prescribed NVIDIA quality standards: | Yes
|
| 172 |
Performance Metrics: | Closed loop success rate on simulated robotic manipulation tasks.
|
| 173 |
Potential Known Risks: | The model might be susceptible to rendering changes on the simulation tasks it was trained on.
|
|
@@ -177,9 +201,6 @@ Licensing:
|
|
| 177 |
Field | Response
|
| 178 |
:---------------------------------------------------|:----------------------------------
|
| 179 |
Model Application Field(s): | Robotics
|
| 180 |
-
Describe the life critical impact (if present). | Not Applicable
|
| 181 |
-
(For GPAI Models): Description of methods implemented in data acquisition or processing, if any, to address other types of potentially harmful data in the training, testing, and validation data: | Not GPAI
|
| 182 |
-
(For GPAI Models): Description of any methods implemented in data acquisition or processing, if any, to address illegal or harmful content in the training data, including, but not limited to, child sexual abuse material (CSAM) and non-consensual intimate imagery (NCII) | Not GPAI
|
| 183 |
Use Case Restrictions: | Abide by [NVIDIA Open Model License Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/)
|
| 184 |
Model and dataset restrictions: | The Principle of least privilege (PoLP) is applied limiting access for dataset generation and model development. Restrictions enforce dataset access during training, and dataset license constraints adhered to.
|
| 185 |
|
|
@@ -188,8 +209,6 @@ Field
|
|
| 188 |
:----------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------
|
| 189 |
Generatable or reverse engineerable personal data? | No
|
| 190 |
Personal data used to create this model? | No
|
| 191 |
-
Was consent obtained for any personal data used? | Not Applicable
|
| 192 |
-
(For GPAI Models): A description of any methods implemented in data acquisition or processing, if any, to address the prevalence of personal data in the training data, where relevant and applicable. | Not Applicable
|
| 193 |
How often is dataset reviewed? | Before Release
|
| 194 |
Is there provenance for all datasets used in training? | Yes
|
| 195 |
Does data labeling (annotation, metadata) comply with privacy laws? | Yes
|
|
|
|
| 11 |
|
| 12 |
### Description:
|
| 13 |
|
| 14 |
+
``mindmap`` is a 3D diffusion policy that generates robot trajectories based on a semantic 3D reconstruction of the environment, enabling robots with spatial memory.
|
|
|
|
| 15 |
|
| 16 |
Trained models are available on Hugging Face: [PhysicalAI-Robotics-mindmap-Checkpoints](https://huggingface.co/nvidia/PhysicalAI-Robotics-mindmap-Checkpoints)
|
| 17 |
|
| 18 |
+
This model is ready for commercial/non-commercial use
|
| 19 |
+
|
| 20 |
### License/Terms of Use
|
| 21 |
|
| 22 |
- Model: [NVIDIA Open Model License Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/)
|
|
|
|
| 34 |
- Developers: Integrate and customize AI for various robotic applications.
|
| 35 |
- Startups & Companies: Accelerate robotics development and reduce training costs.
|
| 36 |
|
| 37 |
+
### Release Date
|
| 38 |
+
|
| 39 |
+
Github 09/26/2025 via github.com/NVlabs/nvblox_mindmap
|
| 40 |
+
|
| 41 |
+
Hugging Face 09/26/2025 via huggingface.co/nvidia/PhysicalAI-Robotics-mindmap-Checkpoints
|
| 42 |
|
| 43 |
## References(s):
|
| 44 |
|
|
|
|
| 96 |
- Gripper: `[PREDICTION_HORIZON, NUM_GRIPPERS, 8]` - consisting of end-effector translation, rotation (quaternion, wxyz) and closedness
|
| 97 |
- Head Yaw: `[PREDICTION_HORIZON, 1]` - only for humanoid embodiments
|
| 98 |
|
| 99 |
+
Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems.
|
| 100 |
+
By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries),
|
| 101 |
+
the model achieves faster training and inference times compared to CPU-only solutions.
|
| 102 |
|
| 103 |
## Software Integration:
|
| 104 |
**Runtime Engine(s):** PyTorch
|
|
|
|
| 117 |
**Preferred/Supported Operating System(s):**
|
| 118 |
* Linux
|
| 119 |
|
| 120 |
+
The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment.
|
| 121 |
+
Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks,
|
| 122 |
+
meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.
|
| 123 |
+
|
| 124 |
## Model Version(s):
|
| 125 |
|
| 126 |
This is the initial version of the model, version 1.0.0
|
|
|
|
| 133 |
- drill_in_box_checkpoint: [GR1 Drill in Box Dataset](https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-mindmap-GR1-Drill-in-Box)
|
| 134 |
- stick_in_bin_checkpoint: [GR1 Stick in Bin Dataset](https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-mindmap-GR1-Stick-in-Bin)
|
| 135 |
|
| 136 |
+
|
| 137 |
+
**Data Modality:** Image, 3D reconstruction, robot states
|
| 138 |
+
|
| 139 |
+
**Image Training Data Size:** Less than a Million Images
|
| 140 |
+
|
| 141 |
+
**3D reconstruction, robot state Data Size:** Less than a Million Samples
|
| 142 |
+
|
| 143 |
+
**Data Collection Method by dataset:**
|
| 144 |
+
* Synthetic
|
| 145 |
+
* Human teleoperation
|
| 146 |
+
* Automatic trajectory generation
|
| 147 |
+
|
| 148 |
+
**Properties:**
|
| 149 |
+
|
| 150 |
+
The models were trained on 100 (GR1) and 130 (Franka) demonstrations.
|
| 151 |
+
The evaluation set consisted of 20 distinct demonstrations.
|
| 152 |
+
Closed loop testing was performed on 100 demonstrations mutually exclusive from the training set.
|
| 153 |
+
The training data is synthetic only and fully generated in Isaac Lab.
|
| 154 |
|
| 155 |
# Inference:
|
| 156 |
|
|
|
|
| 162 |
|
| 163 |
This model is not tested or intended for use in mission critical applications that require functional safety. The use of the model in those applications is at the user's own risk and sole responsibility, including taking the necessary steps to add needed guardrails or safety mechanisms.
|
| 164 |
|
| 165 |
+
- Limitation: This policy is only effective in the exact simulation environment in which it was trained.
|
| 166 |
+
- Mitigation: Recommended to retrain the model in new simulation environments.
|
| 167 |
+
- Limitation: The policy was not tested on a physical robot and likely only works in simulation.
|
| 168 |
- Mitigation: Expand training, testing and validation on physical robot platforms.
|
| 169 |
|
| 170 |
## Ethical Considerations:
|
|
|
|
| 173 |
|
| 174 |
For more detailed information on ethical considerations for this model, please see the Model Card++ Explainability, Bias, Safety & Security, and Privacy Subcards.
|
| 175 |
|
| 176 |
+
Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).
|
| 177 |
|
| 178 |
# Bias
|
| 179 |
Field | Response
|
| 180 |
:---------------------------------------------------------------------------------------------------|:---------------
|
| 181 |
Participation considerations from adversely impacted groups [protected classes](https://www.senate.ca.gov/content/protected-classes) in model design and testing: | Not Applicable
|
| 182 |
Bias Metric (If Measured): | Not Applicable
|
|
|
|
|
|
|
| 183 |
Measures taken to mitigate against unwanted bias: | Not Applicable
|
|
|
|
|
|
|
|
|
|
| 184 |
|
| 185 |
# Explainability
|
| 186 |
Field | Response
|
|
|
|
| 189 |
Model Type: | Denoising Diffusion Probabilistic Model
|
| 190 |
Intended Users: | Roboticists and researchers in academia and industry who are interested in robot manipulation research
|
| 191 |
Output: | Actions consisting of end-effector poses, gripper states and head orientation.
|
|
|
|
| 192 |
Describe how the model works: | ``mindmap`` is a Denoising Diffusion Probabilistic Model that samples robot trajectories conditioned on sensor observations and a 3D reconstruction of the environment.
|
| 193 |
Name the adversely impacted groups this has been tested to deliver comparable outcomes regardless of: | Not Applicable
|
| 194 |
+
Technical Limitations & Mitigation: | - Limitation: This policy is only effective in the exact simulation environment in which it was trained. Mitigation: Recommended to retrain the model in new simulation environments. - Limitation: The policy was not tested on a physical robot and likely only works in simulation. Mitigation: Expand training, testing and validation on physical robot platforms.
|
| 195 |
Verified to have met prescribed NVIDIA quality standards: | Yes
|
| 196 |
Performance Metrics: | Closed loop success rate on simulated robotic manipulation tasks.
|
| 197 |
Potential Known Risks: | The model might be susceptible to rendering changes on the simulation tasks it was trained on.
|
|
|
|
| 201 |
Field | Response
|
| 202 |
:---------------------------------------------------|:----------------------------------
|
| 203 |
Model Application Field(s): | Robotics
|
|
|
|
|
|
|
|
|
|
| 204 |
Use Case Restrictions: | Abide by [NVIDIA Open Model License Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/)
|
| 205 |
Model and dataset restrictions: | The Principle of least privilege (PoLP) is applied limiting access for dataset generation and model development. Restrictions enforce dataset access during training, and dataset license constraints adhered to.
|
| 206 |
|
|
|
|
| 209 |
:----------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------
|
| 210 |
Generatable or reverse engineerable personal data? | No
|
| 211 |
Personal data used to create this model? | No
|
|
|
|
|
|
|
| 212 |
How often is dataset reviewed? | Before Release
|
| 213 |
Is there provenance for all datasets used in training? | Yes
|
| 214 |
Does data labeling (annotation, metadata) comply with privacy laws? | Yes
|