|
|
--- |
|
|
license: apache-2.0 |
|
|
tags: |
|
|
- tensorflow |
|
|
- optimizer |
|
|
- deep-learning |
|
|
- machine-learning |
|
|
- adaptive-learning-rate |
|
|
library_name: tensorflow |
|
|
--- |
|
|
|
|
|
# NEAT Optimizer |
|
|
|
|
|
**NEAT (Noise-Enhanced Adaptive Training)** is a novel optimization algorithm for deep learning that combines adaptive learning rates with controlled noise injection to improve convergence and generalization. |
|
|
|
|
|
## Overview |
|
|
|
|
|
The NEAT optimizer enhances traditional adaptive optimization methods by intelligently injecting noise into the gradient updates. This approach helps: |
|
|
- Escape local minima more effectively |
|
|
- Improve generalization performance |
|
|
- Achieve faster and more stable convergence |
|
|
- Reduce overfitting on training data |
|
|
|
|
|
## Installation |
|
|
|
|
|
### From PyPI (recommended) |
|
|
```bash |
|
|
pip install neat-optimizer |
|
|
``` |
|
|
|
|
|
### From Source |
|
|
```bash |
|
|
git clone https://github.com/yourusername/neat-optimizer.git |
|
|
cd neat-optimizer |
|
|
pip install -e . |
|
|
``` |
|
|
|
|
|
## Quick Start |
|
|
|
|
|
```python |
|
|
import tensorflow as tf |
|
|
from neat_optimizer import NEATOptimizer |
|
|
|
|
|
# Create your model |
|
|
model = tf.keras.Sequential([ |
|
|
tf.keras.layers.Dense(128, activation='relu'), |
|
|
tf.keras.layers.Dense(10, activation='softmax') |
|
|
]) |
|
|
|
|
|
# Use NEAT optimizer |
|
|
optimizer = NEATOptimizer( |
|
|
learning_rate=0.001, |
|
|
noise_scale=0.01, |
|
|
beta_1=0.9, |
|
|
beta_2=0.999 |
|
|
) |
|
|
|
|
|
# Compile and train |
|
|
model.compile( |
|
|
optimizer=optimizer, |
|
|
loss='sparse_categorical_crossentropy', |
|
|
metrics=['accuracy'] |
|
|
) |
|
|
|
|
|
model.fit(x_train, y_train, epochs=10, validation_data=(x_val, y_val)) |
|
|
``` |
|
|
|
|
|
## Key Features |
|
|
|
|
|
- **Adaptive Learning Rates**: Automatically adjusts learning rates per parameter |
|
|
- **Noise Injection**: Controlled stochastic perturbations for better exploration |
|
|
- **TensorFlow Integration**: Drop-in replacement for standard TensorFlow optimizers |
|
|
- **Hyperparameter Flexibility**: Customizable noise schedules and adaptation rates |
|
|
|
|
|
## Parameters |
|
|
|
|
|
- `learning_rate` (float, default=0.001): Initial learning rate |
|
|
- `noise_scale` (float, default=0.01): Scale of noise injection |
|
|
- `beta_1` (float, default=0.9): Exponential decay rate for first moment estimates |
|
|
- `beta_2` (float, default=0.999): Exponential decay rate for second moment estimates |
|
|
- `epsilon` (float, default=1e-7): Small constant for numerical stability |
|
|
- `noise_decay` (float, default=0.99): Decay rate for noise scale over time |
|
|
|
|
|
## Requirements |
|
|
|
|
|
- Python >= 3.7 |
|
|
- TensorFlow >= 2.4.0 |
|
|
- NumPy >= 1.19.0 |
|
|
|
|
|
## Citation |
|
|
|
|
|
If you use NEAT optimizer in your research, please cite: |
|
|
|
|
|
```bibtex |
|
|
@software{neat_optimizer, |
|
|
title={NEAT: Noise-Enhanced Adaptive Training Optimizer}, |
|
|
author={Your Name}, |
|
|
year={2025}, |
|
|
url={https://github.com/yourusername/neat-optimizer} |
|
|
} |
|
|
``` |
|
|
|
|
|
## References |
|
|
|
|
|
- Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. |
|
|
- Neelakantan, A., et al. (2015). Adding gradient noise improves learning for very deep networks. arXiv preprint arXiv:1511.06807. |
|
|
|
|
|
## License |
|
|
|
|
|
This project is licensed under the Apache License 2.0 - see the LICENSE file for details. |
|
|
|
|
|
## Contributing |
|
|
|
|
|
Contributions are welcome! Please feel free to submit a Pull Request. |
|
|
|
|
|
## Support |
|
|
|
|
|
For issues, questions, or feature requests, please open an issue on GitHub. |
