junihealthai/llama-3.1-8B-Instruct-Med-LoRA-v0.5

Update: This model was fine-tuned with LoRA on a 170k dataset. Compared to version 0.4, date information in the dataset was masked, and low-quality data was further filtered out.

Metrics:

• rouge: ~0.48

Dataset:

• 170k data

Training Configuration:

• precision: bf16
• max sequence length: 2048
• effective batch size: 16 (per-device batch size 2 × gradient accumulation 8)
• learning rate: 1 × 10⁻⁴
• epochs: 3
• scheduler: cosine
• warmup steps: 760
• optimizer: AdamW
• max gradient norm: 1.0
• save checkpoint every 5 000 steps
• output directory: saves/Llama-3.1-8B-Instruct/lora/train_2025-06-09-13-48-28

LoRA Hyperparameters:

• rank: 8
• alpha: 32
• dropout: 0.05
• target modules: all
• finetuning type: LoRA

---