|
|
--- |
|
|
library_name: transformers |
|
|
license: mit |
|
|
language: |
|
|
- en |
|
|
metrics: |
|
|
- rouge |
|
|
base_model: |
|
|
- microsoft/Phi-3-mini-4k-instruct |
|
|
pipeline_tag: text-generation |
|
|
--- |
|
|
|
|
|
# Model Card for **Phi3-Lab-Report-Coder (LoRA on Phi-3 Mini 4k Instruct)** |
|
|
|
|
|
A lightweight LoRA-adapter fine-tune of `microsoft/Phi-3-mini-4k-instruct` for **turning structured lab contexts + observations into executable Python code** that performs the target calculations (e.g., mechanics, fluids, vibrations, basic circuits, titrations). Trained with QLoRA in 4-bit, this model is intended as an **assistive code generator** for STEM lab writeups and teaching demos—not as a certified calculator for safety-critical engineering. |
|
|
|
|
|
--- |
|
|
|
|
|
## Model Details |
|
|
|
|
|
### Model Description |
|
|
|
|
|
- **Developed by:** Barghav777 |
|
|
- **Model type:** Causal decoder LM (instruction-tuned) + **LoRA adapter** |
|
|
- **Languages:** English |
|
|
- **License:** MIT |
|
|
- **Finetuned from:** `microsoft/Phi-3-mini-4k-instruct` |
|
|
- **Intended input format:** A structured prompt with: |
|
|
- `### CONTEXT:` (natural-language description of the experiment) |
|
|
- `### OBSERVATIONS:` (JSON-like dict with units, readings) |
|
|
- `### CODE:` (the model is trained to generate the Python solution after this tag) |
|
|
|
|
|
### Model Sources |
|
|
|
|
|
- **Base model:** `microsoft/Phi-3-mini-4k-instruct` |
|
|
- **Training data files:** `train.jsonl` (37 items), `eval.jsonl` (6 items) |
|
|
- **Demo/Colab basis:** Training notebook available at: https://github.com/Barghav777/AI-Lab-Report-Agent |
|
|
|
|
|
--- |
|
|
|
|
|
## Uses |
|
|
|
|
|
### Direct Use |
|
|
- Generate **readable Python code** to compute derived quantities from lab observations (e.g., average \(g\) via pendulum, Coriolis acceleration, Ohm’s law resistances, radius of gyration, Reynolds number). |
|
|
- Produce calculation pipelines with minimal plotting/printing that are easy to copy-paste and run in a notebook. |
|
|
|
|
|
### Downstream Use |
|
|
- Course assistants or lab-prep tools that auto-draft calculation code for **intro undergrad physics/mech/fluids/EE labs**. |
|
|
- Auto-checkers that compare student code vs. a reference implementation (with appropriate guardrails). |
|
|
|
|
|
### Out-of-Scope Use |
|
|
- Any **safety-critical** design decisions (structural, medical, chemical process control). |
|
|
- High-stakes computation without human verification. |
|
|
- Domains far outside the training distribution (e.g., NLP preprocessing pipelines, advanced control systems, large-scale simulation frameworks). |
|
|
|
|
|
--- |
|
|
|
|
|
## Bias, Risks, and Limitations |
|
|
|
|
|
- **Small dataset (37 train / 6 eval)** → plausible overfitting; brittle generalization to unseen experiment formats. |
|
|
- **Formula misuse risk:** The model may pick incorrect constants/units or silently use wrong equations. |
|
|
- **Overconfidence:** Generated code may “look right” while being numerically off or unit-inconsistent. |
|
|
- **JSON brittleness:** If `OBSERVATIONS` keys/units differ from training patterns, the code may break. |
|
|
|
|
|
### Recommendations |
|
|
- Always **review formulas and units**; add assertions/unit conversions in downstream systems. |
|
|
- Run generated code with **test observations** and compare against hand calculations. |
|
|
- For deployment, wrap outputs with **explanations and references** to the formulas used. |
|
|
|
|
|
--- |
|
|
|
|
|
## How to Get Started |
|
|
|
|
|
**Prompt template used in training** |
|
|
```text |
|
|
### CONTEXT: |
|
|
{context} |
|
|
|
|
|
### OBSERVATIONS: |
|
|
{observations} |
|
|
|
|
|
### CODE: |
|
|
``` |
|
|
|
|
|
**Load base + LoRA adapter (recommended)** |
|
|
```python |
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig, TextStreamer |
|
|
from peft import PeftModel |
|
|
import torch |
|
|
|
|
|
base_id = "microsoft/Phi-3-mini-4k-instruct" |
|
|
adapter_id = "YOUR_ADAPTER_REPO_OR_LOCAL_PATH" # e.g., ./phi3-lab-report-coder-final |
|
|
|
|
|
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", |
|
|
bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=False) |
|
|
|
|
|
tok = AutoTokenizer.from_pretrained(base_id, trust_remote_code=True) |
|
|
tok.pad_token = tok.eos_token |
|
|
|
|
|
base = AutoModelForCausalLM.from_pretrained(base_id, quantization_config=bnb, |
|
|
trust_remote_code=True, device_map="auto") |
|
|
model = PeftModel.from_pretrained(base, adapter_id) |
|
|
model.eval() |
|
|
|
|
|
prompt = """### CONTEXT: |
|
|
Experiment to determine acceleration due to gravity using a simple pendulum... |
|
|
|
|
|
### OBSERVATIONS: |
|
|
{'readings': [{'L':0.50,'T':1.42}, {'L':0.60,'T':1.55}], 'unit_L':'m', 'unit_T':'s'} |
|
|
|
|
|
### CODE: |
|
|
""" |
|
|
|
|
|
inputs = tok(prompt, return_tensors="pt").to(model.device) |
|
|
streamer = TextStreamer(tok, skip_prompt=True, skip_special_tokens=True) |
|
|
_ = model.generate(**inputs, max_new_tokens=400, temperature=0.2, do_sample=False, streamer=streamer) |
|
|
``` |
|
|
|
|
|
--- |
|
|
|
|
|
## Training Details |
|
|
|
|
|
### Data |
|
|
- **Files:** `train.jsonl` (list of objects), `eval.jsonl` (list of objects) |
|
|
- **Schema per example:** |
|
|
- `context` *(str)*: experiment description |
|
|
- `observations` *(dict)*: units + numeric readings (lists of dicts) |
|
|
- `code` *(str)*: reference Python solution |
|
|
- **Topical spread (non-exhaustive):** pendulum \(g\), Ohm’s law, titration, density via displacement, Coriolis accel., gyroscopic effect, Hartnell governor, rotating mass balancing, helical spring vibration, bi-filar suspension, etc. |
|
|
|
|
|
**Size & basic stats** |
|
|
- Train: **37** items; Eval: **6** items |
|
|
- Formatted prompt (context+observations+code) length (train): |
|
|
- mean ≈ **222** words (≈ **1,739** chars); 95th pct ≈ **311** words |
|
|
- Reference code length (train): |
|
|
- mean ≈ **34** lines (min **9**, max **71**) |
|
|
|
|
|
### Training Procedure (from notebook) |
|
|
- **Approach:** QLoRA (4-bit) SFT using `trl.SFTTrainer` |
|
|
- **Quantization:** `bitsandbytes` 4-bit `nf4`, compute dtype `bfloat16` |
|
|
- **LoRA config:** `r=16`, `alpha=32`, `dropout=0.05`, `bias="none"`, targets = `q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj` |
|
|
- **Tokenizer:** right padding; `eos_token` as `pad_token` |
|
|
- **Hyperparameters (TrainingArguments):** |
|
|
- epochs: **10** |
|
|
- per-device train batch size: **1** |
|
|
- gradient_accumulation_steps: **4** |
|
|
- optimizer: **paged_adamw_32bit** |
|
|
- learning rate: **2e-4**, weight decay: **1e-3** |
|
|
- warmup_ratio: **0.03**, scheduler: **constant** |
|
|
- bf16: **True** (fp16: False), group_by_length: True |
|
|
- logging_steps: 10, save/eval every 50 steps |
|
|
- report_to: tensorboard |
|
|
- **Saving:** `trainer.save_model("./phi3-lab-report-coder-final")` (adapter folder) |
|
|
|
|
|
### Speeds, Sizes, Times |
|
|
- **Hardware:** Google Colab **T4 GPU** (per notebook metadata) |
|
|
- **Adapter artifact:** LoRA weights only (load with the base model). |
|
|
- **Wall-clock time:** not logged in the notebook. |
|
|
|
|
|
--- |
|
|
|
|
|
## Evaluation |
|
|
|
|
|
### Testing Data, Factors & Metrics |
|
|
- **Eval set:** `eval.jsonl` (**6** items) with same schema. |
|
|
- **Primary metric (planned):** ROUGE-L / ROUGE-1 against reference `code` (proxy for surface similarity). |
|
|
- **Recommended additional checks:** unit tests on numeric outputs; pyflakes/ruff for syntax; run-time assertions. |
|
|
|
|
|
### Results |
|
|
- No automated score recorded in the notebook. |
|
|
- **Suggested protocol:** |
|
|
1) Generate code for each eval item using the same prompt template. |
|
|
2) Execute safely in a sandbox with provided observations. |
|
|
3) Compare computed scalars (e.g., average \(g\), \(R\), Reynolds number) to ground truth tolerances. |
|
|
4) Report pass rate and ROUGE for readability/similarity. |
|
|
|
|
|
--- |
|
|
|
|
|
## Model Examination (optional) |
|
|
- Inspect token-by-token attention to `OBSERVATIONS` keys (ablation: shuffle keys to test robustness). |
|
|
- Add **unit-check helpers** (e.g., `pint`) in prompts to encourage explicit conversions. |
|
|
|
|
|
--- |
|
|
|
|
|
## Environmental Impact |
|
|
- **Hardware Type:** NVIDIA T4 (Colab) |
|
|
- **Precision:** 4-bit QLoRA with `bfloat16` compute |
|
|
- **Hours used:** Not recorded (dataset is small; expected low) |
|
|
- **Cloud Provider/Region:** Colab (unspecified) |
|
|
- **Carbon Emitted:** Not estimated (see [ML CO2 Impact calculator](https://mlco2.github.io/impact#compute)) |
|
|
|
|
|
--- |
|
|
|
|
|
## Technical Specifications |
|
|
|
|
|
### Architecture & Objective |
|
|
- **Backbone:** `Phi-3-mini-4k-instruct` (decoder-only causal LM) |
|
|
- **Objective:** Supervised fine-tuning to continue from `### CODE:` with correct, executable Python. |
|
|
|
|
|
### Compute Infrastructure |
|
|
- **Hardware:** Colab GPU (T4) + CPU RAM |
|
|
- **Software:** |
|
|
- `transformers`, `trl`, `peft`, `bitsandbytes`, `datasets`, `accelerate`, `torch` |
|
|
|
|
|
--- |
|
|
|
|
|
## Citation |
|
|
@article{abdin2024phi3, |
|
|
title = {Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone}, |
|
|
author = {Abdin, Marah and others}, |
|
|
journal = {arXiv preprint arXiv:2404.14219}, |
|
|
year = {2024}, |
|
|
doi = {10.48550/arXiv.2404.14219}, |
|
|
url = {https://arxiv.org/abs/2404.14219} |
|
|
} |
|
|
|
|
|
--- |
|
|
|
|
|
## Glossary |
|
|
- **QLoRA:** Fine-tuning with low-rank adapters on a quantized base model (saves memory/compute). |
|
|
- **LoRA (r, α):** Rank and scaling of low-rank update matrices. |
|
|
|
|
|
--- |
|
|
|
|
|
## More Information |
|
|
- For better robustness, consider augmenting data with **unit-perturbation** and **noise-in-readings** variants, and add examples across more domains (materials, thermo, optics). |
|
|
- Add **eval harness** with numeric tolerances and syntax checks. |
|
|
|
|
|
--- |
|
|
|
|
|
## Model Card Authors |
|
|
- Barghav777 |
|
|
--- |
