yashmarathe/nanochat-checkpoints

This is the reproduction of karpathy's nanochat on AMD Hardware.

nanochat training report

Environment

Hardware

• Platform: Linux
• CPUs: 160 cores (160 logical)
• Memory: 1889.8 GB
• GPUs: 8x AMD Instinct MI300X VF
• GPU Memory: 1533.5 GB total
• CUDA Version: unknown
• Hourly Rate: $16.00/hour

Software

• Python: 3.12.3
• PyTorch: 2.10.0.dev20251028+rocm7.0

---

Tokenizer training

timestamp: 2025-10-29 03:42:34

• max_chars: 2,000,000,000
• doc_cap: 10,000
• vocab_size: 65,536
• train_time: 76.6262
• num_special_tokens: 9
• token_bytes_min: 1
• token_bytes_max: 32
• token_bytes_mean: 6.9151
• token_bytes_std: 2.8736

Tokenizer evaluation

timestamp: 2025-10-29 03:42:37

Comparison with GPT-2

Text Type	Bytes	GPT-2 Tokens	GPT-2 Ratio	Ours Tokens	Ours Ratio	Relative Diff %
news	1819	404	4.50	375	4.85	+7.2%
korean	893	745	1.20	721	1.24	+3.2%
code	1259	576	2.19	493	2.55	+14.4%
math	1834	936	1.96	966	1.90	-3.2%
science	1112	260	4.28	225	4.94	+13.5%
fwe-train	4208518	900364	4.67	856901	4.91	+4.8%
fwe-val	4908443	1059062	4.63	1010356	4.86	+4.6%

Comparison with GPT-4

Text Type	Bytes	GPT-4 Tokens	GPT-4 Ratio	Ours Tokens	Ours Ratio	Relative Diff %
news	1819	387	4.70	375	4.85	+3.1%
korean	893	364	2.45	721	1.24	-98.1%
code	1259	309	4.07	493	2.55	-59.5%
math	1834	832	2.20	966	1.90	-16.1%
science	1112	249	4.47	225	4.94	+9.6%
fwe-train	4208518	874799	4.81	856901	4.91	+2.0%
fwe-val	4908443	1029691	4.77	1010356	4.86	+1.9%

Base model training

timestamp: 2025-10-29 07:47:38

• run: my-llm-training-run-003
• device_type:
• depth: 20
• max_seq_len: 2048
• num_iterations: -1
• target_flops: -1.0000
• target_param_data_ratio: 20
• device_batch_size: 64
• total_batch_size: 1,048,576
• embedding_lr: 0.2000
• unembedding_lr: 0.0040
• weight_decay: 0.0000
• matrix_lr: 0.0200
• grad_clip: 1.0000
• warmup_ratio: 0.0000
• warmdown_ratio: 0.2000
• final_lr_frac: 0.0000
• eval_every: 250
• eval_tokens: 10,485,760
• core_metric_every: 2000
• core_metric_max_per_task: 500
• sample_every: 2000
• model_tag:
• Number of parameters: 560,988,160
• Number of FLOPs per token: 3.491758e+09
• Calculated number of iterations: 10,700
• Number of training tokens: 11,219,763,200
• Tokens : Params ratio: 20.0000
• DDP world size: 8
• warmup_ratio: 0.0000
• warmdown_ratio: 0.2000
• final_lr_frac: 0.0000
• Minimum validation bpb: 0.8119
• Final validation bpb: 0.8119
• CORE metric estimate: 0.2077
• MFU %: 33.97%
• Total training flops: 3.917670e+19
• Total training time: 238.88m
• Peak memory usage: 147486.90MiB

Base model loss

timestamp: 2025-10-29 07:48:10

• train bpb: 0.8146
• val bpb: 0.8120
• sample 0: <|bos|>The capital of France is Paris. It is located in the south of France. It is the second largest
• sample 1: <|bos|>The chemical symbol of gold is Au. It is a soft, silvery-white metal that is malleable and ductile.
• sample 2: <|bos|>If yesterday was Friday, then tomorrow will be Saturday. The day before tomorrow will be Saturday. The day after tomorrow will be
• sample 3: <|bos|>The opposite of hot is cold. The opposite of cold is hot. The opposite of hot is cold.
• sample 4: <|bos|>The planets of the solar system are: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune,
• sample 5: <|bos|>My favorite color is blue. I love the color blue. I love the color blue. I love
• sample 6: <|bos|>If 5x + 3 = 13, then x is a positive integer. If 5x + 3 = 13,

Base model evaluation

timestamp: 2025-10-29 07:51:33

• Model: base_model (step 10700)
• CORE metric: 0.2017
• hellaswag_zeroshot: 0.2547
• jeopardy: 0.1053
• bigbench_qa_wikidata: 0.5239
• arc_easy: 0.5118
• arc_challenge: 0.1251
• copa: 0.2200
• commonsense_qa: 0.0981
• piqa: 0.3765
• openbook_qa: 0.1093
• lambada_openai: 0.3868
• hellaswag: 0.2586
• winograd: 0.2161
• winogrande: 0.0481
• bigbench_dyck_languages: 0.1270
• agi_eval_lsat_ar: 0.0870
• bigbench_cs_algorithms: 0.3689
• bigbench_operators: 0.1524
• bigbench_repeat_copy_logic: 0.0000
• squad: 0.2560
• coqa: 0.1929
• boolq: -0.1597
• bigbench_language_identification: 0.1793

Midtraining

timestamp: 2025-10-29 08:02:20

• run: my-llm-training-run-003
• device_type:
• dtype: bfloat16
• num_iterations: -1
• max_seq_len: 2048
• device_batch_size: 64
• unembedding_lr: 0.0040
• embedding_lr: 0.2000
• matrix_lr: 0.0200
• init_lr_frac: 1.0000
• weight_decay: 0.0000
• eval_every: 150
• eval_tokens: 10,485,760
• total_batch_size: 1,048,576
• dry_run: 0
• Number of iterations: 404
• DDP world size: 8
• Minimum validation bpb: 0.3993

Chat evaluation mid

timestamp: 2025-10-29 08:09:19

• source: mid
• task_name: None
• dtype: bfloat16
• temperature: 0.0000
• max_new_tokens: 512
• num_samples: 1
• top_k: 50
• batch_size: 8
• model_tag: None
• step: None
• max_problems: None
• device_type:
• ARC-Easy: 0.4074
• ARC-Challenge: 0.3157
• MMLU: 0.3236
• GSM8K: 0.0394
• HumanEval: 0.0854
• SpellingBee: 0.9688
• ChatCORE metric: 0.2482

Chat SFT

timestamp: 2025-10-29 08:31:28

• run: my-llm-training-run-003
• source: mid
• device_type:
• dtype: bfloat16
• device_batch_size: 4
• num_epochs: 1
• num_iterations: -1
• target_examples_per_step: 32
• unembedding_lr: 0.0040
• embedding_lr: 0.2000
• matrix_lr: 0.0200
• weight_decay: 0.0000
• init_lr_frac: 0.0200
• eval_every: 100
• eval_steps: 100
• eval_metrics_every: 200
• eval_metrics_max_problems: 1024
• Training rows: 22,439
• Number of iterations: 701
• Training loss: 0.5337
• Validation loss: 1.0260

Chat evaluation sft

timestamp: 2025-10-29 08:49:17

• source: sft
• task_name: None
• dtype: bfloat16
• temperature: 0.0000
• max_new_tokens: 512
• num_samples: 1
• top_k: 50
• batch_size: 8
• model_tag: None
• step: None
• max_problems: None
• device_type:
• ARC-Easy: 0.4192
• ARC-Challenge: 0.3148
• MMLU: 0.3192
• GSM8K: 0.0546
• HumanEval: 0.0671
• SpellingBee: 0.9844
• ChatCORE metric: 0.2517

Summary

• Characters: 395,259
• Lines: 9,643
• Files: 47
• Tokens (approx): 98,814
• Dependencies (uv.lock lines): 1,363

Metric	BASE	MID	SFT	RL
CORE	0.2017	-	-	-
ARC-Challenge	-	0.3157	0.3148	-
ARC-Easy	-	0.4074	0.4192	-
GSM8K	-	0.0394	0.0546	-
HumanEval	-	0.0854	0.0671	-
MMLU	-	0.3236	0.3192	-
ChatCORE	-	0.2482	0.2517	-

Total wall clock time: 5h8m