Some problems at the start
Hi, guys! I'm not an experienced person in DL (and SAE) and perhaps do something wrong, could you help me?
I want to use "google/gemma-scope-2b-pt-transcoders" from here https://huggingface.co/google/gemma-scope-2b-pt-transcoders
I run the command
sae, cfg_dict, sparsity = SAE.from_pretrained(
release = "google/gemma-scope-2b-pt-transcoders",
sae_id = "layer_0/width_16k/average_l0_76",
device=device
)
and have an error
Entry Not Found for url: https://huggingface.co/google/gemma-scope-2b-pt-transcoders/resolve/main/layer_0/width_16k/average_l0_76/cfg.json.
There is no "cfg.json" file. Where I can take it? Can I create it somehow?
Resolved;
David Chanin
Aug 31st at 9:48 PM
@sergeika83 the issue with the GemmaScope transcoders is that you’re not specifying the release correctly. It should be gemma-scope-2b-pt-transcoders, not google/gemma-scope-2b-pt-transcoders . If you give it a path to a huggingface repo it will assume you’re loading a standard SAELens SAE from Huggingface and won’t know how to import it since Gemma Scope wasn’t trained with SAELens. You can find the full list of SAEs/Transcoders at https://jbloomaus.github.io/SAELens/latest/sae_table/#gemma-scope-2b-pt-transcoders. If you click “load this SAE” next to the SAE/Transcoder you want to use it will give you the code to load it
jbloomaus.github.iojbloomaus.github.io
Supported SAEs - SAE Lens
Docs for Sparse Autoencoder Training and Analysis Library
:raised_hands:
2
David Chanin
Aug 31st at 9:52 PM
It looks like the issue in the notebook is that it loads sae-dashboard, but sae-dashboard depends on SAELens v5.x.x (https://github.com/jbloomAus/SAEDashboard/blob/main/pyproject.toml#L22). This makes the colab notebook isntall SAELens v5 instead of v6, but the notebook is written for SAELens v6. We should update the sae-dashboard dependency to be >=5.0.0 instead so it doesn’t force SAELens to downgrade.
:raised_hands:
2
Johnny Lin
Sep 1st at 2:50 AM
Thanks David for the fast response!
re: the first issue, this should be fixed now (update saedashboard dependencies). i reran the notebook and it works fine.
re: second issue - good catch. so it appears there's no issue here.
@Arthur
Conmy Doesn't appear that anything needs to be updated. This issue was isolated to that notebook.
@sergeika83 Please verify that this is fine.
Filed this issue for a more useful error message when this happens: https://github.com/jbloomAus/SAELens/issues/538
Also sent to the channel
sergeika83
Sep 1st at 10:10 PM
Thank you very much, guys! Now it works for me. :slightly_smiling_face:
@Johnny
Lin
@David
Chanin
@Arthur
Conmy
:tada:
3