Skier8402
/

code-search-net-tokenizer

Text Generation

Model card Files Files and versions

Skier8402 commited on Jan 10, 2024

Commit

3e2fb52

·

1 Parent(s): 23c6c52

Update README.md

Files changed (1) hide show

README.md +4 -0

README.md CHANGED Viewed

@@ -3,5 +3,9 @@ license: cc0-1.0
 datasets:
 - code_search_net
 library_name: transformers
 ---
 This is an adapted tokenizer from GPT2 that can recognize tokens to do with Python coding. It is part of the [huggingfaceNLP course exercise](https://huggingface.co/learn/nlp-course/chapter6/2). It uses the method `train_new_from_iterator()`

 datasets:
 - code_search_net
 library_name: transformers
+tags:
+- text-generation
+- code
+- python
 ---
 This is an adapted tokenizer from GPT2 that can recognize tokens to do with Python coding. It is part of the [huggingfaceNLP course exercise](https://huggingface.co/learn/nlp-course/chapter6/2). It uses the method `train_new_from_iterator()`