Commit
·
2f4db23
1
Parent(s):
4d5cbc9
put on bold up to truncation
Browse files
app.py
CHANGED
|
@@ -72,15 +72,18 @@ if __name__ == "__main__":
|
|
| 72 |
gr.Markdown(
|
| 73 |
"""
|
| 74 |
# 👻 Transformers-CFG JSON Demo
|
| 75 |
-
This is a demo of how you can constrain the output of a GPT-2 model to be a valid JSON string(up to truncation).
|
| 76 |
Here we use a simple JSON grammar to constrain the output of the model.
|
| 77 |
-
The grammar is defined in `json_minimal.ebnf` and is written in the Extended Backus-Naur Form (EBNF)
|
| 78 |
|
| 79 |
Internally, it relies on the library [`transformers-cfg`](https://github.com/epfl-dlab/transformers-CFG).
|
| 80 |
For demo purpose, gpt2-large is used, but you can use much larger models for better performance.
|
| 81 |
|
| 82 |
The inference is a bit slow because of the inference is run on **CPU(~20s for 30 tokens)**.
|
| 83 |
The constraint itself **doesn't** introduce significant overhead to the inference.
|
|
|
|
|
|
|
|
|
|
| 84 |
"""
|
| 85 |
)
|
| 86 |
|
|
|
|
| 72 |
gr.Markdown(
|
| 73 |
"""
|
| 74 |
# 👻 Transformers-CFG JSON Demo
|
| 75 |
+
This is a demo of how you can constrain the output of a GPT-2 model to be a *valid* JSON string(*up to max length truncation*).
|
| 76 |
Here we use a simple JSON grammar to constrain the output of the model.
|
| 77 |
+
The grammar is defined in `json_minimal.ebnf` and is written in the **Extended Backus-Naur Form (EBNF)**.
|
| 78 |
|
| 79 |
Internally, it relies on the library [`transformers-cfg`](https://github.com/epfl-dlab/transformers-CFG).
|
| 80 |
For demo purpose, gpt2-large is used, but you can use much larger models for better performance.
|
| 81 |
|
| 82 |
The inference is a bit slow because of the inference is run on **CPU(~20s for 30 tokens)**.
|
| 83 |
The constraint itself **doesn't** introduce significant overhead to the inference.
|
| 84 |
+
|
| 85 |
+
The output may be *truncated* to 30 tokens due to the limitation of the maximum length of the output.
|
| 86 |
+
In practice, with a decent `max_length` parameter, your JSON output will be *complete* and *valid*.
|
| 87 |
"""
|
| 88 |
)
|
| 89 |
|