Commit
·
aa42541
1
Parent(s):
9469cf6
Update README.md
Browse files
README.md
CHANGED
|
@@ -28,6 +28,17 @@ The goal of the Spider challenge is to develop natural language interfaces to cr
|
|
| 28 |
|
| 29 |
This dataset was used to finetune this model.
|
| 30 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 31 |
## Citations
|
| 32 |
|
| 33 |
```
|
|
|
|
| 28 |
|
| 29 |
This dataset was used to finetune this model.
|
| 30 |
|
| 31 |
+
### Spider Skeleton WizardCoder - [test-suite-sql-eval](https://github.com/taoyds/test-suite-sql-eval) Results
|
| 32 |
+
With temperature set to 0.0, top_p set to 0.9, and top_k set to 0, the model achieves **61% execution accuracy** on the Spider dev set.
|
| 33 |
+
|
| 34 |
+
<img src="https://raw.githubusercontent.com/cuplv/text-to-sql-wizardcoder/main/eval/plots/spiderwizard-plus-chatgpt.svg" height="300">
|
| 35 |
+
<img src="https://raw.githubusercontent.com/cuplv/text-to-sql-wizardcoder/main/eval/plots/spiderwizard-vs-chatgpt.svg" height="300">
|
| 36 |
+
|
| 37 |
+
Note:
|
| 38 |
+
- ChatGPT was evaluated with the default hyperparameters and with the system message `You are a sophisticated AI assistant capable of converting text into SQL queries. You can only output SQL, don't add any other text.`
|
| 39 |
+
- Both models were evaluated with `--plug_value` in `evaluation.py` using the Spider dev set with database context.
|
| 40 |
+
- `--plug_value`: If set, the gold value will be plugged into the predicted query. This is suitable if your model does not predict values. This is set to `False` by default.
|
| 41 |
+
|
| 42 |
## Citations
|
| 43 |
|
| 44 |
```
|