Van Tuan DANG
commited on
Commit
·
a2d8783
1
Parent(s):
4afd7f7
feat: Add Readmel
Browse files
README.md
ADDED
|
@@ -0,0 +1,97 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
pipeline_tag: sentence-similarity
|
| 3 |
+
language: fr
|
| 4 |
+
datasets:
|
| 5 |
+
- stsb_multi_mt
|
| 6 |
+
tags:
|
| 7 |
+
- Text
|
| 8 |
+
- Sentence Similarity
|
| 9 |
+
- Sentence-Embedding
|
| 10 |
+
- camembert-base
|
| 11 |
+
license: apache-2.0
|
| 12 |
+
model-index:
|
| 13 |
+
- name: sentence-camembert-base by Van Tuan DANG
|
| 14 |
+
results:
|
| 15 |
+
- task:
|
| 16 |
+
name: Sentence-Embedding
|
| 17 |
+
type: Text Similarity
|
| 18 |
+
dataset:
|
| 19 |
+
name: Text Similarity fr
|
| 20 |
+
type: stsb_multi_mt
|
| 21 |
+
args: fr
|
| 22 |
+
metrics:
|
| 23 |
+
- name: Test Pearson correlation coefficient
|
| 24 |
+
type: Pearson_correlation_coefficient
|
| 25 |
+
value: xx.xx
|
| 26 |
+
---
|
| 27 |
+
|
| 28 |
+
## {Model}
|
| 29 |
+
|
| 30 |
+
Cross-Encoder for sentence-similarity
|
| 31 |
+
|
| 32 |
+
This model was trained using [sentence-transformers](https://www.SBERT.net) Cross-Encoder class.
|
| 33 |
+
|
| 34 |
+
## Training Data
|
| 35 |
+
This model was trained on the [STS benchmark dataset](https://huggingface.co/datasets/stsb_multi_mt/viewer/fr/train). The model will predict a score between 0 and 1 how for the semantic similarity of two sentences.
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
## Usage (Sentence-Transformers)
|
| 39 |
+
|
| 40 |
+
Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
|
| 41 |
+
|
| 42 |
+
```
|
| 43 |
+
pip install -U sentence-transformers
|
| 44 |
+
```
|
| 45 |
+
|
| 46 |
+
Then you can use the model like this:
|
| 47 |
+
|
| 48 |
+
```python
|
| 49 |
+
from sentence_transformers import CrossEncoder
|
| 50 |
+
model = CrossEncoder('model_name', max_length=128)
|
| 51 |
+
scores = model.predict([('Un avion est en train de décoller.', "Un homme joue d'une grande flûte."), ("Un homme étale du fromage râpé sur une pizza.", "Une personne jette un chat au plafond") ])
|
| 52 |
+
|
| 53 |
+
```
|
| 54 |
+
## Evaluation
|
| 55 |
+
The model can be evaluated as follows on the French test data of stsb.
|
| 56 |
+
```python
|
| 57 |
+
from sentence_transformers import SentenceTransformer
|
| 58 |
+
from sentence_transformers.readers import InputExample
|
| 59 |
+
from sentence_transformers.evaluation import EmbeddingSimilarityEvaluator
|
| 60 |
+
from datasets import load_dataset
|
| 61 |
+
def convert_dataset(dataset):
|
| 62 |
+
dataset_samples=[]
|
| 63 |
+
for df in dataset:
|
| 64 |
+
score = float(df['similarity_score'])/5.0 # Normalize score to range 0 ... 1
|
| 65 |
+
inp_example = InputExample(texts=[df['sentence1'],
|
| 66 |
+
df['sentence2']], label=score)
|
| 67 |
+
dataset_samples.append(inp_example)
|
| 68 |
+
return dataset_samples
|
| 69 |
+
|
| 70 |
+
# Loading the dataset for evaluation
|
| 71 |
+
df_dev = load_dataset("stsb_multi_mt", name="fr", split="dev")
|
| 72 |
+
df_test = load_dataset("stsb_multi_mt", name="fr", split="test")
|
| 73 |
+
|
| 74 |
+
# Convert the dataset for evaluation
|
| 75 |
+
|
| 76 |
+
# For Dev set:
|
| 77 |
+
dev_samples = convert_dataset(df_dev)
|
| 78 |
+
val_evaluator = CECorrelationEvaluator.from_input_examples(dev_samples, name='sts-dev')
|
| 79 |
+
val_evaluator(model, output_path="./")
|
| 80 |
+
|
| 81 |
+
# For Test set
|
| 82 |
+
|
| 83 |
+
test_samples = convert_dataset(df_test)
|
| 84 |
+
test_evaluator = CECorrelationEvaluator.from_input_examples(test_samples, name='sts-test')
|
| 85 |
+
test_evaluator(models, output_path="./")
|
| 86 |
+
```
|
| 87 |
+
**Test Result**:
|
| 88 |
+
The performance is measured using Pearson and Spearman correlation:
|
| 89 |
+
- On dev
|
| 90 |
+
| Model | Pearson correlation | Spearman correlation | #params |
|
| 91 |
+
| ------------- | ------------- | ------------- |------------- |
|
| 92 |
+
| [dangvantuan/CrossEncoder-camembert-large](https://huggingface.co/dangvantuan/CrossEncoder-camembert-large)| 90.11 |90.01 | 336M |
|
| 93 |
+
|
| 94 |
+
- On test
|
| 95 |
+
| Model | Pearson correlation | Spearman correlation |
|
| 96 |
+
| ------------- | ------------- | ------------- |
|
| 97 |
+
| [dangvantuan/CrossEncoder-camembert-large](https://huggingface.co/dangvantuan/CrossEncoder-camembert-large)| 88.16 | 87.57|
|