Commit 
							
							·
						
						723775e
	
1
								Parent(s):
							
							58bbcf2
								
Update README
Browse files
    	
        README.md
    CHANGED
    
    | 
         @@ -95,33 +95,33 @@ Performance of this model compared to the top base models on the [MTEB leaderboa 
     | 
|
| 95 | 
         | 
| 96 | 
         
             
            The following datasets were used to evaluate model performance.
         
     | 
| 97 | 
         | 
| 98 | 
         
            -
            - [PubMed QA](https://huggingface.co/datasets/ 
     | 
| 99 | 
         
             
              - Subset: pqa_labeled, Split: train, Pair: (question, long_answer)
         
     | 
| 100 | 
         
            -
            - [PubMed Subset](https://huggingface.co/datasets/ 
     | 
| 101 | 
         
             
              - Split: test, Pair: (title, text)
         
     | 
| 102 | 
         
            -
            - [PubMed Summary](https://huggingface.co/datasets/scientific_papers)
         
     | 
| 103 | 
         
             
              - Subset: pubmed, Split: validation, Pair: (article, abstract)
         
     | 
| 104 | 
         | 
| 105 | 
         
             
            Evaluation results from the original model are shown below for reference. The [Pearson correlation coefficient](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient) is used as the evaluation metric.
         
     | 
| 106 | 
         | 
| 107 | 
         
             
            | Model                                                                         | PubMed QA | PubMed Subset | PubMed Summary | Average   |
         
     | 
| 108 | 
         
             
            | ----------------------------------------------------------------------------- | --------- | ------------- | -------------- | --------- | 
         
     | 
| 109 | 
         
            -
            | [all-MiniLM-L6-v2](https://hf.co/sentence-transformers/all-MiniLM-L6-v2) 
     | 
| 110 | 
         
            -
            | [bge-base-en-v1.5](https://hf.co/BAAI/bge- 
     | 
| 111 | 
         
            -
            | [gte-base](https://hf.co/thenlper/gte-base) 
     | 
| 112 | 
         
            -
            | [**pubmedbert-base-embeddings**](https://hf.co/neuml/pubmedbert-base-embeddings) | **93.27** | **97. 
     | 
| 113 | 
         
            -
            | [S-PubMedBert-MS-MARCO](https://hf.co/pritamdeka/S-PubMedBert-MS-MARCO) 
     | 
| 114 | 
         | 
| 115 | 
         
             
            See the table below for evaluation results per dimension for `pubmedbert-base-embeddings-matryoshka`.
         
     | 
| 116 | 
         | 
| 117 | 
         
             
            | Model               | PubMed QA | PubMed Subset | PubMed Summary | Average   |
         
     | 
| 118 | 
         
             
            | --------------------| --------- | ------------- | -------------- | --------- | 
         
     | 
| 119 | 
         
            -
            | Dimensions =  64    | 92.16     |  
     | 
| 120 | 
         
            -
            | Dimensions = 128    | 92.80     | 96. 
     | 
| 121 | 
         
            -
            | Dimensions = 256    | 93.11     | 96. 
     | 
| 122 | 
         
            -
            | Dimensions = 384    | 93.42     |  
     | 
| 123 | 
         
            -
            | Dimensions = 512    | 93.37     |  
     | 
| 124 | 
         
            -
            | **Dimensions = 768**    | **93.53**     | ** 
     | 
| 125 | 
         | 
| 126 | 
         
             
            This model performs slightly better overall compared to the original model.
         
     | 
| 127 | 
         | 
| 
         | 
|
| 95 | 
         | 
| 96 | 
         
             
            The following datasets were used to evaluate model performance.
         
     | 
| 97 | 
         | 
| 98 | 
         
            +
            - [PubMed QA](https://huggingface.co/datasets/qiaojin/PubMedQA)
         
     | 
| 99 | 
         
             
              - Subset: pqa_labeled, Split: train, Pair: (question, long_answer)
         
     | 
| 100 | 
         
            +
            - [PubMed Subset](https://huggingface.co/datasets/awinml/pubmed_abstract_3_1k)
         
     | 
| 101 | 
         
             
              - Split: test, Pair: (title, text)
         
     | 
| 102 | 
         
            +
            - [PubMed Summary](https://huggingface.co/datasets/armanc/scientific_papers)
         
     | 
| 103 | 
         
             
              - Subset: pubmed, Split: validation, Pair: (article, abstract)
         
     | 
| 104 | 
         | 
| 105 | 
         
             
            Evaluation results from the original model are shown below for reference. The [Pearson correlation coefficient](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient) is used as the evaluation metric.
         
     | 
| 106 | 
         | 
| 107 | 
         
             
            | Model                                                                         | PubMed QA | PubMed Subset | PubMed Summary | Average   |
         
     | 
| 108 | 
         
             
            | ----------------------------------------------------------------------------- | --------- | ------------- | -------------- | --------- | 
         
     | 
| 109 | 
         
            +
            | [all-MiniLM-L6-v2](https://hf.co/sentence-transformers/all-MiniLM-L6-v2)           | 90.40     | 95.92         | 94.07          | 93.46     |
         
     | 
| 110 | 
         
            +
            | [bge-base-en-v1.5](https://hf.co/BAAI/bge-base-en-v1.5)                            | 91.02     | 95.82         | 94.49          | 93.78     |
         
     | 
| 111 | 
         
            +
            | [gte-base](https://hf.co/thenlper/gte-base)                                        | 92.97     | 96.90         | 96.24          | 95.37     |
         
     | 
| 112 | 
         
            +
            | [**pubmedbert-base-embeddings**](https://hf.co/neuml/pubmedbert-base-embeddings) | **93.27** | **97.00**     | **96.58**      | **95.62** |
         
     | 
| 113 | 
         
            +
            | [S-PubMedBert-MS-MARCO](https://hf.co/pritamdeka/S-PubMedBert-MS-MARCO)            | 90.86     | 93.68         | 93.54          | 92.69     |
         
     | 
| 114 | 
         | 
| 115 | 
         
             
            See the table below for evaluation results per dimension for `pubmedbert-base-embeddings-matryoshka`.
         
     | 
| 116 | 
         | 
| 117 | 
         
             
            | Model               | PubMed QA | PubMed Subset | PubMed Summary | Average   |
         
     | 
| 118 | 
         
             
            | --------------------| --------- | ------------- | -------------- | --------- | 
         
     | 
| 119 | 
         
            +
            | Dimensions =  64    | 92.16     | 96.14         | 95.67          | 94.66     |
         
     | 
| 120 | 
         
            +
            | Dimensions = 128    | 92.80     | 96.58         | 96.22          | 95.20     |
         
     | 
| 121 | 
         
            +
            | Dimensions = 256    | 93.11     | 96.82         | 96.53          | 95.49     |
         
     | 
| 122 | 
         
            +
            | Dimensions = 384    | 93.42     | 97.00         | 96.61          | 95.68     |
         
     | 
| 123 | 
         
            +
            | Dimensions = 512    | 93.37     | 97.07         | 96.61          | 95.68     |
         
     | 
| 124 | 
         
            +
            | **Dimensions = 768**    | **93.53**     | **97.13**         | **96.70**          | **95.79**     |
         
     | 
| 125 | 
         | 
| 126 | 
         
             
            This model performs slightly better overall compared to the original model.
         
     | 
| 127 | 
         |