Trainability is not explicitly stated
Hello, with llm.encode_document(text, convert_to_tensor=True) I have
return F.linear(input, self.weight, self.bias)
RuntimeError: Inference tensors cannot be saved for backward. To work around you can make a clone to get a normal tensor and use it in autograd.
which means the encoding cannot propagate gradients.
But this works FYI :
torch.stack(self.llm.encode_document(text, convert_to_numpy=False), dim=0)
Hi @edmond ,
Thanks for bringing this to our attention, the issue is arises because the llm.encode_document() function, when it returns a PyTorch tensor, is likely performing an operation in inference mode, which means it's not set up to track the computational graph needed for back propagation and gradient calculation.
When you try to use the output of llm.encode_document(..., convert_to_tensor=True) directly in an operation like F.linear() within a neural network's forward pass, PyTorch sees that the tensor doesn't have a history for gradient tracking. This is because it was created by an operation that was either marked as requires_grad=False or was performed inside a torch.no_grad() context. The RuntimeError is a safety mechanism to prevent you from getting an unexpected result where no gradients are calculated for that part of your model.
The reason torch.stack(self.llm.encode_document(text, convert_to_numpy=False), dim=0) works is because it's a fundamental difference in how the tensor is being used.
Solution:
tensor.clone() This method creates a full copy of the tensor. This copy has a gradient history and can be used in subsequent operations.
Please find the attached gist file for your reference where we could able to see the embeddings generation after using the .clone() method.
Thanks.