Understanding how pretraining in SMILES lead good performance in predicting molecular propeties

MOLFormer, from this paper: Large-scale chemical language representations, first pre-trained on larger SMILES data and then fine-tuned on other tasks. It can achieve better performance. I wonder if we can use mechanistic interpretability method to understand what information it learned during pretraining is useful for later inference time task. Here is what the paper claims: specifically through the lens of attention, demonstrate that MoLFormer trained on chemical SMILES indeed learns the spatial relationships between atoms within a molecule. Here is the model: https://github.com/IBM/molformer