NAN values in loss, perplexity and next_token_perplexity

[msmmpts]

Hi team,

I was attempting to fine tune a Japanese LLM for a domain specific task. Training completed without any issues but the logs show NAN values for loss, perplexity, ..etc. I am uploading description.json and training_statistics.json for review.

description.json
training_statistics.json

Any help regarding the same would be really appreciated.

Thanks in advance

[arnavgarg1]

Hi @msmmpts, sorry for the delayed response on this discussion thread. Is this still an issue you're running into with the latest stable release of Ludwig (0.9.3)? Happy to help debug!