41 questions
0
votes
0
answers
81
views
Perplexity User Location filter dont work
I have a simple question for people used to Perplexity, this is what appears in the perplexity documentation:
https://docs.perplexity.ai/guides/user-location-filter-guide#examples
import requests
url ...
- 55
0
votes
0
answers
72
views
LDA perplexity with train-test split leads to absurd results (best model = 1 topic)
I'm working with LDA on a Portuguese news corpus (~800k documents with an average of 28 words each after cleaning the data), and I’m trying to evaluate topic quality using perplexity.
When I compute ...
- 1
1
vote
1
answer
401
views
Test a Perplexity API key on python
I don't find an efficient way to check if an api key of perplexity is valid, either on python or anything.
Indeed, for openai I do :
def check_openai_api_key(api_key):
openai.api_key = api_key
...
- 31
0
votes
1
answer
96
views
Why is Perplexity not reliable for open domain text generation tasks?
In the paper here, it says that perplexity as an automated metric is not reliable for open domain text generation tasks, but it instead uses lm-score, a model based metric to produce perplexity like ...
2
votes
2
answers
2k
views
How to get perplexity per token rather than average perplexity?
I can get the perplexity of a whole sentence from here:
device = "cuda"
from transformers import GPT2LMHeadModel, GPT2TokenizerFast
device = "cuda"
model_id = "gpt2"
...
- 2,651
1
vote
0
answers
319
views
Perplexity metric for GPT2 model is lower for non-English text
I am currently working on a project of calculating perplexities of various causal LLMs for different languages to estimate their behaviour if there is an input in a form of the language, that ...
1
vote
1
answer
557
views
Challenges when calculating perplexity: using bidirectional models, and dealing with large text size and values, are my approaches reasonable?
Challenges when calculating perplexity: is my approach reasonable?
I am trying to find a pre-trained language model that will work best for my text. The text is pretty specific in its language and ...
- 29
1
vote
1
answer
848
views
Why is perplexity calculation giving different results for the same input?
I'm following Huggingface doc on calculating the perplexity of fixed-length models. I'm trying to verify that the formula works for various strings and I'm getting odd behavior. In particular, they ...
- 2,651
0
votes
1
answer
134
views
What is the held-out probability in Mallet LDA? How can we calculate Perplexity by the held-out probability?
I am new to mallet. Now I would like to get the perplexity scores for 10-100 topics in my lda model so I run the held-our probability, it gives me the value of -8926490.73103205 for topic=100, which ...
- 1
2
votes
1
answer
1k
views
How to calculate perplexity of BERTopic?
Is there a way to calculate the perplexity of BERTopic? I am unable to find any such thing in the BERTopic library and in other places.
- 124
1
vote
1
answer
644
views
Large Language Model Perplexity
i am currently using GPT-3 and i am trying to compare its capabilities to related language models for my masters thesis.
Unfortunatly GPT-3 is an API based application, so i am not really able to ...
- 83
3
votes
0
answers
239
views
Diagnostics (perplexity, LogLik, etc) for LDA topic model with textmodel_seededLDA package in R
I'm using the seededLDA package to do an LDA topic model. However, all of the packages and functions I've found to compute perplexity, log likelihood, exclusivity, etc (and other diagnostic tools) don'...
- 53
0
votes
2
answers
3k
views
How to find perplexity of bigram if probability of given bigram is 0
Given the formula to calculate the perplexity of a bigram (and probability with add-1 smoothing),
Probability
How does one proceed when one of the probabilities of the word per in the sentence to ...
- 1,159
2
votes
0
answers
438
views
Why am I randomly getting super high perplexities?
A few days ago I finished writing a word prediction program that tests both LSTM and GRU models on a given dataset. I test 4 models - 2 LSTM models and 2 GRU models. I wrote the program on Google ...
- 163
0
votes
0
answers
756
views
Gensim perplexity score increases
I am trying to calculate the perplexity score in Spyder for different numbers of topics in order to find the best model parameters with gensim.
However, the perplexity score is not decreasing as it is ...
- 15