What are context lengths?

- [Instructor] When having a conversation with a large language model, how can you figure out how much of the conversation it remembers? That's what context windows are all about. Now, if you remember, a prompt is the text you input into the model and it's made up of a couple of tokens. The completion is the text outputted from the model, which also makes up a couple of tokens. The sum of the tokens of the prompt and the completions is known as the "context window" or "context length". Now, the longer the context length, the more informational background the model has for generating a response. For a language model to produce a more meaningful and relevant response, it needs to be able to take an entire conversation into consideration. Now, different large language models will have different context lengths. Chat GPT currently has a context length of around 4,000 tokens. GPT-4 has 8,000 tokens. GPT-4-32k has 32,000 tokens. Now, with a context length of 32,000 tokens, you could provide almost 50 pages of text as input to a model and get it to summarize it. Alternatively, you can get it to generate more text if you wanted to create a short story. Claude 2, a large language model from Anthropic, has a context window of 100,000 tokens. Now, what can you do with a context length of 100,000 tokens? Let's take a children's novel like the 50,000 word "Black Beauty" by Anna Sewell. Now, with 50,000 words, this will fit into Anthropic's 100,000 token context window. So, let me go ahead and copy the entire novel. Now, the question is, can we get a large language model to answer a specific question about this novel? So, let's head over to Anthropic's Playground to see how this works. So, I'm now in Anthropic's Playground and I have access to the Claude 2 model. Now, before I do anything, I want to query Anthropic's model about my question. I want to make sure that if Claude 2 answers the question, it isn't in relation to "Black Beauty". So the question is, "Who is Duchess?" Rephrase. So, you can see, the response back is, "Duchess was a character in the 1970 animated film 'The Aristocats', produced by Walt Disney Productions. She was a mother cat with three kittens," and so on. And then, right over at the end, it says, "So, in summary, Duchess was the main mother cat character in the classic Disney animated film, 'The Aristocats', set in the early 20th century Paris." Now, Claude 2 has almost certainly seen "Black Beauty" as part of its training data, but its initial response to my question is not related to "Black Beauty", which is what I wanted to confirm. And let me go ahead and start a new chat and ask the question, "Who is Duchess?" And I'm going to go ahead and paste the entire novel in the context window. And let's give this a couple of seconds. Now, that's pretty impressive. The response back from Claude 2 is, "Based on the contents of the book excerpt, Duchess seems to be the Black Beauty's mother," and some key details. And if I head over to the end, it says, "So, in summary, Duchess has identified as Black Beauty's mother, gives him advice when he is young, and distressed when they're separated after "Black Beauty" is sold," and so on. Now, it would've taken me over three hours to read this book and here I can get my answer in just under a minute. All right, so we've looked at context links of large language models and how they vary from around 4,000 tokens for some models to 100,000 tokens. Go ahead and try and upload a large book or a knowledge base, anything that has less than 75,000 words, into Anthropic's Claude 2 model and query it and post your findings on LinkedIn and tag me.