Running Kafka on the LHM virtual machine - Kafka Tutorial

Welcome to Lesson 1.3, Running Kafka on the LHM Virtual Machine. In this lesson, I'm going to give you an overview of the steps required to get the Linux Hadoop Minimal, or LHM, virtual machine running on your Mac PC or Linux system. And then I'm going to describe how to pull down and extract the notes and example files. Let's begin with the lesson web page I've set up. The URL for this web page should be in the resources section. If it isn't, I've added it right here so you can see it and pause the video and jot it down if need be and enter it into a browser. And that will bring you up to this page. Welcome to Kafka Essentials resource page. And on this page, I've set up as a directory for, or switchboard for everything you'll need to do and work through the examples and get the notes for the course. When you look at this page, it might be slightly different, and that's because I've updated or added a few things. The first thing to call your attention to is the notes files, as I call them. I've made them available in both TGZ, which is tar gzip format and zip format, whichever you prefer to download directly onto your local machine. Now we're going to download these directly into the virtual machine in a few minutes. And if you'd like to look at them yourself, you can do that from here. We're also going to start up and run the virtual machine on, in this case, we're going to use VirtualBox on a Mac mini and the experience will be almost identical to using VirtualBox on a Linux or Windows system with a few additional pieces of software. There is a version for the new Mac M1 and M2 type systems and I'll talk about that in a few minutes. So let's get started with taking a look at this virtual machine. I have two links here and the first one is on a general teaching page where I talk about the Linux Nuke Minimal virtual machine that I use for a lot of different online courses and other live lessons. Basically, the current version is version 2 beta 8. And by the time you download this, it will probably be a newer version, maybe 8.1 or 8.2. Not to worry, that will work with the examples in any case. I've actually listed all the different applications that are installed. And you can see it's quite a long list of data analytics tools that will run on the Linux Hadoop Minimal. I've also included a MD5 to check and make sure your download worked OK. And I want to point out that there's two places to download it. There's a U S location and a European mirror. It's about 11 gigs. So we'll take some time to download next. Let's take a look at the Linux Hadoop minimal installation instructions. And we'll click on this link here. And that brings us up to this page right here. Now I've broken this into two parts, Intel X 86 with virtual box and Apple M1, M2 with UTM. So let's talk about VirtualBox x86 version first. If you have any other issues, you can contact me using this address here and some other addresses I have listed throughout the notes files. And let's scroll down a little bit and take a look at installation steps for Intel x86 based hosts. and this is going to be with VirtualBox. First thing is, you will need a minimum of four cores and four gigs of memory. And actually, I would recommend if you could get, have at least six gigs of memory available for the virtual machine, that will make things go a lot quicker. 70 gigs of disk space will be required as well. In order to make things easy, I have two videos available. one on how to install the LHM on macOS and then one on how to install the LHM on Windows 10. And you will need for Windows something called MobaX Term or PuTTY or you could possibly use PowerShell. I go over all that in the videos and I'll mention that in a little bit. VirtualBox is available for your operating system, whether it be Windows, Mac or Linux from this link here. So you can download and install that. It's entirely free. It's no cost involved. There may be some additional installations for Red Hat systems. You can check this link here. If you're using Red Hat or CentOS, if you're using Windows, you will need something called an SSH client. And that's basically a tool that lets us log into the virtual machine over a text terminal. There are two available that are most popular. The one is PuTTY, and most people have used PuTTY. Although, PuTTY doesn't really provide enough capabilities for everything we want to do, which is why I recommend using MOBAX Term. And like PuTTY, MOBAX Term is freely available, and it will allow us to run X Windows systems on top of our Windows operating system. In addition, if you have Mac OS, you probably want to download Xcorts, which allows you to run Xwindows on top of Mac OS as well. We do have one application that is a Xwindows application, and these tools will let you, MobaXterm and Xcorts will let you run them directly on top of your local operating system. For Linux, you won't need it because it's included with the Linux operating system. A couple other notes here, make sure virtualization is enabled in your BIOS. Most systems do that now. There's a couple other steps and notes here that you can take a look at. Once you download the virtual machine, and this is the OVA version for VirtualBox, then you can start VirtualBox and load the virtual machine. I'm going to go into that in a second here. And a couple other things I mentioned, and I'll show you this. If you have more than four gigs of memory, you can increase the amount of memory for the LHM on your host system. And also some notes about how to shut down the LHM when not in use. And you can take a look at these as well. Also wanted to mention for Apple M1-based hosts, I have a full set of directions here on how to download and load the system. Here is a link to the version that runs on the Mac M1 using UTM, which is freely available, downloadable for the Mac. And this is basically does the same thing that VirtualBox does. It allows you to run virtual machines on the Mac M1 or M2. So I provide complete instructions here and you can follow these along. Once you've got it up and running, then it would work in a similar way on a Mac using the terminal application to log into the virtual machine. Let's go back to our main page. Here we are here. And this link here just took us to where we just were, that secondary instructions page. And let's get the virtual machine started. And then we'll see how we're gonna pull down the Kafka Essentials notes files into the virtual machine. What I'm going to do on my Mac system here is start VirtualBox while VirtualBox is actually running. And you'll notice on the left here are all the virtual machines I have on my system. Now I actually have more than one. I've got several versions, beta nine, beta nine updating. And this is beta 8.1, which is probably the version that you will pull down for this video. When I want to add a virtual machine, I basically go up to file and I click on import appliance. And that's gonna take me to a menu where I can pick a virtual machine or an OVA file, and that's the file that I downloaded. And I go over this in the videos. And once I've got that loaded into my VirtualBox interface here, I can simply start it by clicking this green start button here. Now, the other thing I wanted to mention was if you go into settings, when the machine is powered off, you can see all these machines are powered off. I go into Settings, and I click on System. You can see I can set the base memory higher than the default 4 meg. And in this case, I set it to 6 gigs. I have 32 gigs, so I could actually make this bigger if I wanted to. And I can also click on Processor here. And I've got it set for four processors. and execution cap set to 100%. These are the only two values I recommend that you play with. Setting some of these other things can cause the virtual machine not to run effectively. Okay, so I wanna start this machine and I've highlighted it. I hit start and it's going to open a window, which is actually a terminal into the virtual machine. So this is what I would see when I booted up my machine or a separate machine with a monitor on it. This is what it would show on the monitor. As it starts, it's very quick. And when it's done, I'll get a login prompt up here to log into the machine as a text terminal. And there it is. Now, we're not gonna use this window because it's rather difficult and you can't make it bigger or increase the font size and everything. So we're just gonna ignore this for right now. We'll just actually minimize it. And as a matter of fact, we're not gonna use this interface as we work with a virtual machine. The key thing I wanna point out, it shows that it's running. And now to get to this virtual machine, we need to either open up a terminal on our Mac system or use MOBAX term or PuTTY on Windows. And on Linux, you can just open a terminal window and do the same. So let's do that. I'm going to open up a terminal on my Mac right now. And I've actually already got one open. And you can see this is the application called Terminal. It's default installed on all Macs. And my username on this machine is Deadline. And the name of this machine is Knowther. So this is my desktop Mac that we're looking at now. Since the virtual machine is up and running, It's actually like a separate computer running inside my computer. So I need to connect with it or communicate with it in some way, just like I would a live server out on the internet. And to do that, we're going to use SSH. And this is the SSH client I was talking about. So I'm going to type SSH and the username we're gonna first use is root because this is the first question that everyone asks. what is the root login at, and I'm going to type 127.0.0.1-p2222. Now, if you don't understand what that means, that's okay. It's just a way for me to connect and communicate with the virtual machine that's now running inside my computer. So I hit return, it asked me for a password. The password is Hadoop, all lowercase. And if I've logged in, I get this little banner here telling me the version and so forth. And now I'm running as the root user on the virtual machine. I don't recommend you do this unless you need to change something because you can break a lot of things working as user. I've created a, because you can break a lot of things running as the root user, I've created a user account that we'll work out of for all our examples. What I did want to show you, if I type LS in the root account, you'll see there is a directory called Hadoop Minimal Installation Notes v2 beta 8. If you're interested in how this virtual machine was created and the packages were installed, you can consult this directory. And there's notes and everything that explains how the system was built. That's all I'm going to mention about that. That's just there in case you're interested. So let's exit, type exit, and leave this login. So we've closed the connection. We're back at know their account deadline. And if I scroll this back, and now I type hands-on, because that's the account we're going to use to run our examples in. So I'm going to connect to the virtual machine using user hands-on, and this is a restricted account. It's not as powerful as root, and we won't get ourselves into trouble when we do things. So we do that, and it asks for a password. The password is minimal, and we're logged in to that account. Let me type clear and clear the screen. If I type LS, you'll notice nothing's there. And what we're going to do at this point is pull down the files we need to do the examples. So let's go back to our page right here. And before we do that, I want to mention one thing. If I click on the installation and usage instructions and I scroll down here, Here was the instruction I just did, SSH root. And the first question I get is, what is the root password? And for the x86-based systems, it's Hadoop. For the M1-based systems, it's Hadoop 2023. So the login and passwords for root are right here in the documentation for the LHM. There's some other things you can try when you're root that I don't mention, that I won't mention here. They're here in the instructions if you want to take a look at it. What I did want to show you is here's the hands-on account. Here's the password. So you don't have to worry about remembering passwords. Here's the command I just ran, ssh hands-on, that 127.0.1-p2222. user hands-on, minimal. That's the password. So if you're looking for information about how to do things on the Linux Hadoop minimal, this page is very important and should have everything you're looking for. Now going back again, and this page is on the Kafka Essentials resource page, a link to that page. So let's go back and download the notes. And to do that, we can copy this entire line here. And then we're going to go back to our terminal. And again, we're in the terminal now. And the reason we know is because it says base hands-on local host. It doesn't say know their deadline. We're in the virtual machine. And if we paste in this command and hit return, it's going to download this file, KafkaEssentials.tgz. And it's going to take a few minutes to do that. So we'll just wait till that's all done. OK, so now we're done. And let's clear this out. and I'll be using clear quite a bit to keep our screen nice and neat. And if I type LS to list the local files, now you see there's a file called KafkaEssentials.tgz. What I'm going to want to do is extract this file. This is just like a zip file. It's a little different format, and it contains all the directories and files we're going to need to do these live lessons. So to get that command, let me go back to the main page. and copy this command and go back to my terminal and paste this in and basically what this is going to do is extract this file this TGZ file using the tar command which is stands for tape archive in Unix and there's some other things telling it to extract and uncompress and so forth and we do that and we're going to see all this gobbledygook go across the screen which are the files that are being extracted so let's just clear this and now if I type ls you see there's also this Kafka Essentials v1 which is not this this is a file this is actually a directory and in my system the directories show up as blue. If I change to that directory like that and I type ls now you see I have all these subdirectories lesson 1, 2, 3, 4, 5, 6, 7, 8 with subdirectories so let's take a look at lesson zero one background so lesson 1.3 and 1.4 I have information and data that we're going to use for those lessons and as a matter of fact in lesson 1.3 which is where we are now I have something called a notes file. And if I use the VI editor to examine that file, type VI, it's going to open that. Let me go to the end here. And it's going to show me the contents of that file. This is a totally ASCII text file. Each file will have Kafka Essentials. It will say the lesson, and this is where we are, lesson 1.3, and essentially what it provides here are the notes needed to do what we just did. So it's kind of redundant and covers what I just showed you. And I also have this posted on the main page if you want to look at these in more detail. Just as an aside, all the notes files, the text will start with a hashtag, and any output or commands will be preceded with spaces. So that's a way so we can separate them. And this will make more sense as we go along. So let's quit this. Now, if you're not totally happy with using the VI editor and most people aren't, the Linux to do minimal also includes an editor called nano and I can type nano and the same path to the notes.txt file, and it will open up the file in the nano editor, which is a lot easier. It's a lot easier to navigate. There are control keys. So control X is exit, control G is help, and I can get commands on how to use the editor. So it's a lot more intuitive, a lot more easy to use. you can use either VI or Nano to look at text files on the Linux Hadoop Minimal. So let's control X and quit. So that's where we're going to stop for now. If you get to this point, you were ready to go. Everything should be working. You've got your files downloaded, extracted. We've got our lessons here. There's a couple of readmes I wanted to mention too. If you need to contact me, there's information there, copyright information, and some things about Python libs we're going to add later on. And we'll come to that in a little bit. So with that, we'll finish there and move on to the next lesson where we're actually going to use Kafka.