Observability – devopscorner.id

GenerativeAI: Transforming The Landscape of Observability

Webinars AWS User Group Indonesia

Abstract

In today’s digital ecosystem, the complexity and scale of systems have reached unprecedented levels, presenting immense challenges in ensuring observability and maintaining optimal performance. Generative AI, a cutting-edge technology, is revolutionizing the landscape of observability by offering innovative solutions to these challenges.

This presentation explores the transformative power of Generative AI in enhancing observability across various domains. By leveraging advanced machine learning algorithms, Generative AI enables the creation of synthetic data that closely mimics real-world scenarios, facilitating comprehensive testing and analysis in controlled environments.

Furthermore, Generative AI empowers anomaly detection and predictive analytics by generating synthetic data patterns that represent potential system failures or irregularities. This proactive approach enables organizations to identify and address issues before they escalate, thus minimizing downtime and optimizing performance.

Moreover, Generative AI fosters the development of adaptive monitoring systems capable of learning from dynamic environments and evolving threats. By continuously generating and analyzing synthetic data, these systems adapt in real-time, ensuring robust observability even in the face of rapidly changing conditions.

Through case studies and practical examples, this presentation demonstrates how Generative AI is reshaping the observability landscape, offering unparalleled insights, and driving innovation across industries. From IT infrastructure to cybersecurity and beyond, Generative AI is revolutionizing how we monitor, analyze, and optimize complex systems in the digital age. Join us as we explore the limitless potential of Generative AI in transforming observability and unlocking new frontiers of performance and reliability.

Live Streaming Youtube:

Saturday, May 18th 2024 at 7.30pm (19.30 WIB)
bit.ly/observabilityai

#aws #awscloud #awsusergroup #awscommunitybuilders #generativeai #observability #xapiens #devops #devopscorner #devopscornerid

OpenSearch Operator

Step by step instructions: How to deploy OpenSearch Operator inside Kubernetes Cluster (EKS)

Prerequirements

Install Helm

brew install helm
brew install helmfile
brew install kubectl

Add Plugin Helm

helm plugin install https://github.com/databus23/helm-diff
helm plugin install https://github.com/hypnoglow/helm-s3.git

Add Helm Repository S3 Bucket

### LAB ### helm s3 init s3://devopscorner-helm-chart/lab AWS_REGION=ap-southeast-1 helm repo add devopscorner-lab s3://devopscorner-helm-chart/lab ### STAGING ### helm s3 init s3://devopscorner-helm-chart/staging AWS_REGION=ap-southeast-1 helm repo add devopscorner-staging s3://devopscorner-helm-chart/staging ### PRODUCTION ### helm s3 init s3://devopscorner-helm-chart/prod AWS_REGION=ap-southeast-1 helm repo add devopscorner s3://devopscorner-helm-chart/prod helm repo update

Update Repository

helm repo add stable https://charts.helm.sh/stable
helm repo add opensearch-operator https://opster.github.io/opensearch-k8s-operator/ helm repo update helm repo list

NAME URL
opensearch-operator https://opster.github.io/opensearch-k8s-operator/ stable https://charts.helm.sh/stable

Create Namespace

kubectl create namespace observability

Install OpenSearch Operator

helm install opsearch opensearch-operator/opensearch-operator --create-namespace -n observability

NAME: opsearch LAST DEPLOYED: Sat Nov 4 09:08:25 2023 NAMESPACE: observability STATUS: deployed REVISION: 1 TEST SUITE: None

Install Cluster OpenSearch

Change this default OpenSearch-Cluster.yaml

apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  name: my-cluster
  namespace: default
spec:
  general:
    version: "1.3.0"
    httpPort: 9200
    vendor: opensearch
    serviceName: my-cluster
    monitoring:
      enable: true
    pluginsList: ["repository-s3"]
  dashboards:
    version: "1.3.0"
    enable: true
    replicas: 2
    resources:
      requests:
        memory: "1Gi"
        cpu: "500m"
      limits:
        memory: "1Gi"
        cpu: "500m"
  confMgmt:
    smartScaler: true
  nodePools:
    - component: masters
      replicas: 3
      diskSize: "30Gi"
      nodeSelector:
      resources:
        requests:
          memory: "2Gi"
          cpu: "500m"
        limits:
          memory: "2Gi"
          cpu: "500m"
      roles:
        - "master"
        - "data"
    - component: nodes
      replicas: 3
      diskSize: "30Gi"
      nodeSelector:
      resources:
        requests:
          memory: "2Gi"
          cpu: "500m"
        limits:
          memory: "2Gi"
          cpu: "500m"
      roles:
        - "data"
    - component: coordinators
      replicas: 3
      diskSize: "30Gi"
      nodeSelector:
      resources:
        requests:
          memory: "2Gi"
          cpu: "500m"
        limits:
          memory: "2Gi"
          cpu: "500m"
      roles:
        - "ingest"

LAB Configuration (Simple Cluster)

apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  name: opsearch
  namespace: observability
spec:
  general:
    version: "1.3.0"
    httpPort: 9200
    vendor: opensearch
    serviceName: opsearch
    monitoring:
      enable: true
    pluginsList: ["repository-s3"]
  dashboards:
    version: "1.3.0"
    enable: true
    replicas: 1
    nodeSelector:
      node: devopscorner-monitoring
    resources:
      requests:
        memory: "200Mi"
        cpu: "500m"
      limits:
        memory: "512Mi"
        cpu: "500m"
  confMgmt:
    smartScaler: true
  nodePools:
    - component: masters
      replicas: 2
      diskSize: "10Gi"
      nodeSelector:
        node: devopscorner-monitoring
      resources:
        requests:
          memory: "2Gi"
          cpu: "500m"
        limits:
          memory: "2Gi"
          cpu: "500m"
      roles:
        - "master"
        - "data"

LAB Configuration (HA Cluster)

apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  name: opsearch
  namespace: observability
spec:
  general:
    version: "1.3.0"
    httpPort: 9200
    vendor: opensearch
    serviceName: opsearch
    monitoring:
      enable: true
    pluginsList: ["repository-s3"]
  dashboards:
    version: "1.3.0"
    enable: true
    replicas: 1
    nodeSelector:
      node: devopscorner-monitoring
    resources:
      requests:
        memory: "200Mi"
        cpu: "500m"
      limits:
        memory: "512Mi"
        cpu: "500m"
  confMgmt:
    smartScaler: true
  nodePools:
    - component: masters
      replicas: 2
      diskSize: "30Gi"
      nodeSelector:
        node: devopscorner-monitoring
      resources:
        requests:
          memory: "2Gi"
          cpu: "500m"
        limits:
          memory: "2Gi"
          cpu: "500m"
      roles:
        - "master"
        - "data"
    - component: nodes
      replicas: 2
      diskSize: "30Gi"
      nodeSelector:
        node: devopscorner-monitoring
      resources:
        requests:
          memory: "2Gi"
          cpu: "500m"
        limits:
          memory: "2Gi"
          cpu: "500m"
      roles:
        - "data"
    - component: coordinators
      replicas: 2
      diskSize: "30Gi"
      nodeSelector:
        node: devopscorner-monitoring
      resources:
        requests:
          memory: "2Gi"
          cpu: "500m"
        limits:
          memory: "2Gi"
          cpu: "500m"
      roles:
        - "ingest"

Production Configuration (HA Cluster)

apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  name: opsearch
  namespace: observability
spec:
  general:
    version: "1.3.0"
    httpPort: 9200
    vendor: opensearch
    serviceName: opsearch
    monitoring:
      enable: true
    pluginsList: ["repository-s3"]
  dashboards:
    version: "1.3.0"
    enable: true
    replicas: 2
    nodeSelector:
      node: devopscorner-monitoring
    resources:
      requests:
        memory: "200Mi"
        cpu: "500m"
      limits:
        memory: "1Gi"
        cpu: "500m"
  confMgmt:
    smartScaler: true
  nodePools:
    - component: masters
      replicas: 3
      diskSize: "30Gi"
      nodeSelector:
        node: devopscorner-monitoring
      resources:
        requests:
          memory: "2Gi"
          cpu: "500m"
        limits:
          memory: "2Gi"
          cpu: "500m"
      roles:
        - "master"
        - "data"
    - component: nodes
      replicas: 3
      diskSize: "30Gi"
      nodeSelector:
        node: devopscorner-monitoring
      resources:
        requests:
          memory: "2Gi"
          cpu: "500m"
        limits:
          memory: "2Gi"
          cpu: "500m"
      roles:
        - "data"
    - component: coordinators
      replicas: 3
      diskSize: "30Gi"
      nodeSelector:
        node: devopscorner-monitoring
      resources:
        requests:
          memory: "2Gi"
          cpu: "500m"
        limits:
          memory: "2Gi"
          cpu: "500m"
      roles:
        - "ingest"

Apply manifest

kubectl create -f opensearch-cluster.yaml -n observability

Access Portforward

kubectl get po -n observability  

NAME READY STATUS RESTARTS AGE alertmanager-prometheus-kube-prometheus-alertmanager-0 2/2 Running 0 40m opsearch-coordinators-0 1/1 Running 0 9m57s opsearch-coordinators-1 1/1 Running 0 10m opsearch-dashboards-7fcc5595c7-fhf28 1/1 Running 0 22m opsearch-masters-0 1/1 Running 0 10m opsearch-masters-1 1/1 Running 0 13m opsearch-nodes-0 1/1 Running 0 9m59s opsearch-nodes-1 1/1 Running 0 13m opsearch-opensearch-operator-controller-manager-7cc6dd6fd8qx5xd 2/2 Running 0 37m

kubectl get po opsearch-dashboards-7fcc5595c7-fhf28 -n observability

NAME READY STATUS RESTARTS AGE opsearch-dashboards-7fcc5595c7-fhf28 1/1 Running 0 23m

kubectl get po opsearch-dashboards-7fcc5595c7-fhf28 -n observability

... Containers: dashboards:
Container ID: containerd://02966406e8e5d2c9cef7c1e139b74887537386111374e29cf5c50ab3cbda19ae
Image: docker.io/opensearchproject/opensearch-dashboards:1.3.0
Image ID: docker.io/opensearchproject/opensearch-dashboards@sha256:7dcc706ab6c71ab00013e341246e7a701c11c61a7668e4dbecd298d6d7aef758
Port: 5601/TCP
Host Port: 0/TCP ...

kubectl port-forward opsearch-dashboards-7fcc5595c7-fhf28 5601:8080 -n observability

Forwarding from 127.0.0.1:5601 -> 8080 Forwarding from [::1]:5601 -> 8080

UserName: admin Password: admin

Setup Pattern

Goto Stack Management

Create Index Pattern

Select Index Pattern

Setup Timestamp for sorting Index Pattern

Discover Logs

Using Network LoadBalancer (NLB) via NGINX Controller

Get Deployment Manifest

wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/aws/deploy.yaml

Change Manifest

sed -i 's/externalTrafficPolicy: Local/externalTrafficPolicy: Cluster/g' deploy.yaml

Deploy Manifest

kubectl create -f deploy.yaml

Expose LB for OpenSearch Dashboard

kubectl expose deployment opsearch-dashboards --name=opensearch-lb --type=LoadBalancer --port=80 --target-port=5601 --protocol=TCP --namespace=observability

Helm Release Version

helm list --all-namespaces helm list --namespace=observability

Prometheus Operator

Step by step instructions: How to deploy Prometheus Operator inside Kubernetes Cluster (EKS)

Prerequirements

Install Helm

brew install helm
brew install helmfile
brew install kubectl

Add Plugin Helm

helm plugin install https://github.com/databus23/helm-diff
helm plugin install https://github.com/hypnoglow/helm-s3.git

Add Helm Repository S3 Bucket

Update Repository

helm repo add stable https://charts.helm.sh/stable helm repo add grafana https://grafana.github.io/helm-charts helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update helm repo list

NAME URL prometheus-community https://prometheus-community.github.io/helm-charts grafana https://grafana.github.io/helm-charts stable https://charts.helm.sh/stable

Create Namespace

kubectl create namespace observability

Install Prometheus Operator

helm install prometheus-operator prometheus-community/kube-prometheus-stack --create-namespace -n observability

NAME: prometheus-operator
LAST DEPLOYED: Sun Nov 5 02:03:06 2023
NAMESPACE: observability
STATUS: deployed REVISION: 1 NOTES: kube-prometheus-stack has been installed. Check its status by running: kubectl --namespace observability get pods -l "release=prometheus-operator"

kubectl get po -n observability

NAME READY STATUS RESTARTS AGE alertmanager-prometheus-kube-prometheus-alertmanager-0 2/2 Running 0 32m prometheus-grafana-55fb596bf5-5257r 3/3 Running 0 32m prometheus-kube-prometheus-operator-757f8788d4-v6tk5 1/1 Running 0 32m prometheus-kube-state-metrics-898dd9b88-98qlj 1/1 Running 0 32m prometheus-prometheus-kube-prometheus-prometheus-0 2/2 Running 0 32m prometheus-prometheus-node-exporter-llfn2 1/1 Running 0 32m prometheus-prometheus-node-exporter-nrpkq 1/1 Running 0 32m

kubectl --namespace observability get pods -l "release=prometheus-operator"

NAME READY STATUS RESTARTS AGE
prometheus-operator-kube-p-operator-7cc49d6ffb-ktjnv 1/1 Running 0 2m16s
prometheus-operator-kube-state-metrics-797d9866bd-s4xhb 1/1 Running 0 2m16s
prometheus-operator-prometheus-node-exporter-vg42d 1/1 Running 0 2m16s

kc get po -n observability

kubectl get svc -n observability

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 11m prometheus-operated ClusterIP None <none> 9090/TCP 11m prometheus-operator-grafana ClusterIP 172.20.33.136 <none> 80/TCP 11m prometheus-operator-kube-p-alertmanager ClusterIP 172.20.136.150 <none> 9093/TCP,8080/TCP 11m prometheus-operator-kube-p-operator ClusterIP 172.20.219.78 <none> 443/TCP 11m prometheus-operator-kube-p-prometheus ClusterIP 172.20.195.49 <none> 9090/TCP,8080/TCP 11m prometheus-operator-kube-state-metrics ClusterIP 172.20.214.227 <none> 8080/TCP 11m prometheus-operator-prometheus-node-exporter ClusterIP 172.20.230.46 <none> 9100/TCP 11m

kc get svc -n observability

Edit Prometheus Service

Change to type LoadBalancer from ClusterIP

kubectl edit svc prometheus-kube-prometheus-prometheus -n observability

Edit Grafana Service

Change to type LoadBalancer from ClusterIP

kubectl edit svc prometheus-grafana -n observability

Access Load Balancer

Grafana Access

http://a7754204d8c2e41969dfa8134d4a3d78-2128039805.ap-southeast-1.elb.amazonaws.com

UserName: admin Password: prom-operator

Change Credentials Grafana

kubectl get secret -n observability

NAME TYPE DATA AGE alertmanager-prometheus-operator-kube-p-alertmanager Opaque 1 40h alertmanager-prometheus-operator-kube-p-alertmanager-generated Opaque 1 40h alertmanager-prometheus-operator-kube-p-alertmanager-tls-assets-0 Opaque 0 40h alertmanager-prometheus-operator-kube-p-alertmanager-web-config Opaque 1 40h prometheus-kube-prometheus-admission Opaque 3 4d12h prometheus-operator-grafana Opaque 3 40h prometheus-operator-kube-p-admission Opaque 3 40h prometheus-operator-kube-p-prometheus Opaque 0 40h prometheus-prometheus-operator-kube-p-prometheus Opaque 1 40h prometheus-prometheus-operator-kube-p-prometheus-tls-assets-0 Opaque 1 40h prometheus-prometheus-operator-kube-p-prometheus-web-config Opaque 1 40h

Change Base64 Credentials

Change User Encode (admin-user)

echo “devopscorner-admin” | base64 4oCcZGV2b3BzY29ybmVyLWFkbWlu4oCdCg==

Change Password Encode (admin-password)

echo “devopscorner-secret” | base64 4oCcZGV2b3BzY29ybmVyLXNlY3JldOKAnQo=

Edit admin-password and admin-user from encoded base64 string

kubectl edit secret prometheus-operator-grafana -n observability

Prometheus Access

http://af58dac1154de4b57be3d0d63d60936b-925795512.ap-southeast-1.elb.amazonaws.com:9090

Using Jumpods

References:
https://github.com/devopscorner/devopscorner-helm/tree/master/helmfile/jumppod

Test with Curl (Inside Jumpods)

curl prometheus-kube-prometheus-prometheus.observability.svc.cluster.local:9090

curl prometheus-grafana.observability.svc.cluster.local