Cloud Devops Automation

Orchestrator is a free and opensource mysql high availability and replication management tool whose major functionalities includes MySQL/MariaDB MasterDB failover in seconds (10-20 secs) considering our requirements and managing replication topology (Changing Replication Architecture by drag-drop or via Orchestrator CLI) with ease. Orchestrator has the following features:- 1. Discovery:- It actively crawls through the topologies and maps them and can read the basic replication status and configuration and provides slick visualisation of topologies including replication problems. 2. Refactoring:- Understands replication rules, knows about binlog file:position, GTID, Pseudo GTID, Binlog Servers.Refactoring replication topologies can be a matter of drag & drop a replica under another master. 3. Recovery:- It can detect master and intermediate master failures. It can be configured to perform automated recovery or allow user to choose manual recovery. Master failover ...

Benchmarking helps to establish the performance parameters for a mysql database on different instance sizes in AWS Cloud TOOL:                          sysbench MySQL Version:          5.6.39-log MySQL Community Server (GPL) EC2 Instance type:     r4.xlarge (30 GB RAM, 4 CPU)  DB Size:                      25 GB (10 tables) MySQL 5.6.39-log MySQL Community Server (GPL) sysbench /usr/share/sysbench/oltp_read_write.lua --threads=32 --events=0 --time=120 --mysql-user=root --mysql-password=XXXXXXXXX --mysql-port=3306 --tables=10 --delete_inserts=10 --index_updates=10 --non_index_updates=0 --table-size=10000000 --db-ps-mode=disable --report-interval=5 --mysql-host=10.1.2.3 run sysbench 1.0.15 (using bundled LuaJIT 2.1.0-beta2) Running the test with following options: Number of threads: 32 Report intermediate results every...

Note:   Latest AWS CLI version is required. Once you've installed and configured the AWS CLI, and created an AMI, please follow the steps below: 1) SSH into your Ubuntu instance. 2) Upgrade the kernel on your instance by running the command:     sudo apt-get update && sudo apt-get install linux-image-generic 3) Stop the instance from the console or AWS CLI. 4) Using the AWS CLI, modify the instance attributes to enable  ENA  by running the command below:     aws ec2 modify-instance-attribute --instance-id -- ena -support --region ap-southeast-1 5) Using the AWS CLI, modify the instance attributes to change to the desired instance type (for example: m5.large)     aws ec2 modify-instance-attribute --instance-id --instance-type m5.large --region ap-southeast-1 6) Start your instance from the console or the AWS CLI. Once the instance boots, please confirm if the  ENA  module is in use on the network interface by runni...

Usually when a table is throttled while the consumed capacity is well below the provisioned capacity, it is an indicator of hot partition or a “burst” of read or write activity. Hot partition means one single partition is accessed more than other partitions and hence more capacity is consumed by that partition. Hot partition can be caused by unevenly distributed data across partitions due to hot partition key. A “burst” of read or write activity occurs when workloads rely on short periods of time with high usage, for example batch data upload.

Propagate the tags from a instance in an autoscaling group to the EBS volumes attached. Although autoscaling group allows to apply tags to the instances this doesn't propagate to the instance volumes. So some scripting on user data section is needed during instance launch to properly tag the volumes of instances created by the autoscaling group. You can use the below script to apply the tags to the EBS Volumes in an autoscaling group which should be created in the userdata field in the launch configuration of the autoscaling group. Also you need to attach a IAM role to the instance with the permissions of describe-instances,create-tags etc.

In case you have created an ACM SSL certificate however it is not available in the drop down list to associate with your load balancer. The reason that you are unable to attach it to your ELB is because the certificate has a key length size of RSA-4096. Although it is possible to import a SSL certificate of 4096 bits into ACM, currently the ELB supported Certificate types are RSA_1024 and RSA_2048. If you are using any other type of Certificate, it will unfortunately not be eligible for attachment to your ELB, which means that you won't be able to select it during the ELB creation process. ACM supports 4096-bit RSA (RSA_4096)  certificates but integrated services (such as ELBs) allow only algorithms and key sizes they support to be associated with their resources. Note that ACM certificates (including certificates imported into ACM), are regional resources. Therefore, you must import your certificate into the same region that your ELB is in in order to associate it with th...

Check the status of the particular FS by df -ih Check for the recently created files by entering the FS which is showing high inodes find $1 -type f -print0 | xargs -0 stat --format '%Y :%y %n' | sort -nr | cut -d: -f2- | head Check the directory which is having most of the files find . -type d -print0 | xargs -0 -n1 count_files | sort -n

If you are running your servers in different VPC than based on the cidr range of the ip addresses and different username its difficult to remember all the keys  and username while connecting to the servers.

Rolling deployment strategy follows wherein servers are taken out in a batch for deployment instead of deploying onto all servers at once. This helps in lowering the HTTP 5xx and other such issues when the traffic is high on the application and the deployment needs to be performed. What Applications Qualifies for this Deployment: Applications running behind an Elastic load balancer. Application which have been moved to build and deploy model for NodeJS Applications (Wherein we build the deployment package once instead of doing on the fly compilation/installation)

If you are having the Infrastructure running in different AWS Accounts than you don't need to logout and login individually to each AWS Account or use the different browser. You can simply switch the roles between the different account 1. Login to your primary account this would be the entry level account through which you can switch to any of your other aws account. 2.  Click on your username at the top of the screen and choose Switch Role > Then choose switch role again. It will ask you for the following information Account Number:- (Account number in which you want to switch from existing account) Role:- (Role name which has been assigned to your user in IAM) Display Name:- (Easy to recognise name e.g. Production-Appname) 3. Click on switch role and you should be in the other account without loging out from your current account. 4. When done editing revert back to your account. 5. You can add any number of accounts under the switch role and move between d...

Kubernetes Master:- The master nodes in the kubernetes is a node which controls all the other nodes in the cluster. There can be several of those master nodes in a cluster if its said to be highly available or there can only be 1 if you got a single node cluster than that node will always going to be a master node. Kube-Apiserver:- Is the main part of the cluster which answer all the apicalls. This uses the key value store for storing the configuration of other persistent storage such as ETCD. ETCD:- Etcd is open source distributed key value store that provides the shared onfiguration and service discovery for the containers linux clusters. etcd runs on each machine in a cluster and gracefully handles the leader election during the network partitions and loss of the current leader. It is responsible for storing and replicating all kubernetes cluster state. Service Discovery:- It is the automatic detection of the devices and services offerentby the devices on a computer networ...

The kafka queue is a highly scalable and extreme fast queuing service which can really come in handy when you have to handle large amount of messages and have to build and services to work in async mode with an option to handle the fault in the services but not loosing the data at the same time you need the system to be scalable which can meet the ever growing demand of messages which would be pushed to this cluster. Following are some of the important points to consider while creating the highly available kafka zookeeper cluster:- 1. If you want to scale your kafka nodes you should consider keeping the zookeeper on the separate nodes. This is particularly useful for environment where kafka messages throughput is extremely large and more number of brokers would be required after certain period of time to deal with the fault tolerance while maintaining the system to be scalable. Kafka in itself is very scalable solution and in case you are not getting the data in TBs you can cons...