Solved: How to lock Terraform provider version

While working with terraform you would have noticed that every time you execute a terraform plan it will download the latest version of terraform available for that provider.

While this is good if you are testing as you get the latest features but, it can create trouble in production if a buggy version gets deployed. So, it is always recommended that you lock down the version of provider. In this post we will show you how to do that.

It’s really very simple to lock down the provider version. You just have to add a snippet like below in your main.tf file .

provider "aws" {

     region="us-east-1"
 
     version="<=2.6.0"

}

In the above example we have specified that version 2.6.0 or older can be used.

The version argument value may either be a single explicit version or a version constraint string. Constraint strings use the following syntax to specify a range of versions that are acceptable:

>= 2.4.0: version 2.4.0 or newer
<= 2.4.0: version 2.4.0 or older
~> 2.4.0: any non-beta version >= 2.4.0 and < 2.5.0, e.g. 2.4.X
~> 2.4: any non-beta version >= 2.4.0 and < 3.0.0, e.g. 2.X.Y
>= 2.0.0, <= 3.0.0: any version between 2.0.0 and 3.0.0 inclusive

Give it a try and let us know if you have any query or suggestion.

Solved: How to configure Terraform backend on AWS S3

Terraform is a very useful tool for IaaS. As you would have already known that it create a .tfstate file to save the status of infra. If you are doing testing you can save the .tfstate locally on your laptop. But, if you are working in prod environment with team then it’s best that you save the .tfstate remotely so that it’s secure and can be used by other team members.

Here we will show you two ways of configuring AWS S3 as backend to save the .tfstate file.

  1. First way of configuring .tfstate is that you define it in the main.tf file. You will just have to add a snippet like below in your main.tf file.
terraform {

      backend "s3" {

          bucket="cloudvedas-test123"

          key="cloudvedas-test-s3.tfstate"

          region="us-east-1"

      }

}

Here we have defined following things.

bucket = The S3 bucket in which the .tfstate should be saved

key = The name of the .tfstate file

region = The region in which S3 backend bucket exists.

2 Another way of specifying the S3 backend is that you define it when you initialize the terraform using the init command. This can be useful when you want to invoke the terraform from a jenkins file.

  • Here is an example that you can  execute in windows command prompt. This will do the same thing as we did in first example.
terraform init -no-color -reconfigure -force-copy -backend-config="region="us-east-1"" -backend-config="bucket="cloudvedas-test123"" -backend-config="key="cloudvedas-test1-win-s3.tfstate""
  • If you want to execute from a linux shell use below syntax.
 terraform init -no-color -reconfigure -force-copy \
-backend-config="region=us-east-1" \
-backend-config="bucket=cloudvedas-test123" \
-backend-config="key=cloudvedas-test-s3.tfstate"

Give it a try and let us know in comments section if you have any query or suggestion.

Solved : How to simply install python pip on windows?

The latest version of python is shipped with pip. But, if you are using older version of python 2(<2.7.9) or python 3(<3.4) and still need pip on windows please follow the instruction below.

If pip is not installed or the path is not set correctly you will get an error like below while invoking it in windows command prompt.

pip install sklearn
'pip' is not recognized as an internal or external command,
operable program or batch file.

Here are the steps to get pip in your windows box.

  • Download get-pip.py . Copy and paste the contents of the link in a notepad and save it as get-pip.py, remember not in .txt format but .py format.
  • Next install it.
python get-pip.py
  • Find the recently installed pip.exe in your machine. It’s generally in C:\Python27\Scripts or similar folder depending on the python version you installed. You can set the path variable in windows using these instructions.

– Search for “Advanced System Settings” in your windows machine.
– Click on the tab “Advanced”. In it click on Environment Variables.
– In the new window click on System Variables.  Select “Path” and click on “Edit”.
– Click on new and enter C:\Python27\Scripts\ in the space.
– Finally click ok to all windows and re-open command prompt.

  • If you have followed the instructions correctly you will now be able to install packages using pip.
pip install sklearn

That’s all you need to install pip in windows.

Tip:- Some people make mistake of executing pip from Python interpreter and get the below error.

>>> pip install sklearn
File "<stdin>", line 1
pip install sklearn
^
SyntaxError: invalid syntax
>>>

But, from the above tutorial you now know that pip is installed and executed from command prompt and not from the interpreter.

Hope you find this simple tutorial useful. Let us know in comment section if you face any issue.

Solved : How to check disk usage in windows like Linux

Identifying the files or directories which are hogging space in your windows machine can be very difficult specially if you have hidden files or they are in nested directories.

If you have worked on Linux you will know it has a very useful command du -sh. When i work on windows  I really miss the Linux command which is very handy.

Thankfully you can find a utility at sysinternals on microsoft site which do the same job as du command in linux.

The name of utility is simply “du” and you can download it from here .

Once you download the utility you just have unzip it and no installation needed.

After that open the command prompt as admin user and go to the directory where you have unzipped the “du” and execute it as below to find the usage of any directory.

du -l 1 "C:\Program Files\Microsoft SQL Server"

It will show you usage of even the hidden files. The usage is shown in KB.

Solved : Check without telnet if port is open on remote host – Linux

Earlier many system admins were using telnet to check if a port is open on remote machine. But, because of security concerns telnet is now not recommended to be installed on most linux boxes.

But hey we still need to check the remote port. This is part of normal troubleshooting in a linux admin’s life.

So, instead of using telnet you can use  nc.  netcat or nc is a  simple tool which is generally installed by default in linux distributions or you can install it with instructions below.

Ubuntu

$ sudo apt-get update

$ sudo apt-get install netcat

Redhat or CentOS

yum install nc.x86_64

Amazon Linux

 sudo yum install nc

Once you are done with installation simply run it as below:-

nc -z 100.4.78.39 22
Connection to 100.4.78.39 22 port [tcp/ldaps] succeeded!

Here we can see that port 22 is open on remote server.

Tip :- If you want to check the same from windows machine, follow this post on How to check remote port status in windows without telnet.

nc is a swiss knife kind of tool. It comes with very useful options.  You can explore the below options also.

Tag Description
4 Forces nc to use IPv4 addresses only.
6 Forces nc to use IPv6 addresses only.
D Enable debugging on the socket.
d Do not attempt to read from stdin.
h Prints out nc help.
i interval
Specifies a delay time interval between lines of text sent and received. Also causes a delay time between connections to multiple ports.
k Forces nc to stay listening for another connection after its current connection is completed. It is an error to use this option without the –l option.
l Used to specify that nc should listen for an incoming connection rather than initiate a connection to a remote host. It is an error to use this option in conjunction with the –p , –s , or –z options. Additionally, any timeouts specified with the –w option are ignored.
n Do not do any DNS or service lookups on any specified addresses, hostnames or ports.
p source_port
Specifies the source port nc should use, subject to privilege restrictions and availability. It is an error to use this option in conjunction with the –l option.
r Specifies that source and/or destination ports should be chosen randomly instead of sequentially within a range or in the order that the system assigns them.
S Enables the RFC 2385 TCP MD5 signature option.
s source_ip_address
Specifies the IP of the interface which is used to send the packets. It is an error to use this option in conjunction with the –l option.
T ToS
Specifies IP Type of Service (ToS) for the connection. Valid values are the tokens “lowdelay”, “throughput”, “reliability”, or an 8-bit hexadecimal value preceded by “0x”.
t Causes nc to send RFC 854 DON’T and WON’T responses to RFC 854 DO and WILL requests. This makes it possible to use nc to script telnet sessions.
U Specifies to use Unix Domain Sockets.
u Use UDP instead of the default option of TCP.
v Have nc give more verbose output.
w timeout
If a connection and stdin are idle for more than timeout seconds, then the connection is silently closed. The –w flag has no effect on the –l option, i.e. nc will listen forever for a connection, with or without the –w flag. The default is no timeout.
X proxy_version
Requests that nc should use the specified protocol when talking to the proxy server. Supported protocols are “4” (SOCKS v.4), “5” (SOCKS v.5) and “connect” (HTTPS proxy). If the protocol is not specified, SOCKS version 5 is used.
x proxy_address[: port]
Requests that nc should connect to hostname using a proxy at proxy_address and port. If port is not specified, the well-known port for the proxy protocol is used (1080 for SOCKS, 3128 for HTTPS).
z Specifies that nc should just scan for listening daemons, without sending any data to them. It is an error to use this option in conjunction with the –l option.

Real shell scripting interview questions

Q What is Shell?

Ans: Shell is a command interpreter, which interprets the command which the user gives to the kernel. It can also be defined as an interface between a user and operating system.

Q How to debug the problems encountered in shell script/program?

Ans: Two options
1) Execute the script as “sh -x script.sh”
2) Put “set -x” in the script

Q Which is the symbol used for comments in bash shell scripting ?
Ans:

#

Q What is the difference between = and ==?

Ans:

=      -> It is used for assigning value to the variable.

==    -> It is used for string comparison.

Q How to get 4th element from each line of a file ?

Ans:

awk '{print $4}'

Q What needs to be done before you can execute a shell script?

Ans:
You need to make the shell script executable using the chmod command.

This chmod command makes the shell script file “file1” executable for the user (owner) only:
$ chmod u+x file1

Below syntax makes it executable for all (everyone):

$ chmod a+x file1

Q How to pass argument to a script ?
Ans:

./script argument

Q How do you terminate an if statement?

Ans: Using “fi” . Check example below.

Q Give an example of if else statement
Ans:

#Testfile.sh script to test if the file exists
#!/bin/ksh
cvfile=$1
if [ -f $cvfile ]
then
echo "$cvfile exists"
else
"$cvfile does not exists"
fi
exit 0

So you will execute the script as “./Testfile.sh file1” .

Q How to check if a directory exists?

Ans:

if [ -d $mydir ] 
then
echo "Directory exists"
fi

Q How to calculate number of passed arguments ?

Ans:

$#

Q How to check if previous command run successful ?

Ans:

echo $?

If exit code is 0 it means command ran successfully

Q How to get last line from a file ?

Ans:

tail -1

Q How to redirect stdout and stderr streams to log.txt file from inside the script ?
Ans:

Add “exec >log.txt 2>&1” put this as the first command in the script

Q How to remove blank lines from a file?

Ans :

grep -v '^

Q Write a command to find all the files modified in less than 3 days and print the record count of each?

find . –mtime -3 –exec wc –l {} \;

Q How to find a process name from process ID?

ps -p PID

 testcv.txt > testcv2.txt

Q Write a command to find all the files modified in less than 3 days and print the record count of each?

find . –mtime -3 –exec wc –l {} \;

Q How to find a process name from process ID?

ps -p PID

Must have skills for DevOps Engineer and Free methods to learn them

Many people have asked us how to become a DevOps engineer so in this post we will show you the skills and technologies which can get you there.

Before we start we should know what is “DevOps”. As you can see the word is combination of “Developer” and “Operations”. So a DevOps engineer is suppose to be a bridge between the development and operations processes.

Let’s see which skills do you need to become a DevOps engineer. In this post we will also discuss how you can acquire these skills for free. We have also mentioned some advanced paid courses which can help you acquire these skills quickly.

Skill 1 :-  Make Linux your friend.

Nowadays whole world is moving towards opensource, and Linux is the poster boy of opensource world.  Most of the new enterprise level tools and services are now compatible with Linux and people are adopting them fast. So, as a devops engineer you will definitely have to work on/with linux in your job. Even Microsoft has come up with their own Linux version. So, you can understand the importance of Linux. You can learn Linux for free from Linux Foundation courses. But, if you are short of time you can quickly learn Linux through this course Learn Linux in 5 days .

Skill 2 :- Learn scripting

Automation is a big part of a DevOps engineer job and you will be expected to automate mundane tasks. This will not only help you work efficiently but also improve the overall maintainability of your company’s setup. Easiest way to automate things is through scripting. You should learn either shell scripting on Linux/Unix or powershell on windows. Free source to learn scripting is Learn Shell. Else, you can check out this around 6 hours course on Shell Scripting to solve real-world problems like a pro.

Skill 3 :- Understand the Cloud

Companies are shifting to cloud like never before. Applications are being re-built to leverage the advantage of cloud computing. So it’s always good that you know cloud really well. AWS, Azure and GCP are three market leaders and it will be good to know about at least one of them. All these vendors have tons of free material on their sites which can help you learn their technology. Else you can checkout these awesome Udemy courses for AWS, Azure or GCP .

Skill 4:- Source code Management

Tools like git are used to manage the version of code base and avoid any conflicts in code. As a devops engineer you should have good understanding of tools like git or bitbucket since they will be useful in build phase. Later you will use them to build a codepipeline. One of the good source to learn git for free is Try Git. To become zero to hero of Git in hours, one of the best course is Git complete.

Skill 5 :- Configuration management

Configuration management is used to manage the state of your infrastructure. You can use them to patch your linux servers or manage the services.  Some of the tools in config management are Ansible, Chef and Puppet. To get basic understanding of Ansible the best free course is from Redhat called Ansible essentials. If you want to get deeper understanding you can try Mastering Ansible.

Skill 6: – Continuous Integration/Continuous Delivery

CI/CD is an important aspect in DevOps engineers job profile. As we are moving towards cloud automated deployments are now essential. They avoid human errors and are fast. You can learn tools like Jenkins for CI/CD. Jenkins will specially be very useful in building a codepipeline, you can learn it for free here. Since CI/CD consists of multiple tools we think that it’s better you learn it from a single source. We found course from Edward Learn Devops: Continuously Deliver Better Software  to have covered most of the topics.

Skill 7 :- Infra as a Code

The latest “in thing” in cloud is Infra as a Code. Yes you read it right, gone are the days where you have to buy the server, install OS and other softwares on it. With tools like Terraform or Cloudformation you can automate the build of your entire infrastructure in cloud right from VPC, Security groups,  servers etc. to application installation and configuration. To get basic free understanding of terraform you can learn it from Katacoda . If you want to deploy AWS Infrastructure using Terraform we found Edward’s another course on Terraform to be really useful. For cloudformation we found the free AWS cloudformation tutorials to be useful. If you are looking for advance practical guidance AWS CloudFormation Master Class is a good course.

Skill 8:- Containers

Servers are now shrinking every day. With the advent of VMs, hardware anyways became less important. Dockers, a containerization technology is taking it to one level up. Now with Dockers you can have a miniature copy of your OS image with the application code. Docker will only run for a few mili seconds when a query is executed and once it’s job is done it will just stop. If you want to learn Docker for free the best source is Play with Docker Classroom. To gain more advanced skills you can learn from Docker Mastery.

Skill 9 :-  Container orchestration

Tools like Kubernetes or Elastic container service (ECS) are used to manage cluster of Docker containers. These tools provides you mechanisms for deploying, maintaining, and scaling application. You can learn about Kubernetes with practical labs for free from Katacoda. For advanced knowledge of Kubernetes try the complete Kubernetes course.

Skill 10 : – Serverless

Technologies like AWS lambda are removing the need to manage the servers at all. You just have to upload you code in the AWS Lambda and set your run time environment. Rest all the issues of memory, CPU, OS patching will be taken care by AWS Lambda. It’s important as a DevOps engineer that you understand how the lambda works and how you can set it up. You can learn about AWS Lambda for free from this AWS Lambda Tutorial. For more advanced real world example you can refer how to build a serverless App with AWS Lambda .

We hope this post is helpful to you in understanding the skills which are important for a DevOps engineer and how you can master these skills. You can expect questions related to these skills in the interview of a DevOps Engineer.

Solved: How to start or stop Jenkins in Linux and check it’s port number

In this post we will see how to start or stop Jenkins on different Linux distributions. Also we will see how to check the status of Jenkins service and identify the port on which it is running or listening.

If you have installed Jenkins on a Fedora or Debian based distribution like Ubuntu, you can use the below commands:

Start Jenkins

$ sudo service jenkins start

Stop Jenkins

$ sudo service jenkins stop

Restart Jenkins

$ sudo service jenkins restart

Check Jenkins status

$ sudo service jenkins status

For other Linux distribution like RHEL or CentOS use:

Start Jenkins

$ sudo systemctl start jenkins.service

Stop Jenkins

$ sudo systemctl stop jenkins.service

Restart Jenkins

$ sudo systemctl restart jenkins.service

Check Jenkins status

$ sudo systemctl status jenkins.service

Check Jenkins Port

To check the port on which Jenkins is listening we can use “lsof”(check installation instructions at the end) command.

[ec2-user@cloudvedas ~]$ sudo /usr/sbin/lsof -i -P | grep -i jenkins
java 1030 jenkins 161u IPv6 27986 0t0 TCP *:8080 (LISTEN)

Above we can see Jenkins is listening on port 8080.

If you do not have lsof you can simply install it with following commands

Ubuntu

sudo apt-get install lsof

RHEL or CentOS

sudo yum install lsof

Solved: How to change from EFI to SMI label and vice-versa

Hello!

In this post we will see how to change the disk label from EFI to SMI in a Solaris server.

Before you decide to change the disk label ensure that the disk doesn’t have any important data. If needed take backup of the disk. As during label change all the data will be removed from the disk.

We will use “format -e” command to change the label on disk.

root@cldvds# format -e c0t6006023764A62A00C5H174886B5BC267d0

format> la
 [0] SMI Label
 [1] EFI Label
 Specify Label type[1]: 0
 Auto configuration via format.dat[no]?
 Auto configuration via generic SCSI-2[no]?

Geometry: 256 heads, 10 sectors 40960 cylinders result in 104857600 out of 104857600 blocks.
 Do you want to modify the device geometry[no]? yes
 format> p

The disk partition should now change from 0 to 9 slices(EFI) to 0 to 7 slices (SMI).

If you want to change from SMI to EFI you can follow the same steps and instead of choosing option 0 choose option 1.

That’s all folks! Do comment if you have any concern or query!

Solved: Create virtual environment with python 3 in linux

Once you have installed python 3.6 as described in the last post you can create virtual environment to work with it.

  • First let’s install the package of virtual environment for python 3.

CentOS/RHEL

sudo yum install -y python3-venv

Ubuntu

sudo apt-get install -y python3-venv 

  • Create a directory for you virtual environment and get inside it.
mkdir virtualenvs
cd virtualenvs
  • Now let’s create our virtual environment.
python3.6 -m venv cloudvedasenv
  • Finally activate the new virtual environment. Notice the new environment name in brackets once you run the source command. That means you are now inside the virtual environment.
[ec2-user@cloudvedas ~]$ source cloudvedasenv/bin/activate

(cloudvedasenv) [ec2-user@cloudvedas ~]$
  • If you want to install a package inside the virtual environment you can use pip .
(cloudvedasenv) [ec2-user@cloudvedas ~]$ pip install zappa requests flask