November 2018
M T W T F S S
« Oct    
 1234
567891011
12131415161718
19202122232425
2627282930  

Categories

WordPress Quotes

A real decision is measured by the fact that you've taken a new action. If there's no action, you haven't truly decided.
Tony Robbins

Recent Comments

November 2018
M T W T F S S
« Oct    
 1234
567891011
12131415161718
19202122232425
2627282930  

Short Cuts

2012 SERVER (64)
2016 windows (9)
AIX (13)
Amazon (34)
Ansibile (18)
Apache (133)
Asterisk (2)
cassandra (2)
Centos (209)
Centos RHEL 7 (258)
chef (3)
cloud (2)
cluster (3)
Coherence (1)
DB2 (5)
DISK (25)
DNS (9)
Docker (28)
Eassy (11)
ELKS (1)
EXCHANGE (3)
Fedora (6)
ftp (5)
GIT (3)
GOD (2)
Grub (1)
Hacking (10)
Hadoop (6)
horoscope (23)
Hyper-V (10)
IIS (15)
IPTABLES (15)
JAVA (7)
JBOSS (32)
jenkins (1)
Kubernetes (2)
Ldap (5)
Linux (189)
Linux Commands (167)
Load balancer (5)
mariadb (14)
Mongodb (4)
MQ Server (22)
MYSQL (84)
Nagios (5)
NaturalOil (13)
Nginx (30)
Ngix (1)
openldap (1)
Openstack (6)
Oracle (34)
Perl (3)
Postfix (19)
Postgresql (1)
PowerShell (2)
Python (3)
qmail (36)
Redis (12)
RHCE (28)
SCALEIO (1)
Security on Centos (29)
SFTP (1)
Shell (64)
Solaris (58)
Sql Server 2012 (4)
squid (3)
SSH (10)
SSL (14)
Storage (1)
swap (3)
TIPS on Linux (28)
tomcat (60)
Uncategorized (29)
Veritas (2)
vfabric (1)
VMware (28)
Weblogic (38)
Websphere (71)
Windows (19)
Windows Software (2)
wordpress (1)
ZIMBRA (17)

WP Cumulus Flash tag cloud by Roy Tanck requires Flash Player 9 or better.

Who's Online

16 visitors online now
4 guests, 12 bots, 0 members

Hit Counter provided by dental implants orange county

Migrating to Amazon Linux 2

AWS also announced that Amazon Linux 2018.03 is the last release for the current generation of Amazon Linux and will be supported until June 30, 2020. Therefore, you have to come up with a migration plan.

Amazon Linux 2 comes with the same benefits as Amazon Linux, but it adds some new capabilities:

  • long-term support: Amazon Linux 2 supports each LTS release for five years
  • on-premises support: virtual machine images for on-premises development and testing are available
  • systemd: replacing SystemVinit
  • extras library: provides up-to-date versions of software bundles such as nginx

Let’s dive into some of the changes in more detail. At the end of the post, I will also outline some pitfalls I encountered when migrating our Free Templates for AWS CloudFormation to Amazon Linux 2.

Further reading: Release Notes, FAQs, AWS Blog Post, Announcement

Long-term support

The Amazon Linux delivers a continuous flow of updates that allow you to roll from one version of the Amazon Linux AMI to the most recent. A yum update always moves your system to the latest Amazon Linux version. There were no versions of Amazon Linux available, only snapshots.

Amazon Linux 2 changes this. You will have Amazon Linux 2 versions that are supplied with updates for five years. Once a new Amazon Linux 2 LTS release becomes available, no breaking changes will be introduced by AWS for this release.

systemd

Amazon Linux uses SysVinit to bootstrap the Linux user space and to manage system processes after booting. This procedure is usually called init. One of the major drawbacks of SysVinit is that it starts tasks serially, waiting for each to finish loading before moving on to the next. This can result in long delays during boot.

Amazon Linux 2 uses systemd as the init system. systemd executes elements of its startup sequence in parallel, which is faster than the traditional serial approach from SysVinit. systemd can also ensure that a service is running (e.g., it restarts a service if it crashed).

systemd is not just the name of the init system daemon but also refers to the entire software bundle around it, which includes:

  • journald: responsible for event logging (replaces syslog)
  • udevd: device manager for the Linux kernel, which handles the /dev directory and all user space actions when adding/removing devices
  • logind: manages user logins and seats in various ways.

I will not cover udevd and logind in this post. You should not get in touch with them as a normal user like me. Keep in mind that networking configuration is not controlled by networkd (also part of systemd software bundle). Instead, networking configuration is controlled by cloud-init which is triggered by systemd several times during boot. cloud-init handles early initialization of an EC2 instance (also works with other vendors).

Further reading: systemd man page

Reading logs from journald

To read all system logs (journal in journald terminology), starting with the oldest entry, run journalctl. The output is paged through less by default. Which means you can scroll down / up an entry with the DOWN / UP arrow keys, or scroll a full page down/up with the SPACE / b keys. Press the q key to quit. To reverse the order, run journalctl -r.

To show only the most recent journal entries, and continuously print new entries, run journalctl -f (like a tail -f).

There are many ways to filter the output. Based on priority, run journalctl -p err to get levels alert, crit, and err (using syslog log levels). Based on the unit, run journalctl -u sshd to get all entries for sshd. Check the further reading links for more information.

Keep in mind that some applications still write logs to /var/log. Journald also forwards logs to rsyslog which is configured (/etc/rsyslog.conf) to write some of them to files in /var/log.

Further reading: journalctl man page

Controlling systemd services

To start a service (unit in systemd terminology), you run:

systemctl start awslogsd.service

To make sure a service (unit in systemd terminology) is started during boot/reboot, you run:

systemctl enable awslogsd.service

There are many other commands. E.g., you can also reboot the system:

systemctl reboot

Further reading: systemctl man page

Extras Library

The Extras Library (aka Amazon Linux Extras Repository or Extras mechanism), provides a way to install up-to-date software bundles (topics in Amazon Linux 2 terminology) without impacting the stability of the rest of the operating system.

Extras Library is not covered by LTS!

To get a list of available topics, run:

$ amazon-linux-extras list
  0  ansible2                 available  [ =2.4.2 ]
  1  emacs                    available  [ =25.3 ]
  2  memcached1.5             available  [ =1.5.1 ]
  3  nginx1.12                available  [ =1.12.2 ]
  4  postgresql9.6            available  [ =9.6.6  =9.6.8 ]
  5  python3                  available  [ =3.6.2 ]
  6  redis4.0                 available  [ =4.0.5 ]
  7  R3.4                     available  [ =3.4.3 ]
  8  rust1                    available  [ =1.22.1  =1.26.0 ]
  9  vim                      available  [ =8.0 ]
 10  golang1.9                available  [ =1.9.2 ]
 11  ruby2.4                  available  [ =2.4.2  =2.4.4 ]
 12  nano                     available  [ =2.9.1 ]
 13  php7.2                   available  [ =7.2.0  =7.2.4  =7.2.5 ]
 14  lamp-mariadb10.2-php7.2  available  [ =10.2.10_7.2.0  =10.2.10_7.2.4  =10.2.10_7.2.5 ]
 15  libreoffice              available  [ =5.0.6.2_15 ]
 16  gimp                     available  [ =2.8.22 ]
 17  docker=latest            enabled    [ =17.12.1  =18.03.1 ]
 18  mate-desktop1.x          available  [ =1.19.0  =1.20.0 ]
 19  GraphicsMagick1.3        available  [ =1.3.29 ]
 20  tomcat8.5                available  [ =8.5.31 ]

To install an topic, run amazon-linux-extras install <topic> (e.g., amazon-linux-extras install ruby2.4).

If you install (or only enable) a topic, a new repository (plus two for sources and debuginfo) is configured in /etc/yum.repos.d/amzn2-extras.repo.

Pitfalls

I migrated Free Templates for AWS CloudFormation to Amazon Linux 2. In the following, I will outline the problems I was faced with and how I worked around them.

The awslogs agent was renamed

The awslogs agent was renamed to awslogsd but you still install it via yum install awslogs.

You can start (activate in systemd terminology) awslogs with systemctl start awslogsd.service (shortcut: systemctl start awslogsd).

The awslogs agent does not support journald

awslogs agent cannot read logs directly from the journal. journald fowards all logs to rsyslog which is configured (/etc/rsyslog.conf) to write some of the logs to files in /var/log from where the awslogs agent can pick them up.

Where are the log files?

/var/log does not contain all system logs anymore.

If in doubt, you can access all system logs with journalctl.

Ruby is missing

Ruby is no longer installed by default. This breaks cfn-init if you want to install RubyGems.

You can install Ruby 2.0 with yum install ruby or Ruby 2.4 with amazon-linux-extras install ruby2.4.

netcat is missing

netcat (or nc) is no longer installed by default.

You can install ncat with yum install nmap-ncat, but this will install nmap based ncat which behaves differently (e.g., no -z flag anymore). Learn more

Nginx package not available by default

nginx is no longer part of the default repository.

$ yum install nginx
Failed to set locale, defaulting to C
Loaded plugins: langpacks, update-motd
No package nginx available.
Error: Nothing to do

To install nginx, use the new Amazon Linux Extras Repository amazon-linux-extras install nginx1.12.

EPEL repository is missing

The EPEL repository (Extra Packages for Enterprise Linux) is no longer installed by default or available to install. The Extras Library replaces the EPEL repository but contains only a fraction of the packages which may causes troubles during your migration.

NAT and ECS optimized AMIs are missing

NAT and ECS optimized AMI are not available. You can replace your NAT instances with NAT Gateways to get around this problem. But for ECS workloads there is no easy workaround. I advise waiting for news from AWS regarding the ECS optimized AMI.

cfn-init is not integrated with the Extras Library

You can not install packages from the Extras Library with the package mechanism in cfn-init easily. cfn-init is the way how you can install software onto EC2 instances managed by CloudFormation.

There can either run amazon-linux-extras enable <topic> before running cfn-init which than can install the package by using the package mechanism. Or you can use two config sets. The first config sets uses the command mechanism to enable the topic. The second config set uses the package mechanism to install the enabled package. You have to use two config sets because commands run after package installation. Here is an example:

AutoScalingGroup:
  Type: 'AWS::AutoScaling::AutoScalingGroup'
  Properties:
    # [...]
LaunchConfiguration:
  Type: 'AWS::AutoScaling::LaunchConfiguration'
  Metadata:
    'AWS::CloudFormation::Init':
      configSets:
        default: [extras, config]
      extras:
        commands:
          a_enable_nginx:
            command: 'amazon-linux-extras enable nginx1.12'
            test: "[ ! grep -Fxq '[amzn2extra-nginx1.12]' /etc/yum.repos.d/amzn2-extras.repo ]"
      config:
        packages:
          yum:
            nginx: [] # will install nginx1.12
  Properties:
    # [...]
    UserData:
      'Fn::Base64': !Sub |
        #!/bin/bash -x
        /opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource LaunchConfiguration --region ${AWS::Region}
        /opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackName} --resource AutoScalingGroup --region ${AWS::Region}

Summary

Amazon Linux 2 is the new default for running Linux workloads on AWS. Amazon Linux 2 benefits from systemd, LTS, and a new extras library. There are a few pain points when migrating, most notably the missing EPEL repository. Besides that, you should spend some time to understand how systemd works, because that’s central in modern Linux operating systems.

It’s time to plan your migration from Amazon Linux now!

mysql dump issue utf8_unicode_520_ci

i use this in linux :

sed -i 's/utf8mb4/utf8/g' your_file.sql
sed -i 's/utf8_unicode_ci/utf8_general_ci/g' your_file.sql
sed -i 's/utf8_unicode_520_ci/utf8_general_ci/g' your_file.sql

then restore your_file.sql

mysql -uyourdbuser -pyourdbpasswd yourdb < your_file.sql

Install LDAP-slapd.conf mode

Install LDAP-slapd.conf mode
——————start installation———————
Query whether to install

# rpm -qa openldap-servers

Remove ldap
# yum remove openldap

# yum remove openldap-servers

Installation Environment
Centos7
Apache/2.4.6 (CentOS)
PHP 7.1.11

Install LDAP Server

Openldap-servers-2.4.44-5.el7.x86_64

# yum install openldap-servers openldap-clients migrationtools

# rpm -qa | grep openldap

Delete all files in the slapd.d folder and copy a copy of slapd.conf
# rm -rvf /etc/openldap/slapd.d
# mkdir /etc/openldap/slapd.d

Generate password

# /sbin/slappasswd
New password:
Re-enter new password:

{SSHA}XXXXXXXXXXXXXXXXX….

Ldap-server configuration file
# cp /usr/share/openldap-servers/slapd.ldif /etc/openldap/slapd.conf

Modify the slapd.conf file
# vi /etc/openldap/slapd.conf

Put

olcAccess: to * by dn.base=”gidNumber=0+uidNumber=0,cn=peercred,cn=external,c n=auth” read by dn.base=”cn=Manager,dc=my-domain,dc= Com” read by * none
olcSuffix: dc=my-domain,dc=com olcRootDN: cn=Manager,dc=my-domain,dc=com

Change to

olcAccess: to * by dn.base=”gidNumber=0+uidNumber=0,cn=peercred,cn=external,c n=auth” read by dn.base=” cn=Manager,dc=rmohan,dc=com ” read by * none olcSuffix: dc=rmohan,dc=com
olcRootDN: cn=Manager,dc=rmohan,dc=com
olcRootPW: {SSHA}XXXXXXXXXXXXXXXXXXXXX

(olcRootPW please copy from password.txt just)

Start converting configuration files
# rm -rvf /var/lib/ldap/*
# /usr/libexec/openldap/convert-config.sh
# head -20 /etc/openldap/slapd.d/cn\=config/olcDatabase\=\{[12]}*

Test if the LDAP configuration file is normal
# slaptest -u
Config file testing succeeded

Clear the database sample file and copy the database default profile
Copy database preset profile #
Cp /usr/share/openldap-servers/DB_CONFIG.example /var/lib/ldap/DB_CONFIG

# rm -rvf /etc/openldap/certs
# mkdir /etc/openldap/ certs

Create a certs DB profile
# /usr/libexec/openldap/create-certdb.sh
Creating certificate database in ‘/etc/openldap/certs’.
# /usr/libexec/openldap/generate-server-cert.sh
Creating new server certificate in ‘/etc/openldap/certs’.
# chown ldap:ldap -R /var/lib/ldap/
# systemctl start slapd
# slaptest
Config file testing succeeded
# systemctl enable slapd
# firewall-cmd –permanent –zone=public –add-port=389/tcp
# firewall-cmd –reload
# ldapwhoami -WD cn=Manager,dc=rmohan,dc=com
Enter LDAP Password:
Dn :cn=Mana g er,dc=rmohan,dc=com

Import schema
# ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/openldap/schema/cosine.ldif
# ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/openldap/schema/nis.ldif
# ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/openldap/schema/inetorgperson.ldif

Edit root node

# vim base.ldif
Dn: dc=rmohan,dc=com
objectClass: top
objectClass: dcObject
Objectclass: organization
o: rmohan
Dc: rmohan

Dn: cn=Manager,dc=rmohan,dc=com
objectClass: organizationalRole
Cn: Manager
Description: Directory Manager

Dn: ou=STU, dc=rmohan,dc=com
objectClass: organizationalUnit
Ou: STU
Description:student

Dn: ou=TEA,dc=rmohan,dc=com
objectClass: organizationalUnit
Ou: TEA
Description:teacher

# /bin/ldapadd -x -D “cn=Manager,dc=rmohan,dc=com” -W -f base.ldif
Enter LDAP Password:
Adding new entry “dc=rmohan,dc=com”
Adding new entry “cn=Manager,dc=rmohan,dc=com”
Adding new entry “ou=People,dc=rmohan,dc=com”
Adding new entry “ou=Group,dc=rmohan,dc=com”

Firewall settings

# /bin/firewall-cmd –permanent –add-service=ldap
# /bin/firewall-cmd –reload

How to reset your root MySQL password

Stop the MySQL process
# service mysqld stop

Once MySQL has stopped, restart it with the --skip-grant-tables option
# mysqld_safe --skip-grant-tables &
or edit your /etc/my.cnf file to add the line
skip-grant-tables

Connect to MySQL using the root user.
mysql -u root

Once logged in, you should see the following prompt:
mysql>
Enter the following commands:
mysql> use mysql;
mysql> UPDATE user SET password=PASSWORD("YOUR NEW PASSWORD HERE") WHERE User='root';
mysql> flush privileges;
mysql> quit

Now stop MySQL again:
# service mysqld stop

If you edited your /etc/my.cnf file, delelete the skip-grant-tables line.

Now restart MySQL and test your new login.
# service mysqld restart
# mysql -u root -p


 

MariaDB [(none)]> CREATE USER xxxx;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> CREATE USER xxxx@hostname ;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> GRANT ALL PRIVILEGES ON dbane.* To ‘xxxxx@hostname’ IDENTIFIED BY ‘P@ssw0rd5768#’;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> FLUSH PRIVILEGES;

GRANT ALL PRIVILEGES ON db.* To ‘usernam’@’localhost’ IDENTIFIED BY ‘P@ssw0rd5768#’;
GRANT USAGE ON *.* TO ‘usernam’@’localhost’ IDENTIFIED BY ‘P@ssw0rd5768#’;

 

Reset root password

UPDATE user SET authentication_string=password(‘password’) WHERE user=’root’;

update mysql.user set password_expired = ‘N’, authentication_string=PASSWORD(‘password’) where user = ‘root’;

GRANT USAGE ON *.* TO ‘root’@’localhost’ IDENTIFIED BY ‘password’;

Reset SonarQube Admin User Password

Reset SonarQube Admin User Password

by default sonar creates admin account : user: admin, password : admin

Start root access to mysql database

mysql -uroot -p
Bash

Check if sonar database exists.

show databases;
SQL
use sonar;
SQL

Reset admin user to admin password.

update users set crypted_password = '88c991e39bb88b94178123a849606905ebf440f5', salt='6522f3c5007ae910ad690bb1bdbf264a34884c6d' where login = 'admin';

git cmd

git pull # will get you repo updates
git add . # will add files in your dir
git add [dirname]/* # will add files under a new dir
git commit -m your comment # will commit your code to your changes
git push # will push your code to a reposatory
git pull # will pull down new reposatory updates

Git Fundamentals

Git Fundamentals

 

Topics To be Covered

  1. Installation

2. Setup

3. Creating a Project

4. Checking the status of the repository

5. Making changes

6. Staging the changes

7. Staging and committing

8. Commiting the changes

9. Changes, not files

10. History

11. Aliases

12. Getting older versions

13. Tagging versions

14. Discarding local changes (before staging)

15. Cancel Staged changes (before committing)

16. Cancelling commits


1. Installing Git

Installing on Linux

If you want to install the basic Git tools on Linux via a binary installer, you can generally do so through the basic package-management tool that comes with your distribution. If you’re on Fedora for example, you can use yum:

$ sudo yum install git-all

If you’re on a Debian-based distribution like Ubuntu, try apt-get:

$ sudo apt-get install git-all

For more options, there are instructions for installing on several different Unix flavors on the Git website, at http://git-scm.com/download/linux.

Installing on Mac

There are several ways to install Git on a Mac. The easiest is probably to install the Xcode Command Line Tools. On Mavericks (10.9) or above you can do this simply by trying to run git from the Terminal the very first time. If you don’t have it installed already, it will prompt you to install it.

If you want a more up to date version, you can also install it via a binary installer. An OSX Git installer is maintained and available for download at the Git website, at https://sourceforge.net/projects/git-osx-installer/

You can also install it as part of the GitHub for Mac install. Their GUI Git tool has an option to install command line tools as well. You can download that tool from the GitHub for Mac website, athttp://mac.github.com.

Installing on Windows

There are also a few ways to install Git on Windows. The most official build is available for download on the Git website. Just go to https://git-for-windows.github.io/

Another easy way to get Git installed is by installing GitHub for Windows. The installer includes a command line version of Git as well as the GUI. It also works well with Powershell, and sets up solid credential caching and sane CRLF settings. We’ll learn more about those things a little later, but suffice it to say they’re things you want. You can download this from the GitHub for Windows website, at http://windows.github.com

2. One time Git Environment Setup

Now that you have Git on your system, you’ll want to do a few things to customize your Git environment. You should have to do these things only once; they’ll stick around between upgrades. You can also change them at any time by running through the commands again.

Git comes with a tool called git config that lets you get and set configuration variables that control all aspects of how Git looks and operates. These variables can be stored in three different places:

$ git config?—?global user.name “Aravind G V”

$ git config?—?global user.email aravind_gv@intuit.com

$ git config?—?global core.editor “edit -w”

3. Creating a Project

Create a new repository

Create a New Folder or Do in Existing Folder if you want to add it to vcs

mkdir training
cd training
touch hello.txt

Create a repository

So you have a directory that contains one file. Run the git init in order to create a git repo from that directory.

RUN:

git init

RESULT:

$ git init
Initialized empty Git repository in /Users/agv/git-training

This creates a new subdirectory named .git that contains all of your necessary repository files?—?a Git repository skeleton. At this point, nothing in your project is tracked yet

Add the page to the repository

Now let’s add the “Hello, World” page to the repository.

RUN:

git add hello.txt
git commit -m "First Commit"

You will see …

RESULT:

$ git add hello.txt
$ git commit -m "First Commit"
[master (root-commit) 2fc4372] First Commit
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 hello.txt

Checking the status of the repository

Use the git status command, to check the current state of the repository.

RUN:

git status

RESULT:

$ git status
# On branch master
nothing to commit (working directory clean)

The command checks the status and reports that there’s nothing to commit, meaning the repository stores the current state of the working directory, and there are no changes to record.

We will use the git status, to keep monitoring the states of both the working directory and the repository.

Making changes

Let’s add some thing in text file

vi hello.txt
Test status

Checking the status

Check the working directory’s status.

RUN:

git status

You will see …

RESULT:

$ git status
On branch master
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)
modified:   hello.txt
no changes added to commit (use "git add" and/or "git commit -a")

The first important aspect here is that git knows hello.txt file has been changed, but these changes are not yet committed to the repository.

Another aspect is that the status message hints about what to do next. If you want to add these changes to the repository, use git add. To undo the changes use git checkout.

6. Staging the changes

Adding changes

Now command git to stage changes. Check the status

RUN:

git add hello.txt
git status

You will see …

RESULT:

$ git add hello.txt
$ git status
# On branch master
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#   modified:   hello.txt
#

Changes to the hello.txt have been staged. This means that git knows about the change, but it is not permanent in the repository. The next commit will include the changes staged.

Should you decide not to commit the change, the status command will remind you that you can use the git reset command to unstage these changes.

Staging and committing

A staging step in git allows you to continue making changes to the working directory, and when you decide you wanna interact with version control, it allows you to record changes in small commits.

Suppose you have edited three files (a. html, b. html, and c. html). After that you need to commit all the changes so that the changes to a. html and b. html were a single commit, while the changes to c. html were not logically associated with the first two files and were done in a separate commit.

In theory you can do the following:

touch a.html b.html c.html
git add a.html
git add b.html
git commit -m "Changes for a and b"
git add c.html
git commit -m "Unrelated change to c"

Separating staging and committing, you get the chance to easily customize what goes into a commit

8. Committing the changes

Well, enough about staging. Let’s commit the staged changes to the repository.

When you previously used git commit for committing the first hello.html version to the repository, you included the -m flag that gives a comment on the command line. The commit command allows interactively editing comments for the commit. And now, let’s see how it works.

If you omit the -m flag from the command line, git will pop you into the editor of your choice from the list (in order of priority):

  • GIT_EDITOR environment variable
  • core.editor configuration setting
  • VISUAL environment variable
  • EDITOR environment variable

I have the EDITOR variable set to emacsclient (available for Linux and Mac).

Let us commit now and check the status.

RUN:

git commit

You will see the following in your editor:

RESULT:

|
# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
# On branch master
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#   modified:   hello.html
#

On the first line, enter the comment: “Added hi tag”. Save the file and exit the editor (to do it in default editor, press ESC and then type:wq and hit Enter). You should see …

RESULT:

git commit
Waiting for Emacs...
[master 569aa96] Added h1 tag
 1 files changed, 1 insertions(+), 1 deletions(-)

“Waiting for Emacs…” is obtained from the emacsclient program sending the file to a running emacs program and waiting for it to be closed. The rest of the data is the standard commit messages.

02Checking the status

At the end let us check the status.

RUN:

git status

You will see …

RESULT:

$ git status
# On branch master
nothing to commit (working directory clean)

The working directory is clean, you can continue working

Changes, not files

Understanding that git works with the changes, not the files.

Most version control systems work with files. You add the file to source control and the system tracks changes from that moment on.

Git concentrates on the changes to a file, not the file itself. A git add file command does not tell git to add the file to the repository, but to note the current state of the file for it to be commited later.

We will try to investigate the difference in this lesson.

First Change:

FILE: HELLO.txt

First Change

Add this change

Now add this change to the git staging.

RUN:

git add hello.html

Second change:

FILE: HELLO.txt

Second Change

Check the current status

RUN:

git status

You will see …

RESULT:

$ git status
# On branch master
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#   modified:   hello.txt
#
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#   modified:   hello.txt
#

Please note that hello.txt is listed in the status twice. The first change (the addition of default tags) is staged and ready for a commit. The second change (adding HTML headers) is unstaged. If you were making a commit right now, headers would not have been saved to the repository.

Let’s check.

05Commit

Commit the staged changes (default values), then check the status one more time.

RUN:

git commit -m "Added second "
git status

You will see …

RESULT:

$ git commit -m "Added second"
[master 8c32287] Added standard HTML page tags
 1 files changed, 3 insertions(+), 1 deletions(-)
$ git status
# On branch master
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#   modified:   hello.txt
#
no changes added to commit (use "git add" and/or "git commit -a")

The status command suggests that hello.html has unrecorded changes, but is no longer in the buffer zone.

Adding the second change

Add the second change to the staging area, after that run the git status command.

RUN:

git add .
git status

Note: The current directory (‘.’) will be our file to add. This is the most convenient way to add all the changes to the files of the current directory and its folders. But since it adds everything, it is a good idea to check the status prior to doing an add ., to make sure you don’t add any file that should not be added.

I wanted you to see the “add .” trick, and we will continue adding explicit files later on just in case.

You will see …

RESULT:

$ git status
# On branch master
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#   modified:   hello.html
#

The second change has been staged and is ready for a commit.

Commit the second change

RUN:

git commit -m "Added second change"

History

To learn to view the project’s history.

Getting a list of changes made is a function of the git log command.

RUN:

git log

You will see …

RESULT:

commit ef44671e9b7aef27027f9d2b438e366d3133102b
Author: Aravind G V <aravind_g@?—?— >
Date: Wed Apr 6 13:09:55 2016 +0530

XIADMIN windows scripts

commit 70d87e94e7937f8a6ce89d4cd6001d99abfb4e77
Author: Neelam Malik <Neelam_Malik@?—?>
Date: Wed Apr 6 11:21:54 2016 +0530

added app_stop

commit 77ac1140018714a6b26c27535c6279eff40c87e9
Author: Neelam Malik <aravind_gv@?—?>
Date: Wed Apr 6 11:21:32 2016 +0530

added app_status

commit 36fb85091deabfe3e590a101aa1a1aca1adfa8c2
Author: Neelam Malik ?@?—?.om>
Date: Wed Apr 6 11:21:04 2016 +0530

added app_start

added app_stop

commit e4be69a36c3a37203e2b539f97eb77b3f253fe99
Author: Neelam Malik <aravind_gv@?—?>
Date: Wed Apr 6 11:18:42 2016 +0530

One line history

You fully control over what the log shows. I like the single line format:

RUN:

git log --pretty=oneline

You will see …

RESULT:

$ git log --pretty=oneline
fa3c1411aa09441695a9e645d4371e8d749da1dc Added HTML header
8c3228730ed03116815a5cc682e8105e7d981928 Added standard HTML page tags
43628f779cb333dd30d78186499f93638107f70b Added h1 tag
911e8c91caeab8d30ad16d56746cbd6eef72dc4c First Commit

Controlling the display of entries

There are many options to choose which entries appear in the log. Play around with the following parameters:

git log --pretty=oneline --max-count=2
git log --pretty=oneline --since='5 minutes ago'
git log --pretty=oneline --until='5 minutes ago'
git log --pretty=oneline --author=<your name>
git log --pretty=oneline --all

Details are provided in the git-log instruction.

Getting fancy

This is what I use to review the changes made within the last week. I will add?—?author=alex if I want to see only the changes made by me.

git log --all --pretty=format:"%h %cd %s (%an)" --since='7 days ago'

The ultimate format of the log

Over time, I found the following log format to be the most suitable.

RUN:

git log --pretty=format:"%h %ad | %s%d [%an]" --graph --date=short

It looks like this:

RESULT:

$ git log --pretty=format:"%h %ad | %s%d [%an]" --graph --date=short
* fa3c141 2011-03-09 | Added HTML header (HEAD, master) [Alexander Shvets]
* 8c32287 2011-03-09 | Added standard HTML page tags [Alexander Shvets]
* 43628f7 2011-03-09 | Added h1 tag [Alexander Shvets]
* 911e8c9 2011-03-09 | First Commit [Alexander Shvets]

Let’s look at it in detail:

  • — pretty=”…” defines the output format.
  • %h is the abbreviated hash of the commit
  • %d commit decorations (e.g. branch heads or tags)
  • %ad is the commit date
  • %s is the comment
  • %an is the name of the author
  • — graph tells git to display the commit tree in the form of an ASCII graph layout
  • — date=short keeps the date format short and nice

So, every time you want to see a log, you’ll have to do a lot of typing. Fortunately, we will find out about the git aliases in the next lesson.

Other tools

Both gitx (for Mac) and gitk (for any platform) can help to explore log history.

Aliases

Command aliases (optional)

Common aliases

For Windows users:

RUN:

git config --global alias.co checkout
git config --global alias.ci commit
git config --global alias.st status
git config --global alias.br branch
git config --global alias.hist 'log --pretty=format:"%h %ad | %s%d [%an]" --graph --date=short'
git config --global alias.type 'cat-file -t'
git config --global alias.dump 'cat-file -p'

Also, for users of Unix/Mac:

git status, git add, git commit, and git checkout are common commands so it is a good idea to have abbreviations for them.

Add the following to the .gitconfig file in your $HOME directory.

.GITCONFIG

[alias]
  co = checkout
  ci = commit
  st = status
  br = branch
  hist = log --pretty=format:\"%h %ad | %s%d [%an]\" --graph --date=short
  type = cat-file -t
  dump = cat-file -p

We’ve already talked about commit and status commands. In the previous lesson we covered the log command and will get to know the checkout command very soon. The most important thing to learn from this lesson is that you can type git st wherever you had to typegit status. Best of all, the git hist command will help you avoid the really long log command.

Go ahead and try using the new commands.

If your shell supports aliases, or shortcuts, you can add aliases on this level, too. I use:

alias gs='git status '
alias ga='git add '
alias gb='git branch '
alias gc='git commit'
alias gd='git diff'
alias go='git checkout '
alias gk='gitk --all&'
alias gx='gitx --all'
alias got='git '
alias get='git '

The go abbreviation for git checkout is very useful, allowing me to type:

go <branch>

to checkout a particular branch.

Also, I often mistype git as get or got so I created aliases for them too.

Getting older versions

To learn how to checkout any previous snapshot into the working directory.

Going back in history is very simple. The checkout command can copy any snapshot from the repo to the working directory.

Getting hashes for the previous versions

RUN:

git hist

Note: Do not forget to define hist in your .gitconfig file? If you do not remember how, review the lesson on aliases.

RESULT:

$ git hist
* 7358571 2016-06-23 | Second Test (HEAD -> master) [Aravind G V]
* 2fc4372 2016-06-23 | First Commit [Aravind G V]

Check the log data and find the hash for the first commit. You will find it in the last line of the git hist data. Use the code (its first 7 chars are enough) in the command below. After that check the contents of the hello.html file.

RUN:

git checkout <hash>
cat hello.txt

Note: Many commands depend on the hash values in the repository. Since my hash values will be different from yours, substitute in the appropriate hash value for your repository everytime you see <hash> or <treehash> in the command.

You will see …

RESULT:

$ git checkout 911e8c9
Note: checking out '911e8c9'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
  git checkout -b new_branch_name
HEAD is now at 911e8c9... First Commit
$ cat hello.txt
Hello, World

The checkout command output totally clarifies the situation. Older git versions will complain about not being on a local branch. But you don’t need to worry about that right now.

Note that the content of the hello.html file is the default content.

Returning to the latest version in the master branch

RUN:

git checkout master
cat hello.txt

‘master’ is the name of the default branch. By checking out a branch by name, you go to its latest version.

Tagging versions

To learn how to tag commits for future references

Let’s call the current version of the hello program version 1 (v1).

Creating a tag of the first

RUN:

git tag v1

Now, the current version of the page is referred to as v1.

Tags for previous versions

Let’s tag the version prior to the current version with the name v1-beta. First of all we will checkout the previous version. Instead of looking up the hash, we are going to use the ^ notation indicating “the parent of v1”.

If the v1^ notation causes troubles, try using v1~1, referencing the same version. This notation means “the first version prior to v1”.

RUN:

git checkout v1^
cat hello.html

RUN:

$ git checkout v1^
Note: checking out 'v1^'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
  git checkout -b new_branch_name
HEAD is now at 8c32287... Added standard HTML page tags
$ cat hello.html

.

RUN:

git tag v1-beta

Check out by the tag name

Now try to checkout between the two tagged versions.

RUN:

git checkout v1
git checkout v1-beta

RESULT:

$ git checkout v1
Previous HEAD position was 8c32287... Added standard HTML page tags
HEAD is now at fa3c141... Added HTML header
$ git checkout v1-beta
Previous HEAD position was fa3c141... Added HTML header
HEAD is now at 8c32287... Added standard HTML page tags

04Viewing tags with the tag command

You can see the available tags using the git tag command.

RUN:

git tag

RESULT:

$ git tag
v1
v1-beta

Viewing tags in logs

You can also check for tags in the log.

RUN:

git hist master --all

RESULT:

$ git hist master --all
* fa3c141 2011-03-09 | Added HTML header (v1, master) [Alexander Shvets]
* 8c32287 2011-03-09 | Added standard HTML page tags (HEAD, v1-beta) [Alexander Shvets]
* 43628f7 2011-03-09 | Added h1 tag [Alexander Shvets]
* 911e8c9 2011-03-09 | First Commit [Alexander Shvets]

You can see tags (v1 and v1-beta) listed in the log together with the name of the branch (master). The HEAD shows the commit you checked out (currently v1-beta).

Discarding local changes (before staging)

To learn how to discard the working directory changes

Checking out the Master branch

Make sure you are on the lastest commit in the master brach before you continue.

RUN:

git checkout master

Change hello.txt

It happens that you modify a file in your local working directory and sometimes wish just to discard the committed changes. Here is when the checkout command will help you.

Make changes to the hello.html file in the form of an unwanted comment.

FILE: HELLO.txt

test-1
test-2
no need of this line

Check the status

First of all, check the working directory’s status.

RUN:

git status

Undoing the changes in the working directory

Use the checkout command in order to checkout the repository’s version of the hello.html file.

RUN:

git checkout hello.txt
git status
cat hello.txt

Cancel Staged changes (before committing)

To learn how to undo changes that have been staged

Edit file and stage changes

Make changes to the hello.html file in the form of an unwanted comment

FILE: HELLO.txt

test-1
test-2
no need of this line

Stage the modified file.

RUN:

git add hello.txt

Check the status

Check the status of unwanted changes .

RUN:

git status

RESULT:

$ git status
# On branch master
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#   modified:   hello.txt
#

Status shows that the change has been staged and is ready to commit.

Reset the buffer zone

Fortunately, the displayed status shows us exactly what we should do to cancel staged changes.

RUN:

git reset HEAD hello.txt

RESULT:

$ git reset HEAD hello.html
Unstaged changes after reset:
M   hello.html

The reset command resets the buffer zone to HEAD. This clears the buffer zone from the changes that we have just staged.

The reset command (default) does not change the working directory. Therefore, the working directory still contains unwanted comments. We can use the checkout command from the previous tutorial to remove unwanted changes from working directory.

Switch to commit version

RUN:

git checkout hello.html
git status

RESULT:

$ git status
# On branch master
nothing to commit (working directory clean)

Our working directory is clean again.

Removing a commit from a branch

To learn to delete the branch’s latest commits

Revert is a powerful command of the previous section that allows you to cancel any commits to the repository. However, both original and cancelled commits are seen in the history of the branch (when using git log command).

Often after commit is already made, we realize that it was a mistake. It would be nice to have undo option command allowing deleting incorrect commit immediately. This command would prevent the appearance of unwanted commit in the git log history.

The reset command

We have already used reset command to match the buffer zone and the selected commit (HEAD commit was used in the previous lesson).

When a commit reference is given (ie, a branch, hash, or tag name), the reset command will…

  1. Overwrite the current branch so it will point to the correct commit
  2. Optionally reset the buffer zone so it will comply with the specified commit
  3. Optionally reset the working directory so it will match the specified commit

02Check our history

Let us do a quick scan of our commit history.

RUN:

git hist

We see that the last two commits in this branch are “Oops” and “Revert Oops”. Let us remove them with reset command.

Mark this branch first

Let us mark the last commit with tag, so you can find it after removing commits.

RUN:

git tag oops

Reset commit to previous Oops

At the history log (see above), the commit tagged «v1» is committing previous wrong commit. Let us reset the branch to that point. As the branch has a tag, we can use the tag name in the reset command (if it does not have a tag, we can use the hash value).

RUN:

git reset --hard v1
git hist

RESULT:

$ git reset --hard v1
HEAD is now at fa3c141 Added HTML header

Our master branch is pointing at commit v1 and “Revert Oops” and “Oops” commits no longer exist in the branch. The?—?hard parameter points out that the working directory must be updated to reflect the new branch head.

Nothing is ever lost

What happens to the wrong commits? They are still in the repository. Actually, we still can refer to them. At the beginning of the lesson, we created «oops» tag for the canceled commit. Let us take a look at all commits.

RUN:

git hist --all

We can see that the wrong commits haven’t gone. They are not listed in the master branch anymore but still remain in the repository. They would be still in the repository if we did not tag them, but then we could reference them only by their hash names. Unreferenced commits remain in the repository until the garbage collection software is run by system.

Reset dangers

Resets on local branches are usually harmless. The consequences of any “accident” can be reverted by using the proper commit.

Though, other users sharing the branch can be confused if the branch is shared on remote repositories.

Removing the tag

Oops tag has performed it’s function. Let us remove that tag and permit the garbage collector to delete referenced commit.

RUN:

git tag -d oops
git hist --all

Changing commits

Goals

  • To learn how to modify an already existing commit

Change the page and commit

Post an author comment on the page.

git add hello.html
git commit -m "Add an author comment"

Change the previous commit

We do not want to create another commit for some update. Let us change the previous commit and add an update.

git add hello.html
git commit --amend -m "Add an author/email comment"

View history

git hist

Moving files

Now we will create the structure of our repository. Let us move the page in the lib directory

RUN:

mkdir lib
git mv hello.html lib
git status

Moving files with git, we notify git about two things

  1. The hello.html file was deleted.
  2. The lib/hello.html file was created.

Both facts are staged immediately and ready for a commit. Git status command reports the file has been moved.

One more way to move files

A positive fact about git is that you don’t need to remember about version control to the moment when you need to commit code. What could happen if we were using the operating system command line instead of the git command to move files?

The following set of commands turned out to be identical to our last actions. It requires more work with same result.

We can do:

mkdir lib
mv hello.txt lib
git add lib/hello.txt
git rm hello.txt

Commit new directory

Let us commit this movement.

RUN:

git commit -m "Moved hello.html to lib"

Creating a Branch

Goals

  • To learn how to create a local branch in the repository

It is time to make our hello world more expressive. Since it may take some time, it is best to move these changes into a new branch to isolate them from master branch changes.

Create a branch

Let us name our new branch «testbranch».

RUN:

git checkout -b testbranch
git status

Note: git checkout -b <branch name> is the shortcuts for git branch <branch name> followed by agit checkout <branch name>.

Note that the git status command reports that you are in the testbranch branch.Make some changes and stage the changes.

Navigating Branches

Now your project has two branches:

RUN:

git hist --all

Switching to the Master branch

To switch between branches simply use the git checkout command.

RUN:

git checkout master
cat lib/hello.txt

Changes to master branch

To learn how to work with several branches with different (sometimes conflicting) changes.

At the time you are changing the style branch, someone decided to change the master branch. He added a README file.

Commit changes of README file in the master branch.

RUN:

git checkout master
git add README
git commit -m "Added README"

View the different branches

Now we have a repository with two different branches. To view branches and their differences use log command as follows.

RUN:

git hist --all

We have opportunity to see?—?graph of git hist in action. Adding the?—?graph option to git log causes the construction of a commit tree with the help of simple ASCII characters. We see both branches (style and master) and that the current branch is master HEAD. The Added index.html branch goes prior to both branches.

The?—?all flag guarantees that we see all the branches. By default, only the current branch is displayed.

Merging to a single branch

Merging brings changes from two branches into one. Let us go back to the testbranch branch and merge it with master.

RUN:

git checkout style
git merge master
git hist --all

Creating and Resolving conflict

Return to the master and create conflict

Return to the master branch and make some changes:

git checkout master
vi lib/hello.txt
Test Status
Creating conflict
git add lib/hello.html
git commit -m 'Life is great!'

(Warning: make sure you’ve used single-quotes to avoid problems with bash and ! character)

View branches

git hist --all

Merge the master branch with testbranch

Let us go back to the testbranch branch and merge it with a new master branch.

git checkout testbranch
git merge master

If you open the lib/hello.html you will see conflict errors

Resolution of the conflict

You need to resolve the conflict manually. Make changes to lib/hello.html to achieve the following result.

Now that you’ve got a good handle on Git, let’s look at GitHub. I’m keen not to overwhelm you, so I’ve made an annotated screenshot of a GitHub project, so that you can quickly become familiar with the most common features. Yes, GitHub is more than simply a project repository, but that’s where you’re likely going to spend most of your time on the site.

Setup and Run Jenkins

Installing Jenkins:

Download latest or specific version you want to install from here.

For example:

$ wget https://updates.jenkins-ci.org/download/war/2.123/jenkins.war

Starting Jenkins

The easiest way to execute Jenkins is through the built in Jetty servlet container. You can execute Jenkins like this:

$ java -jar jenkins.war

Of course, you probably want to send the output of Jenkins to a log file, and if you’re on Unix, you probably want to use nohup:

$ nohup java -jar jenkins.war > $LOGFILE 2>&1 &

Accessing Jenkins

To see Jenkins, simply bring up a web browser and go to URL http://myServer:8080 where myServer is the name of the system running Jenkins.

How Remove Files completely from git repository history

It is straight forward to remove a file from the current commit or HEAD but if you want remove entirely from the repository’s history then you need to run couple of commands.

It took some trial and error for me to find something that worked for me

  • Find out the path fo the file in your repo
  • Execute Below command with file path
git filter-branch --index-filter "git rm -rf --cached --ignore-unmatch path_to_file" HEAD

It’s a time intensive task might takes good amount of time to complete. As it has to check each commit and remove. For my repo which is ~10 years old took almost 10hours with 90K commits.

  • If you want to push it to remote repo just do git push
git push -all

If you like it and want to spread this please clap it as mush as possible.

Create a HTTPS proxy for jenkins using NGINX

Create a HTTPS proxy for jenkins using NGINX

In situations where you have existing web sites on your server, you may find it useful to run Jenkins (or the servlet container that Jenkins runs in) behind Nginx, so that you can bind Jenkins to the part of a bigger website that you may have. This document discusses some of the approaches for doing this.

 

When a request arrives for certain URLs, Nginx becomes a proxy and further forward that request to Jenkins, then it forwards the response back to the client. A typical set up for mod_proxy would look like this:

When using SSL, you might want to use something like the below nginx config.

  • Terminate SSL connection at nginx
  • Proxy it internally to Jenkins on port 8080
  • Replace the Location Header of Jenkins with https instead of http

Note that the third point is pretty tricky. We use proxy_redirect http:// https://; that corresponds to Apaches’s ProxyPassReverse


upstream jenkins {
  server 127.0.0.1:8080 fail_timeout=0;
}

server {
  listen 80 default;
  server_name 127.0.0.1 *.mydomain.com;
  rewrite ^ https://$server_name$request_uri? permanent;
}

server {
  listen 443 default ssl;
  server_name 127.0.0.1 *.mydomain.com;

  ssl_certificate           /etc/ssl/certs/my.crt;
  ssl_certificate_key       /etc/ssl/private/my.key;

  ssl_session_timeout  5m;
  ssl_protocols  SSLv3 TLSv1;
  ssl_ciphers HIGH:!ADH:!MD5;
  ssl_prefer_server_ciphers on;

  # auth_basic            "Restricted";
  # auth_basic_user_file  /home/jenkins/htpasswd;

  location / {
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-Proto https;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_redirect http:// https://;

    add_header Pragma "no-cache";

    proxy_pass http://jenkins;
  }
}