Mohan WebSite

2012 SERVER (64)
2016 windows (9)
AIX (13)
Amazon (34)
Ansibile (19)
Apache (133)
Asterisk (2)
cassandra (2)
Centos (228)
- ftp (5)
- IPTABLES (15)
- SFTP (1)
- swap (3)
Centos RHEL 7 (264)
chef (3)
cloud (2)
cluster (3)
Coherence (1)
DB2 (5)
DISK (25)
DNS (9)
Docker (30)
Eassy (11)
ELKS (1)
EXCHANGE (3)
Fedora (6)
GIT (3)
GOD (2)
Grub (1)
Hacking (10)
Hadoop (6)
horoscope (23)
Hyper-V (10)
IIS (15)
JAVA (7)
JBOSS (32)
jenkins (1)
Kubernetes (3)
Ldap (5)
Linux (188)
Linux Commands (166)
Load balancer (5)
mariadb (14)
Mongodb (4)
MQ Server (24)
MYSQL (84)
Nagios (5)
NaturalOil (13)
Nginx (32)
Ngix (1)
openldap (1)
Openstack (6)
Oracle (34)
Perl (3)
Postfix (19)
Postgresql (1)
PowerShell (2)
Python (3)
qmail (36)
Redis (12)
RHCE (28)
SCALEIO (1)
Security on Centos (29)
Shell (64)
Solaris (58)
Sql Server 2012 (4)
squid (3)
SSH (10)
SSL (14)
Storage (1)
TIPS on Linux (28)
tomcat (61)
Uncategorized (29)
Veritas (2)
vfabric (1)
VMware (28)
Weblogic (38)
Websphere (71)
Windows (19)
Windows Software (2)
wordpress (1)
ZIMBRA (17)

WP Cumulus Flash tag cloud by Roy Tanck requires Flash Player 9 or better.

How to Fix Redis CLI Error Connection Reset by Peer

Overview

Recently we faced an issue with an AWS ElastiCache Redis instance when trying to test the connections from EC2 Instance using Redis CLI, we faced the following error

$ ./redis-cli -c -h my-redis-server -p 6379 my-redis-server:6379> set a “hello” Error: Connection reset by peer

Problem

On investigation, we found that the ElastiCache Redis Instance is using Encryption in-transit and Encryption at-rest and by design, the Redis CLI is not compatible with the encryption.

Solution

The solution to test the connectivity and to use the Redis CLI with ElastiCache In-Transit encryption, we needed to configure ‘stunnel’

Stunnel is a proxy designed to add TLS encryption functionality to existing clients and servers without any changes in the programs’ code

With stunnel client would create a SSL tunnel to the Redis nodes and use redis-cli to connect through the tunnel to access data from encrypted redis nodes.

Here is how to setup everything, we are using Amazon Linux in this example but same steps should work on Redhat Linux

1. Install stunnel


$ sudo yum install stunnel  -y

2. Configure SSL tunnel for redis-cli


$ sudo vi /etc/stunnel/redis-cli.conf
 Set the following properties in redis-cli.conf file 
fips = no
setuid = root
setgid = root
pid = /var/run/stunnel.pid
debug = 7
options = NO_SSLv2
options = NO_SSLv3
[redis-cli]
client = yes
accept = 127.0.0.1:6379
connect = my-redis-server:6379

3. Start Stunnel


$ sudo stunnel /etc/stunnel/redis-cli.conf

4. Verify the tunnel is running


$ sudo netstat -tulnp | grep -i stunnel
 You might see following output from the above command 
tcp 0 0 127.0.0.1:6379 0.0.0.0:* LISTEN 1314 stunnel

5. Last is to connect to Redis cluster using Redis CLI using SSL tunnel (Yes it is connecting using localhost tunnel)


redis-cli -h localhost -p 6379

Note: To install Redis CLI on Linux check this AWS documentation

6. Run few Redis commands to see if it works



$ ./redis-cli -h localhost -p 6379
localhost:6379> set a hello
OK
localhost:6379> get a
"hello"
localhost:6379>

Hope you find this post useful, please leave a comment and let us know what topics you would like to see.

Elasticsearch cluster deployment under CentOS7.3

The bottom layer of Elastic is the open source library Lucene. However, you can’t use Lucene directly, you have to write your own code to call its interface. Elastic is a package of Lucene that provides an operational interface to the REST API, out of the box. The bottom layer of Elastic is the open source library. However, you can’t use Lucene directly, you have to write your own code to call its interface. Elastic is a package of Lucene that provides an operational interface to the REST API, out of the box.

First, the basic concept in ES
Cluster
Represents a cluster. There are multiple nodes in the cluster. One of them is the master node. This master node can be elected. The master and slave nodes are internal to the cluster. A concept of es is decentralization. The literal understanding is that there is no central node. This is for the outside of the cluster, because from the outside, the es cluster is logically a whole, and you communicate with any node and the whole. Es cluster communication is equivalent.

Shards
On behalf of index shards, es can divide a complete index into multiple shards. This has the advantage of splitting a large index into multiple nodes and distributing them to different nodes. Form a distributed search. The number of shards can only be specified before the index is created, and cannot be changed after the index is created.

Replica
On behalf of the index copy, es can set a copy of multiple indexes. The role of the copy is to improve the fault tolerance of the system. When a certain piece of a node is damaged or lost, it can be recovered from the copy. The second is to improve the efficiency of es query, es will automatically load balance the search request.

Recovery
Representing data recovery or data redistribution, es will re-allocate index fragments according to the load of the machine when a node joins or exits, and data recovery will be performed when the suspended node restarts.

River
A data source representing es, and a method of synchronizing data to es by other storage methods (such as databases). It is an es service that exists as a plugin. By reading the data in the river and indexing it into es, the official river has couchDB, RabbitMQ, Twitter, and Wikipedia.

Gateway
Represents the storage mode of the es index snapshot. es defaults to storing the index in memory first, and then persisting to the local hard disk when the memory is full. The gateway stores the index snapshot. When the es cluster is closed and restarted, the index backup data is read from the gateway. Es supports multiple types of gateways, including local file system (default), distributed file system, Hadoop HDFS and amazon s3 cloud storage service.

Discovery.zen
On behalf of es’ automatic discovery node mechanism, es is a p2p-based system. It first searches for existing nodes through broadcast, and then communicates between nodes through multicast protocols, and also supports peer-to-peer interaction.

Transport
Represents the internal node of es or the interaction between the cluster and the client. The default internal interaction is using tcp protocol. At the same time, it supports the transmission protocol of http protocol (json format), thrift, servlet, memcached, zeroMQ, etc. (integrated through plug-in).

Second, the deployment environment
The deployment of Elasticsearch clusters with three CentOS 7.3s requires index fragmentation when deploying Elasticsearch clusters. The following is a brief introduction to index sharding.

system Node name IP
CentOS7.3 Els1 172.18.68.11
CentOS7.3 Els2 172.18.68.12
CentOS7.3 Els3 172.18.68.13
An index in an ES cluster may consist of multiple shards, and each shard can have multiple copies. By dividing a single index into multiple shards, we can handle large indexes that cannot be run on a single server. Simply put, the size of the index is too large, causing efficiency problems. The reason you can’t run is probably memory or storage. Since each shard can have multiple copies, you can increase the load capacity of the query by assigning copies to multiple servers.

Third, deploy Elasticsearch cluster
1. Install JDK
Elasticsearch is based on Java development and is a Java program that runs in Jvm, so the first step is to install JDK.

yum install -y java-1.8.0-openjdk-devel
2. Download elasticsearch
Https://artifacts.elastic.co/downloads/elasticsearch/ is the official site of ELasticsearch. If you need to download the latest version, you can download it from the official website. It can be downloaded to a local computer and then imported into CentOS, or it can be downloaded directly from CentOS.

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.0.1.rpm
3. Configuration directory
After the installation is complete, many files will be generated, including configuration file log files, etc. The following are the most important configuration file paths.

/etc/elasticsearch/elasticsearch.yml # els
/etc/elasticsearch/jvm.options # JVM
/etc/elasticsearch/log4j2.properties
/var/lib/elasticsearch
4. Create a directory for storing data and logs
The data file will grow rapidly with the system running, so the default log file and data file path can not meet our needs, then manually create the log and data file path, you can use NFS, you can use Raid, etc. to facilitate future management and Expansion

mkdir /els/{log,date}
chown -R elasticsearch.elasticsearch /els/*
5. Cluster configuration
The most important cluster configuration is two node.nameand network.hosteach node must be unreasonable. Among them node.nameis the node name mainly in Elasticsearch’s own log to distinguish each node information.
discovery.zen.ping.unicast.hostsIt is the node information in the cluster. You can use the IP address and you can use the host name (you must be able to resolve it).

vim /etc/elasticsearch
cluster.name: aubin-cluster
node.name: els1

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

network.host: 192.168.0.1
http.port: 9200

discovery.zen.ping.unicast.hosts: [“172.18.68.11”, “172.18.68.12”,”172.18.68.13″]

discovery.zen.minimum_master_nodes: 2
6.JVM configuration
Since Elasticsearch is developed in Java, you can /etc/elasticsearch/jvm.optionsset the JVM settings through a configuration file. If there is no special requirement, you can press the default.
However, there are still two of the most important minimum memory -Xmx1gwith the -Xms1gJVM. If it is too small, it will cause Elasticsearch to stop as soon as it starts. Too slow to slow down the system itself

vim /etc/elasticsearch/jvm.options
-Xms1g # JVM
-Xmx1g
7. Start Elasticsearch
Since launching Elasticsearch will automatically start daemon-reload, the last item can be omitted.

systemctl enable elasticsearch.service
systemctl start elasticsearch
systemctl daemon-reload
8. Testing
Elasticsearch has directly heard the http interface, so you can view some cluster-related information directly using the curl command.

You can use the curl command to get information about the cluster.

_cat stands for viewing information
The nodes are for viewing node information, the default will be displayed as one line, so use the knife? Preety to make the information better display
?preety makes the output information more friendly display
curl -XGET ‘http://172.18.68.11:9200/_cat/nodes?pretty’
172.18.68.12 18 68 0 0.07 0.06 0.05 mdi – els2
172.18.68.13 25 67 0 0.01 0.02 0.05 mdi * els3
172.18.68.11 7 95 0 0.02 0.04 0.05 mdi – els1
If you want to see more about cluster information, current node statistics, etc., you can use the following command to get all the information you can view.

curl -XGET ‘http://172.18.68.11:9200/_cat?pretty’

apache redirect

Create your own short links with the mod_rewrite feature of the Apache HTTP server.

A long time ago, people started sharing links on Twitter. The 140-character limit means that the URL may consume most (or all) of a tweet, so people use URLs to shorten the service. In the end, Twitter added a built-in URL shortening service ( t.co ).

The number of characters is not important now, but there are other reasons to shorten the link. First, shortening the service can provide analytics—you can see the popularity of the links you share. It also simplifies making URLs that are easy to remember. For example, bit.ly/INtravel is easier to remember than https://www.in.gov/ai/appfiles/dhs-countyMap/dhsCountyMap.html . If you want to share a link in advance, but you don’t know the final address, then the URL shortening service can come in handy. .

As with any technology, URL shortening services are not all positive. By blocking the final address, shortened links can be used to point to malicious or offensive content. However, if you go online, the URL shortening service is a useful tool.

We previously posted an article on shortening the service on the website , but maybe you want to run some shortened services supported by simple text files. In this article, we’ll show you how to set up your own URL shortening service using the mod_rewrite feature of the Apache HTTP server. If you are not familiar with the Apache HTTP server, check out David Both ‘s article on installing and configuring it.

Create a VirtualHost
In this tutorial, I assume that you have purchased a cool domain name that you specifically use for the URL shortening service. For example, my website is funnelfiasco.com , so I bought funnelfias.co for my URL shortening service (well, it’s not very short, but it can satisfy my vanity). If you are not running the shortened service as a separate domain, skip to the next section.

The first step is to set up the VirtualHost that will be used for the URL shortening service. For more information on VirtualHost, see the article by David Both . This step only takes a few lines:

<VirtualHost *:80>
ServerName rmohan.com
</VirtualHost>

Create a rewrite rule
This service uses the HTTPD rewrite engine to rewrite the URL. If you created VirtualHost in the section above, the following configuration jumps to your VirtualHost section. Otherwise jump to the server’s VirtualHost or primary HTTPD configuration.

RewriteEngine on
RewriteMap shortlinks txt:/data/web/shortlink/links.txt
RewriteRule^/(.+)$ ${shortlinks:$1}[R=temp,L]
The first line just enables the rewrite engine. The second line builds a short link mapping in a text file. The path above is just an example. You need to use a valid path on your system (make sure it can be read by a user account running HTTPD). The last line rewrites the URL. In this case, it accepts any characters and looks them up in the rewrite map. You may want to use a specific string when rewriting. For example, if you want all shortened links are “slX” (where X is a number), then the above (.+)is replaced (sl\d+).

I am using a temporary redirect (HTTP 302) here. This will allow me to update the target URL later. If you want short links to always point to the same destination, you can use a permanent redirect (HTTP 301). By permanentreplacing the third row temp.

Build your map
Edit the configuration file RewriteMapspecifies the file line. The format is a space-separated key-value store. Put a link on each line:

osdc https://opensource.com/users/bcotton
twitter https://twitter.com/funnelfiasco
swody1 https://www.spc.noaa.gov/products/outlook/day1otlk.html

Restart HTTPD
The final step is to restart the HTTPD process. This is done by systemctl restart httpdsimilar command or completed (commands and daemons names may differ due to release). Your link shortening service is now up and running. There is no need to restart the web server when you are ready to edit the map. All you have to do is save the file and the web server will get the difference.

future career
This example gives you a basic URL shortening service. If you want to develop your own management interface as a learning project, it can be a good starting point. Or you can use it to share easy-to-remember links to URLs that are easy to forget.

GC Details

Third, the option parameters explain
1, heap size settings
1 -Xmx3550m -Xms3550m -Xmn2g -Xss128k

-Xmx 3550m: Set the maximum available memory of the JVM to 3550M.

-Xms 3550m: Set the JVM initial memory to 3550m. This value can be set to be the same as -Xmx to avoid the JVM reallocating memory after each garbage collection.

-Xmn 2g: Set the young generation size to 2G. The entire heap size = young generation size + age generation size + permanent generation size. The permanent generation has a fixed size of 64m, so increasing the young generation will reduce the size of the old generation. This value has a great impact on system performance. Sun officially recommends 3/8 of the entire heap.

-Xss 128k: Set the stack size for each thread . After JDK 5.0, the stack size of each thread is 1M, and the previous stack size of each thread is 256K. The memory size required for more application threads is adjusted. Under the same physical memory, reducing this value can generate more threads. However, the operating system has a limit on the number of threads in a process, and cannot be generated indefinitely. The empirical value is around 3000~5000.

2 -XX:NewRatio=4 -XX:SurvivorRatio=4 -XX:MaxPermSize=16m -XX:MaxTenuringThreshold=0

-XX:NewRatio =4: Set the ratio of the young generation (including Eden and the two Survivor areas) to the old generation (remove the permanent generation). Set to 4, the ratio of young to old generation is 1:4, and younger generation is 1/5 of the whole stack.

-XX:SurvivorRatio =4: Set the size ratio of the Eden area to the Survivor area in the young generation . Set to 4, the ratio of the two Survivor areas to an Eden area is 2:4, and one Survivor area accounts for 1/6 of the entire young generation.

-XX:PermSize : Sets the permanent value (perm gen) initial value. The default is 1/64 of physical memory.

-XX:MaxPermSize : Sets the maximum value of the persistent generation. 1/4 of physical memory.

– XX: MaxTenuringThreshold =0: Set the maximum age of garbage . If set to 0, the young generation object does not go through the Survivor area and directly enters the old generation. For applications with more old generations, efficiency can be improved. If this value is set to a larger value, the younger generation object will be replicated multiple times in the Survivor area, which will increase the survival time of the younger generation of the object and increase the generalization of recycling in the younger generation.

2, recycler selection
(1) Throughput priority parallel collector

1 -XX:+UseParallelGC -XX:ParallelGCThreads=20

-XX:+UseParallelGC : Select the garbage collector as a parallel collector . This configuration is only valid for younger generations . In the above configuration, the young generation uses concurrent collection, while the old generation still uses serial collection.

-XX:ParallelGCThreads =20: Configure the number of threads in the parallel collector , that is: how many threads are garbage collected together. This value is preferably configured to be equal to the number of processors.

2 -XX:+UseParallelOldGC

-XX:+UseParallelOldGC : Configures the old generation garbage collection method to be collected in parallel . JDK 6.0 supports parallel collection of old generations.

3 -XX:MaxGCPauseMillis=100

-XX:MaxGCPauseMillis =100: Set the maximum time for each young generation garbage collection . If this time is not met, the JVM will automatically adjust the young generation size to meet this value.

4 -XX:+UseAdaptiveSizePolicy

-XX:+UseAdaptiveSizePolicy : When this option is set, the parallel collector will automatically select the size of the young generation area and the corresponding proportion of the Survivor area to achieve the minimum corresponding time or collection frequency specified by the target system. This value is recommended when using the parallel collector. , always open.

(2) Response time priority concurrent collector

1 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC

-XX:+UseConcMarkSweepGC : Set the old generation to concurrent collection . After configuring this in the test, the configuration of -XX:NewRatio=4 is invalid, and the reason is unknown. Therefore, at this time, the size of the young generation is best set with -Xmn.

-XX:+UseParNewGC : Set the young generation to collect in parallel. Can be used simultaneously with CMS collection. JDK5.0 or above, the JVM will be set according to the system configuration, so there is no need to set this value.

2 -XX:CMSFullGCsBeforeCompaction=5 -XX:+UseCMSCompactAtFullCollection

-XX:CMSFullGCsBeforeCompaction : Since the concurrent collector does not compress or organize the memory space, it will generate “fragmentation” after running for a period of time, which reduces the running efficiency. This value sets how many times the GC is run to compress and organize the memory space.

-XX:+UseCMSCompactAtFullCollection : Opens the compression of the old generation. May affect performance, but can eliminate fragmentation

3. Auxiliary information
The JVM provides a number of command line arguments to print information for debugging purposes. There are mainly the following:

1 -XX:+PrintGC

Output form:

[GC 118250K->113543K (130112K), 0.0094143 secs]

[Full GC 121376K->10414K (130112K), 0.0650971 secs]

2 -XX:+PrintGCDetails

Output form:

[GC [DefNew: 8614K->781K (9088K), 0.0123035 secs] 118250K->113543K (130112K), 0.0124633 secs]

[GC [DefNew: 8614K->8614K(9088K), 0.0000665 secs][Tenured: 112761K->10414K (121024K), 0.0433488 secs] 121376K->10414K (130112K), 0.0436268 secs]

3 -XX:+PrintGCTimeStamps -XX:+PrintGC:PrintGCTimeStamps can be mixed with the above two

Output form:

11.851: [GC 98328K->93620K (130112K), 0.0082960 secs]

4 -XX:+PrintGCApplicationConcurrentTime: Prints the execution time of the program before it is garbage collected. Can be mixed with the above

Output form:

Application time: 0.5291524 seconds

5 -XX:+PrintGCApplicationStoppedTime: Prints the time the program was paused during garbage collection. Can be mixed with the above

Output form:

Total time for which application threads were stopped: 0.0468229 seconds

6 -XX: PrintHeapAtGC: Print detailed stack information before and after GC

Output form:

34.702: [GC {Heap before gc invocations=7:

Def new generation total 55296K, used 52568K [0x1ebd0000, 0x227d0000, 0x227d0000)

Eden space 49152K, 99% used [0x1ebd0000, 0x21bce430, 0x21bd0000)

From space 6144K, 55% used [0x221d0000, 0x22527e10, 0x227d0000)

To space 6144K, 0% used [0x21bd0000, 0x21bd0000, 0x221d0000)

Tenured generation total 69632K, used 2696K [0x227d0000, 0x26bd0000, 0x26bd0000)

The space 69632K, 3% used [0x227d0000, 0x22a720f8, 0x22a72200, 0x26bd0000)

Compacting perm gen total 8192K, used 2898K [0x26bd0000, 0x273d0000, 0x2abd0000)

The space 8192K, 35% used [0x26bd0000, 0x26ea4ba8, 0x26ea4c00, 0x273d0000)

Ro space 8192K, 66% used [0x2abd0000, 0x2b12bcc0, 0x2b12be00, 0x2b3d0000)

Rw space 12288K, 46% used [0x2b3d0000, 0x2b972060, 0x2b972200, 0x2bfd0000)

34.735: [DefNew: 52568K->3433K (55296K), 0.0072126 secs] 55264K->6615K (124928K) Heap after gc invocations=8:

Def new generation total 55296K, used 3433K [0x1ebd0000, 0x227d0000, 0x227d0000)

Eden space 49152K, 0% used [0x1ebd0000, 0x1ebd0000, 0x21bd0000)

From space 6144K, 55% used [0x21bd0000, 0x21f2a5e8, 0x221d0000)

To space 6144K, 0% used [0x221d0000, 0x221d0000, 0x227d0000)

Tenured generation total 69632K, used 3182K [0x227d0000, 0x26bd0000, 0x26bd0000)

The space 69632K, 4% used [0x227d0000, 0x22aeb958, 0x22aeba00, 0x26bd0000)

Compacting perm gen total 8192K, used 2898K [0x26bd0000, 0x273d0000, 0x2abd0000)

The space 8192K, 35% used [0x26bd0000, 0x26ea4ba8, 0x26ea4c00, 0x273d0000)

Ro space 8192K, 66% used [0x2abd0000, 0x2b12bcc0, 0x2b12be00, 0x2b3d0000)

Rw space 12288K, 46% used [0x2b3d0000, 0x2b972060, 0x2b972200, 0x2bfd0000)

}

, 0.0757599 secs]

7 -Xloggc:filename: Used in conjunction with the above, log related log information to a file for analysis.

Fourth, common configuration summary
1, heap settings
1 -Xms: initial heap size

2 -Xmx: maximum heap size

3 -XX:NewSize=n: set the size of the young generation

4 -XX:NewRatio=n: Set the ratio of young and old generations. For example, it is 3, indicating that the ratio of young to old is 1:3, and the young generation accounts for 1/4 of the young generation.

5 -XX: SurvivorRatio=n: The ratio of the Eden area to the two Survivor areas in the young generation. Note that there are two in the Survivor area. Such as: 3, said Eden: Survivor = 3: 2, a Survivor area accounted for 1/5 of the entire young generation

6 -XX:MaxPermSize=n: Set the permanent generation size

2, collector settings
1 -XX:+UseSerialGC: Set the serial collector

2 -XX:+UseParallelGC: Set the parallel collector

3 -XX:+UseParalledlOldGC: Set parallel old generation collector

4 -XX:+UseConcMarkSweepGC: Set Concurrent Collector

3, garbage collection statistics
1 -XX:+PrintGC

2 -XX:+PrintGCDetails

3 -XX:+PrintGCTimeStamps

4 -Xloggc:filename

4, parallel collector settings
1 -XX:ParallelGCThreads=n: Sets the number of CPUs used for parallel collector collection. Collect threads in parallel.

2 -XX:MaxGCPauseMillis=n: Set the maximum pause time for parallel collection

3 -XX:GCTimeRatio=n: Set the garbage collection time as a percentage of the program runtime. The formula is 1/(1+n)

5, concurrent collector settings
1 -XX:+CMSIncrementalMode: Set to incremental mode. Applicable to single CPU situations.

2 -XX:ParallelGCThreads=n: Set the number of CPUs used by the concurrent collector collection method for parallel collection. Collect threads in parallel.

V. Summary of tuning
1, young generation size selection
1 Response time-first application: Set as large as possible until the system’s minimum response time limit (according to the actual situation). In this case, the frequency of young generation collections is also minimal. At the same time, reduce the number of people reaching the old generation.

2 throughput-priority applications: as large as possible, may reach the level of Gbit. Because there is no requirement for response time, garbage collection can be performed in parallel, and is generally suitable for applications with more than 8 CPUs.

2, the age of old generation choice
1 Response time-first application: The old generation uses the concurrent collector, so its size needs to be carefully set, generally considering some parameters such as concurrent session rate and session duration. If the heap is set small, it can cause memory fragmentation, high recovery frequency, and application pauses using traditional markup cleanup; if the heap is large, it takes longer to collect. The optimized solution generally needs to be obtained by referring to the following data:

Concurrent garbage collection information

Persistent generation concurrent collections

Traditional GC information

Proportion of time spent on recycling young and old generations

Reducing the time spent by young and old generations generally increases the efficiency of the application

2 Throughput-first applications: General throughput-first applications have a large young generation and a smaller older generation. The reason is that this way, most of the short-term objects can be recovered as much as possible, and the medium-term objects can be reduced.

3. Fragmentation problems caused by smaller heaps
Because the older collector’s concurrent collector uses the markup and cleanup algorithms, the heap is not compressed. When the collector recycles, it merges the adjacent spaces so that it can be assigned to larger objects. However, when the heap space is small, after running for a while, “fragmentation” will occur. If the concurrent collector cannot find enough space, the concurrent collector will stop and then use traditional markup and cleanup methods for recycling. If “fragmentation” occurs, you may need to do the following:

1 -XX:+UseCMSCompactAtFullCollection: Turns on compression for the old generation when using the concurrent collector.

2 -XX:CMSFullGCsBeforeCompaction=0: In the case where the above configuration is enabled, how many times the Full GC is set here, the old generation is compressed.

Six, related concepts
1, generational garbage recycling details
(1) Young (young generation)

The young generation is divided into three districts . One Eden area, two Survivor areas . Most of the objects are generated in the Eden area. When the Eden area is full, the surviving objects will be copied to the Survivor area (one of the two). When the Survivor area is full, the surviving objects in this area will be copied to another Survivor area, when the Survivor goes When it is full, the object copied from the first Survivor area and still alive at this time will be copied “Tenured”. It should be noted that the two areas of Survivor are symmetrical and have no relationship, so there may be objects copied from Eden and objects copied from the previous Survivor in the same area, and only the first one copied to the old district. The object that Survivor came over. Moreover, there is always one empty in the Survivor area.

(2) Tenured (old generation)

The old generation stores objects that survive from the young generation . In general, older generations store objects that have a long life span.

(3) Perm (persistent generation)

Used to store static files , Java classes, methods, and more. Persistent generation has no significant impact on garbage collection, but some applications may dynamically generate or call some classes, such as Hibernate, etc. In this case, you need to set a larger persistent generation space to store the new classes in the running process. The persistent generation size is set by -XX:MaxPermSize=.

2, GC type
There are two types of GC: Scavenge GC and Full GC.

(1) Scavenge GC

In general, when a new object is generated and the Eden application space fails, the Scavenge GC is triggered, the heap Eden area is GC, the non-viable objects are cleared, and the surviving objects are moved to the Survivor area. Then organize the two areas of Survivor.

(2) Full GC

Organize the entire heap , including Young, Tenured, and Perm. Full GC is slower than the Scavenge GC, so you should minimize the Full GC . The Full GC may be caused by the following reasons:

Tenured is filled
Perm domain is filled
System.gc() is called to be called
Dynamic change of Heap’s domain allocation strategies after the last GC
3, garbage collector
There are three main types of collectors: serial collectors, parallel collectors, and concurrent collectors.

(1) Serial collector

Use single-threaded processing of all garbage collection work, because it does not require multi-threaded interaction, so it is more efficient. However, the advantages of multiprocessors cannot be used, so this collector is suitable for single processor machines. Of course, this collector can also be used on multiprocessor machines with small data volumes ( around 100M ). It can be opened with -XX:+UseSerialGC .

(2) Parallel collector

Parallel garbage collection for younger generations can reduce garbage collection time. Generally used on multi-threaded multiprocessor machines. Open with -XX:+UseParallelGC . The parallel collector was introduced in the J6SE5.0 sixth 6 update, and was enhanced in Java SE 6.0 – it can be collected in parallel for older generations. If the old generation does not use concurrent collection, it uses single-threaded garbage collection , which will restrict the expansion capability. Open with -XX:+UseParallelOldGC .
Use -XX:ParallelGCThreads=Set the number of threads for parallel garbage collection . This value can be set equal to the number of machine processors .
This collector can be configured as follows:
Maximum garbage collection pause: Specify the maximum pause time for garbage collection, specified by -XX:MaxGCPauseMillis= . In milliseconds. If this value is specified, the heap size and garbage collection related parameters are adjusted to the specified value. Setting this value may reduce the throughput of your application.
Throughput: The ratio of throughput to garbage collection time and non-garbage recovery time is set by -XX:GCTimeRatio= , and the formula is 1/(1+N) . For example, when -XX:GCTimeRatio=19, it means 5% of the time is used for garbage collection. The default is 99, which is 1% of the time for garbage collection.
(3) Concurrent collector

It can guarantee that most of the work is carried out concurrently (the application does not stop), and garbage collection is only suspended for a small amount of time. This collector is suitable for medium and large-scale applications with high response time requirements. Open with -XX:+UseConcMarkSweepGC .

The concurrent collector mainly reduces the pause time of the old generation. He uses a separate garbage collection thread to track the reachable objects without stopping the application. In each old generation garbage collection cycle, the concurrent collector will briefly pause the entire application at the beginning of the collection, and will pause again during the collection. The second pause will be slightly longer than the first one, during which multiple threads will perform garbage collection at the same time.
The concurrent collector uses the processor for a short pause . On a system with N processors, the concurrent collection part is recovered using K/N available processors, typically 1<=K<=N/4 .
Using a concurrent collector on a host with only one processor, a shorter pause can also be achieved by setting to incremental mode .
Floating garbage : Since garbage is collected while the application is running, some garbage may be generated when the garbage collection is completed, which results in “Floating Garbage”, which needs to be recycled in the next garbage collection cycle. Therefore, concurrent collectors typically require 20% of the reserved space for these floating garbage.
Concurrent Mode Failure : The concurrent collector collects when the application is running, so you need to ensure that the heap has enough space for the program to use during the garbage collection. Otherwise, the garbage collection is not completed and the heap space is full. In this case, “concurrency mode failure” will occur, and the entire application will be suspended for garbage collection.
Start Concurrent Collector : Because concurrent collection is collected while the application is running, you must ensure that there is enough memory space for the program to use before the collection is complete, otherwise “Concurrent Mode Failure” will occur. Start concurrent collection by setting -XX:CMSInitiatingOccupancyFraction= Specifying how many remaining heaps there are
(4) Summary

1 serial processor:

— Applicable situation: The data volume is relatively small (about 100M) ; the application under single processor and no response time.

— Disadvantages: only for small applications

2 parallel processor:

— Applicable situation: “High requirements for throughput” , multi-CPU, medium and large-scale applications that do not require application response time. Examples: background processing, scientific computing.

— Disadvantages: Application response time may be longer

3 concurrent processor:

— Applicable situation: “High requirements for response time” , multi-CPU, medium and large-scale applications with high requirements on application response time. For example: web server / application server, telecommunications exchange, integrated development environment.

4, the basic recovery algorithm
(1) Reference Counting

Older recycling algorithm. The principle is that this object has a reference, which is to increment a count, and to delete a reference to reduce a count. In garbage collection, only objects with a count of 0 are collected. The most deadly of this algorithm is the inability to handle circular references.

(2) Mark-Sweep (Mark-Sweep)

This algorithm is executed in two phases. The first stage marks all referenced objects starting from the reference root node, and the second stage traverses the entire heap, clearing the unmarked objects. This algorithm needs to pause the entire application and, at the same time, generate memory fragmentation.

(3) Copying

This algorithm divides the memory space into two equal regions, using only one of the regions at a time. In garbage collection, traverse the current usage area and copy the object in use to another area. The secondary algorithm only processes the objects in use at a time, so the copying cost is relatively small, and the corresponding memory can be sorted after copying in the past, but the “fragmentation” problem occurs. Of course, the shortcoming of this algorithm is also obvious, that is, it requires twice the memory space.

(4) Mark-Compact

This algorithm combines the advantages of both the “mark-clear” and “copy” algorithms. It is also divided into two phases. The first phase marks all referenced objects from the root node. The second phase traverses the entire heap, clears the unlabeled objects and “compresses” the surviving objects into one of the heaps, and discharges them in order. This algorithm avoids the “flag-clear” fragmentation problem and also avoids the space problem of the “copy” algorithm.

(5) Incremental Collecting

Implement a garbage collection algorithm that performs garbage collection while the application is running. I don’t know why the collector in JDK 5.0 does not use this algorithm.

(6) Generational Collecting

Based on the garbage collection algorithm after analyzing the life cycle of the object. The objects are divided into young, old, and permanent generations, and different algorithms (one of the above methods) are used to recover objects of different life cycles. The current garbage collector (starting with J2SE 1.2) uses this algorithm.

COPY and ADD commands in Dockerfile

COPY and ADD commands in Dockerfile

Two very similar commands COPY and ADD are provided in the Dockerfile. This article attempts to explain the basic functions of these two commands, as well as their similarities and differences, and then summarize their respective suitable application scenarios.

Build context concept
When you create a mirror from a Dockerfile using the docker build command, a build context is generated. The so-called build context is the collection of files in the path specified by the PATH or URL of the docker build command. Any file in the context can be referenced during the image build process, such as the COPY and ADD commands we will introduce, to reference the files in the context.

By default, the docker build -t testx . command in the command indicates that the build context is the current directory. Of course we can specify a directory as the context, such as the following command:

$ docker build -t testx /home/mohan/hc

We specify the /home/mohan/hc directory as the build context. By default, docker will use the Dockerfile found in the root of the context.

The COPY and ADD commands cannot copy local files outside the context.
For COPY and ADD commands, if you want to copy a local file to an image, the local file must be a file in the context directory.
In fact, this is a good explanation, because when the build command is executed, the docker client will send all the files in the context to the docker daemon . Considering that the docker client and the docker daemon are not on the same machine,

the build command can only get the file from the context. If we reference a file that is not in the context in the COPY and ADD commands of the Dockerfile, we receive an error similar to the following:

Work with WORKDIR

The WORKDIR command configures the working directory for subsequent commands such as RUN, CMD, COPY, and ADD. After the WORKDIR command is set, the relative path in the next COPY and ADD commands is the path specified relative to WORKDIR. For example, we add the following command to the Dockerfile:

WORKDIR /app
COPY checkredis.py .

Then build a container image named testx and run a container to view the file path:

The checkredis.py file is copied to the WORKDIR /app directory.

The simplicity of the COPY command
If you just copy the local file to the container image, the COPY command is the most appropriate. The format of the command is:
COPY <src> <dest>

In addition to specifying the full file name, the COPY command also supports Go-style wildcards, such as:

COPY check* /testdir/ # Copy all files at the beginning of
check COPY check?.log /testdir/ # ? is a placeholder for a single character, such as matching file check1.log

For directories, the COPY and ADD commands have the same characteristics: only copy the contents of the directory and not the directory itself. For example, we add the following command to the Dockerfile:

WORKDIR /app
COPY mohandir .

The structure of the mohandir directory is as follows:

There are only file1 and file2, and there is one less directory mohandir. If you want file1 and file2 to be saved in the mohandir directory, you need to specify the name of the directory in the target path, for example:

WORKDIR /app
COPY mohandir ./mohandir

One use of the COPY command from the ADD command is in a multistage scenario. For an introduction and usage of multistage, please refer to the author’s article ” multi-stage in Dockerfile “. In the multistage usage, you can use the COPY command to copy the product from the previous stage to another image, such as:

FROM golang: 1.7.3
WORKDIR /go/src/github.com/sparkdevo/href-counter/
RUN go get -d -v golang.org/x/net/html
COPY app.go .
RUN CGO_ENABLED=0 GOOS=linux Go build -a -installsuffix cgo -o app .

FROM alpine:latest
RUN apk –no-cache add ca-certificates
WORKDIR /root/
COPY –from=0 /go/src/github.com/sparkdevo/href-counter/app .
CMD [“./app”]

This code is referenced in the article ” multi-stage in Dockerfile “, where the COPY command copies the product of the previous stage build into the current image by specifying the –from=0 parameter.

The ADD command can also do other things.
The format of the ADD command is the same as the COPY command, which is also:
ADD <src> <dest>

In addition to the fact that it can’t be used in multistage scenarios, the ADD command can do all the functions of the COPY command, and it can also do two cool features:

Extract the compressed files and add them to the image
Copy files from url to image
Of course, these features also make the ADD command more complicated and less intuitive than the COPY command.

Extract the compressed files and add them to the image.
If we have a compressed file package, we need to add the files from this compressed package to the image. Do you need to unpack the package and then execute the COPY command? Of course not needed! We can do this once with the ADD command:

WORKDIR /app
ADD mohandir.tar.gz .

This should be the best use case for the ADD command!

Copying files from url to images
is a much more cool usage! However, in the best practices of the official documentation of docker , it is strongly recommended not to use this! ! The docker officially recommends that when we need to copy files from a remote location, it is best to use the curl or wget commands instead of the ADD command. The reason is that when using the ADD command, more mirror layers are created, and of course the size of the image will be larger (the two pieces of code below are from the docker official documentation):

ADD http://example.com/big.tar.xz /usr/src/things/
RUN tar -xJf /usr/src/things/big.tar.xz -C /usr/src/things
RUN make -C / Usr/src/things all

If you use the following command, not only does the number of layers in the image decrease, but the big.tar.xz file is also not included in the image:

RUN mkdir -p /usr/src/things \
&& curl -SL http://example.com/big.tar.xz \
| tar -xJC /usr/src/things \
&& make -C /usr/src/things All

Well, it seems that the ADD command is only needed when extracting compressed files and adding them to the image!

Tips for speeding up image construction
When using the COPY and ADD commands, we can use some tricks to speed up the mirror build process. For example, put the copy operation of the files that are least likely to change in the lower mirror layer, so that the cache generated by the previous build will be used when rebuilding the image. For example, the author needs to use the following files when building a mirror:

As shown in the figure above, the second step and the third step do not rebuild the mirror layer, but use the previous cache. From the fourth step, the mirror layer is rebuilt. When the file size is large and the number of files is large, especially when you need to perform operations such as installation, such a design is still very obvious for the speed of the build. So we should try to choose the Dockerfile method that can use the cache.

to sum up
When you first see the COPY and ADD commands, you can’t help wondering. But after analysis, you will find that the COPY command is designed for the most basic usage, with clear concepts and simple operation. The ADD command is basically a superset of the COPY command (except for the multistage scenario), which allows for some convenient and cool copy operations. The ADD command adds complexity to its use, such as copying compressed files from urls. I hope this article can solve everyone’s doubts about the COPY and ADD commands in the Dockerfile

Docker to build a Tomcat runtime environment

1 Prepare the host system

Prepare a CentOS 7 operating system with the following specific requirements:

Must be a 64-bit operating system. The
kernel is above 3.8 to
view your CentOS kernel with the following command:

# uname -r

2 Install Docker

# yum install docker
Use the following command to see if Docker is installed successfully:

# docker version
If the version number of Docker is output, the installation is successful. You can start the Docker service with the following command:

# systemctl start docker.service
Once the Docker service has started, you can start using Docker.

3 download image

Take CentOS as an example, download a CentOS image:

# docker pull centos After
downloading, use the command to view the local mirror list:

# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
centos latest e934aafc2206 2 weeks ago 199MB

4 The host creates the /root/software/ directory and places the installation package in this directory.

5 start the container

The container is run on a mirrored basis. Once the container is started, we can log into the container and install the software or applications we need.

Start the container with the following command:

# docker run -i -t -v /root/software/:/mnt/software/ The e934 /bin/bash
command contains the following three sections:

Docker run <related parameters> <mirror ID> <initial command>
Among them, related parameters include:

-i: indicates
that the container is run in “interactive mode” -t: indicates that the container will enter its command line after startup
-v: indicates which directory needs to be mounted locally to the container, format: -v <host directory>:<container Directory > In
this example, all installers are placed in the /root/software/ directory of the host, and now you need to mount them in the /mnt/software/ directory of the container.

# cd /mnt/software/
# pwd
/mnt/software

# ls
apache-tomcat-7.0.81.tar.gz jdk-8u121-linux-x64.tar.gz The
initial command means that once the container is started, you need to run the command. Use “/bin/bash” to indicate that you directly enter the bash shell after booting.

6 Installing the software

In order to build the Java Web runtime environment, you need to install JDK and Tomcat. The following procedures are performed inside the container. In this example, select the /opt/ directory as the installation directory. You must first enter the directory using the cd /opt/ command.

6.1 Installing the JDK

First, unzip the JDK package:

# tar -zxf /mnt/software/jdk-8u121-linux-x64.tar.gz -C .

Then, move the JDK directory:

# mv jdk1.8.0_121/ /opt/jdk/

6.2 Installing Tomcat

First, extract the Tomcat package:

# tar -zxf /mnt/software/apache-tomcat-7.0.81.tar.gz -C .

Then, move the Tomcat directory:

# mv apache-tomcat-7.0.81/ /opt/tomcat/

6.3 Writing a run script

Write a run script that, when the container is started, runs the script and starts Tomcat.

First, create a run script:

# touch /root/run.sh

# vi /root/run.sh
Then, edit the script as follows:

#!/bin/bash

export JAVA_HOME=/opt/jdk/
export PATH=$JAVA_HOME/bin:$PATH

sh /opt/tomcat/bin/catalina.sh run
Finally, add execute permission for running the script:

# chmod u+x /root/run.sh

7 Exit the container

When the above steps are all completed, you can use the exit command to exit the container.

You can then view the running container using the following command:

Docker ps
At this point, you should not see any running programs, because the container that you just exited with the exit command, the container is stopped, you can use the following command to view all containers:

# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d4e3a062ab05 e934 “/bin/bash” 38 minutes ago Exited (0) About a minute ago lucid_einstein
Remember the above CONTAINER ID (container ID), which will then be created through the container One can run a Tomcat image.

8 Create a Tomcat image

Use the following command to create a new “mirror” based on a “container ID”:

# Docker the commit d4e3 mytomcat: 1.0
SHA256: c5ef8dacbf3eead5ea2b9fc3c1050a395863c6db0abd0eb7d6dee8ed46a31ffd

# Docker Images
the REPOSITORY the TAG the IMAGE ID CREATED SIZE
mytomcat 1.0 c5ef8dacbf3e 18 is seconds The ago Member 583MB
CentOS Latest e934aafc2206 2 weeks ago Member 199MB
ID of this container is d4e3, image name created is “mytomcat: 1.0 “, then you can use the image to start the Tomcat container.

9 Start the Tomcat container

First, create a new /root/webapps/ROOT directory,

Cd # / root
# mkdir webapps
# cd webapps /
# mkdir ROOT
# cd ROOT /
# vi index.html
and create an index.html file in the directory, file reads as follows:

<html>
<body>
<h2>Hello World!</h2>
</body>
</html>
As described above, the container can be started with the “mirror name” or “mirror ID”, and the last time it was started. The difference between the containers is that instead of entering the command line of the container, the Tomcat service inside the container is started directly. In this case, you need to use the following command:

# docker run -d -p 58080:8080 -v /root/webapps/:/opt/tomcat/webapps/ –name mytomcat_1 mytomcat:1.0 /root/run.sh
where related parameters include:

-d: Indicates that the /root/run.sh script is executed in “daemon mode”, at which point the Tomcat console does not appear on the output terminal.
-p: indicates the port mapping between the host and the container. At this time, the 8080 port inside the container is mapped to the port 58080 of the host. This exposes the port 58080 to the outside world. The Docker bridge can be used to access the port 8080 inside the container. .
-v: Indicates which local directory needs to be mounted to the container. Format: -v <host directory>: <container directory>
–name: indicates the container name, which can be named with a meaningful name.
In the browser, enter the host IP and port number to access Tomcat:

11 Stop the Tomcat container

# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
54215923125b mytomcat:1.0 “/root/run.sh” 3 minutes ago Up 3 minutes 0.0.0.0:58080->8080/tcp mytomcat_1

# docker stop 5421

12 Remove the container

# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
54215923125b mytomcat:1.0 “/root/run.sh” 3 minutes ago Exited (137) 2 seconds ago mytomcat_1

# docker rm 5421
5421

# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

13 Remove the image

# Docker Images
the REPOSITORY the TAG the IMAGE ID CREATED SIZE
mytomcat 1.0 c5ef8dacbf3e 20 is minutes ago Member 583MB
CentOS Latest e934aafc2206 2 weeks ago Member 199MB

# Docker RMI c5ef
the Untagged: mytomcat: 1.0
the Untagged: mytomcat @ SHA256: d949cbb93a58de27eec4c911f27b9f09edeb3d3ce57cf5ce77d4745211c947f6
Deleted: SHA256: c5ef8dacbf3e7ce916f57c52c16de3892c724996b5e30ca0d141c81897d9a06c
Deleted: SHA256: 7e62d1c2f904e8de3fadc6b01edea96bcb324634f0df514cc9297b1bf11d2f06

# Docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
Centos latest e934aafc2206 2 weeks ago 199MB

IBM MQ 7.5 Developer Edition installation configuration

IBM MQ 7.5 Developer Edition installation configuration

Download the development version here

Download address: https://www.ibm.com/developerworks/cn/downloads/ws/wmq/

Environment CentOS 7.4 x64

1. Preparation before installation

[root@236 ~]# mkdir mq
#Create a new installation directory [root@236 ~]# tar -xzf mqadv_dev75_linux_x86-64.tar.gz -C mq #Unpack
[root@236 ~]# ls mq
copyright MQSeriesFTAgent-7.5. 0-2.x86_64.rpm MQSeriesMan-7.5.0-2.x86_64.rpm MQSeriesMsg_ko-7.5.0-2.x86_64.rpm MQSeriesSDK-7.5.0-2.x86_64.rpm
crtmqpkg MQSeriesFTBase-7.5.0-2.x86_64 .rpm MQSeriesMsg_cs-7.5.0-2.x86_64.rpm MQSeriesMsg_pl-7.5.0-2.x86_64.rpm MQSeriesServer-7.5.0-2.x86_64.rpm
lap MQSeriesFTLogger-7.5.0-2.x86_64.rpm MQSeriesMsg_de-7.5 .0-2.x86_64.rpm MQSeriesMsg_pt-7.5.0-2.x86_64.rpm MQSeriesXRClients-7.5.0-2.x86_64.rpm
licenses MQSeriesFTService-7.5.0-2.x86_64.rpm MQSeriesMsg_es-7.5.0-2.x86_64.rpm MQSeriesMsg_ru-7.5.0-2.x86_64.rpm MQSeriesXRService-7.5.0-2.x86_64.rpm
mqlicense.sh MQSeriesFTTools-7.5.0-2.x86_64.rpm MQSeriesMsg_fr-7.5.0-2.x86_64.rpm MQSeriesMsg_Zh_CN-7.5.0-2.x86_64.rpm PreReqs
MQSeriesAMS-7.5.0-2.x86_64.rpm MQSeriesGSKit-7.5.0-2.x86_64.rpm MQSeriesMsg_hu-7.5.0-2.x86_64.rpm MQSeriesMsg_Zh_TW-7.5.0-2.x86_64.rpm READMEs
MQSeriesClient-7.5.0-2.x86_64.rpm MQSeriesJava-7.5.0-2.x86_64.rpm MQSeriesMsg_it-7.5.0-2.x86_64.rpm MQSeriesRuntime-7.5.0-2.x86_64.rpm repackage
MQSeriesExplorer-7.5.0-2.x86_64.rpm MQSeriesJRE-7.5.0-2.x86_64.rpm MQSeriesMsg_ja-7.5.0-2.x86_64.rpm MQSeriesSamples-7.5.0-2.x86_64.rpm

Run the license, choose 1 to agree

./mqlicense.sh

Install MQ Server

[root@236 mq]# rpm -ivh MQSeriesRuntime-7.5.0-2.x86_64.rpm #??MQ Runtime
Preparing… ################################# [100%]
Creating group mqm
Creating user mqm
Updating / installing…
1:MQSeriesRuntime-7.5.0-2 ################################# [100%]

[root@236 mq]# rpm -ivh MQSeriesSamples-7.5.0-2.x86_64.rpm ##??MQ Samples
Preparing… ################################# [100%]
Updating / installing…
1:MQSeriesSamples-7.5.0-2 ################################# [100%]

[root@236 mq]# rpm -ivh MQSeriesServer-7.5.0-2.x86_64.rpm #??MQ server
Preparing… ################################# [100%]
Updating / installing…
1:MQSeriesServer-7.5.0-2 ################################# [100%]

After the installation has completed, run the ‘/opt/mqm/bin/mqconfig’
command, using the ‘mqm’ user ID.

For example, execute the following statement when running as the ‘root’ user:

su mqm -c “/opt/mqm/bin/mqconfig”

The ‘mqconfig’ command validates that the system configuration satisfies the
requirements for WebSphere MQ, and ensures that the settings for the ‘mqm’
user ID are suitably configured. Other WebSphere MQ administrators in the
‘mqm’ group can run this command to ensure their user limits are also
properly configured to use WebSphere MQ.

If ‘mqconfig’ indicates that any of the requirements have not been met,
consult the installation section within the WebSphere MQ Information Center
for details about how to configure the system and user limits.

Then follow the prompts and execute the command to check if the environment is allowed.

First check, prompting for missing bc

[root@236 mq]# su mqm -c “/opt/mqm/bin/mqconfig”
mqconfig: Analyzing CentOS Linux release 7.4.1708 (Core) settings for
WebSphere MQ V7.5
mqconfig: The bc program was not found on this system. Please install bc
and try running mqconfig again.

Install bc

[root@236 mq]# yum install -y bc

Second inspection

There are several fail to resolve, refer to the documentation: https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.0.0/com.ibm.mq.ins.doc/q008550_.htm

Modify kernel parameters

Edit /sysctl.conf and add the following configuration

[root @ 236 mq] # vim /etc/sysctl.conf

kernel.sem = 500 256000 250 1024

net.ipv4.tcp_keepalive_time = 300

fs.file-max = 524288

Write configuration

[root@236 mq]# sysctl -p

Third check

2 files left to be solved

Edit limit.conf

[root @ 236 mq] # vim /etc/security/limits.conf

Add two lines

mqm hard capsule 10240
mqm soft capsule 10240

The fourth inspection passed

Modify environment variables

Since mq is installed in the /opt/mqm directory by default, you will not find the mq related command after the installation is complete. You need to configure the environment variable to find it.

Vim /etc/profile #Add the following line

PATH=/opt/mqm/bin:/opt/mqm/samp/bin/:$PATH

The installation is complete

2, start the instance

Switch to mqm user startup

[root @ 236 mq] # su mqm
bash-4.2 $

Create a default instance

bash-4.2$ crtmqm -q oe
WebSphere MQ queue manager created.
Directory ‘/var/mqm/qmgrs/oe’ created.
The queue manager is associated with installation ‘Installation1’.
Creating or replacing default objects for queue manager ‘oe’.
Default objects statistics : 74 created. 0 replaced. 0 failed.
Completing setup.
Setup completed.

View the instance, the status is extended

bash-4.2$ dspmq
QMNAME(oe) STATUS(Ended immediately)

Start instance

bash-4.2$ strmqm oe
WebSphere MQ queue manager ‘oe’ starting.
The queue manager is associated with installation ‘Installation1’.
5 log records accessed on queue manager ‘oe’ during the log replay phase.
Log replay for queue manager ‘oe’ complete.
Transaction manager state recovered for queue manager ‘oe’.
WebSphere MQ queue manager ‘oe’ started using V7.5.0.2.

Create a queue named test

define qlocal(test)
1 : define qlocal(test)
AMQ8006: WebSphere MQ queue created.
end
2 : end
One MQSC command read.
No commands have a syntax error.
All valid MQSC commands were processed.

Send message test, error 2085

bash-4.2$ amqsput Test oe
Sample AMQSPUT0 start
target queue is Test
MQOPEN ended with reason code 2085
unable to open queue for output
Sample AMQSPUT0 end

Later, I found that the queue could not be lowercase. The test queue was converted to uppercase. It is recommended that the queue name be set to uppercase, resend the message test, and press twice to confirm the input.

bash-4.2$ amqsput TEST oe
Sample AMQSPUT0 start
target queue is TEST
hello world!

Sample AMQSPUT0 end

Receive a message and accept success

bash-4.2$ amqsget TEST oe
Sample AMQSGET0 start
message <hello world!>

Start port listening

Successful startup

bash-4.2$ netstat -tpln | grep 2424
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp6 0 0 :::2424 :::* LISTEN 5067/runmqlsr

docker centos

[root@ docker]# cat /etc/yum.repos.d/docker-main.repo
[docker-main-repo]
name=Docker main Repository
baseurl=https://get.daocloud.io/docker/yum-repo/main/CentOS/7
enabled=1
gpgcheck=1
gpgkey=https://get.daocloud.io/docker/yum/gpg

[root@ docker]# yum install docker-engine –y

[root@ docker]# docker –version
Docker version 17.05.0-ce, build 89658be

firewall-cmd –add-port=2377/tcp –permanent

firewall-cmd –add-port=7946/tcp –permanent

firewall-cmd –add-port=7946/udp –permanent

firewall-cmd –add-port=4789/udp –permanent

firewall-cmd –reload**

[root@ docker]# cat /etc/docker/daemon.json
{
“insecure-registries”:[“docker-registry.rmohan.com:5000”],
“log-driver”:”json-file”,
“log-opts”:{“max-size”:”1024m”,”max-file”:”2″}
}

[root@ ~]# systemctl restart docker.service

[root@master ~]# docker swarm join-token worker
To add a worker to this swarm, run the following command:

docker swarm join \
–token SWMTKN-1-4p4djbee1kqcss8x5prfzg6v01x0y7hfa7rqob6rffg6e2p2wq-278qafb9ptpqtfebr0kyngi0b \
192.168.191.6:2377

[root@filters ~]# docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
0y4ga216x49nd9zt821afe45p * mohan_191_6 Ready Active Reachable
56mgklq3iyf0ajioytmtr0b44 mohan_191_10 Ready Active
b5mljua4e0tchrrk871ggbw7r mohan_191_7 Ready Active Reachable
gxm1gvh8lx2eja36a4cms9w98 mohan_182_212 Ready Active
hgweoxc5vmy9g30uyozmbumhj mohan_182_213 Ready Active
lluom23ugtfuiwphcye2frbmd mohan_182_215 Ready Active
om0ezdbqzdvavn8z8d844osbo mohan_182_214 Ready Active
t00tu1qs7wbll2fg2accsx7wf mohan_191_4 Ready Active Leader
tk89xphvsg16rz7b7l6lam4xv mohan_191_9 Ready Active
xbcx4hx2lq3ji8zog2ayzctat mohan_191_8 Ready Active Reachable
y3pnrw9jt69vkfiuxrs0co7ke mohan_191_5 Ready Active Reachable

docker swarm join –token manger node ip:port

[root@dockernode~]#docker swarm join –token SWMTKN-1-4p4djbee1kqcss8x5prfzg6v01x0y7hfa7rqob6rffg6e2p2wq-278qafb9ptpqtfebr0kyngi0b 192.168.191.6:2377

$ docker run Ubuntu:14.04 /bin/echo ‘Hello world’
Hello world

$ docker run -t -i ubuntu:14.04 /bin/bash
root@af8bae53bdd3:/#

$ docker run -d ubuntu /bin/sh -c “while true; do echo hello world; sleep 1; done”
cb30b87566d0550ec5f1232d148c5ffed6546c347889e58a6405579f2af73f2a

$ docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
cb30b87566d0 ubuntu “/bin/sh -c ‘while t…” 2 minutes ago Up 2 minutes goofy_mcclintock

$ docker container logs goofy_mcclintock
hello world
hello world
hello world

$ docker container stop goofy_mcclintock
goofy_mcclintock

$ docker container ls -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
cb30b87566d0 ubuntu “/bin/sh -c ‘while t…” 20 minutes ago Exited (137) 23 seconds ago goofy_mcclintock

$ docker attach e1ff
root@e1ffd4f792fe:/#

Note: If exit from this stdin, it will cause the container to stop.

The exec command
-i -t parameter
docker exec can be followed by multiple parameters, here mainly the -i -t parameter.
When only the -i parameter, since there is no allocation of pseudo-terminals, the interface is not familiar Linux command prompt, the command execution
line results can still be returned.
When the -i -t parameter is used together, you can see the familiar Linux command prompt.

$ docker run -dit ubuntu
16168d4b66b115b5afac5836db3ff93304774e98489f628ac625fff2bcd640ba

$ docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
16168d4b66b1 ubuntu “/bin/bash” 58 seconds ago Up 57 seconds happy_bardeen

$ docker exec -it 16168 bash
root@16168d4b66b1:/#

Exit from this stdin will not cause the container to stop. That’s why the docker exec is recommended.
For more parameter descriptions, please use docker exec –help to view.

Fifth, delete the container

Delete a container that is in a terminated state in the format:
docker container rm [options] CONTAINER [CONTAINER…]

$ docker container rm awesome_payne
awesome_payne

If you want to delete a running container, you can add the -f parameter. Docker will send a SIGKILL signal to the container.

Clean up all containers in the terminated state. Use the docker container ls -a command to view all the containers that have been created, including the termination status. If the number is too large, it may be cumbersome to delete them one by one. You can use the following command to clear all the termination status. Container.

$ docker container prune
WARNING! This will remove all stopped containers.
Are you sure you want to continue? [y/N] y
Deleted Containers:
545f8f6d19286efae28307d06ed1acc034d07f109e907c01892471a6f89e772d
cb30b87566d0550ec5f1232d148c5ffed6546c347889e58a6405579f2af73f2a
……

Export and import containers

Exporting a container
If you want to export a local container, you can use the docker export command.

$ docker container ls -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
16168d4b66b1 ubuntu “/bin/bash” 18 minutes ago Up 18 minutes happy_bardeen

$ docker export 16168d4b66b1 > ubuntu.tar

This will export the container snapshot to a local file.

Import container snapshots
can be imported as mirrors from the container snapshot file using docker import, for example

$ cat ubuntu.tar | docker import – test/ubuntu:v1.0
sha256:91b174fec9ed55d7ebc3d2556499713705f40713458e8594efa114f261d7369a

$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
test/ubuntu v1.0 91b174fec9ed 10 seconds ago 69.8MB
ubuntu latest 735f80812f90 3 weeks ago 83.5MB

Alternatively, you can import it by specifying a URL or a directory, such as
$ docker import http://example.com/exampleimage.tgz example/imagerepo

Note: Users can either use the docker load to import the image storage file to the local image library, or use docker import to import a container snapshot to the local image library. The difference between the two is that the container snapshot file will discard all history and metadata information (that is, only the snapshot state of the container at the time), and the image storage file will save the full record and be large. In addition, metadata information such as tags can be reassigned when importing from a container snapshot file.

MySQL: Calculate the free space in IBD files

If you use MySQL with InnoDB, chances are you’ve seen growing IBD data files. Those are the files that actually hold your data within MySQL. By default, they only grow — they don’t shrink. So how do you know if you still have free space left in your IBD files?

There’s a query you can use to determine that:

SELECT round((data_length+index_length)/1024/1024,2)
FROM information_schema.tables
WHERE
  table_schema='zabbix'
  AND table_name='history_text';

The above will check a database called zabbix for a table called history_text. The result will be the size that MySQL has “in use” in that file. If that returns 5.000 as a value, you have 5GB of data in there.

In my example, it showed the data size to be 16GB. But the actual IBD file was over 50GB large.

$ ls -alh history_text.ibd
-rw-r----- 1 mysql mysql 52G Sep 10 15:26 history_text.ibd

In this example I had 36GB of wasted space on the disk (52GB according to the OS, 16GB in use by MySQL). If you run MySQL with innodb_file_per_table=ON, you can individually shrink the IBD files. One way, is to run an OPTIMIZE query on that table.

Note: this can be a blocking operation, depending on your MySQL version. WRITE and READ I/O can be blocked to the table for the duration of the OPTIMIZE query.

MariaDB [zabbix]> OPTIMIZE TABLE history_text;
Stage: 1 of 1 'altering table'   93.7% of stage done
Stage: 1 of 1 'altering table'    100% of stage done

+---------------------+----------+----------+-------------------------------------------------------------------+
| Table               | Op       | Msg_type | Msg_text                                                          |
+---------------------+----------+----------+-------------------------------------------------------------------+
| zabbix.history_text | optimize | note     | Table does not support optimize, doing recreate + analyze instead |
| zabbix.history_text | optimize | status   | OK                                                                |
+---------------------+----------+----------+-------------------------------------------------------------------+
2 rows in set (55 min 37.37 sec)

The result is quite a big file size savings:

$ ls -alh history_text.ibd
-rw-rw---- 1 mysql mysql 11G Sep 10 16:27 history_text.ibd

The file that was previously 52GB in size, is now just 11GB.

Apache 2.4 AH01762 & AH01760: failed to initialize shm (Shared Memory Segment)

Mattias Geniar, Tuesday, January 12, 2016

I recently ran into the following problem on an Apache 2.4 server, where after server reboot the service itself would no longer start.

This was the error whenever I tried to start it:

$ tail -f error.log
[auth_digest:error] [pid 11716] (2)No such file or directory:
   AH01762: Failed to create shared memory segment on file /run/httpd/authdigest_shm.11716
[auth_digest:error] [pid 11716] (2)No such file or directory:
   AH01760: failed to initialize shm - all nonce-count checking, one-time nonces,
   and MD5-sess algorithm disabled

Systemd reported the same problem;

$ systemctl status -l httpd.service
 - httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since ...

The cause is shown in the first error message: Failed to create shared memory segment on file /run/httpd/authdigest_shm.11716.

If I traced this, I noticed the directory /run/httpd no longer existed. The simple fix in this case, was to re-create that missing directory.

$ mkdir /run/httpd
$ chown root:httpd /run/httpd
$ chmod 0710 /run/httpd

The directory should be owned by root and writeable for the root user. The Apache group (in my case, httpd), needs executable rights to look into the directory.

iven facility which allows non-dependent subsystems to be started, controlled, or stopped in parallel. Here we explain how to add a custom script to the systemd facility.

1. Write And Debug The Custom Script

Typically a systemd script is written as a shell script. Begin by writing your custom script using the normal conventions. We will call our script my-custom-script.sh and is straightforward:

#!/bin/sh
echo "I am a custom script" > /var/tmp/script.out
echo "The script was run at : `date`" >> > /var/tmp/script.out

The script must be executable.

# chmod 0755 /var/tmp/my-custom-script.sh

2. Describe The Custom Script To systemd

With the script written and tested manually, the script is ready to be described to the systemd system. To do this, a [name].service file is needed. The syntax uses the INI format commonly used for configuration files. Continuing our example, we need a my-custom-script.service file. The executable will run exactly once for each time the service is started. The service will not be started until the networking layer is up and stable.

Create a new service unit file at /etc/systemd/system/my-custom-script.service with below content. The name of the service unit is user defined and can be any name of your choice.

# This is my-custom-script.service, which describes the my-custom-script.sh file
[Unit]
Description=This is executed on shutdown or reboot
DefaultDependencies=no
Wants=network-pre.target                                                                   # (if network is required before running the script)
Before=network-pre.target shutdown.target reboot.target halt.target                        # Defines the order in which units are stoped. #(REQUIRED)

[Service]
Type=oneshot                                                                               # enables specifying multiple custom commands that are then executed sequentially. (REQUIRED)
RemainAfterExit=true                                                                       # required by the oneshot setting (REQUIRED)
Environment=ONE='one' "TWO='2"                                                             # you can set some environment variables, that may be necessary to pass as arguments
ExecStart=/bin/true                                                                        # because is a shutdown script nothing is done when this service is started
ExecStop=/bin/bash /var/tmp/my-custom-script.sh ${ONE} ${TWO}                              # < --*********** change to the script full path ************ (REQUIRED)
TimeoutStopSec=1min 35s                                                                    # Configures the time to wait for stop. 

[Install]
WantedBy=multi-user.target                                                                 # When this unit is enabled, the units listed in WantedBy gain a Want dependency on the unit. (REQUIRED)

3. Enable The Script For Future Reboots

Similar to the chkconfig from earlier versions, the service must be enabled. Since a new service was added, notify the systemd daemon to reconfigure itself:

# systemctl enable my-custom-script.service
# systemctl daemon-reload

Hit Counter provided by dental implants orange county

Recent Posts

Pages

Categories

Archives

WordPress Quotes

Recent Comments

Short Cuts

Who's Online

Categories

My Clock

How to Fix Redis CLI Error Connection Reset by Peer

Overview

Problem

Solution

Elasticsearch cluster deployment under CentOS7.3

apache redirect

GC Details

COPY and ADD commands in Dockerfile

Docker to build a Tomcat runtime environment

IBM MQ 7.5 Developer Edition installation configuration

docker centos

MySQL: Calculate the free space in IBD files

Apache 2.4 AH01762 & AH01760: failed to initialize shm (Shared Memory Segment)

Apache 2.4 AH01762 & AH01760: failed to initialize shm (Shared Memory Segment)

1. Write And Debug The Custom Script

2. Describe The Custom Script To systemd

3. Enable The Script For Future Reboots

Recent Posts

Blogroll

Meta