Docker basic commands

A Linux Container is a lightweight application isolation mechanism that allows the kernel to run groups of processes in their own isolated user spaces, separate from the host system. The container has its own process list, network stack, file systems, and other resource, but shares the kernel with the host and other containers running on the system. Linux containers are implemented through a combination of three kernel features: namespaces for isolation, control groups for resource control, and SELinux for security.

Docker is used to create, control and manage containers. Docker adds an API, an image format, and a delivery and sharing model to Linux containers. A Docker image contains an application and all its dependencies. When a container is started, a read-write layer for that container is combined with the read-only image using LVM thin provisioning. Docker images are portable and can be saved and exported to other systems and users. (Source: RH254 – RedHat System Administration III)

  • Install docker:
    • yum install docker
    • systemctl start docker
    • docker search apache                   (Search for container)
    • docker pull docker.io/httpd         (Retrieve the container)
    • docker images [–all]     (List installed containers and components (–all))
    • docker run httpd                            (Starts the container)
    • docker ps [-a | -l]                            (List Container or process)
    • docker info                                       (Display system-wide information)
    • curl http://172.17.0.2                      (IP is shown in docker httpd output)
  • Stop and uninstall docker:
    • ctrl+c
    • docker stop httpd
    • docker rmi -f docker.io/httpd   (Removes container image)
    • systemctl stop docker
    • yum remove docker
  • Issues:
    • Error: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
    • Solution: Run the command with root privileges or sudo.
Advertisements
Posted in docker, LFCE, LFCS

Amazon Web Services (AWS) Concetps

  • Regions -> Availability Zone -> Data Center
  • VPS (Virtual Private Cloud)  (OpenStack: Project / Tenant)
  • EC2 (Elastic Compute Cloud) (OpenStack: Compute Service/Nova)
    • instance:
  • RDS (Relational Database Service) (OpenStack: DBaaS/Trove)
    • MySQL, Oracle
  • S3 (Simple Storage Service) (OpenStack: Object Storage/Swift)
Posted in aws

tmux and screen basic commands

tmux:

  • tmux | tmux new -s “mysession”          (Start a new session)
  • tmux ls                       (List sessions)
  • ctrl+b d | exit            (Detach or exit from a session)
  • ctrl+b ”                        (Divides screen horizontally / create a new pane)
  • ctrl+b %                      (Divides screen vertically / create a new pane)
  • ctrl+b x | exit            (Kill a pane)

screen:

  • screen                                   (Create a new session)
  • screen -ls                              (List session)
  • screen -r [screen-name]     (Attach a session)
  • exit                                        (exit or kill a session)
  • ctrl+a d |screen -d             (Detach a session)
  • ctrl+a ”                                 (Lists all running commands)
Posted in LFCE, LFCS, Linux, Other

OpenStack CLI (commands) reference

(i) OpenStack Services health check: (Verify installation)

  • openstack host list | openstack host show $(hostname)
  • openstack hypervisor list | openstack hypervisor show $(hostname)
  • openstack catalog list                                                      (Show all service endpoints)
  • openstack compute service list | nova service-list    (Status of Compute Services)
  • openstack volume service list   | cinder service-list  (Status of volume services)
  • openstack network agent list    | neutron agent-list  (Status of Network Services)
  • openstack usage list                                                          (Show usage per project)
  • openstack command list                                                  (Show all commands)
  • openstack server list –all-project | nova list –all-tenants | virsh list –all
  • swift stat -v | df -ha | grep swift                                    (Disk space utilization)

(ii) OpenStack Identity Service (Keystone):

  • Service:
    • openstack service list
    • openstack service create –name service1  type1 
    • openstack service show service1
    • openstack service set –description “My Service1” service1
    • openstack service delete service1
  • Endpoint:
    • openstack endpoint list  [–service service1] | openstack catalog list
    • openstack endpoint create service1 –region RegionOne \
      –adminurl  <url>  –publicurl  <url> –internalurl <url>
    • openstack endpoint show <endpoint_id>
    • openstack endpoint set  –url http://10.0.0.1:456  <endpoint_id>
    • openstack endpoint delete <endpoint_id>
  • Project:
    • openstack project list  [–long]
    • openstack project create –name prj1   [–domain default]
    • openstack project show  prj1
    • openstack project set prj1 –description “My Project1”
    • openstack project delete prj1
  • User
    • openstack user list   [–long|–project prj1]
    • openstack user create –password password  user1
    • openstack user show user1
    • openstack user set –email “user1@example.comuser1
    • openstack user delete user1
    • keystone user-create –pass pass123 –name user1
  • Role:
    • openstack role list
    • openstack role create role1
    • openstack role show role1
    • openstack role delete role1
    • openstack role add|remove –user user1 –project prj1  Member
    • openstack role add|remove role1 –group grp1 –project p1|–domain d1
    • openstack role assignment list –names [–user usr|–group grp|–project prj]
  • Group:   (Use Dashboard to manage groups because its not included in ‘Newton’)
    • openstack group list
    • openstack group create grp1   [–domain domain1]
    • openstack group show grp1
    • openstack group set –description “My Group1” grp1
    • openstack group delete grp1
    • openstack group add|remove grp1 user1
    • openstack group contains user grp1 user1
  • Domain: (Not included in ‘Newton’ release)
    • openstack domain list
    • openstack domain create domain1
    • openstack domain show domain1
    • openstack domain set –disable domain1
    • openstack domain delete domain1   (Domain must be disabled before delete)
  • Logs:
    • /var/log/keystone/keystone.log

(iii)  OpenStack Networking (Neutron):

  • network: (External networks can be created with admin user only)
    • openstack network list  [–long|–external]
    • openstack network create net1  [–project prj1]   (Private/Internal network)
    • openstack network create net1-ext –share –external –project prj1  \
      –provider-network-type  flat  –provider-physical-network default
    • openstack network show net1
    • openstack network delete net1
  • subnet:
    • openstack subnet list
    • openstack subnet pool list
    • openstack subnet create –network net1 —subnet-range 10.0.0.0/24  net1-subnet
    • openstack subnet show net1-subnet
    • openstack subnet delete net1-subnet
  • router:
    • openstack router list  [–long]
    • openstack router create router1
    • openstack router show router1
    • openstack router add|remove subnet  router1  net1-subnet
    • openstack router add|remove port  router1  port1
    • neutron router-gateway-set router1 public  (set External Network/Gateway)
      • openstack port list  –router router1      (Verify router interfaces)
    • openstack router delete router1
  • floating ip (Public IP):
    • openstack floating ip list
    • openstack floating ip pool list
    • openstack floating ip create public
    • openstack floating ip show  <floating-ip>
    • openstack floating ip delete <floating-ip>
    • /etc/nova/nova.confauto_assign_floating_ip=true
    • P.S. To assign a floating ip the instances’s private network must be connected with a router and that router must be connected with a External network via Set Gateway.
  • port (Private/Fixed IP):
    • openstack port list  [–router router1]   (Shows router interfaces)
    • openstack port create –network net1  port1
    • openstack port show port1
    • openstack port delete port2
  • security group:
    • openstack security group list | grep <project_id>
    • openstack security group create sg1  [–project prj1]
    • openstack security group show sg1
    • openstack security group delete sg1
    • nova help [secgroup-create|secgroup-list|secgroup-delete]
    • security group rule:
      • openstack security group rule list  sg1  [–long]
      • openstack security group rule create –protocol icmp  sg1
      • openstack security group rule create –dst-port 22 sg1 [–src-ip 0.0.0.0/0] 
      • openstack security group rule show <Rule_id>
      • openstack security group rule delete <Rule_id>
      • nova help [secgroup-add-rule|secgroup-list-rules|secgroup-delete-rule]
  • Load balancer:
    • neutron help | grep lb-
    • lb-pool:
      • neutron lb-pool-create  –lb-method ROUND_ROBIN  –protocol HTTP \
        –subnet-id  <subnet-id> –name lb1
      • neutron lb-pool-list
      • neutron lb-pool-stats lb1
      • neutron lb-pool-delete lb1
    • lb-member:
      • neutron lb-member-create –address 172.24.4.8 –protocol-port 80  lb1
      • neutron lb-member-create –address 172.24.4.9 –protocol-port 80  lb1
      • neutron lb-member-list
    • lb-vip:
      • neutron lb-vip-create –protocol-port 80 –protocol HTTP \
        –subnet-id <subnetid> –name lb1vip  lb1
      • neutron lb-vip-list
      • neutron lb-vip-delete lb1vip
    • lb-healthmember:
      • neutron lb-healthmonitor-create –type HTTP \
        –delay –max-retries 3 –timeout 3
      • neutron lb-healthmonitor-associate <healthmonitor–id> lb1
      • neutron lb-healthmonitor-list
      • neutron lb-healthmonitor-disassociate <healthmonitor-id> lb1
      • neutron lb-healthmonitor-delete <healthmonitor-id>
    • Setup instances/servers for load balancer:
      • openstack server create –image cirros –flavor m1.tiny  vm1
      • openstack server create –image cirros –flavor m1.tiny  vm2
      • openstack floating ip create public
      • openstack floating ip create public
      • openstack server add floating vm1  172.24.4.8
      • openstack server add floating vm2  172.24.4.9
      • openstack security group rule create –dst-port 80 default
      • openstack security group rule create –dst-port 22 default
      • ssh cirros@172.24.4.8
        • nohup sudo nc -lk -p 80  -e echo -ne \
          “HTTP/1.0 200 OK\r\n Content-Length: 5\r\n\r\nvm1\r\n” &
        • curl http://172.24.4.8  -> vm1
      • ssh cirros@172.24.4.9
        • nohup sudo nc -lk -p 80 -e echo -ne \
          “HTTP/1.0 200 OK\r\n Content-Length: 5\r\n\r\nvm2\r\n” &
        • curl http://172.24.4.9  -> vm2
      • curl http://&lt;mylbvip-ip> | http://
  • neutron (client):
    • neutron help | grep gateway
    • neutron help [net-create|net-list|net-show|net-delete]
    • neutron help [subnet-create|subnet-list|subnet-show|subnet-delete]
    • neutron help [router-create|router-interface-add|router-gateway-set]
    • neutron help [security-group-create|security-group-list|show|delete]
    • neutron help [security-group-rule-create|security-group-rule-list|show|delete]
    • neutron help [floatingip-create|floatingip-associate|
    • neutron net-create ext-net –tenant-id <prj-id> –shared –router:external True \
      –provider:network_type flat –provider:physical_network default  (admin role)
  • Logs:
    • /var/log/neutron/*.log
    • /var/lib/neutron/*

(iv) OpenStack Compute (Nova):

  • keypair:
    • openstack keypair list
    • openstack keypair create keypair1  > ~/.ssh/keypair1.pem
      • chmod 600 ~/.ssh/keypair1.pem
    • openstack keypair show keypair1   [–public-key]
    • openstack keypair delete keypair1
    • ssh-keygen -qf  ~/.ssh/key
      • openstack keypair create –public-key ~/.ssh/key.pub  keypair1
    • nova help [keypair-list|keypair-add|keypair-show|keypair-delete]
  • flavor:   (Only admin user can create it and by default are public)
    • openstack flavor list  [–all |–long|–public|–private]
    • openstack flavor create –ram 512 –disk 1 –vcpus 1 –id 7  m1.demo
    • openstack flavor show m1.demo
    • openstack flavor set –private –project prj1  m1.demo
    • openstack flavor delete m1.demo
    • nova help [flavor-list|flavor-create|flavor-show|flavor-delete]
  • server (instance/VM):
    • openstack server list  [–all-projects|–project prj1]
    • openstack server create  vm1 –image cirros –flavor m1.tiny \
      –key-name key1 –security-group default –nic net-id=private,v4-fixed-ip=10.0.0.9
    • openstack server show vm1
    • openstack server delete vm1
    • openstack server add|remove floating ip  vm1  172.168.4.23
    • openstack server add|remove volume  vm1  vol1  [–device /dev/xvdb]
    • openstack server image create –name vm1_snapshot  vm1 (glance image-list)
    • openstack server backup create –name vm1_backup  vm1  (glance image-list)
    • openstack server stop|start|pause|unpause|suspend|resume  vm1
    • Connect to instance:
      • openstack server ssh –login cirros  vm1  [–public|–private]
      • ssh -i .ssh/keypair.pem  cirros@172.24.4.3     (Connect over Public IP)
      • ip netns list                                                          (Connect over Private IP)
      • ip netns exec qrouter-<routerid> ssh -i keypair.pem  cirros@10.0.0.3
  • quota:
    • openstack quota show prj1 [–default]
    • openstack quota set –cores 5 prj1
    • nova help [quota-show|quota-update|quota-defaults|quota-delete]
  • Policies:
    • ls -l /etc/*/policy.json:
      • /etc/cinder/policy.json
      • /etc/glance/policy.json
      • /etc/nova/policy.json
      • /etc/neutron/policy.json
      • /etc/keystone/policy.json
    • /etc/cinder/policy.json:
      • “volume:create”: “rule:admin”   (Limits volume creation to admin)
      • systemctl restart devstack@c-api    (Restart cinder API service)
    • /etc/glance/policy.json:
      • “add_image”: “rule:admin”   (Limits image creation to admin)
      • Restart glance API service:
        • su – stack; screen -ls; script /dev/null; screen -r stack
        • ctrl+a “-> g-api -> ctrl+c -> UP-Arrow+Enter -> ctrl+a d -> exit
  • nova (client):
    • nova help |grep flavor
    • nova help [list|boot|quota-*|keypair-*|flavor-*|
      floating-ip-associate|volume-attach
      ]
    • nova [hypervisor-list|host-list|service-list|usage-list|version-list]
    • nova boot vm1 –image cirros –flavor m1.tiny \
      –key-name keypair1 –security-groups default –nic net-id=<network-id>
  • Logs:
    • /var/log/nova/*.log
    • /var/lib/nova/instances/<instance-id>/        (Ephemeral disk)
    • /var/lib/nova/instances/snapshot/                (Snapshots location)

(v) OpenStack Image Service (Glance)(Default visibility is private)

  • openstack image list  [–long]   (List OS images and server snapshots/backups)
  • openstack image create –file cirros.img  cirros  [–disk-format qcow2]
  • openstack image show cirros
  • openstack image set –min-ram 512  –public  cirros
  • openstack image save –file cirros-v2.img   cirros
  • openstack image delete cirros
  • openstack image add|remove project cirros  prj2   (Add private image into prj)
  • wget https://10.9.7.8/cirros.img               (Download an image)
  • glance (client):
    • glance help [image-list|image-create|image-show|image-delete]
  • Logs:
    • /var/log/glance/*.log
    • /var/lib/glance/images/<image-id>/       (Image store location)

(vi) OpenStack Block Storage (Cinder):

  • volume:
    • openstack volume service list
    • openstack volume type list
    • openstack volume list  [–all-projects]
    • openstack volume create –size 1  vol1
    • openstack volume set –size 2 vol1 
    • openstack volume show vol1
    • openstack volume delete vol1
    • backup:
  • snapshot:
    • openstack snapshot list
    • openstack snapshot create –name vol1_snapshot  vol1
    • openstack snapshot show  vol1_snapshot
    • openstack snapshot delete  vol1_snapshot
  • Encrypt volume:
    • openstack volume type create LUKS
    • cinder encryption-type-create  –cipher aes-xts-plain64  –key_size 256 \
      –control_location front-end  LUKS nova.volume.encryptors.luks.LuksEncryptor
    • openstack volume create  –type LUKS  –size 1  crypt-vol
    • openstack volume show crypt-vol
    • openstack server volume add vm1 crypt-vol
  • cinder (client):
    • cinder help [list|create|snapshot-list|backup-list|type-list|version-list]
  • Logs:
    • /var/log/cinder/*.log
    • /var/lib/cinder/*

(vii) OpenStack Object Storage (Swift):

  • container:
    • openstack container list  [–long|-all]
    • openstack container create con1
    • openstack container show cont
    • openstack container delete con1
  • object:
    • openstack object list con1
    • openstack object store account show
    • openstack object create con1  file.txt    (echo “File” > file.txt)
    • openstack object show con1  file.txt
    • openstack object save con1  file.txt  –file file2.txt
    • openstack object delete cont file.txt
    • swift (client):
      • swift [list|post|stat|upload|download|delete]  –help
      • swift post con1   [-m Color:Red]                (Creates container)
      • swift upload con1  file.jpg   [–object-name pic]  [–segment-size]
      • swift download con1 file.jpg -o file2.jpg
      • swift delete con1  file.jpg
      • swift stat con1 |swift stat con1  file.jpg
      • swift list [con1]
      • Set ACLs (Containers):
        • swift post con1 -r  .r:*,.rlistings                 (Read and List access for all) 
        • swift post con1 -r  .r:*,.r:-example.com   (Deny access for a domain)
        • swift post con1 -w|-r prj1:*   (Write/Read access for prj1 members)
        • swift stat cont | openstack container show con1
      • Set Expiry (Objects):
        • swift post con1 file.txt -H “X-Delete-At:1516”  (date -d 1/1/2018 +%s)
        • swift post con1  file.txt  -H “X-Remove-Delete-At:”     (Removes date)
        • swift post con1  file.txt  -H “X-Delete-After:100”  (Delete after 100s)
        • swift post con1  file.txt  -H “X-Remove-Delete-After:” (Removes secs)
        • swift stat cont file.txt
      • Allow web access:
        • swift post -m ‘web-listings: true con1
        • swift post -m ‘web-listings-css:listings.css’ con1
  • Logs:
    • /var/log/swift/swift.log
    • /var/lib/swift/*

(viii) OpenStack Orchestration (Heat):

  • openstack stack list  [–long|–all-projects]
  • openstack stack create  -t hello_world.yaml   stack1  \
    –parameter image=cirros –parameter flavor=m1.tiny –parameter vm-name=vm1 \
    –parameter net=public –parameter volume=<vol-id> 
  • openstack stack show stack1
  • openstack stack delete stack1
  • openstack stack update -t stack1.yaml   stack1
  • openstack stack resource list stack1
  • openstack stack event list stack1
  • output:
    • openstack output list stack1 | heat output-list stack1
    • openstack output show stack1 private_ip [-all]
  • snapshot:
    • openstack stack snapshot list stack1 | heat snapshot-list stack1
    • openstack stack snapshot create stack1  [–name stack1-snap]
    • openstack stack snapshot show stack1
    • openstack stack snapshot restore stack1  <snaphost_id>
    • openstack stack snapshot delete stack1  <snaphost_id>
  • heat (client):
    • heat help [service-list|stack-list|stack-create|stack-show|stack-delete]
  • Template (hello_world.yaml):
    heat_template_version: 2018-01-01
    description: Example hello_world.yaml template to deploy a single instance
    resources:
    server:
    type: OS::Nova::Server
    properties:
    image: cirros
    flavor: m1.tiny
    networks:
    – network: 8f3a3cbe-c68d-442e-af6a-d347757c38a5
  • P.S. The private network referenced in the template must be Shared.

(ix) OpenStack Ceilometer (Telemetry):

  • ceilometer (client):
    • ceilometer help | grep alarm
    • ceilometer meter-list | less
    • ceilometer sample-list -m memory| head -10
    • ceilometer statistics -m memory
    • ceilometer meter-list -q resource-id=<server-id>
    • ceilometer sample-list -m cpu -q resource-id=<server-id>
    • ceilometer statistics -m cpu_util -p 60 -q resource-id=<server-id>
    • alarm:
      • ceilometer alarm-threshold-create –name cpu-alarm \
        –description “cpu alarm high” -m cpu_util –statistic avg –period 60 \
        –evaluation-periods 3 –comparison-operator gt –threshold 50.0 \
        –alarm-action ‘log://’ -q resource_id=<server_id>
      • ceilometer alarm-list
      • ceilometer alarm-update –enable false <alarm-id>
      • ceilometer alarm-history <alarm-id>
      • ceilometer alarm-delete <alarm-id>

(ix) Credential file(credrc|demo-openrc|keystone_admin)

  • Sample credential file myproject-openrc or keystonerc_myproject:
    export OS_AUTH_URL=http://127.0.0.1:5000/v2.0  (http://127.0.0.1:5000/identity)
    export OS_TENANT_NAME=”myproject
    export OS_USERNAME=”admin
    export OS_PASSWORD=”openstack”
  • P.S. Make sure admin user is assigned either member or admin role in myproject.

(x) Mariadb (CentOS) / Mysql(Ubuntu):

  • systemctl status|start|stop  mariadb|mysql
  • mysql -u root [-p password]; quit
  • show databases; show tables; use mysql|nova|neutron|glance
  • select user, host, password from user;

 

 

Posted in coa, openstack

OpenStack common issues and solutions

  1. /var/log/syslog-> ERROR cinder.service [-] Manager for service cinder-volume devstack@lvmdriver-1 is reporting problems, not sending heartbeat. Service will appear “down”.
    • Soluton:
      1. sudo losetup -f /opt/stack/data/stack-volumes-default-backing-file
      2. sudo losetup -f /opt/stack/data/stack-volumes-lvmdriver-1-backing-file
      3. ip addr add 172.24.4.1/24 dev br-ex
      4. sudo route add -net 10.0.0.0/24 gw 172.24.4.1
      5. sudo systemctl restart devstack@c-vol
    • For boot persistence add the above two lines in /etc/rc.local
      sudo losetup -f /opt/stack/data/stack-volumes-default-backing-file
      sudo losetup -f /opt/stack/data/stack-volumes-lvmdriver-1-backing-file
    • And the floating ip address for br-ex in /etc/network/interfaces
      auto br-ex
      iface br-ex inet static
      address 172.24.4.1
      netmask 255.255.255.0
       
  2. ERROR: Can’t connect on private IP address of a server (10.0.0.3) from OpenStack host:
    1. Solution:
      • Use any of the following commands to find out the public ip address of the router on public network for demo project:
        • openstack router show | grep ip
        • openstack port list –device-owner network:router_gateway  (admin)
        • ip netns exec qrouter-<router-id> ip a | grep qg
      • Then, add the route for guest server private network where gateway would be the router ip:
        • ip route add 10.0.0.0/24 via 172.24.4.231 dev br-ex
      • P.S. Make sure icmp and ssh are added into security group rule in the vm. 
  3. ERROR: The connection is refused because git is blocked by firewall:
    • SOLUTION: Run the following cmd to force git to use https://
      1. git  config  –global url.”https://&#8221;.insteadOf git://
    • P.S. This error is thrown during devstack/stack.sh command when VM is behind the firewall.
  4. Task: Move an Instance/VM from project A to B:
    1. Create a snapshot/image of the instance in project A:
      openstack server image create –name vm1-snapshot  vm1
    2. Set snapshot/image visibility to Public with admin user:
      openstack image set –public vm1-snapshot
    3. Create an instance/VM from the snapshot in project B:
      openstack server create –image vm1-snapshot –flavor m1.tiny  vm2-clone
Posted in coa, openstack, Other

Python basics

(i) Install and configure git:

  1. yum install git && vim-enhanced
  2. git config –global user.name “User”
  3. git config –global user.email “user@python.local”
  4. cur https://raw.githubusercontent.com/linuxacademy/content-python-for-sys-admins/master/helpers/bashrc -o ~/.bashrc
  5. cur https://raw.githubusercontent.com/linuxacademy/content-python-for-sys-admins/master/helpers/vimrc -o ~/.vimrc
  6. mkdir sample; touch sample.txt; cd sample
  7. git init
  8. git add –all .
  9. git commit -m ‘Initial commit’

(ii) Data types:

  • Comments:
    • # This is a comment
    • print(“Hello World”)    #This is also  a comment
  • Strings:
    • str1 = “Hi”
    • str2 =’Hi’
    • “pass” + “word”  -> ‘password’
    • “Ha” * 3 -> ‘HaHaHa’
  • Numbers:
    • 1 + 2  -> 3, 2.2 + 2 -> 4.2, int(“1”), float(“1.1”)
  • Booleans:
    • True; False
  • Variables:
    • my_int = 0, my_int = ” int”, print(my_int)
  • None:
  • List and Tuples:
    • Lists:
      • list = [1, “str”, 1.2, ‘g’, False]; print (list)
      • list[0] -> 1; list[3] -> ‘g’; list[-1] -> False
      • list[0:3] = list[:3] -> [1, ‘str’, 1.2]
      • list[2:] -> [1.2, ‘g’, False]
      • list[::2]-> len[1, 1.2, False]   (Steps two, i.e indexes 0, 2, 4)
      • list[1::2] -> [‘str’, ‘g’]              (Starts at index 1 and steps 2)
      • list[1] = ‘abc’; print(list) -> [1, ‘abc’, 1.2, ‘g’, False]
      • list[1, 3] = [‘a’, ‘c’]
      • list[1,4] = []
      • list += [1, 3] ->print(list) -> [1, ‘abc’, 1.2, ‘g’, False, True, 1, 3
      • list.append(True); print(list) -> [1, ‘abc’, 1.2, ‘g’, False, True]
      • list.remove(“str”)
      • list.pop() -> False; list.pop(0) -> 1
      • len(list) -> 5
    • Tuples:
      • point = (2.0, 3.0)
      • point(0) -> 2.0
  • Dictionaries:
    • ages = { ‘Ishtiaq’ : 33, ‘Ali’: 35}
    • print(ages) -> {‘Ishtiaq’: 33, ‘Ali’: 35}
    • ages[‘Ishtiaq’] = 39
    • ages.values() -> [33, 35]
    • ages.keys() -> [‘Ishtiaq’, ‘Ali’]
    • ages.items() -> [(‘Ishtiaq’, 33), (‘Ali’, 35)]
    • ages.pop(‘Ali’) -> 35; ages-> {‘Ishtiaq’: 33}

(iii) Control Flow – Conditionals, Comparisons and Logic Operations:

  • if, elif, else:            (==, !=, >, >=, <, <=)
    • 2 > 1 -> True; 2 < 1 -> False; 2 ==2.0-> True; 1 != 1.0 -> False
    • 2 in [1, 2, 3] -> True
    • num = 0;
      if num: print(“Number is not zero %s ” %num)  (If num NOT zero)
      else:  print(“Number is zero %s ” %num)  (This would be executed)
    • str = “‘
      if str: print(“String is not empty: %s ” %str)  (If str NOT empty)
      else:  print(“String is empty %s ” %str)  (This would be executed)
    • num = 3
      if num % 2 == 0:
      print(“Number is even.”)
      elif num % 2 != 0:
      print(“Num is odd”)
      else:
      print(“Invalid number”)
  • and operator:
    1. 1 and 2 -> 2
    2. 0 and 1 -> 0
    3. 1 and 1 -> 1
  • or operator:
    1. 1 or 2 -> 1
    2. 1 or 1 -> 1
    3. 0 or 1 -> 1

(iv) Control Flow – Loops:

  • for loop:
    1. for i in range(3):         (By default starts at 0 -> range(0,3))
      print(“%s ” %i)            (Output-> 0, 1, 2)
    2. colors = {‘blue’, ‘green’, ‘red’}
      for color in colors:
      print(color)            (Insert 4 white-spaces here)
    3. point = (1, 3.2, 4)
      for p in point:
      print(p)                    (Insert 4 white-spaces here)
    4. ages = {‘David’: 33, ‘Alisa’: 35}       (Dictionary)
      for key in ages:
      print(key)               (Output: David  Alisa)
    5. for key, value in ages.items():
      print(key)                (Output-> David 33 Alisa 35)
      print(value)
    6. numbers = [1, 2, 3, 4]
      print([x * x for x in numbers])   (Output-> 1, 4, 9, 16)
  • while loop:
    1. count = 0
      while (count < 10):
      if count%2 == 0:
      count += 1
      continue
      print(“Counting odd numbers %s ” %count)
      count += 1

(v) Scripts – Reading User Input:

  1. Execute normal scripts:
    1. cat > hello.py <<EOF
      #! /usr/bin/env python           (Can also use: #!/bin/python)

      name = raw_input(“What’s your name? “)    (Input strings)
      age = input(“What’s your age? “)                     (Input numbers)
      print(“Your name is ‘%s’ and age is ‘%s'” % (name, age))
      EOF
    2. python hello.py
  2. Executes scripts from ~/bin directory:
    1. chmod +x hello.py
    2. mkdir bin
    3. cp hello.py bin/hello
    4. export PATH=$HOME/bin:$PATH
    5. hello
  3. User defined functions:
    #!/bin/python
    def calc_fact(num):
    fact = 1
    while num > 1:
    fact *= num
    num -= 1
    return fact
    num = input(“Enter a number: “)
    print(“%s factorial is: %s” % (num, calc_fact(num)))
  4. Standard library ~/bin/timer
    #!/bin/python
    import time
    start_time = time.localtime()
    print(“Timer starts at %s ” % time.strftime(“%X”, start_time))
    raw_input(“Please press Enter to continue…”)
    stop_time = time.localtime()
    difference = time.mktime(stop_time) – time.mktime(start_time)
    print(“Timer stops at %s ” % time.strftime(“%X”, stop_time))
    print(“Total time %s seconds: ” % difference)

Python Standard Library:

  • Built-in Functions:
    • exit()            (Exits from python editor)
    • print():
      • print(“Name=%s” %name)
      • print(“Name=%s \t Age=%s” % (name, age))
    • bool():
      • bool(1) -> True, bool(0) -> False, bool(None) -> False, bool(“test”) -> True
    • len():
      • len(“Hi”)   -> 2, var = “Hi”; len(var)    -> 2
    • lower():
      •  “Hi”.lower()   ->  hi,  var = “Hi”; var.lower()    ->  hi
    • find():
      • “double”.find(‘s’)     -> -1    (Couldn’t find)
      • “double”.find(“bl”)  -> 3     (Indexes starts at 0, so bl=3)
    • strip()             (Strips white-spaces)
    • str.startswith(‘ch’)
    • append():
    • list.append(‘abc’)
    • remove(): list.remove(‘a’)
    • pop():
      • list.pop(), dic.pop(), list.pop(list_index), ages.pop(dic_key)
    • del():
      • del list|dic
    • items():
      • ages = {‘David’: 33, ‘Alisa’: 35}    (Dictionary)
      • ages.items()     ( Output-> [(‘David’, 33), (‘Alisa’, 35)] )
    • str():
      • str(1) -> ‘1’, str(False) -> ‘False’
    • int(“1”-> 1
    • float(“1.0”-> 1.0
    • bool():
      • bool(“1”) | bool(1) -> True, bool(“0”) | bool(0) | bool (“”) -> True
    • time:
      • localtime(), strftime(), mktime()

(vi) Files:

  • open/close:
    • file = open(‘/home/user/file.txt’)
    • file = open(‘/home/user/file.txt’, ‘r+’)
    • file = open(‘/home/user/file.txt’, ‘w’)
    • file = open(‘/home/user/file.txt’, ‘a’)
    • file.close()
  • read:
    • for line in file: print(line)
    • print(file.read())
  • write/seek:
    • file.seek(-1, 2)
    • file.write(“Hello\n”)
    • file.writelines([‘Hello\n’, ‘World\n’])
    • with open(‘/home/user/file.txt’, ‘a’) as file:
      file.write(“Hello World\n”)

Passing environment variables and command line parameters:

  • Environment variables:
    • pi.python:
      #!/usr/bin/env python
      from os import getenv
      from math import pi
      digits = getenv(“DIGITS”) or 10
      print(“%.*f” % (int(digits), pi))
    • DIGITS=5 python pi.python -> 3.14159
  • Command line argument and Handling Errors with try/except/else/finally:
    • #!/bin/python
      from sys import argv
      for arg in argv: print arg        (python test.py abc 123 -> abc 123)
    • #!/bin/python
      import argparse
      parser = argparse.ArgumentParser(description=’Read a file in reverse’)
      parser.add_argument(‘filename’, help=’the files to read’)
      parser.add_argument(‘–limit’, ‘-l’, type=int, help=’the number of lines to read’)
      parser.add_argument(‘–version’, ‘-v’, action=’version’, version=’%(prog)s 1.0′)
      args = parser.parse_args()
      try:
      f = open(args.filename)
      except IOError as err:
      print(“\n”File not found!\n”)
      else:
      with f:
      limit = args.limit
      lines = f.readlines()
      lines.reverse()
      if limit:
      lines = lines[:limit]
      for line in lines:
      print(line.strip()[::-1]
      finally:
      printf”\nWe are finished.\n”)

Execute Shell Command from Python:

#!/bin/bash
import subprocess
output = subprocess.check_output(‘ls -a’, stderr=subprocess.STDOUT)
code = subprocess.call([‘ls’, ‘-l’])      (If successful code=0)
print(output )
print(code)

Exit Statuses:

#!/bin/python
import sys
print(“Hello”)
sys.exit(1)            #echo $?-> 1

Iteration with List Comprehension:

#!/bin/python
import argparse

parser = argparse.ArgumentParser(description=’Search for words’)
parser.add_argument(‘snippet’, help=’Search for a word in /usr/share/dict/words file’)

args = parser.parse_args()
snippet = args.snippet.lower()

words = open(‘/usr/share/dict/words’).readlines()
print([word.strip() for word in words if snippet in word.lower()])

Python Package Manager: (pip replaced easy_install)

  • pip install:
    • sudo yum install epel-release
    • sudo yum update
    • sudo yum upgrade python-setuptools
    • sudo yum install python-pip python-wheel
  • pip commands:
    • pip -V                                                        (List pip version)
    • sudo pip install –upgrade pip             (Update pip)
    • sudo pip install boto3  [–user] [-r requirements.txt]   (Install a package)
    • pip list [–format=columns]  [–user]               (List installed pip packages)
    • pip freeze | grep boto3 > requirements.txt   (List pip packages)
      • pip install boto 3 –user                (Installs in /home/user/.local/)
      • python -> import boto3 -> boto3
        <module ‘boto3’ from ‘/home/user/.local/lib/python2.7/site-packages/boto3/*’>
      • python -> import time -> time
        <module ‘time’ from ‘/usr/lib64/python2.7/lib-dynload/timemodule.so’>
    • sudo pip uninstall boto3  [-r requriements.txt]    (Uninstall packages)

Virtualenv:
Virtualenvs are Pythons way of sandboxing dependencies to work with multiple versions of Python packages.

  • pip install virtualenv  [–user]      (which virtualenv | virtualenv –version)
  • mkdir venvs
  • virtualenv venvs/experiment
  • source venvs/experiment/bin/activate   (Activate virutalenv)
  • pip list
  • pip install flask     (Install a package in vritualenv and don’t need root permissions)
  • deactivate                                                   (Deactivate virutalenv)
  • Remove virtual environment:
    • deactivate
    • rm -rf venvs
    • pip uninstall virtualenv

requests library:
Its a popular library used for utilizing the third-party packages in your own scripts.

  • source venvs/experiment/bin/activate
  • pip install requests     (Verify-> pip list|freeze)
  • bin/weather:   (chmod +x bin/weather)
    #!/usr/bin/env python
    import requests
    print(requests.__file__)
  • weather:
    /home/user/venvs/experiment/lib/python2.7/site-packages/requests/__init__.pyc
  • deactivate && weather:
    /usr/lib/python2.7/site-packages/requests/__init__.pyc

PostgreSQL:

Structuring python project:

  • pip install pipenv –user
  • mkdir -p ~/code/pgbackup
  • pipenv install –two
  • pipenv shell
  • Create a in ~/code/pgbackup/README.rst
  • mkdir -p src/pgbackup tests docs
  • touch src/pgbackup/__init__.py  docs/.keep  tests/.keep
  • Create ~/code/pgbackup/setup.py
  • git init
  • curl https://raw.githubusercontent.com/github/gitignore/master/Python.gitignore \
    -o .gitignore
  • git add -all .       (git status)
  • git commit -m ‘Initial commit’
  • Create ~/code/pgbackup/tests/test_cli.py
  • pipenv install –dev pytest
  • make
  • create ~/code/pgbackup/src/pgbackup/cli.py

Build and Install Python 3 from source on CentOS7:

  • sudo su –
  • Install Development Tools and some extra libraries:
    • yum groupinstall “development tools” -y
    • yum install -y libffi-devel zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel expat-devel wget
  • Download Python 3 source package and extract it:
    • cd /usr/src
    • wget https://www.python.org/ftp/python/3.6.4/Python-3.6.4.tar.xz
    • tar xvf Python-3.6.4.tar.xz
  • Build and Install Python 3:
    • cd /usr/src/Python-3.6.4
    • ./configure –enable-optimizations
    • make && make altinstall            (Note: altinstall will leave Python 2.7 intact)
    • python3.6
    • quit()

 

Python Built distribution: (wheel replaced egg)

Posted in Other, python

Setup OpenStack (Pike) using Devstack in Ubuntu 16.04

Install OpenStack (Pike) release using Devstack:

  1. Install and configure Ubuntu 16.04 in VMware and set interface to Bridge:
    • Add the following into /etc/network/interfaces
      auto eth0
      iface eth0 inet static
      address 192.168.9.6
      netmask 255.255.255.0
      #gateway 192.168.9.1
      auto eth1
      iface eth1 inet dhcp
    • sudo systemctl restart networking
    • P.S. There are two network adapter added into VM, eth0 (Host-only) to allow access from guest and eth1 (NAT) to allow access to Internet.
  2. Update and Install required packages:
    1. sudo apt-get update
    2. sudo apt-get install git vim
  3. Setup a user:
    1. sudo useradd -s /bin/bash -d /opt/stack -m stack
    2. echo “stack ALL=(ALL) NOPASSWD: ALL” | sudo tee /etc/sudoers
    3. echo stack | sudo passwd –stdin stack
    4. su – stack
  4. git clone https://git.openstack.org/openstack-dev/devstack -b stable/pike
  5. Add the following into by using “vim ~/devstack/local.conf”
    [[local|localrc]]
    HOST_IP=192.168.9.6
    FLAT_INTERFACE=eth0
    ADMIN_PASSWORD=openstack

    DATABASE_PASSWORD=$ADMIN_PASSWORD
    RABBIT_PASSWORD=$ADMIN_PASSWORD
    SERVICE_PASSWORD=$ADMIN_PASSWORD
  6. ~/devstack/stack.sh
  7. To uninstall or reinstall run:
    1. ~/devstack/unstack.sh
    2. ~/devstack/clean.sh

Useful commands:

  • Openstack RC files:
    • source /opt/stack/devstack/openrc username [<projectname>]
    • cat /opt/stack/devstack/.localrc.auto | grep -i password
    • Sample basic Openstack RC/openrc file:
      export OS_AUTH_URL=http://127.0.0.1/identity   [http://127.0.0.1:5000/v2.0]
      export OS_TENANT_NAME=”demo
      export OS_USERNAME=”demo
      export OS_PASSWORD=”openstack
  • Logs and samples:
    1. /opt/stack/logs/stack.sh.log
    2. /opt/stack/devstack/samples/local.conf
  • openstack services status:
    • openstack [host|hypervisor|catalog] list
    • openstack [compute service |volume service | network agent] list
  • Connect to instance/VM:
    • ip netns list
    • ip netns exec qrouter-<router-id> ssh cirros@10.0.0.71

Restart services via systemctl:

  • sudo systemctl -a  | grep devstack        (Status for All services)
  • sudo systemctl status devstack@*        (All services status)
  • sudo systemctl restart devstack@*       (Restart all)
  • sudo journalctl -f –unit devstack@*|n-*|q-*|c-*|g-*  (Logs)
  • Nova:
    • sudo systemctl status devstack@n-*
    • devstack@n-cpu.service (nova-compute)
    • devstack@n-sch.service (nova-scheduler)
    • devstack@n-cauth.service (nova-consoleaut)
    • devstack@n-cond-cell1.service  (nova-conductor)
    • devstack@n-super-cond.service (nova-conductor)
    • devstack@n-api-meta.service  (uwsgi)
    • devstack@n-api.service  (uwsgi)
    • devstack@n-novnc.service  (nova-novncproxy)
  • Neutron:
    • sudo systemctl status devstack@q-*
    • devstack@q-svc.service (neutron-server)
    • devstack@q-l3.service  (neutron-l3-agen)
    • devstack@q-dhcp.service (neutron-dhcp-ag)
    • devstack@q-agt.service (neutron-openvsw)
    • devstack@q-meta.service (neutron-metadat)
  • Cinder:
    • sudo systemctl status devstack@c-*
    • devstack@c-vol.service  (cinder-volume)
    • devstack@c-api.service (uwsgi)
    • devstack@c-sch.service (cinder-schedule)
  • Glance:
    • sudo systemctl status devstack@c-*
    • devstack@g-reg.service  (glance-registry)
    • devstack@g-api.service  (uwsgi)
  • sudo systemctl status devstack@keystone.service  (Keystone service)
  • sudo systemctl status devstack@placement-api.service  (uwsgi)
  • sudo systemctl status devstack@dstat.service (dstat.sh)
  • devstack@etcd.service  (etcd)

Restart Services via Screen:

  1. su – stack                          (Connect to stack user)
  2. screen -ls                           (Lists screens)
  3. script /dev/null           (Resolution for Error: ‘Cannot open your terminal ‘/dev/pts/0’)
  4. screen -r stack                  (Attach into the screen)
  5. ctrl+a ”                               (List all services)
  6. g-api                                   (e.g select Glance API service)
  7. ctrl+c                                  (Stop the service)
  8. Click UP-Arrow (key) and press Enter  (Recall and execute previous service Start cmd)
  9. ctr+a d                                (Detach from the screen)
  10. exit
Posted in coa, openstack, Other