AWS Networking and Content Delivery Services

  1. Virtual Private Cloud (VPC):
    • A VPC (virtual data center) is an isolated portion of the AWS cloud populated by AWS objects, such as Amazon EC2 instances. You must specify an IPv4 address range for your VPC. Specify the IPv4 address range as a Classless Inter-Domain Routing (CIDR) block; for example, 10.0.0.0/16. Block sizes must be between a /16 netmask and /28 netmask.
    • It consists of an Internet Gateway (IGW), multiple Subnets, one or more Route Tables, multiple Security Groups.
    • Max 5 VPC are allowed per account but you can increase this limit by contacting AWS support.
    • You can’t change or increase/decrease CIDR block of the VPC and subnets inside it.
    • You can add more than one CIDR block in a single VPC.
    • The CIDR block must not overlap with an existing CIDR blocks in the VPC.
    • Troubleshooting:
      • To enable access to or from internet for instances in a VPC, you must attach an Internet Gateway to your VPC, ensure that your subnet’s route table points to the Internet Gateway, ensure that instances in your subnet have a public IP address or Elastic IP address, and ensure that your network access control list and security group rules allow relevant traffic to and from your instance.
      • Traffic flow in VPC: Route 53Internet Gateway > ELB > Route Table > Network ACL > Subnet > Security Group > Instance
    • Subnets: to provision AWS resources (Such as EC2 instances)
      When you create a VPC, it spans all of the availability zones in the region. After creating a VPC, you can add one or more subnets into it.

      • A subnet must reside entirely within one availability zone and can’t span zones i.e 1 subnet = 1 AZ (You can have multiple subnets in a single AZ).
      • CIDR block sizes must be between a /16 netmask and /28 netmask. Also, note that a subnet can be the same size as your VPC.  You can’t change CIDR block. (http://cidr.xyz/       Selecting CIDR block range)
      • Subnets must be associated with a route table.
      • A subnet can be associated with only one Network ACL at a time.
      • The different subnets in a VPC can’t overlap (i.e sub1=10.0.1.0/24, sub2=10.0.1.0/28).
      • All IPv6 addresses are public and are allocated by AWS.
      • By default all traffic is allowed between subnets in a VPC via a local route. That route is created by default when a new VPC is created and it can’t be modified or deleted.
      • In each subnet the first 4 and last IP addresses are reserved by AWS:
        • 10.0.0.0 (Network address)
        • 10.0.0.1 (VPC router, reserved by AWS)
        • 10.0.0.2 (DNS server, reserved by AWS)
        • 10.0.0.3 (Future use, reserved by AWS for )
        • 10.0.0.255 (Network broadcast address)
      • Public subnetHave a route to the internet (Enable auto-assign public IP settings on the subnet), in other words the route table of which this subnet is part of, have IGW assigned for default route (0.0.0.0/0).
      • Private subnet: Don’t have a route to the internet (But you can access the internet via NAT gateway or instance). In other words the route table of which this subnet is part of have no IGW assigned instead you can assign NAT gateway for default route (0.0.0.0/0).
      • VPN Only subnet: Its same as private subnet but doesn’t require access to the internet (i.e no NAT gateway) and is accessed via VPN.
      • Auto-assign IPs or Enable auto-assign public IPv4 address: Automatically request a public IPv4 address for instances launched into this subnet.
    • Internet Gateway (IGW): Its a virtual router that connects a VPC to the internet.
      • Only one Internet Gateway can be attached to a VPC at a time.
      • It can’t be detached from a VPC which have active AWS resources in it (such as EC2 instance or RDS database).
      • In order to delete, it must be detached from the VPC.
      • It perform network address translation (NAT) for instances that have been assigned public IPv4 addresses.
      • It supports IPv4 and IPv6 traffic.
      • Your default VPC already has a IGW attached.
    • NAT Gateway: Network Address Translation (NAT) Gateway is designed to provide EC2 instances (in a private subnet) access to the internet (to download software and updates).
      • A NAT gateway must be provisioned into a public subnet so that they can have a pubic IP address, and it must part of the private subnet’s route table (by adding 0.0.0.0/0 route to NAT Gateway).
      • It allows only incoming traffic through if a request is originated from a instance in a private subnet.
      • An Elastic IP (EIP) is associated with NAT gateways, and are manged by AWS, and they can scale up to 10 Gbps.
      • When you have resources in multiple AZs and they share one NAT gateway, if the AZ in which NAT gateway was provision goes down, the resources in the other AZs will also lose internet access. To create AZ fail-over architecture, create a NAT gateway in each AZ and configure your routing to ensure that resources use the NAT gateway in the same AZ.
    • NAT Instance: Its an EC2 instance provisioned in a public subnet (i.e having public IP) and used to provide internet access for EC2 instances in private subnets. It used to do the same job as NAT Gateway (Elastic IPs only) and is a legacy feature of AWS. It can function on both Public or Elastic IP.
      • You need to disable Source/Destination Check flag.
      • It need to be created from an AMI image that allows NAT.
      • You can use it as Bastion server as well.
      • For HA and Failover, create a script to manage failover between multiple NAT instances.
      • NAT instance can be searched by typing “amzn-ami-vpc-nat” in Community AMIs.
      • When you launch an instance from a NAT AMI, folloing configuration occurs on the instance:
        • IPv4 forwarding is enabled and ICMP redirects are disabled in /etc/sysctl.d/10-nat-settings.conf
        • A script located at /usr/sbin/configure-pat.sh runs at startup and configures iptables IP masquerading.
    • Route Table: (with predefined routes to the default subnets) A route table contains a set of rules (routes), that are used to determine where network traffic is directed.
      • It specifies how packets are forwarded between the subnets within your VPC, the Internet, and your VPN connection.
      • By default, all subnet’s traffic is allowed to each other available subnet within your VPC via local route (e.g 10.0.0.0/16 -> local) and the local route can’t be modified or deleted.
      • Each subnet in your VPC must be associated with only one route table (By default are associated with Main route table).
      • Main route table is created by default along with VPC and it can be deleted.
      • You can have multiple “active” route tables in a VPC.
      • You can’t delete a route table if it have dependencies (associated subnets)
      • You can have max 200 route tables in a single VPC and you can have 50 routes in a single route table.
      • A route table’s rules are comprised of two main components:
        • Destination: The CIDR block range of target where data is routed to, e.g 10.0.0.0/16.
        • Target: A name identifies where the data is being routed to, e.g local or igw-bd4567hj.
    • Network Access Control List (ACL): (with predefined rules for access)
      network access control list (ACL) is an optional layer of security for your VPC that acts as a firewall for controlling traffic in and out of one or more subnets.

      • ACLs operate on the network/subnet level whereas security groups operate at instance level.
      • They support allow and deny rules for traffic travelling in and out of a subnet.
      • They are stateless, so return traffic must also be allowed explicitly through an outbound rule.
      • The last rule in every ACL is catch all deny rule.
      • The default Network ACL (from default VPC) Allow All inbound and outbound traffic to the default subnets.
      • Whereas a newly created custom Network ACL Deny All inbound and outbound traffic by default.
      • Rules are evaluated from lowest to highest based on Rule number. The first rule found that applies to the traffic type is immediately applied, regardless of any rule that comes after it
      • A Network ACL may contain multiple subnets, but a single subnet can only be associated with only one Network ACL at a time.
      • Allow outbound traffic for all ephemeral ports i.e 1024-65353.
      • You can block IP addresses using NACL not Security Groups.
      • Inbound: Rule #|Type|Protocol|Port Range|Source|Allow/Deny
      • Outbound: Rule #|Type|Protocol|Port Range|Destination|Allow/Deny
    • Security Groups: They are similar to Network ACL in that they allow or deny traffic to EC2 instances.
      • Security groups operate at instance level whereas Network ACLs operate at subnet level.
      • They are stateful, so return traffic requests are allowed regardless of the outbound rules are added or not.
      • You can add only allow rules, but there are NO deny rules.
      • By default all traffic is denied, unless there’s an explicit allow rule for it. And all rules are evaluated before deciding to allow traffic.
      • Outbound traffic is always allowed both for default and custom security groups.
      • Inbound traffic is denied on custom security group, and for default security group its allowed from the sources that have the same (itself) security group applied.
      • You can use security group names in Inbound Rules (as source) and in Outbound Rules (as destination).
      • Up to five security groups can be applied onto an EC2 instance
      • You can’t block (because there’s no Deny rules in SGs) traffic from a specific IP/network address using security groups instead use Network ACL for it.
      • The default security group created with VPC, can’t be deleted. The security group have a scope of VPC in which its created i.e a security group created in one VPC can’t be used in another VPC.
      • Should be used to group resources with similar functions.
      • Inbound: Type|Protocol|Port Range|Source
      • Outbound: Type|Protocol|Port Range|Destination
    • VPC Flow logs: It enables you to capture information about the IP traffic going to and from network interfaces in your VPC.
      • Flow log data is stored in CloudWatch Logs and can be viewed from there.
      • Flow logs can help you with a number of tasks; for example, to troubleshoot why specific traffic is not reaching an instance, which in turn helps you diagnose overly restrictive security group rules. You can also use flow logs as a security tool to monitor the traffic that is reaching your instance.
      • There is no additional charge for using flow logs; however, standard CloudWatch Logs charges apply.
    • Bastion/Jump HostEC2 instances that sit within your public subnet and are typically accessed using SSH or RDP. Once remote connectivity has been established with a bastion host, it then acts as a jump server, allowing you to use SSH or RDP to log in to other EC2 instances (within private subnets) deeper within your VPC.
    • Peering Connections: VPC Peering is used to extend your private network from one VPC (or one subnet or specifically one instance) to another VPC. This is for sharing internal resources via private IP addresses.
      • VPC peering supports in the same regions as well as in Inter-Regions and in the same accounts and Inter-accounts.
      • Once VPC peering is done, then you need to update the Route Tables in both VPCs to point to VPC Peering ID as gateway to route the VPCs traffic.
      • Overlapping CIDR blocks are not allowed.
        • i.e VPC1=192.168.0.0/16, and VPC2=192.168.1.0/24
      • Transitive peering is not allowed.
        • i.e if VPC1 is peered with VPC2 and VPC3, then connection between VPC2 and VPC3 is not allowed automatically, VPC2 and VPC3 must be peered explicitly.
      • Edge to edge routing through a gateway or private connection is not allowed.
    • VPC Endpoints: Allows you to securely (internally) connect your VPC to another AWS service.
      • Applicable in the same region and IPv4 only.
      • An interface can be access via DirectConnect (Not through VPC or VPC Peering).
      • An Interface endpoint (powered by PrivateLink) uses Elastic Network Interface (ENI) as an entry point for traffic destined to the AWS service. ENI with a private address serves as endpoint, e.g:
        • Elastic Load Balancer
        • EC2 API, EC2 System Manager
        • Service Catalog
        • Kinesis Streams
      • A Gateway Endpoint servers as target for a route in your route table for traffic destined for the service. A target for route table in your environment.
        • DyamoDB
        • S3
    • Virtual Private Network (VPN)It enables the ability to extend a  subnet from one geographic location to another geographic location on two separate networks. This is essentially extending on-premise network to the cloud or vice versa.
      • For AWS this allows to communicate all resources like (EC2 instances) internally without the need for public IP addresses and an internet gateway.
      • Both a Virtual Private Gateway (VPC) and Customer Gateway are required to setup a VPN Connection.
      • A VPC can have both a VPG and IGW attached at the same time.
      • Customer Gateway: Its either a physical hardware or software application that’s located on-premise data center and acts as a VPN connector on on-premise data center side and must be configured with a static public IP address.
      • Virtual Private Gateway: It acts as a connector on the AWS data center VPC side of the VPN connection. The VPG is connected to VPC and only one VPG can be attached to a VPC (just like IGW). It provides to VPN endpoints for automatic failover.
      • VPN Connection: Its a actual link between the VPG and the Customer Gateway. This connection is setup and managed in AWS. It uses two parallel routes (IPSec+BGP tunnels), which is for redundancy and fail-over.
      • Router: The route table (for subnet you wish to extend) must include routes for the on-premise network that are used by the VPN, and point them to the Virtual Private Gateway.
      • You have a soft limit of 10 VPN connection per VGW.
    • Enhanced Networking:
      • EC2 instance must support SR-I/OV (Single Root I/O Virtualization)
        • Its virtualizing the network adapter on the physical host.
        • SR-I/OV is supported on R4, X1, P2, C3/4, I2, M4, D2
      • Should be created from HVM AMIs and be launched in default VPC.
      • Can be enabled on EBS and Instance backed instances.
  2. Route 53 (Domain & DNS):
    • Its a domain management services (DNS hosting solution) provided by AWS. It can manage external DNS for domain routing. Route 53 is commonly used with ELB to direct traffic from domain to ELB. You may use it configure fail over to S3 or CloudFront.
    • Like IAM, its a global service applies to all AWS regions. Its named 53 cause DNS service listens on port 53. By default you can have 50 domain names limit which can be increased by contacting to AWS support.
    • ELBs don’t have a pre-defined IPv4 addresses, you always resolve them using a DNS name.
    • Route53 has a security feature that prevents internal DNS from being read by external sources. The work around is to create a EC2 hosted DNS instance that does zone transfers from the internal DNS, and allows itself to be queried by external servers.
    • Key features and functions include:
      • Domain Registration: You can use route 53 to registers a domain name with on it.
        • Creates a hosted zone that has the same name as your domain.
        • It assigns a set of four name servers (unique to your account) to the hosted zone.
        • It gets the name severs from the hosted zone and adds them to the domain.
      • DNS Service: As a DNS, it routes Internet traffic to the resources for your domain name by translating the domain into IP address. Route 53 responds to DNS queries using a global network of authoritative DNS servers, which reduces latency.
        • If you registered your domain with a different registrar (e.g GoDaddy), then you need to configure the Route 53 NS server list in your registrar’s DNS database for your domain.  It may take up to 48 hrs to update the changes.
        • Once you update the Route 53 NS settings with your domain registrar, then Route 53 will be responsible to respond to DNS queries for the hosted zone.
      • Health Checking: It checks the health of your resource/domain by sending automated requests over the Internet to your application, to verify that it’s reachable, available, and functional. It tracks the health status of your resources, such as web or mail server, and take action when an outage occurs.
    • Hosted Zones: Its a collection of records for a specified domain. Its a container that holds information about how you want to route traffic for a domain, such as example.com, and its subdomains.
      • It stores DNS record for your domain and it contain all rules that tells Route 53 what to do with DNS request.
      • It comes pre-populated with NS (name server) and SOA (start of authority) record sets in AWS, don’t change these records.
      • Public hosted zones: Is a container that holds information about how you want to route traffic for a domain and its sub-domains on the internet.
      • Private hosted zones: Is a container that holds information about how you want to route traffic for a domain and its sub-domains within one or more VPCs.
    • Record SetsThey are instructions that actually match domain names to IP addresses. Record sets compromised of various options:
      • Record type, Standard alias, Routing policy, Evaluate target health.
      • You can’t set TTL (time to live) in Alias record instead it will use the TTL in Elastic Load Balancer, CloudFront, Elastic Beanstalk or S3.
      • Common record types includes:
        • A: Resolves a domain name to an IPv4 address.
        • AAAA: Resolves a domain name to an IPv6 address.
        • CNAME: Resolves a domain name to another domain (hostname).
          • web  IN  CNAME   www.example.com
          • The DNS protocol doesn’t allow you to create CNAME record for apex/root/naked domain (example.com). In Route 53 you use Alias record to do that.
        • NS: Used for delegating zone to a nameserver
          • example.com   IN   NS   ns1.example.com
        • SOA: Start of Authority
          • Every single zone has only one SOA resource record at the beginning of the zone.
        • MX: Used to route email (mail exchange)
          • example.com    IN   MX  10  mail01.example.com
      • Alias Record: It contains a pointer to an AWS specific resource i.e an ELB, S3 bucket (configured as static website), CloudFront distribution, Elastic Beanstalk environment.
        • Queries to Alias records are free.
        • Alias record sets are used only for A and AAAA DNS record types. The are similar to CNAME record set, one of the difference is that you can create an alias for the zone apex (root/naked domain e.g example.com).
          • Normally, you can use CNAME for sub-domain e.g:
            http://www.example.com   CNAME  123.ec2.amazonaws.com
          • But you CAN’T use CNAME for root/naked domain e.g:
            example.com  CNAME   123.elb1.amazonaws.com
            So you have to use Aliases for this.
      • Routing Policies:
        • Simple (round robin): Route all traffic to one endpoint. Use for a single resource that performs a given function for your domain, for example, a web server that serves content for the http://www.example.com website.
        • Weighted: Route traffic to multiple endpoints (manual load balancing). It route traffic to multiple resources (e.g ELBs in separate or same regions) in proportions to weights that you specify. (Valid values 0-255, 0-> disable routing to a resource)
        • Latency: Route traffic to an endpoint based on users latency to various endpoints. Use when you have resources in multiple locations and you want to route traffic to the resource that provides the best latency i.e fastest response time.
        • Failover: Route traffic to secondary endpoint, if the primary is unavailable.
          • Use when you want to configure active-passive failover. If the main resource is not healthy, then route traffic to a different resource.
          • You can use Route 53 with failover policy to failover to a static S3 website or CloudFront distribution.
        • Goelocation: Route traffic to an endpoint based on the geographical location of the user.
          • If you don’t create a default record for the IP addresses which aren’t mapped to geographic locations, then Route 53 returns a “no answer” response for the queries from those IPs.
        • Geoproximity: Use when you want to route traffic based on the location of your resources and optionally shift traffic from resources in one location to resources in another.
        • Multivalue answer: Use when you want Route 53 to respond to DNS queries with up to eight healthy records selected at random.
    • S3 for DNS failover: By using failover routing policy in a Route 53 DNS record set, an S3 bucket can be used as a failover endpoint.
      • S3 bucket name must be same as the domain name in DNS record.
      • It can provide an extremely reliable backup solution if your primary endpoint fails.
      • An S3 bucket can also be used as a primary endpoint, if you just want to host a simple static site.
  3. CloudFront:
    • Its a global Content Delivery Network (CDN) which delivers contents from origin location to an edge location. Edge location allows the caching of static objects from the origin locations (S3 bucket or ELB).
    • Users experience lower latency and contents load time.
    • Reduces load on application resources (origin services), thus reducing costs.
    • They are not just read-only you can write objects on Edge Locations as well.
    • It can integrate with Route 53 for alternate CNAMEs that will allow to create URLs such as http://cdn.mydomain.com
    • It has builtin Distributed Denial of Service (DDOS) attack protection and mitigation.
    • To serve a new version of object, either create a new object with a new name or create an invalidation request on the CloudFront distribution.
    • You can clear cached items via Invalidation but it has a cost, so if you have to invalidate a large CloudFront distribution, then perhaps it would be economical to create a new distribution and move DNS names.
    • Caching is done based on object names.
    • Cache objects can also be set specific expiration time (TTL/Time to Live) or to not cache at all.
    • To log CloudFront usage data (for PCI and HIPPA compliant) enable CloudFront access logs and capture requests that are sent to the CloudFront API.
    • Content Delivery Network (CDN): Its a system of distributed servers (network) that deliver web contents to a user based on his geographic locations, the origin of the web content and a content delivery server.
    • Edge Location: An Amazon Data center which doesn’t contain AWS services. Instead its used to deliver contents to parts of the world.
      • Cached items such as pdf files can be cached at an edge locations which  reduces the amount of space/time/latency required for a request from that part of the world.
    • Distribution: A collection of Edge Locations, it allows you to distribute content using a worldwide network of edge locations that provide low latency and high data transfer speeds. Following are types of distribution:
      • Web: Static or dynamic websites content (.html, .css, .php, and images), origin can be either an S3 bucket or a web server.
      • RTMP: Streaming media files using Adobe Flash Media Server’s RTMP protocol.
    • Origin: Its the source of  the content (static objects). This can be either S3 bucket, EC2 instance, ELB, Route53 or custom web server.
    • Signed URLs: Signed URLs allow access to private content by creating a temporary, one time use URL based off of the number of seconds you want it to be accessible. Signed with X.509 certificates.
    • Performance:
      • Its performance can be affected by file size and type
      • Slow DNS issues can cause performance issues, because its depends upon DNS check to determine edge location of the user request
      • Longer cache periods increases performance.
      • Query strings reduces performance (cache hits), because they are often unique so it reduces the cache hits and also require extra work in order to forward it to the origin.
  4. Direct Connect (DX):
    • It provides a dedicated network connection (1Gbps or 10 Gbps) between your on-premise data center network and one of the AWS Direct Connect locations. Its done through a authorized Director Connect Provider (i.e Verizon or other ISP).
    • It provides dedicated private network connection (no internet) between on-premise network and AWS network.
    • AWS Direct Connect location provides access to only the AWS region (plus all AZs in it) its associated with, not with other AWS regions.
    • A virtual interface is basically a 802.1Q VLAN mapped from customer router to Direct Connect. You need to use multiple virtual interfaces to connect to multiple VPCs.
    • Virtual Private Gateway (VPG) is used at AWS end, and you can’t use NAT gateway (to access internet) in Direct Connect connection.
    • Cross network connection: The physical connection between your network and the Direct Connect authorized partner, which then handles the routes and connections to AWS cloud network.
    • Public Virtual Interface: It allows you to connect to public AWS services (i.e S3, Glacier and DynamoDB).
      • Required public (IP) CIDR block range.
      • Connection maintains consistent traffic consistency as its sent over dedicated network.
      • Using IPSec VPN you can connect over the public VIF to remote AWS regions as well.
    • Private Virtual Interface: It allow you to connect to AWS VPC via automatic route discovery using BGP and requires a public or private ASN number.
      • It can only communicate with VPC (private IPs of EC2 instances).
      • Can’t access public IP addresses as Direct Connect is not an internet provider.
      • Its a dedicated private connection that works like a VPN.
      • For best practice use two Direct Connect connections for active-active or active-failover availability.
      • Can also use VPN as a backup connection.
      • Can create multiple private PVIs to multiple VPCs at the same time.
Advertisements
Posted in aws, cloud

Docker basic commands

Virtual machines are an abstraction of physical hardware, where hypervisor allow multiple VMs running on the same server. Whereas, Containers are a further level of abstraction on application layer (OS), it packages code and dependencies together.

A Linux Container is a lightweight application isolation mechanism that allows the kernel to run groups of processes in their own isolated user spaces, separate from the host system. The container has its own process list, network stack, file systems, and other resource, but shares the kernel with the host and other containers running on the system. Linux containers are implemented through a combination of three kernel features: namespaces for isolation, control groups for resource control, and SELinux for security.

Docker is used to create, control and manage containers. Docker adds an API, an image format, and a delivery and sharing model to Linux containers. A Docker image contains an application and all its dependencies. When a container is started, a read-write layer for that container is combined with the read-only image using LVM thin provisioning. Docker images are portable and can be saved and exported to other systems and users. (Source: RH254 – RedHat System Administration III)

  • Install docker:
    • yum install docker
    • systemctl start docker
    • docker search apache                   (Search for container)
    • docker pull docker.io/httpd         (Retrieve the container)
    • docker images [–all]     (List installed containers and components (–all))
    • docker run httpd                            (Starts the container)
    • docker ps [-a | -l]                            (List Container or process)
    • docker info                                       (Display system-wide information)
    • curl http://172.17.0.2                      (IP is shown in docker httpd output)
  • Stop and uninstall docker:
    • ctrl+c
    • docker stop httpd
    • docker rmi -f docker.io/httpd   (Removes container image)
    • systemctl stop docker
    • yum remove docker
  • Issues:
    • Error: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
    • Solution: Run the command with root privileges or sudo.
Posted in docker, LFCE, LFCS

tmux and screen basic commands

tmux:

  • tmux | tmux new -s “mysession”          (Start a new session)
  • tmux ls                       (List sessions)
  • ctrl+b d | exit            (Detach or exit from a session)
  • ctrl+b ”                        (Divides screen horizontally / create a new pane)
  • ctrl+b %                      (Divides screen vertically / create a new pane)
  • ctrl+b x | exit            (Kill a pane)

screen:

  • screen                                   (Create a new session)
  • screen -ls                              (List session)
  • screen -r [screen-name]     (Attach a session)
  • exit                                        (exit or kill a session)
  • ctrl+a d |screen -d             (Detach a session)
  • ctrl+a ”                                 (Lists all running commands)
Posted in LFCE, LFCS, Linux, Other

OpenStack CLI (commands) reference

(i) OpenStack Services health check: (Verify installation)

  • openstack host list | openstack host show $(hostname)
  • openstack hypervisor list | openstack hypervisor show $(hostname)
  • openstack catalog list                                                      (Show all service endpoints)
  • openstack compute service list | nova service-list    (Status of Compute Services)
  • openstack volume service list   | cinder service-list  (Status of volume services)
  • openstack network agent list    | neutron agent-list  (Status of Network Services)
  • openstack usage list                                                          (Show usage per project)
  • openstack command list                                                  (Show all commands)
  • openstack server list –all-project | nova list –all-tenants | virsh list –all
  • swift stat -v | df -ha | grep swift                                    (Disk space utilization)

(ii) OpenStack Identity Service (Keystone):

  • Service:
    • openstack service list
    • openstack service create –name service1  type1 
    • openstack service show service1
    • openstack service set –description “My Service1” service1
    • openstack service delete service1
  • Endpoint:
    • openstack endpoint list  [–service service1] | openstack catalog list
    • openstack endpoint create service1 –region RegionOne \
      –adminurl  <url>  –publicurl  <url> –internalurl <url>
    • openstack endpoint show <endpoint_id>
    • openstack endpoint set  –url http://10.0.0.1:456  <endpoint_id>
    • openstack endpoint delete <endpoint_id>
  • Project:
    • openstack project list  [–long]
    • openstack project create –name prj1   [–domain default]
    • openstack project show  prj1
    • openstack project set prj1 –description “My Project1”
    • openstack project delete prj1
  • User
    • openstack user list   [–long|–project prj1]
    • openstack user create –password password  user1
    • openstack user show user1
    • openstack user set –email “user1@example.comuser1
    • openstack user delete user1
    • keystone user-create –pass pass123 –name user1
  • Role:
    • openstack role list
    • openstack role create role1
    • openstack role show role1
    • openstack role delete role1
    • openstack role add|remove –user user1 –project prj1  Member
    • openstack role add|remove role1 –group grp1 –project p1|–domain d1
    • openstack role assignment list –names [–user usr|–group grp|–project prj]
  • Group:   (Use Dashboard to manage groups because its not included in ‘Newton’)
    • openstack group list
    • openstack group create grp1   [–domain domain1]
    • openstack group show grp1
    • openstack group set –description “My Group1” grp1
    • openstack group delete grp1
    • openstack group add|remove grp1 user1
    • openstack group contains user grp1 user1
  • Domain: (Not included in ‘Newton’ release)
    • openstack domain list
    • openstack domain create domain1
    • openstack domain show domain1
    • openstack domain set –disable domain1
    • openstack domain delete domain1   (Domain must be disabled before delete)
  • Logs:
    • /var/log/keystone/keystone.log

(iii)  OpenStack Networking (Neutron):

  • network: (External networks can be created with admin user only)
    • openstack network list  [–long|–external]
    • openstack network create net1  [–project prj1]   (Private/Internal network)
    • openstack network create net1-ext –share –external –project prj1  \
      –provider-network-type  flat  –provider-physical-network default
    • openstack network show net1
    • openstack network delete net1
  • subnet:
    • openstack subnet list
    • openstack subnet pool list
    • openstack subnet create –network net1 —subnet-range 10.0.0.0/24  net1-subnet
    • openstack subnet show net1-subnet
    • openstack subnet delete net1-subnet
  • router:
    • openstack router list  [–long]
    • openstack router create router1
    • openstack router show router1
    • openstack router add|remove subnet  router1  net1-subnet
    • openstack router add|remove port  router1  port1
    • neutron router-gateway-set router1 public  (set External Network/Gateway)
      • openstack port list  –router router1      (Verify router interfaces)
    • openstack router delete router1
  • floating ip (Public IP):
    • openstack floating ip list
    • openstack floating ip pool list
    • openstack floating ip create public
    • openstack floating ip show  <floating-ip>
    • openstack floating ip delete <floating-ip>
    • /etc/nova/nova.confauto_assign_floating_ip=true
    • P.S. To assign a floating ip the instances’s private network must be connected with a router and that router must be connected with a External network via Set Gateway.
  • port (Private/Fixed IP):
    • openstack port list  [–router router1]   (Shows router interfaces)
    • openstack port create –network net1  port1
    • openstack port show port1
    • openstack port delete port2
  • security group:
    • openstack security group list | grep <project_id>
    • openstack security group create sg1  [–project prj1]
    • openstack security group show sg1
    • openstack security group delete sg1
    • nova help [secgroup-create|secgroup-list|secgroup-delete]
    • security group rule:
      • openstack security group rule list  sg1  [–long]
      • openstack security group rule create –protocol icmp  sg1
      • openstack security group rule create –dst-port 22 sg1 [–src-ip 0.0.0.0/0] 
      • openstack security group rule show <Rule_id>
      • openstack security group rule delete <Rule_id>
      • nova help [secgroup-add-rule|secgroup-list-rules|secgroup-delete-rule]
  • Load balancer:
    • neutron help | grep lb-
    • lb-pool:
      • neutron lb-pool-create  –lb-method ROUND_ROBIN  –protocol HTTP \
        –subnet-id  <subnet-id> –name lb1
      • neutron lb-pool-list
      • neutron lb-pool-stats lb1
      • neutron lb-pool-delete lb1
    • lb-member:
      • neutron lb-member-create –address 172.24.4.8 –protocol-port 80  lb1
      • neutron lb-member-create –address 172.24.4.9 –protocol-port 80  lb1
      • neutron lb-member-list
    • lb-vip:
      • neutron lb-vip-create –protocol-port 80 –protocol HTTP \
        –subnet-id <subnetid> –name lb1vip  lb1
      • neutron lb-vip-list
      • neutron lb-vip-delete lb1vip
    • lb-healthmember:
      • neutron lb-healthmonitor-create –type HTTP \
        –delay –max-retries 3 –timeout 3
      • neutron lb-healthmonitor-associate <healthmonitor–id> lb1
      • neutron lb-healthmonitor-list
      • neutron lb-healthmonitor-disassociate <healthmonitor-id> lb1
      • neutron lb-healthmonitor-delete <healthmonitor-id>
    • Setup instances/servers for load balancer:
      • openstack server create –image cirros –flavor m1.tiny  vm1
      • openstack server create –image cirros –flavor m1.tiny  vm2
      • openstack floating ip create public
      • openstack floating ip create public
      • openstack server add floating vm1  172.24.4.8
      • openstack server add floating vm2  172.24.4.9
      • openstack security group rule create –dst-port 80 default
      • openstack security group rule create –dst-port 22 default
      • ssh cirros@172.24.4.8
        • nohup sudo nc -lk -p 80  -e echo -ne \
          “HTTP/1.0 200 OK\r\n Content-Length: 5\r\n\r\nvm1\r\n” &
        • curl http://172.24.4.8  -> vm1
      • ssh cirros@172.24.4.9
        • nohup sudo nc -lk -p 80 -e echo -ne \
          “HTTP/1.0 200 OK\r\n Content-Length: 5\r\n\r\nvm2\r\n” &
        • curl http://172.24.4.9  -> vm2
      • curl http://&lt;mylbvip-ip> | http://
  • neutron (client):
    • neutron help | grep gateway
    • neutron help [net-create|net-list|net-show|net-delete]
    • neutron help [subnet-create|subnet-list|subnet-show|subnet-delete]
    • neutron help [router-create|router-interface-add|router-gateway-set]
    • neutron help [security-group-create|security-group-list|show|delete]
    • neutron help [security-group-rule-create|security-group-rule-list|show|delete]
    • neutron help [floatingip-create|floatingip-associate|
    • neutron net-create ext-net –tenant-id <prj-id> –shared –router:external True \
      –provider:network_type flat –provider:physical_network default  (admin role)
  • Logs:
    • /var/log/neutron/*.log
    • /var/lib/neutron/*

(iv) OpenStack Compute (Nova):

  • keypair:
    • openstack keypair list
    • openstack keypair create keypair1  > ~/.ssh/keypair1.pem
      • chmod 600 ~/.ssh/keypair1.pem
    • openstack keypair show keypair1   [–public-key]
    • openstack keypair delete keypair1
    • ssh-keygen -qf  ~/.ssh/key
      • openstack keypair create –public-key ~/.ssh/key.pub  keypair1
    • nova help [keypair-list|keypair-add|keypair-show|keypair-delete]
  • flavor:   (Only admin user can create it and by default are public)
    • openstack flavor list  [–all |–long|–public|–private]
    • openstack flavor create –ram 512 –disk 1 –vcpus 1 –id 7  m1.demo
    • openstack flavor show m1.demo
    • openstack flavor set –private –project prj1  m1.demo
    • openstack flavor delete m1.demo
    • nova help [flavor-list|flavor-create|flavor-show|flavor-delete]
  • server (instance/VM):
    • openstack server list  [–all-projects|–project prj1]
    • openstack server create  vm1 –image cirros –flavor m1.tiny \
      –key-name key1 –security-group default –nic net-id=private,v4-fixed-ip=10.0.0.9
    • openstack server show vm1
    • openstack server delete vm1
    • openstack server add|remove floating ip  vm1  172.168.4.23
    • openstack server add|remove volume  vm1  vol1  [–device /dev/xvdb]
    • openstack server image create –name vm1_snapshot  vm1 (glance image-list)
    • openstack server backup create –name vm1_backup  vm1  (glance image-list)
    • openstack server stop|start|pause|unpause|suspend|resume  vm1
    • Connect to instance:
      • openstack server ssh –login cirros  vm1  [–public|–private]
      • ssh -i .ssh/keypair.pem  cirros@172.24.4.3     (Connect over Public IP)
      • ip netns list                                                          (Connect over Private IP)
      • ip netns exec qrouter-<routerid> ssh -i keypair.pem  cirros@10.0.0.3
  • quota:
    • openstack quota show prj1 [–default]
    • openstack quota set –cores 5 prj1
    • nova help [quota-show|quota-update|quota-defaults|quota-delete]
  • Policies:
    • ls -l /etc/*/policy.json:
      • /etc/cinder/policy.json
      • /etc/glance/policy.json
      • /etc/nova/policy.json
      • /etc/neutron/policy.json
      • /etc/keystone/policy.json
    • /etc/cinder/policy.json:
      • “volume:create”: “rule:admin”   (Limits volume creation to admin)
      • systemctl restart devstack@c-api    (Restart cinder API service)
    • /etc/glance/policy.json:
      • “add_image”: “rule:admin”   (Limits image creation to admin)
      • Restart glance API service:
        • su – stack; screen -ls; script /dev/null; screen -r stack
        • ctrl+a “-> g-api -> ctrl+c -> UP-Arrow+Enter -> ctrl+a d -> exit
  • nova (client):
    • nova help |grep flavor
    • nova help [list|boot|quota-*|keypair-*|flavor-*|
      floating-ip-associate|volume-attach
      ]
    • nova [hypervisor-list|host-list|service-list|usage-list|version-list]
    • nova boot vm1 –image cirros –flavor m1.tiny \
      –key-name keypair1 –security-groups default –nic net-id=<network-id>
  • Logs:
    • /var/log/nova/*.log
    • /var/lib/nova/instances/<instance-id>/        (Ephemeral disk)
    • /var/lib/nova/instances/snapshot/                (Snapshots location)

(v) OpenStack Image Service (Glance)(Default visibility is private)

  • openstack image list  [–long]   (List OS images and server snapshots/backups)
  • openstack image create –file cirros.img  cirros  [–disk-format qcow2]
  • openstack image show cirros
  • openstack image set –min-ram 512  –public  cirros
  • openstack image save –file cirros-v2.img   cirros
  • openstack image delete cirros
  • openstack image add|remove project cirros  prj2   (Add private image into prj)
  • wget https://10.9.7.8/cirros.img               (Download an image)
  • glance (client):
    • glance help [image-list|image-create|image-show|image-delete]
  • Logs:
    • /var/log/glance/*.log
    • /var/lib/glance/images/<image-id>/       (Image store location)

(vi) OpenStack Block Storage (Cinder):

  • volume:
    • openstack volume service list
    • openstack volume type list
    • openstack volume list  [–all-projects]
    • openstack volume create –size 1  vol1
    • openstack volume set –size 2 vol1 
    • openstack volume show vol1
    • openstack volume delete vol1
    • backup:
  • snapshot:
    • openstack snapshot list
    • openstack snapshot create –name vol1_snapshot  vol1
    • openstack snapshot show  vol1_snapshot
    • openstack snapshot delete  vol1_snapshot
  • Encrypt volume:
    • openstack volume type create LUKS
    • cinder encryption-type-create  –cipher aes-xts-plain64  –key_size 256 \
      –control_location front-end  LUKS nova.volume.encryptors.luks.LuksEncryptor
    • openstack volume create  –type LUKS  –size 1  crypt-vol
    • openstack volume show crypt-vol
    • openstack server volume add vm1 crypt-vol
  • cinder (client):
    • cinder help [list|create|snapshot-list|backup-list|type-list|version-list]
  • Logs:
    • /var/log/cinder/*.log
    • /var/lib/cinder/*

(vii) OpenStack Object Storage (Swift):

  • container:
    • openstack container list  [–long|-all]
    • openstack container create con1
    • openstack container show cont
    • openstack container delete con1
  • object:
    • openstack object list con1
    • openstack object store account show
    • openstack object create con1  file.txt    (echo “File” > file.txt)
    • openstack object show con1  file.txt
    • openstack object save con1  file.txt  –file file2.txt
    • openstack object delete cont file.txt
    • swift (client):
      • swift [list|post|stat|upload|download|delete]  –help
      • swift post con1   [-m Color:Red]                (Creates container)
      • swift upload con1  file.jpg   [–object-name pic]  [–segment-size]
      • swift download con1 file.jpg -o file2.jpg
      • swift delete con1  file.jpg
      • swift stat con1 |swift stat con1  file.jpg
      • swift list [con1]
      • Set ACLs (Containers):
        • swift post con1 -r  .r:*,.rlistings                 (Read and List access for all) 
        • swift post con1 -r  .r:*,.r:-example.com   (Deny access for a domain)
        • swift post con1 -w|-r prj1:*   (Write/Read access for prj1 members)
        • swift stat cont | openstack container show con1
      • Set Expiry (Objects):
        • swift post con1 file.txt -H “X-Delete-At:1516”  (date -d 1/1/2018 +%s)
        • swift post con1  file.txt  -H “X-Remove-Delete-At:”     (Removes date)
        • swift post con1  file.txt  -H “X-Delete-After:100”  (Delete after 100s)
        • swift post con1  file.txt  -H “X-Remove-Delete-After:” (Removes secs)
        • swift stat cont file.txt
      • Allow web access:
        • swift post -m ‘web-listings: true con1
        • swift post -m ‘web-listings-css:listings.css’ con1
  • Logs:
    • /var/log/swift/swift.log
    • /var/lib/swift/*

(viii) OpenStack Orchestration (Heat):

  • openstack stack list  [–long|–all-projects]
  • openstack stack create  -t hello_world.yaml   stack1  \
    –parameter image=cirros –parameter flavor=m1.tiny –parameter vm-name=vm1 \
    –parameter net=public –parameter volume=<vol-id> 
  • openstack stack show stack1
  • openstack stack delete stack1
  • openstack stack update -t stack1.yaml   stack1
  • openstack stack resource list stack1
  • openstack stack event list stack1
  • output:
    • openstack output list stack1 | heat output-list stack1
    • openstack output show stack1 private_ip [-all]
  • snapshot:
    • openstack stack snapshot list stack1 | heat snapshot-list stack1
    • openstack stack snapshot create stack1  [–name stack1-snap]
    • openstack stack snapshot show stack1
    • openstack stack snapshot restore stack1  <snaphost_id>
    • openstack stack snapshot delete stack1  <snaphost_id>
  • heat (client):
    • heat help [service-list|stack-list|stack-create|stack-show|stack-delete]
  • Template (hello_world.yaml):
    heat_template_version: 2018-01-01
    description: Example hello_world.yaml template to deploy a single instance
    resources:
    server:
    type: OS::Nova::Server
    properties:
    image: cirros
    flavor: m1.tiny
    networks:
    – network: 8f3a3cbe-c68d-442e-af6a-d347757c38a5
  • P.S. The private network referenced in the template must be Shared.

(ix) OpenStack Ceilometer (Telemetry):

  • ceilometer (client):
    • ceilometer help | grep alarm
    • ceilometer meter-list | less
    • ceilometer sample-list -m memory| head -10
    • ceilometer statistics -m memory
    • ceilometer meter-list -q resource-id=<server-id>
    • ceilometer sample-list -m cpu -q resource-id=<server-id>
    • ceilometer statistics -m cpu_util -p 60 -q resource-id=<server-id>
    • alarm:
      • ceilometer alarm-threshold-create –name cpu-alarm \
        –description “cpu alarm high” -m cpu_util –statistic avg –period 60 \
        –evaluation-periods 3 –comparison-operator gt –threshold 50.0 \
        –alarm-action ‘log://’ -q resource_id=<server_id>
      • ceilometer alarm-list
      • ceilometer alarm-update –enable false <alarm-id>
      • ceilometer alarm-history <alarm-id>
      • ceilometer alarm-delete <alarm-id>

(ix) Credential file(credrc|demo-openrc|keystone_admin)

  • Sample credential file myproject-openrc or keystonerc_myproject:
    export OS_AUTH_URL=http://127.0.0.1:5000/v2.0  (http://127.0.0.1:5000/identity)
    export OS_TENANT_NAME=”myproject
    export OS_USERNAME=”admin
    export OS_PASSWORD=”openstack”
  • P.S. Make sure admin user is assigned either member or admin role in myproject.

(x) Mariadb (CentOS) / Mysql(Ubuntu):

  • systemctl status|start|stop  mariadb|mysql
  • mysql -u root [-p password]; quit
  • show databases; show tables; use mysql|nova|neutron|glance
  • select user, host, password from user;

 

 

Posted in cloud, coa, openstack

OpenStack common issues and solutions

  1. /var/log/syslog-> ERROR cinder.service [-] Manager for service cinder-volume devstack@lvmdriver-1 is reporting problems, not sending heartbeat. Service will appear “down”.
    • Soluton:
      1. sudo losetup -f /opt/stack/data/stack-volumes-default-backing-file
      2. sudo losetup -f /opt/stack/data/stack-volumes-lvmdriver-1-backing-file
      3. ip addr add 172.24.4.1/24 dev br-ex
      4. sudo route add -net 10.0.0.0/24 gw 172.24.4.1
      5. sudo systemctl restart devstack@c-vol
    • For boot persistence add the above two lines in /etc/rc.local
      sudo losetup -f /opt/stack/data/stack-volumes-default-backing-file
      sudo losetup -f /opt/stack/data/stack-volumes-lvmdriver-1-backing-file
    • And the floating ip address for br-ex in /etc/network/interfaces
      auto br-ex
      iface br-ex inet static
      address 172.24.4.1
      netmask 255.255.255.0
  2. ERROR: Can’t connect on private IP address of a server (10.0.0.3) from OpenStack host:
    1. Solution:
      • Use any of the following commands to find out the public ip address of the router on public network for demo project:
        • openstack router show | grep ip
        • openstack port list –device-owner network:router_gateway  (admin)
        • ip netns exec qrouter-<router-id> ip a | grep qg
      • Then, add the route for guest server private network where gateway would be the router ip:
        • ip route add 10.0.0.0/24 via 172.24.4.231 dev br-ex
      • P.S. Make sure icmp and ssh are added into security group rule in the vm.
  3. ERROR: The connection is refused because git is blocked by firewall:
    • SOLUTION: Run the following cmd to force git to use https://
      1. git  config  –global url.”https://&#8221;.insteadOf git://
    • P.S. This error is thrown during devstack/stack.sh command when VM is behind the firewall.
  4. Task: Move an Instance/VM from project A to B:
    1. Create a snapshot/image of the instance in project A:
      openstack server image create –name vm1-snapshot  vm1
    2. Set snapshot/image visibility to Public with admin user:
      openstack image set –public vm1-snapshot
    3. Create an instance/VM from the snapshot in project B:
      openstack server create –image vm1-snapshot –flavor m1.tiny  vm2-clone
Posted in cloud, coa, openstack

Python basics

(i) Install and configure git:

  1. yum install git && vim-enhanced
  2. git config –global user.name “User”
  3. git config –global user.email “user@python.local”
  4. cur https://raw.githubusercontent.com/linuxacademy/content-python-for-sys-admins/master/helpers/bashrc -o ~/.bashrc
  5. cur https://raw.githubusercontent.com/linuxacademy/content-python-for-sys-admins/master/helpers/vimrc -o ~/.vimrc
  6. mkdir sample; touch sample.txt; cd sample
  7. git init
  8. git status
  9. git add –all .
  10. git commit -m ‘Initial commit’

(ii) Data types:

  • Comments:
    • # This is a comment
    • print(“Hello World”)    #This is also  a comment
  • Strings:
    • str1 = “Hi”
    • str2 =’Hi’
    • “pass” + “word”  -> ‘password’
    • “Ha” * 3 -> ‘HaHaHa’
  • Numbers:
    • 1 + 2  -> 3, 2.2 + 2 -> 4.2, int(“1”), float(“1.1”)
  • Booleans:
    • True; False
  • Variables:
    • my_int = 0, my_int = ” int”, print(my_int)
  • None:
  • List and Tuples:
    • Lists:
      • list = [1, “str”, 1.2, ‘g’, False]; print (list)
      • list[0] -> 1; list[3] -> ‘g’; list[-1] -> False
      • list[0:3] = list[:3] -> [1, ‘str’, 1.2]
      • list[2:] -> [1.2, ‘g’, False]
      • list[::2]-> len[1, 1.2, False]   (Steps two, i.e indexes 0, 2, 4)
      • list[1::2] -> [‘str’, ‘g’]              (Starts at index 1 and steps 2)
      • list[1] = ‘abc’; print(list) -> [1, ‘abc’, 1.2, ‘g’, False]
      • list[1, 3] = [‘a’, ‘c’]
      • list[1,4] = []
      • list += [1, 3] ->print(list) -> [1, ‘abc’, 1.2, ‘g’, False, True, 1, 3
      • list.append(True); print(list) -> [1, ‘abc’, 1.2, ‘g’, False, True]
      • list.remove(“str”)
      • list.pop() -> False; list.pop(0) -> 1
      • len(list) -> 5
    • Tuples:
      • point = (2.0, 3.0)
      • point(0) -> 2.0
  • Dictionaries:
    • ages = { ‘Ishtiaq’ : 33, ‘Ali’: 35}
    • print(ages) -> {‘Ishtiaq’: 33, ‘Ali’: 35}
    • ages[‘Ishtiaq’] = 39
    • ages.values() -> [33, 35]
    • ages.keys() -> [‘Ishtiaq’, ‘Ali’]
    • ages.items() -> [(‘Ishtiaq’, 33), (‘Ali’, 35)]
    • ages.pop(‘Ali’) -> 35; ages-> {‘Ishtiaq’: 33}

(iii) Control Flow – Conditionals, Comparisons and Logic Operations:

  • if, elif, else:            (==, !=, >, >=, <, <=)
    • 2 > 1 -> True; 2 < 1 -> False; 2 ==2.0-> True; 1 != 1.0 -> False
    • 2 in [1, 2, 3] -> True
    • num = 0;
      if num: print(“Number is not zero %s ” %num)  (If num NOT zero)
      else:  print(“Number is zero %s ” %num)  (This would be executed)
    • str = “‘
      if str: print(“String is not empty: %s ” %str)  (If str NOT empty)
      else:  print(“String is empty %s ” %str)  (This would be executed)
    • num = 3
      if num % 2 == 0:
      print(“Number is even.”)
      elif num % 2 != 0:
      print(“Num is odd”)
      else:
      print(“Invalid number”)
  • and operator:
    1. 1 and 2 -> 2
    2. 0 and 1 -> 0
    3. 1 and 1 -> 1
  • or operator:
    1. 1 or 2 -> 1
    2. 1 or 1 -> 1
    3. 0 or 1 -> 1

(iv) Control Flow – Loops:

  • for loop:
    1. for i in range(3):         (By default starts at 0 -> range(0,3))
      print(“%s ” %i)            (Output-> 0, 1, 2)
    2. colors = {‘blue’, ‘green’, ‘red’}
      for color in colors:
      print(color)            (Insert 4 white-spaces here)
    3. point = (1, 3.2, 4)
      for p in point:
      print(p)                    (Insert 4 white-spaces here)
    4. ages = {‘David’: 33, ‘Alisa’: 35}       (Dictionary)
      for key in ages:
      print(key)               (Output: David  Alisa)
    5. for key, value in ages.items():
      print(key)                (Output-> David 33 Alisa 35)
      print(value)
    6. numbers = [1, 2, 3, 4]
      print([x * x for x in numbers])   (Output-> 1, 4, 9, 16)
  • while loop:
    1. count = 0
      while (count < 10):
      if count%2 == 0:
      count += 1
      continue
      print(“Counting odd numbers %s ” %count)
      count += 1

(v) Scripts – Reading User Input:

  1. Execute normal scripts:
    1. cat > hello.py <<EOF
      #! /usr/bin/env python           (Can also use: #!/bin/python)

      name = raw_input(“What’s your name? “)    (Input strings)
      age = input(“What’s your age? “)                     (Input numbers)
      print(“Your name is ‘%s’ and age is ‘%s'” % (name, age))
      EOF
    2. python hello.py
  2. Executes scripts from ~/bin directory:
    1. chmod +x hello.py
    2. mkdir bin
    3. cp hello.py bin/hello
    4. export PATH=$HOME/bin:$PATH
    5. hello
  3. User defined functions:
    #!/bin/python
    def calc_fact(num):
    fact = 1
    while num > 1:
    fact *= num
    num -= 1
    return fact
    num = input(“Enter a number: “)
    print(“%s factorial is: %s” % (num, calc_fact(num)))
  4. Standard library ~/bin/timer
    #!/bin/python
    import time
    start_time = time.localtime()
    print(“Timer starts at %s ” % time.strftime(“%X”, start_time))
    raw_input(“Please press Enter to continue…”)
    stop_time = time.localtime()
    difference = time.mktime(stop_time) – time.mktime(start_time)
    print(“Timer stops at %s ” % time.strftime(“%X”, stop_time))
    print(“Total time %s seconds: ” % difference)

Python Standard Library:

  • Built-in Functions:
    • exit()            (Exits from python editor)
    • print():
      • print(“Name=%s” %name)
      • print(“Name=%s \t Age=%s” % (name, age))
    • bool():
      • bool(1) -> True, bool(0) -> False, bool(None) -> False, bool(“test”) -> True
    • len():
      • len(“Hi”)   -> 2, var = “Hi”; len(var)    -> 2
    • lower():
      •  “Hi”.lower()   ->  hi,  var = “Hi”; var.lower()    ->  hi
    • find():
      • “double”.find(‘s’)     -> -1    (Couldn’t find)
      • “double”.find(“bl”)  -> 3     (Indexes starts at 0, so bl=3)
    • strip()             (Strips white-spaces)
    • str.startswith(‘ch’)
    • append():
    • list.append(‘abc’)
    • remove(): list.remove(‘a’)
    • pop():
      • list.pop(), dic.pop(), list.pop(list_index), ages.pop(dic_key)
    • del():
      • del list|dic
    • items():
      • ages = {‘David’: 33, ‘Alisa’: 35}    (Dictionary)
      • ages.items()     ( Output-> [(‘David’, 33), (‘Alisa’, 35)] )
    • str():
      • str(1) -> ‘1’, str(False) -> ‘False’
    • int(“1”-> 1
    • float(“1.0”-> 1.0
    • bool():
      • bool(“1”) | bool(1) -> True, bool(“0”) | bool(0) | bool (“”) -> True
    • time:
      • localtime(), strftime(), mktime()

(vi) Files:

  • open/close:
    • file = open(‘/home/user/file.txt’)
    • file = open(‘/home/user/file.txt’, ‘r+’)
    • file = open(‘/home/user/file.txt’, ‘w’)
    • file = open(‘/home/user/file.txt’, ‘a’)
    • file.close()
  • read:
    • for line in file: print(line)
    • print(file.read())
  • write/seek:
    • file.seek(-1, 2)
    • file.write(“Hello\n”)
    • file.writelines([‘Hello\n’, ‘World\n’])
    • with open(‘/home/user/file.txt’, ‘a’) as file:
      file.write(“Hello World\n”)

Passing environment variables and command line parameters:

  • Environment variables:
    • pi.python:
      #!/usr/bin/env python
      from os import getenv
      from math import pi
      digits = getenv(“DIGITS”) or 10
      print(“%.*f” % (int(digits), pi))
    • DIGITS=5 python pi.python -> 3.14159
  • Command line argument and Handling Errors with try/except/else/finally:
    • #!/bin/python
      from sys import argv
      for arg in argv: print arg        (python test.py abc 123 -> abc 123)
    • #!/bin/python
      import argparse
      parser = argparse.ArgumentParser(description=’Read a file in reverse’)
      parser.add_argument(‘filename’, help=’the files to read’)
      parser.add_argument(‘–limit’, ‘-l’, type=int, help=’the number of lines to read’)
      parser.add_argument(‘–version’, ‘-v’, action=’version’, version=’%(prog)s 1.0′)
      args = parser.parse_args()
      try:
      f = open(args.filename)
      except IOError as err:
      print(“\n”File not found!\n”)
      else:
      with f:
      limit = args.limit
      lines = f.readlines()
      lines.reverse()
      if limit:
      lines = lines[:limit]
      for line in lines:
      print(line.strip()[::-1]
      finally:
      printf”\nWe are finished.\n”)

Execute Shell Command from Python:

#!/bin/bash
import subprocess
output = subprocess.check_output(‘ls -a’, stderr=subprocess.STDOUT)
code = subprocess.call([‘ls’, ‘-l’])      (If successful code=0)
print(output )
print(code)

Exit Statuses:

#!/bin/python
import sys
print(“Hello”)
sys.exit(1)            #echo $?-> 1

Iteration with List Comprehension:

#!/bin/python
import argparse

parser = argparse.ArgumentParser(description=’Search for words’)
parser.add_argument(‘snippet’, help=’Search for a word in /usr/share/dict/words file’)

args = parser.parse_args()
snippet = args.snippet.lower()

words = open(‘/usr/share/dict/words’).readlines()
print([word.strip() for word in words if snippet in word.lower()])

Python Package Manager: (pip replaced easy_install)

  • pip install:
    • sudo yum install epel-release
    • sudo yum update
    • sudo yum upgrade python-setuptools
    • sudo yum install python-pip python-wheel
  • pip commands:
    • pip -V                                                        (List pip version)
    • sudo pip install –upgrade pip             (Update pip)
    • sudo pip install boto3  [–user] [-r requirements.txt]   (Install a package)
    • sudo pip install ./pkg.whl                                   (Installs a wheel package locally)
    • sudo pip install http://example.com/pkg.whl  (Installs package from a URL)
    • pip list [–format=columns]  [–user]                (List installed pip packages)
    • pip freeze | grep boto3 > requirements.txt    (List pip packages)
      • pip install boto 3 –user                (Installs in /home/user/.local/)
      • python -> import boto3 -> boto3
        <module ‘boto3’ from ‘/home/user/.local/lib/python2.7/site-packages/boto3/*’>
      • python -> import time -> time
        <module ‘time’ from ‘/usr/lib64/python2.7/lib-dynload/timemodule.so’>
    • sudo pip uninstall boto3  [-r requriements.txt]    (Uninstall packages)

Virtualenv:
Virtualenvs are Pythons way of sandboxing dependencies to work with multiple versions of Python packages.

  • pip install virtualenv  [–user]      (which virtualenv | virtualenv –version)
  • mkdir venvs
  • virtualenv  venvs/experiment   [–python=python3.6]
  • source venvs/experiment/bin/activate   (Activate virutalenv)
  • pip list
  • pip install flask     (Install a package in vritualenv and don’t need root permissions)
  • deactivate                                                   (Deactivate virutalenv)
  • Remove virtual environment:
    • deactivate
    • rm -rf venvs
    • pip uninstall virtualenv

requests library:
Its a popular library used for utilizing the third-party packages in your own scripts.

  • source venvs/experiment/bin/activate
  • pip install requests     (Verify-> pip list|freeze)
  • bin/weather:   (chmod +x bin/weather)
    #!/usr/bin/env python
    import requests
    print(requests.__file__)
  • weather:
    /home/user/venvs/experiment/lib/python2.7/site-packages/requests/__init__.pyc
  • deactivate && weather:
    /usr/lib/python2.7/site-packages/requests/__init__.pyc

PostgreSQL:

Structuring python project:

  • pip install pipenv –user
  • mkdir -p ~/code/pgbackup
  • pipenv install –two
  • pipenv shell
  • Create a in ~/code/pgbackup/README.rst
  • mkdir -p src/pgbackup tests docs
  • touch src/pgbackup/__init__.py  docs/.keep  tests/.keep
  • Create ~/code/pgbackup/setup.py
  • git init
  • curl https://raw.githubusercontent.com/github/gitignore/master/Python.gitignore \
    -o .gitignore
  • git add -all .       (git status)
  • git commit -m ‘Initial commit’
  • Create ~/code/pgbackup/tests/test_cli.py
  • pipenv install –dev pytest
  • make
  • create ~/code/pgbackup/src/pgbackup/cli.py

Build and Install Python 3 from source on CentOS7:

  • sudo su –
  • Install Development Tools and some extra libraries:
    • yum groupinstall “development tools” -y
    • yum install -y libffi-devel zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel expat-devel wget
  • Download Python 3 source package and extract it:
    • cd /usr/src
    • wget https://www.python.org/ftp/python/3.6.4/Python-3.6.4.tar.xz
    • tar xvf Python-3.6.4.tar.xz
  • Build and Install Python 3:
    • cd /usr/src/Python-3.6.4
    • ./configure –enable-optimizations
    • make && make altinstall            (Note: altinstall will leave Python 2.7 intact)
    • python3.6
    • quit()

 

Python Built distribution: (wheel replaced egg)

Posted in Other, python

Setup OpenStack (Pike) using Devstack in Ubuntu 16.04

Install OpenStack (Pike) release using Devstack:

  1. Install and configure Ubuntu 16.04 in VMware and set interface to Bridge:
    • Add the following into /etc/network/interfaces
      auto eth0
      iface eth0 inet static
      address 192.168.9.6
      netmask 255.255.255.0
      #gateway 192.168.9.1
      auto eth1
      iface eth1 inet dhcp
    • sudo systemctl restart networking
    • P.S. There are two network adapter added into VM, eth0 (Host-only) to allow access from guest and eth1 (NAT) to allow access to Internet.
  2. Update and Install required packages:
    1. sudo apt-get update
    2. sudo apt-get install git vim
  3. Setup a user:
    1. sudo useradd -s /bin/bash -d /opt/stack -m stack
    2. echo “stack ALL=(ALL) NOPASSWD: ALL” | sudo tee /etc/sudoers
    3. echo stack | sudo passwd –stdin stack
    4. su – stack
  4. git clone https://git.openstack.org/openstack-dev/devstack -b stable/pike
  5. Add the following into by using “vim ~/devstack/local.conf”
    [[local|localrc]]
    HOST_IP=192.168.9.6
    FLAT_INTERFACE=eth0
    ADMIN_PASSWORD=openstack

    DATABASE_PASSWORD=$ADMIN_PASSWORD
    RABBIT_PASSWORD=$ADMIN_PASSWORD
    SERVICE_PASSWORD=$ADMIN_PASSWORD
  6. ~/devstack/stack.sh
  7. To uninstall or reinstall run:
    1. ~/devstack/unstack.sh
    2. ~/devstack/clean.sh

Useful commands:

  • Openstack RC files:
    • source /opt/stack/devstack/openrc username [<projectname>]
    • cat /opt/stack/devstack/.localrc.auto | grep -i password
    • Sample basic Openstack RC/openrc file:
      export OS_AUTH_URL=http://127.0.0.1/identity   [http://127.0.0.1:5000/v2.0]
      export OS_TENANT_NAME=”demo
      export OS_USERNAME=”demo
      export OS_PASSWORD=”openstack
  • Logs and samples:
    1. /opt/stack/logs/stack.sh.log
    2. /opt/stack/devstack/samples/local.conf
  • openstack services status:
    • openstack [host|hypervisor|catalog] list
    • openstack [compute service |volume service | network agent] list
  • Connect to instance/VM:
    • ip netns list
    • ip netns exec qrouter-<router-id> ssh cirros@10.0.0.71

Restart services via systemctl:

  • sudo systemctl -a  | grep devstack        (Status for All services)
  • sudo systemctl status devstack@*        (All services status)
  • sudo systemctl restart devstack@*       (Restart all)
  • sudo journalctl -f –unit devstack@*|n-*|q-*|c-*|g-*  (Logs)
  • Nova:
    • sudo systemctl status devstack@n-*
    • devstack@n-cpu.service (nova-compute)
    • devstack@n-sch.service (nova-scheduler)
    • devstack@n-cauth.service (nova-consoleaut)
    • devstack@n-cond-cell1.service  (nova-conductor)
    • devstack@n-super-cond.service (nova-conductor)
    • devstack@n-api-meta.service  (uwsgi)
    • devstack@n-api.service  (uwsgi)
    • devstack@n-novnc.service  (nova-novncproxy)
  • Neutron:
    • sudo systemctl status devstack@q-*
    • devstack@q-svc.service (neutron-server)
    • devstack@q-l3.service  (neutron-l3-agen)
    • devstack@q-dhcp.service (neutron-dhcp-ag)
    • devstack@q-agt.service (neutron-openvsw)
    • devstack@q-meta.service (neutron-metadat)
  • Cinder:
    • sudo systemctl status devstack@c-*
    • devstack@c-vol.service  (cinder-volume)
    • devstack@c-api.service (uwsgi)
    • devstack@c-sch.service (cinder-schedule)
  • Glance:
    • sudo systemctl status devstack@c-*
    • devstack@g-reg.service  (glance-registry)
    • devstack@g-api.service  (uwsgi)
  • sudo systemctl status devstack@keystone.service  (Keystone service)
  • sudo systemctl status devstack@placement-api.service  (uwsgi)
  • sudo systemctl status devstack@dstat.service (dstat.sh)
  • devstack@etcd.service  (etcd)

Restart Services via Screen:

  1. su – stack                          (Connect to stack user)
  2. screen -ls                           (Lists screens)
  3. script /dev/null           (Resolution for Error: ‘Cannot open your terminal ‘/dev/pts/0’)
  4. screen -r stack                  (Attach into the screen)
  5. ctrl+a ”                               (List all services)
  6. g-api                                   (e.g select Glance API service)
  7. ctrl+c                                  (Stop the service)
  8. Click UP-Arrow (key) and press Enter  (Recall and execute previous service Start cmd)
  9. ctr+a d                                (Detach from the screen)
  10. exit
Posted in cloud, coa, openstack