Wiki

This will setup a basic functional assessment platform (a learning "academy") ala Sandbox prior to getting a BGS environment that we can use. As compared to Sandbox, the goal of this activity is rather to install the actual OPNFV components as listed for BGS, under a single node (e.g. a laptop with lots of memory) or multi-node environment.

The procedure below has been used to create a 3-node OPNFV install using Intel NUC i7 nodes with 16GB RAM, 250MB SSD, and 1 TB HDD. The install uses one NUC for the Jumphost, and one each for the controller (OpenStack + ODL) and the compute nodes.

• Includes instructions from http://artifacts.opnfv.org/genesis/foreman/docs/installation-instructions.html
• install Centos 7 x86_64 minimal server: CentOS-7-x86_64-Minimal-1503-01.iso
• during install, add user opnfv
• once active, login to opnfv account from console

ip addr

• note IP address of enp0s25 - will set it in ifcfg-enp0s25 per the notes below

• add opnfv account to sudoers after "root ALL=(ALL) ALL"

su
visudo
opnfv   ALL=(ALL)       ALL

• update

sudo yum -y update

• reboot and select updated kernel so correct kernel headers can be obtained in virtualbox setup

sudo shutdown -r 0

• Select boot option: CentOS Linux (3.10.0-229.14.1.e17.x86_64) 7 (Core)

• configure hostname

sudo vi /etc/hostname
jumphost1.opnfv.org

• Disable NetworkManager

sudo systemctl disable NetworkManager

• configure single NIC as static per IP assigned during install

sudo vi /etc/sysconfig/network-scripts/ifcfg-enp0s25
TYPE="Ethernet"
BOOTPROTO="static"
IPADDR=192.168.1.201
NETMASK=255.255.255.0
GATEWAY=192.168.1.1
NM_CONTROLLED="no"
(rest as-is)

• Restart networking

sudo service network restart

• Edit /etc/resolv.conf and add a nameserver

sudo vi /etc/resolv.conf
8.8.8.8

• Disable selinux:

sudo setenforce 0
sudo sed -i 's/SELINUX=.*/SELINUX=permissive/' /etc/selinux/config

• Disable firewalld:

sudo systemctl stop firewalld
sudo systemctl disable firewalld

• install and start ntp

sudo yum -y install ntp
sudo systemctl start ntpd
date

• verify time is correct

• Set NUCs to PXE boot on wake-on-lan
• https://help.ubuntu.com/community/WakeOnLan
• In BIOS options (F2), Power tab, set "Wake on LAN from S4/S5" to "Power On - PXE Boot" and save (F10)
• Boot NUCs and note MAC addresses, so they can be included in the wakenodes.sh script referenced below
• The "1"and "2" below refer to the 3-node installs ("PODS") which must be in independent networks (e.g. behind distinct internet gateway routers) so they don't conflict. Within each POD the node names and assigned IP addresses etc are the same.
• NUC1 JUMPHOST1 B8:AE:ED:76:FC:WB
• NUC2 OSCONTROLLER1 B8:AE:ED:76:FB:45
• NUC3 OSCOMPUTE1 B8:AE:ED:76:FB:C4
• NUC4 JUMPHOST2 B8:AE:ED:76:A4:E9
• NUC5 OSCONTROLLER2 B8:AE:ED:76:F9:FF
• NUC6 OSCOMPUTE2 B8:AE:ED:76:C5:ED

• install ether-wake if needed to test of wakenodes.sh

sudo yum -y install net-tools

• install git

sudo yum -y install git
cd ~

• clone genesis

git clone https://blsaws@gerrit.opnfv.org/gerrit/genesis

• (for testing, downloaded trozet's patch fork snapshot from https://gerrit.opnfv.org/gerrit/gitweb?p=genesis.git;a=commit;h=756ee8c81cfac9a69e8f67811429e63da9af6480

curl "https://gerrit.opnfv.org/gerrit/gitweb?p=genesis.git;a=snapshot;h=756ee8c81cfac9a69e8f67811429e63da9af6480;sf=tgz" -o genesis-756ee8c.tar.gz
gzip -d genesis-756ee8c.gz
tar -xvf genesis-756ee8c.tar

• modify to clone khaleesi from my fork (trozet⇒blsaws) per the patches below
• note: if you want to use the wakenodes.sh script, you will need to fork my repo and mod wakenodes.sh for your MACs

vi ~/genesis-756ee8c/foreman/ci/bootstrap.sh
if ! git clone -b opnfv https://github.com/blsaws/khaleesi.git; then

• modify ~/genesis-756ee8c/foreman/ci/opnfv_ksgen_settings_no_HA.yml for my specific config

vi ~/genesis-756ee8c/foreman/ci/opnfv_ksgen_settings_no_HA.yml
# Jumphost1:
  # (for compute1:)
    name: oscompute1.{{ domain_name }}
    hostname: oscompute1.{{ domain_name }}
    short_name: oscompute1
    mac_address: "B8:AE:ED:76:FB:C4"
  # (for controller1:)
    name: oscontroller1.{{ domain_name }}
    hostname: oscontroller1.{{ domain_name }}
    short_name: oscontroller1
    mac_address: "B8:AE:ED:76:FB:45"
    private_mac: "B8:AE:ED:76:FB:45"
# Jumphost2:
  # (for compute1:)
    name: oscompute1.{{ domain_name }}
    hostname: oscompute1.{{ domain_name }}
    short_name: oscompute1
    mac_address: "B8:AE:ED:76:C5:ED"
  # (for controller1:)
    name: oscontroller1.{{ domain_name }}
    hostname: oscontroller1.{{ domain_name }}
    short_name: oscontroller1
    mac_address: "B8:AE:ED:76:F9:FF"
    private_mac: "B8:AE:ED:76:F9:FF"

• clone my fork of trozet's khaleesi, updated with patches
• it will later be cloned by the foreman vm, so these patches will be available in the foreman vm

cd /opt
sudo git clone -b opnfv https://github.com/blsaws/khaleesi.git

• during initial debugging, the patches were manually created as below; later they were pulled from the repo fork after being committed
• patch 1: add wakenodes.sh to root of khaleesi repo
  • alternative is to manually power-on the nodes, which will go into PXE-boot mode

sudo vi /opt/khaleesi/wakenodes.sh
# !/bin/bash
yum -y install net-tools
ether-wake B8:AE:ED:76:FB:C4
ether-wake B8:AE:ED:76:FB:45
ether-wake B8:AE:ED:76:F9:FF
ether-wake B8:AE:ED:76:C5:ED

• During initial debugging wakenodes.sh was copied to the shared folder for the foreman vm and later copied to the root of the cloned repo in the foreman vm.
  • (in jumphost) sudo cp /opt/khaleesi/wakenodes.sh /var/opt/opnfv/foreman_vm
  • (after foreman vm is up)
    • su; cd /var/opt/opnfv/foreman_vm; vagrant ssh; cp /vagrant/wakenodes.sh /opt/khaleesi/wakenodes.sh
• patch 2: in main.yml add call to wakenodes.sh script as shown in http://docs.ansible.com/ansible/script_module.html
  • In khaleesi/roles/get_nodes/foreman/tasks/main.yml add call to node wakeup script after block "- name: Provision nodes"

sudo vi /opt/khaleesi/roles/get_nodes/foreman/tasks/main.yml 
- script: /opt/khaleesi/wakenodes.sh

• During initial debugging main.yml was copied to the shared folder for the foreman vm and later copied to the cloned repo in the foreman vm.
  • (in jumphost) sudo cp /opt/khaleesi/roles/get_nodes/foreman/tasks/main.yml /var/opt/opnfv/foreman_vm/main.yml
  • (after foreman vm is up)
    • su; cd /var/opt/opnfv/foreman_vm; vagrant ssh; cp /vagrant/main.yml /opt/khaleesi/roles/get_nodes/foreman/tasks/main.yml
• patch 3: in foreman.py skip IPMI related code in node bringup
• In khaleesi/library/foreman.py exit early to avoid IPMI code
• change "elif ipmi_host is None:" to "if ipmi_host is None:" and add module.exit before i

sudo vi /opt/khaleesi/library/foreman.py
# bryan_att modified to skip IPMI stuff
module.exit_json(changed=True, msg="Rebuilding Node")
# change elif to if so the module.exit is outside the previous if block
if ipmi_host is None:

• During initial debugging foreman.py was copied to the shared folder and then to the foreman VM as above.
  • (in jumphost) sudo cp /opt/khaleesi/library/foreman.py /var/opt/opnfv/foreman_vm/
  • (after foreman vm is up)
    • su; cd /var/opt/opnfv/foreman_vm; vagrant ssh; cp /vagrant/foreman.py /opt/khaleesi/library/foreman.py

• kickoff deploy.sh

cd ~/genesis-756ee8c/foreman/ci/
sudo ./deploy.sh -single_baremetal_nic enp0s25 -base_config /home/opnfv/genesis-756ee8c/foreman/ci/opnfv_ksgen_settings_no_HA.yml

• if errors, before retrying wipe using clean.sh or

su
cd /var/opt/opnfv/foreman_vm
vagrant destroy -f
cd -
rm -rf /var/opt/opnfv
exit

OpenStack running in a VM had touble being reachable from compute hosts outside the main host machine. So dropping back to running OpenStack on the host machine directly, until I figure out how to resolve the connectivity issue (maybe the bridged networking config below would help with that - it's a recent change which was needed to allow the compute hosts running in VMs to connect to OpenStack…).

• installed with SSH server, LAMP, Tomcat Java, and Virtualization as options
  • In the process you will be asked to setup a MYSQL password - remember it for below
• Get the basic server setup
  • sudo apt-get update
  • sudo apt-get upgrade
  • sudo apt-get install ubuntu-desktop
  • sudo apt-get install virtinst
  • sudo apt-get install virt-manager
  • sudo apt-get install hal-info
  • sudo apt-get install qemu-system
  • sudo apt-get install git -y
  • Setup bridged networking for local VMs
    • See https://help.ubuntu.com/community/KVM/Networking#bridgednetworking
    • sudo invoke-rc.d networking stop
    • sudo vi /etc/network/interfaces

    • auto lo
      iface lo inet loopback
      
      auto eth0
      iface eth0 inet manual
      
      auto br0
      iface br0 inet dhcp
               bridge_ports eth0
               bridge_stp off
               bridge_fd 0
               bridge_maxwait 0

  • sudo /etc/init.d/networking restart
  • to be safe, reboot
• Install OpenStack
  • git clone https://git.openstack.org/openstack-dev/devstack
  • cd devstack
  • customize the configurtion file that will used to setup OpemStack
  • cp samples/local.conf .
  • vi local.conf
    • FLAT_INTERFACE=(your ethernet interface)
    • ADMIN_PASSWORD=(something you will remember - you will use this to login to the Horizon UI)
    • MYSQL_PASSWORD=(mysql root password)
    • RABBIT_PASSWORD=(something you will remember)
    • SERVICE_PASSWORD=(something you will remember)
  • ./stack.sh
  • If all goes well, you will get an output of key items at the end of the script e.g.
    • Horizon is now available at http://192.168.1.132/
    • Keystone is serving at http://192.168.1.132:5000/v2.0/
    • Examples on using novaclient command line is in exercise.sh
    • The default users are: admin and demo
    • The password: opnfv
    • This is your host ip: 192.168.1.132
• Setup Compute controller service
  • Make sure compute nodes can connect to the rabbitmq service. After setting up compute hosts, an "invalid credentials" error may be found in the logs: "grep credentials /var/log/rabbitmq/* -B 3"
    • sudo rabbitmqctl change_password guest opnfv
• Setup compute hosts to connect to OpenStack
  • Prerequisite: see Basic Compute Host Install below
    • also see http://docs.openstack.org/icehouse/install-guide/install/apt/content/nova-compute.html
  • sudo apt-get install nova-compute-kvm
  • Customize the nova compute config file
    • sudo vi /etc/nova/nova.conf

    • glance_host = 192.168.1.132
      my_ip = 192.168.122.209
      vnc_enabled = True
      vncserver_listen = 192.168.1.132
      vncserver_proxyclient_address = 192.168.1.132
      novncproxy_base_url = http://192.168.1.132:6080/vnc_auto.html
      rpc_backend = rabbit
      rabbit_host = 192.168.1.132
      rabbit_password = opnfv
      auth_strategy = keystone
      
      [keystone_authtoken]
      auth_uri = http://192.168.1.132:5000
      auth_host = ubuntu-1404-openstack
      auth_port = 35357
      auth_protocol = http
      admin_tenant_name = service
      admin_user = nova
      admin_password = opnfv
      
      [database]
      # The SQLAlchemy connection string used to connect to the database
      connection = mysql://nova:opnfvmysql@192.168.1.132/nova

  • verify that your machine supports hardware acceleration (returns 1)
    • egrep -c '(vmx|svm)' /proc/cpuinfo
    • if it returns 0 see further instructions linked above
  • sudo rm /var/lib/nova/nova.sqlite
  • sudo service nova-compute restart
• In Horizon, verify that your compute hosts show up in the System / Hypervisors view
• After restarting your host machine
  • sh ./devstack/rejoin-stack.sh

• The following assumes you will use Ubuntu 14.04 LTS Server for the VM host
• Create a new Virtual Machine Manager image for the VM
  • If the image is not in qcow2 format already
    • Starting from a vmdk (VMWare image): example for JeOS 14.04 (very minimal Ubuntu image)
      • qemu-img convert ub1404lts-disk1.vmdk -O qcow2 jeos1404.qcow2
    • Starting from ISO: example for Ubuntu 14.04
      • qemu-img create -f qcow2 ubuntu-1404.qcow2 10G
      • virt-install –virt-type kvm –name trusty –ram 1024 –cdrom=ubuntu-14.04-server-amd64.iso –disk ubuntu-1404.qcow2,format=qcow2 –network network=default –graphics vnc,listen=0.0.0.0 –noautoconsole –os-type=linux –os-variant=ubuntutrusty
  • If when you run Virtual Machine Manager, your CM is not aleady there (due to virt-install above)
    • Create new VM and select import existing image. Hit Forward.
    • browse ("local") to the qcow2 file folder
    • select OS type "linux". Hit Forward
    • select RAM (min 1048) and CPUs (leave at 1 is OK). Hit Forward.
    • Select "customize configuration". Hit Finish.
    • Under Disk 1 select advanced options and select Storage Format qcow2. Hit Apply.
    • Hit Begin Installation.
• Run Virtual Machine Manager
• Select the VM you want to initialize, and select Run
• If the VM is from an install disk (ISO), you will get the Ubuntu installer.
  • Customize the options if you want; the below assumes no options selected
• When install is complete, reboot the VM and login
• sudo apt-get update
• sudo apt-get upgrade
• sudo apt-get install openssh-server
  • this will make it easier to manage hosts later, e.g. to enable copy/paste paste
• How to customize the VM with LAMP support
  • sudo apt-get update
  • sudo apt-get install lamp-server^
    • the carat is important for some reason - needs to be in the command…
  • select a mysql password when prompted

• Use Virtual Machine Manager to clone into the desired compute hosts, e.g. I started with ubuntu-1404 and cloned into ubuntu-1404-h1 and ubuntu-1404-h2
• Select "Clone" (not share disk)
• Boot the VMs, and update the configs
  • Hostname (replace ubuntu with ubuntu-1404-h1 or ubuntu-1404-h2 or etc as applicable)
    • sudo hostname ubuntu-1404-h1
    • sudo vi /etc/hostname
    • sudo vi /etc/hosts
      • Add ubuntu-1404-h1 to 127.0.0.1
      • If 127.0.1.1 is there (? why), delete that row

To address some of the potential snags occurring due to conflicts between OpenStack, ODL, OVS, libvirt, … I am switching to running OpenStack and compute nodes under KVM VMs, with the addition of other compute nodes on other machines. At this point ODL is not in the picture as I have to figure out how the whole flat network approach works first. And the goal for this initial sandbox will be to get off the ground with policy feature assessment in OpenStack, so ODL can wait until I learn it better or get guidance on how to factor it into this setup.

At this point I have most of the above working. What is not currently working is getting the external compute node connected to OpenStack nova, likely some firewall issue with KVM that I need to fix.

I'll post details on how this was setup soon.

Returning to a more direct approach, this time will ensure component independence by installing: * OpenStack on host OS * ODL in VM managed by OpenStack

Decided pretty quickly that as some of the earlier issues may have been with running OpenStack on the host directly, I would switch to trying it in a VM (take 4).

Trying out Foreman to see if it helps get past some of the basic issues with the manual install.

• sudo rpm -ivh http://yum.puppetlabs.com/puppetlabs-release-el-7.noarch.rpm
• sudo yum -y install epel-release http://yum.theforeman.org/releases/1.7/el7/x86_64/foreman-release.rpm
• sudo yum -y install foreman-installer
• sudo yum -y install epel-release http://yum.theforeman.org/releases/1.7/el7/x86_64/foreman-release.rpm
• sudo foreman-installer
• setup admin through foreman UI: https://opnfv.bkaj.net/foreman_setup/provisioners
• repeat foreman-installer
• …
• only options seem to be bare metal install. It´s unclear how to install OpenStack in a VM, so had to return to devstack install. But ran into the same issues as found in the original Centos7 effort below (httpd for keystone fails to start due to a configuration error related to a module - no examples of how to resolve this error are found anywhere).

. . .

Here is a graphic of the concept. This is very draft and leaves many things unclear partly because I'm not sure how to do them yet.

This procedure is not being further pursued at the moment… too many undocumented things about the overall setup of OpenStack, ODL, etc…

Restart process for For sandbox based upon host OS: Unbuntu 14.04 Server LTS

• devstack/rejoin-stack.sh
• from Horizon, launch mininet-vm created earlier
• ssh to mininet VM and enter
  • sudo mn –controller=remote,ip=192.168.1.132
• back in ubuntu terminal
• cd distribution-karaf-0.2.1-Helium-SR1.1
• ./bin/karaf clean
• (currently here in the debug process) install enough ODL features to get ODL UX working (ports 8080 and 8181)
• feature:install odl-restconf odl-l2switch-switch odl-mdsal-apidocs odl-dlux-core

Restart process for sandbox based upon host OS: Unbuntu 14.04 Server LTS

1. Ensure you have enable virtualization support in your BIOS
2. Install Ubuntu. I selected options for SSH, LAMP, VM host.
   • Make sure you remember the mysql root password - you will need it below
3. install updates
   • sudo apt-get update
   • sudo apt-get upgrade
4. install ubuntu desktop: sudo apt-get install ubuntu-desktop
   • Qemu requires a window system… something lighter than desktop env wil
5. install Java 7
   • Note that apparently Java 8 is not compatible with Karaf
   • I used the notes from ubuntuhandbook
   • sudo add-apt-repository ppa:webupd8team/java
   • sudo apt-get update
   • sudo apt-get install oracle-java7-installer
   • sudo apt-get install oracle-java7-set-default
   • export JAVA_HOME=/usr/lib/jvm/java-7-oracle
   • sudo vi /etc/environment
     • (add a line: JAVA_HOME=/usr/lib/jvm/java-7-oracle)
6. install git: sudo apt-get install git -y
7. Install OpenStack (DevStack)
   • git clone https://git.openstack.org/openstack-dev/devstack
   • cd devstack
   • Create local.conf: cp samples/local.conf .
   • Add/replace these lines: vi local.conf
     • FLAT_INTERFACE=(your ethernet interface)
     • ADMIN_PASSWORD=(something you will remember - you will use this to login to the Horizon UI)
     • MYSQL_PASSWORD=(mysql root password)
     • RABBIT_PASSWORD=(something you will remember)
     • SERVICE_PASSWORD=(something you will remember)
   • Start the devstack install: ./stack.sh
   • If all goes well, you will get an output of key items at the end of the script
   • If you hit an error related to the mysql root password, you can reset it and try again
     • sudo mysql –defaults-file=/etc/mysql/debian.cnf
     • mysql> UPDATE mysql.user SET Password=PASSWORD('mysql root password') WHERE User='root';
     • mysql> FLUSH PRIVILEGES;
     • mysql> quit;
     • mysql -u root -p
   • At this point you should be able to login to Horizon at your installed server e.g. http://192.168.1.132/ using the account "admin" and the ADMIN_PASSWORD you set above.
8. install and configure mininet
   • download mininet vm: wget http://downloads.mininet.org/mininet-2.2.0-150106-ubuntu-14.04-server-amd64.zip -O mininet.zip
   • convert to QCOW2 format
   • unzip mininet.zip
   • qemu-img convert -O qcow2 mn-2.2.0-trusty64server-150106-01-30-00/mininet-vm-x86_64.vmdk mn-2.2.0-trusty64server-150106-01-30-00/mininet-vm-x86_64.qcow2
   • In Horizon, create image for mininet-vm
     • create from file in qcow2 format
     • min RAM = 1024
9. Install openvswitch: sudo apt-get install openvswitch-switch
10. Install OpenDaylight Controller (see Install Guide)
    • Get the ODL Helium distribution: curl https://nexus.opendaylight.org/content/repositories/public/org/opendaylight/integration/distribution-karaf/0.2.1-Helium-SR1.1/distribution-karaf-0.2.1-Helium-SR1.1.zip -o Helium-SR1.1.zip
    • unzip Helium-SR1.1.zip
    • cd distribution-karaf-0.2.1-Helium-SR1.1
    • ./bin/karaf
    • You should get the OpenDaylight prompt "opendaylight-user@root>"
    • verify it's working with: feature:list
    • install the ODL components
      • feature:install odl-base-all odl-aaa-authn odl-restconf odl-nsf-all odl-adsal-northbound odl-mdsal-apidocs odl-ovsdb-openstack odl-ovsdb-northbound odl-dlux-core
        • start with this minimal set of features per the video ¨ODL & OpenStack Discussion
        • will pull these additional features as I find they are essential
          • feature:install odl-mdsal-all odl-restconf-all odl-ovsdb-all odl-openflowplugin-flow-services-ui odl-sfc-all odl-groupbasedpolicy-ofoverlay odl-dlux-all odl-config-persister-all odl-aaa-all odl-ttp-all odl-openflowplugin-all odl-adsal-compatibility-all odl-tcpmd5-all odl-adsal-all odl-config-all odl-netconf-all odl-base-all odl-yangtools-all odl-integration-compatible-with-all odl-netconf-connector-all odl-akka-all odl-l2switch-switch odl-mdsal-apidocs odl-aaa-authn
          • (can't find a feature called ML2-plugin)
    • At this point you should be able to login to DLUX at your installed server e.g. http://192.168.1.132:8181/dlux/index.html using the account "admin" and the In Hidefault password "admin".
11. In Horizon, launch mininet-vm created earlier
12. ssh to mininet VM and enter
    • sudo mn –controller=remote,ip=192.168.1.132
13. Bring up a VM and validate basic functionality: this will be initially a manual sequence and eventually scripted using OpenStack APIs

Tried CentOS 7 but continually getting errors:

1. Install CentOS 7 "DVD ISO"
2. When installing, select the "Server with GUI" profile, leaving all options blank (it's unclear whether any are needed at this point)
3. As su, add your username to /etc/sudoers via "visudo"
4. Log out of root!
5. Install git: sudo yum install git
6. Download devstack: git clone https://git.openstack.org/openstack-dev/devstack
7. Create devstack/local.conf, adding/replacing these lines
   • FLAT_INTERFACE=(your ethernet interface)
   • ADMIN_PASSWORD=(random string)
   • MYSQL_PASSWORD=(random string)
   • RABBIT_PASSWORD=(random string)
   • SERVICE_PASSWORD=(random string)
8. Start the devstack install: cd devstack; ./stack.sh
9. A variety of errors received - no reason why, not time to debug