Table of Contents

CONFIG FILES

CONFIG FILE TEMPLATES

For Bare Metal and Virtual deployment, template files are available

fuel/deploy/templates/virtual_environment/conf/<dea.yaml,dha.yaml> fuel/deploy/templates/hardware_environment/conf/<ericsson_montreal_lab, linux_foundation_lab, opnfv_box>/<dea.yaml,dha.yaml>

VM definition files

fuel/deploy/templates/virtual_environment/vms/<controller.xml, compute.xml, fuel.xml> fuel/deploy/templates/hardware_environment/vms/fuel.xml

Network config for Virtual deployment

fuel/deploy/templates/virtual_environment/networks/fuel<x>.xml

DEA CONFIG FILES

There are 2 config files wired out from fuel: DEA, DHA

Deployment Environment Adapter: dea.yaml
mode: ha
net_segment_type: gre
wanted_release: Juno on Ubuntu 14.04.1
Node roles
Fuel networks, ips, interfaces, bridge configs
OS components settings
...

DHA CONFIG FILES

There are 2 config files wired out from fuel: DEA, DHA

Deployment Hardware Adapter: dha.yaml
adapter: libvirt
nodes:
- id: 1
  libvirtName: controller1
  libvirtTemplate: templates/virtual_environment/vms/controller.xml
- id: 5
  libvirtName: fuel-master
  libvirtTemplate: templates/virtual_environment/vms/fuel.xml
  isFuel: yes
  username: root
  password: r00tme

  virtNetConfDir: templates/virtual_environment/networks
  disks:
  fuel: 50G
  controller: 50G
  compute: 50G

DEPLOYMENT

AUTO DEPLOYER

fuel/ci/deploy.sh

Typically:

sudo ./deploy.sh -iso opnfv.iso -dea dea.yaml -dha dha.yaml -s /mnt/images -p PLUGIN
-> deploy OPNFV Cloud env with Fuel

-iso	-> OPNFV iso downloaded or built from source
-p	-> plugins for FUEL, like ODL puppet files to install ODL on nodes
           * Not ODL itself
           * Default ODL rpm for ARNO SR1 is integrated in OPNFV iso
           * Can be downloaded or built from source if different version required
-s	-> storage location for generated VMs, user defined
-nf	-> no fuel (redeployment)
-nh	-> no health check

PREPARE ARNO ISO

DOWNLOAD ARNO ISO FILE: http://artifacts.opnfv.org/arno.2015.2.0/fuel/arno.2015.2.0.fuel.iso

It contains a Centos FUEL image containing all the requirements for OPNFV deployments (e.g. OpenStack Juno, ODL Lithium, Fuel plugins

USE CORRECT NTP SETTINGS IN DEA.YAML

NTP1: 10.118.34.219
NTP2:
NTP3:
value: 10.118.34.219

Unless using a correct value, deployment will fail - FUEL GUI ERROR:

Deployment has failed. Method granular_deploy. Failed to execute hook 'puppet' Puppet run failed. Check puppet logs for details
---
priority: 200
fail_on_error: true
type: puppet
uids:
- '3'
parameters:
  puppet_modules: "/etc/puppet/modules"
  puppet_manifest: "/etc/puppet/modules/osnailyfacter/modular/ntp/ntp-check.pp"
  timeout: 600
  cwd: "/"
.
Inspect Astute logs for the details

OPTIONAL ODL BUILD STEP

One should either use the ODL integrated in ARNO SR1 (iso downloaded or built manually in previour step already contains it, no manual step required) or build rpm from source, and add it to the deploy script as argument.

BUILD: (readme: https://github.com/openstack/fuel-plugin-opendaylight)

git clone https://github.com/stackforge/fuel-plugin-opendaylight
pip install fuel-plugin-builder
gem install fpm
fpb --build fuel-plugin-opendaylight/

INCLUDE ODL PLUGIN IN DEPLOYMENT (by using argument for deploy.sh on next slide):

-p <fuel-plugin-opendaylight-rmp-dir>

EXCLUDE ODL PLUGIN ENTIRELY If ODL is not required and the plugin deployment is causing the entire deployment to fail then the ODL plugin can be disabled entirely by editing /fuel/deploy/templates/virtual_environment/conf/plugins_conf/opendaylight.yaml

 opendaylight:
   metadata:
     enabled: false      <<<<< disable ODL
     label: OpenDaylight plugin

AUTOMATED DEPLOYMENT

Typical script parameters:

sudo ~/opnfv/fuel/fuel/ci/deploy.sh \
-iso ~/opnfv/fuel/fuel/build/release/opnfv-P0000.iso \
-dea ~/opnfv/fuel/fuel/deploy/templates/virtual_environment/conf/dea.yaml \
-dha ~/opnfv/fuel/fuel/deploy/templates/virtual_environment/conf/dha.yaml \
-s ~/opnfv/fuel/fuel/deploy/images \
-nh # Would fail at Live-Migraition test phase because of a known ODL bug anyway, let’s test it afterwards

Console output:

Fuel Plugins Directory not specified! No external plugins will be installed!
Using OPNFV ISO file: ~/opnfv/fuel/fuel/build/release/opnfv-P0000.iso
Using image directory: ~/opnfv/fuel/fuel/deploy/images
...
<Cleanup, Create Networks, power on VMs>
... 
Install Fuel Master
Insert ISO ~/opnfv/fuel/fuel/deploy/fueltmp/deploy-opnfv-P0000.iso
Inject our own astute.yaml settings
Let the Fuel deployment continue
Installing Fuel Plugins
...

MANUAL CLEANUP

May be useful if deployment script was stopped unexpectedly, and system ends in undefined state

sudo ~/opnfv/fuel/fuel/ci/deploy.sh -co \
-dha ../deploy/templates/virtual_environment/conf/dha.yaml

VERIFICATION

Deploy verification

Verify FUEL deployment on GUI: From Jumphost (Fuel Host):

$ firefox http://10.20.0.2:8000 & (admin/admin)

Check fuel version and plugins:

ssh root@10.20.0.2 (pwd: r00tme)
[root@opnfv_virt ~]# fuel plugins
fuel --versionid | name         | version | package_version
---|--------------|---------|----------------
1  | opendaylight | 0.6.2   | 2.0.0
[root@opnfv_virt ~]# fuel --version
6.1.0

Verify OpenStack GUI: From Jumphost (Fuel Host):

$ firefox http://172.16.0.3/horizon/ & (admin/admin)

Verify OpenDaylight:

Is GUI accessible - Point your browser to the following URL: 172.16.0.3:8181/index.html (admin/admin)

Verify services: SSH to Controller:

root@node-2:~# nova-manage service list
Binary           Host                                 Zone             Status     State Updated_At
nova-consoleauth node-2.opnfvericsson.ca              internal         enabled    :-)   2015-10-22 06:08:20
nova-scheduler   node-2.opnfvericsson.ca              internal         enabled    :-)   2015-10-22 06:08:21
nova-conductor   node-2.opnfvericsson.ca              internal         enabled    :-)   2015-10-22 06:08:21
nova-cert        node-2.opnfvericsson.ca              internal         enabled    :-)   2015-10-22 06:07:43
nova-compute     node-1.opnfvericsson.ca              nova             enabled    :-)   2015-10-22 06:07:29
nova-compute     node-3.opnfvericsson.ca              nova             enabled    :-)   2015-10-22 06:07:36
nova-compute     node-4.opnfvericsson.ca              nova             enabled    :-)   2015-10-22 06:08:11
root@node-2:~# neutron agent-list
+--------------------------------------+----------------+-------------------------+-------+----------------+------------------------+
| id                                   | agent_type     | host                    | alive | admin_state_up | binary                 |
+--------------------------------------+----------------+-------------------------+-------+----------------+------------------------+
| 11709a29-24f8-46a2-b196-8b5545a6e0a5 | Metadata agent | node-2.opnfvericsson.ca | :-)   | True           | neutron-metadata-agent |
| 2cb0a373-c08a-4ea2-a819-13d7bf892f2e | L3 agent       | node-2.opnfvericsson.ca | :-)   | True           | neutron-l3-agent       |
| 42d08a62-83b5-4f12-a24c-8d375cde63f4 | DHCP agent     | node-2.opnfvericsson.ca | :-)   | True           | neutron-dhcp-agent     |
+--------------------------------------+----------------+-------------------------+-------+----------------+------------------------+

CHALLENGES

CHALLENGE #1 - SOLVED

sudo ./deploy.sh  ...
error: Failed to start domain controller1
error: unsupported configuration: guest and host CPU are not compatible: Host CPU does not provide required features: erms, smep, fsgsbase, rdrand, f16c, avx, osxsave, xsave, tsc-deadline, x2apic

I am using Intel(R) Xeon(R) CPU E5620 @ 2.40GHz – Westmere architecture

I had to change arcthitecture in VM definition XMLs:

cd ../deploy/templates/virtual_environment/vms
sed -i 's/SandyBridge/Westmere/g' compute.xml controller.xml fuel.xml
cd -

I had to comment out flags not supported by my CPU in VM definition XMLs:

cd ../deploy/templates/virtual_environment/vms
for flag in erms smep fsgsbase rdrand f16c avx osxsave xsave tsc-deadline x2apic;do
sed -i "/'${flag}'/s/^/<\!-- /" compute.xml controller.xml fuel.xml &&
sed -i "/'${flag}'/s/$/ -->/" compute.xml controller.xml fuel.xml
done
cd –

CHALLENGE #2 - SOLVED

Deployment seems to be pretty error-prone, if you CTRL+C, and see error „OSError: [Errno 38] Function not implemented: '~/opnfv/fuel/fuel/deploy/fueltmp/origiso/boot.catalog‚”, iso is probably still mounted, check it with:

$ ~/opnfv/fuel/fuel/ci$ cat /etc/mtab
fuseiso ~/opnfv/fuel/fuel/deploy/fueltmp/origiso fuse.fuseiso rw,nosuid,nodev 0 0

Unmount it with:

$ ~/opnfv/fuel/fuel/ci$ sudo fusermount -u ~/opnfv/fuel/fuel/deploy/fueltmp/origiso

CHALLENGE #3 - DISAPPEARED AFTER CLEANUP WITH MAKE CLEAN

BUILD fails with build.sh:

Installing Fuel Plugins
Fofakeroot /opt/fuel-createmirror-6.1/fuel-createmirror
 * INFO: Started /opt/fuel-createmirror-6.1/deb-mirror /opt/fuel-createmirror-6.1/config/mos-ubuntu-updatesonly.cfg
 * FATAL: Upstream mirror mirror.fuel-infra.org does not support rsync protocol, aborting
 * INFO: Started /opt/fuel-createmirror-6.1/deb-mirror /opt/fuel-createmirror-6.1/config/ubuntu.cfg

 * FATAL: Upstream mirror archive.ubuntu.com does not support rsync protocol, aborting
 * FATAL: Creation of MOS mirror FAILED, check logs at /var/log/mirror-sync
 * FATAL: Creation of Ubuntu mirror FAILED, check logs at /var/log/mirror-sync
make[2]: Leaving directory `~/opnfv/fuel/fuel/build/f_isoroot/f_repobuild'
make[2]: *** [.nailgun] Error 1
semop(1): encountered an error: Invalid argument
make[1]: *** [f_repobuild] Error 2
make: *** [f_isoroot] Error 2
make[1]: Leaving directory `~/opnfv/fuel/fuel/build/f_isoroot'
make: *** [all] Error 2
Build: make all failed, exiting ...
Exiting ...
und plugin /root/plugins/opendaylight-0.6-0.6.1-1.noarch.rpm, installing ...
Eject ISO file ~/opnfv/fuel/fuel/deploy/fueltmp/deploy-opnfv-P0000.iso
Remove ISO directory ~/opnfv/fuel/fuel/deploy/fueltmp
Deleting directory ~/opnfv/fuel/fuel/deploy/fueltmp
Installation of Fuel Plugin /root/plugins/opendaylight-0.6-0.6.1-1.noarch.rpm failed: Traceback (most recent call last):
...
    plugin_tar = tarfile.open(plugin_path, 'r')
  File "/usr/lib64/python2.6/tarfile.py", line 1657, in open
    raise ReadError("file could not be opened successfully")
tarfile.ReadError: file could not be opened successfully

CHALLENGE #4 - WORKAROUND: REDEPLOY WITH "-nf" FLAG

Trying to SSH into Fuel VM 10.20.0.2 ... sleeping 10 seconds
Wait until Fuel menu is up
Inject our own astute.yaml settings
Modifying Fuel astute
Let the Fuel deployment continue
Found FUEL menu as PID 1292, now killing it
Wait until installation complete
Waiting for one minute for Fuel to stabilize
Delete deprecated fuel client config /etc/fuel/client/config.yaml
Installing Fuel Plugins
Found plugin /root/plugins/opendaylight-0.6-0.6.2-1.noarch.rpm, installing ...
Eject ISO file ~/opnfv/fuel/fuel/deploy/fueltmp/deploy-opnfv-P0000.iso
Remove ISO directory ~/opnfv/fuel/fuel/deploy/fueltmp
Deleting directory ~/opnfv/fuel/fuel/deploy/fueltmp

Installation of Fuel Plugin /root/plugins/opendaylight-0.6-0.6.2-1.noarch.rpm failed: 
http://mirror.fuel-infra.org/mos/centos-6/mos6.1/security/repodata/repomd.xml: [Errno 14] PYCURL ERROR 6 - "Couldn't resolve host 'mirror.fuel-infra.org'"
Trying other mirror.
http://mirror.fuel-infra.org/mos/centos-6/mos6.1/updates/repodata/repomd.xml: [Errno 14] PYCURL ERROR 6 - "Couldn't resolve host 'mirror.fuel-infra.org'"
Trying other mirror.

TRY TO REDEPLOY WITH „-nf” FLAG (NO FUEL): Verification of fuel plugin install looks to be failed but in reality it is successfully installed, verify it:

ssh root@10.20.0.2 (pwd: r00tme)
[root@opnfv_virt ~]# fuel plugins

Fuel will not de redeployed with this flag, so the deployment can be continued.

A fix for this is already under review.

CHALLENGE #5 - HANGS... WORKAROUND: REDEPLOY WITH FUEL (WITHOUT „-nf” FLAG)

~/opnfv/fuel/fuel/ci$ sudo ./deploy.sh -iso /home/egbohal/opnfv/fuel/fuel/build/release/opnfv-P0000.iso -dea /home/egbohal/opnfv/fuel/fuel/deploy/templates/virtual_environment/conf/dea.yaml -dha /home/egbohal/opnfv/fuel/fuel/deploy/templates/virtual_environment/conf/dha.yaml -s /home/egbohal/opnfv/fuel/fuel/deploy/images -nf

Namespace(cleanup=False, cleanup_only=False, dea_file='/home/egbohal/opnfv/fuel/fuel/deploy/templates/virtual_environment/conf/dea.yaml', dha_file='/home/egbohal/opnfv/fuel/fuel/deploy/templates/virtual_environment/conf/dha.yaml', fuel_only=False, fuel_plugins_dir=None, iso_file='/home/egbohal/opnfv/fuel/fuel/build/release/opnfv-P0000.iso', no_fuel=True, no_health_check=False, pxe_bridge='pxebr', storage_dir='/home/egbohal/opnfv/fuel/fuel/deploy/images')

Fuel Plugins Directory not specified! No external plugins will be installed!
Power OFF Node controller1
Power OFF Node compute1
Power OFF Node compute2
Power OFF Node compute3
Set boot order ['network', 'hd'] on Node controller1
Set boot order ['network', 'hd'] on Node compute1
Set boot order ['network', 'hd'] on Node compute2
Set boot order ['network', 'hd'] on Node compute3
Power ON Node controller1
Power ON Node compute1
Power ON Node compute2
Power ON Node compute3
Check prerequisites
Check supported release: Juno on Ubuntu 14.04.1
Check previous installation
Deleting environment 1

CHALLENGE #6

If deployment hangs for a while, you can ssh to fuel and verify which state the deployment currently is to know where to dig in:

[root@opnfv_virt ~]# watch -n 1 fuel task
id | status  | name           | cluster | progress | uuid
---|---------|----------------|---------|----------|-------------------------------------
1  | ready   | check_networks | 1       | 100      | e0734875-e5b9-4b05-9eb5-0737001c0ad0
4  | running | deployment     | 1       | 0        | 119ff982-48f1-40cd-91a5-63f5ce02ae7d
2  | running | deploy         | 1       | 29       | 4a37d617-8752-41b2-b056-afc87dc8d022
3  | ready   | provision      | 1       | 100      | 5ec218aa-3e8d-4dae-aa46-de8fcaef5e90

CHALLENGE #7 - SOLVED

If there is a timeout error relating to recreate-neutron-db in /var/log/docker-logs/astute/astute.log on fuel-master, this means the task for recreating neutron database during ODL plugin deployment is taking too much time and deployment will fail.

Solutions: * Use a later version of the ODL plugin than that packaged with SR1 .iso (by cloning & building the ODL plugin and adding the -p flag to the deploy.sh command line). * Disable ODL plugin as described in a section above.

MANUAL ISO BUILD

PREREQUISITES

SSH KEY

UPLOAD SSH KEYS TO:

FIREWALL RULES

ALLOW FIREWALL for HTTP, HTTPS, SSH, GIT(9418), RSYNC(873) protocols to:

PKG DEPENDENCIES
sudo apt-get install -y libvirt-bin qemu-kvm python-pip fuseiso mkisofs
sudo apt-get install -y python-dev libz-dev libxml2-dev libxslt-dev
sudo pip install pyyaml netaddr paramiko lxml scp pycrypto ecdsa

curl -sSL https://get.docker.com/gpg | sudo apt-key add -
curl -sSL https://get.docker.com/ | sh
sudo usermod -aG docker <user>

logout & login again

BUILD ARNO ISO

git clone https://gerrit.opnfv.org/gerrit/fuel
cd fuel/fuel/build
make all

It will generate a Centos FUEL image containing all the requirements for OPNFV deployments (e.g. OpenStack Juno, ODL Lithium, Fuel plugins):

End of console output:

Making ISO...
...
Product ISO is ~/opnfv/fuel/fuel/build/release/opnfv-P0000.iso

PROPOSED INVESTIGATION TOPICS

OVS PUPPET PLUGIN

https://review.openstack.org/#/c/222066/

From Guo Ruijing ruijing.guo@intel.com

Install OVS from specified URL in ODL plugin

Uses pre_build_hook and puppet

Uploaded Sep 10, 2015 10:17 AM

Not investigated, To Be Checked!

DEVELOPERS PLAYGROUND

Fuel@OPNFV devel pipeline/sandbox

Jonas Bjurel jonas.bjurel@ericsson.com

Devel pipeline homepage: http://jonasbjurel.github.io/OPNFV-Playground/

Wiki: https://github.com/jonasbjurel/OPNFV-Playground/wiki

FUEL PLUGIN OVS

https://github.com/lingyu1/fuel-plugin-ovs

https://github.com/lingyu1/fuel-rpm

REFERENCES

genesis/fuel/ci/README: with examples for BM/VIRTUAL/FUEL_ONLY DEPLOYMENT

genesis/fuel/deploy/README: same as above, more verbose, many template configs

genesis/fuel/build/README: build container and .iso

https://gerrit.opnfv.org/gerrit/fuel

http://artifacts.opnfv.org/arno.2015.2.0/fuel/install-guide.arno.2015.2.0.pdf

Genesis doc: http://artifacts.opnfv.org/genesis/fuel/docs/build-instructions.html

Genesis doc: http://artifacts.opnfv.org/genesis/fuel/docs/installation-instructions.html

Docker doc: https://docs.docker.com/installation/ubuntulinux/