This is an old revision of the document!
Table of Contents
CONFIG FILES
CONFIG FILE TEMPLATES
For Bare Metal and Virtual deployment, template files are available
fuel/deploy/templates/virtual_environment/conf/<dea.yaml,dha.yaml> fuel/deploy/templates/hardware_environment/conf/<ericsson_montreal_lab, linux_foundation_lab, opnfv_box>/<dea.yaml,dha.yaml>
VM definition files
fuel/deploy/templates/virtual_environment/vms/<controller.xml, compute.xml, fuel.xml> fuel/deploy/templates/hardware_environment/vms/fuel.xml
Network config for Virtual deployment
fuel/deploy/templates/virtual_environment/networks/fuel<x>.xml
DEA CONFIG FILES
There are 2 config files wired out from fuel: DEA, DHA
Deployment Environment Adapter: dea.yaml
mode: ha net_segment_type: gre wanted_release: Juno on Ubuntu 14.04.1 Node roles Fuel networks, ips, interfaces, bridge configs OS components settings ...
DHA CONFIG FILES
There are 2 config files wired out from fuel: DEA, DHA
Deployment Hardware Adapter: dha.yaml
adapter: libvirt nodes: - id: 1 libvirtName: controller1 libvirtTemplate: templates/virtual_environment/vms/controller.xml - id: 5 libvirtName: fuel-master libvirtTemplate: templates/virtual_environment/vms/fuel.xml isFuel: yes username: root password: r00tme virtNetConfDir: templates/virtual_environment/networks disks: fuel: 50G controller: 50G compute: 50G
DEPLOYMENT
AUTO DEPLOYER
fuel/ci/deploy.sh
Typically:
sudo ./deploy.sh -iso opnfv.iso -dea dea.yaml -dha dha.yaml -s /mnt/images -p PLUGIN
-> deploy OPNFV Cloud env with Fuel
-iso -> OPNFV iso downloaded or built from source
-p -> plugins for FUEL, like ODL puppet files to install ODL on nodes
* Not ODL itself
* Default ODL rpm for ARNO SR1 is integrated in OPNFV iso
* Can be downloaded or built from source if different version required
-s -> storage location for generated VMs, user defined
-nf -> no fuel (redeployment)
-nh -> no health check
PREPARE ARNO ISO
DOWNLOAD ARNO ISO FILE: http://artifacts.opnfv.org/arno.2015.2.0/fuel/arno.2015.2.0.fuel.iso
It contains a Centos FUEL image containing all the requirements for OPNFV deployments (e.g. OpenStack Juno, ODL Lithium, Fuel plugins
USE CORRECT NTP SETTINGS IN DEA.YAML
- Section „fuel”:
NTP1: 10.118.34.219 NTP2: NTP3:
- Section „ntp_list”:
value: 10.118.34.219
Unless using a correct value, deployment will fail - FUEL GUI ERROR:
Deployment has failed. Method granular_deploy. Failed to execute hook 'puppet' Puppet run failed. Check puppet logs for details --- priority: 200 fail_on_error: true type: puppet uids: - '3' parameters: puppet_modules: "/etc/puppet/modules" puppet_manifest: "/etc/puppet/modules/osnailyfacter/modular/ntp/ntp-check.pp" timeout: 600 cwd: "/" . Inspect Astute logs for the details
OPTIONAL ODL BUILD STEP
One should either use the ODL integrated in ARNO SR1 (iso downloaded or built manually in previour step already contains it, no manual step required) or build rpm from source, and add it to the deploy script as argument.
BUILD: (readme: https://github.com/openstack/fuel-plugin-opendaylight)
git clone https://github.com/stackforge/fuel-plugin-opendaylight pip install fuel-plugin-builder gem install fpm fpb --build fuel-plugin-opendaylight/
INCLUDE ODL PLUGIN IN DEPLOYMENT (by using argument for deploy.sh on next slide):
-p <fuel-plugin-opendaylight-rmp-dir>
EXCLUDE ODL PLUGIN ENTIRELY If ODL is not required and the plugin deployment is causing the entire deployment to fail then the ODL plugin can be disabled entirely by editing /fuel/deploy/templates/virtual_environment/conf/plugins_conf/opendaylight.yaml
opendaylight:
metadata:
enabled: false <<<<< disable ODL
label: OpenDaylight plugin
AUTOMATED DEPLOYMENT
Typical script parameters:
sudo ~/opnfv/fuel/fuel/ci/deploy.sh \ -iso ~/opnfv/fuel/fuel/build/release/opnfv-P0000.iso \ -dea ~/opnfv/fuel/fuel/deploy/templates/virtual_environment/conf/dea.yaml \ -dha ~/opnfv/fuel/fuel/deploy/templates/virtual_environment/conf/dha.yaml \ -s ~/opnfv/fuel/fuel/deploy/images \ -nh # Would fail at Live-Migraition test phase because of a known ODL bug anyway, let’s test it afterwards
Console output:
Fuel Plugins Directory not specified! No external plugins will be installed! Using OPNFV ISO file: ~/opnfv/fuel/fuel/build/release/opnfv-P0000.iso Using image directory: ~/opnfv/fuel/fuel/deploy/images ... <Cleanup, Create Networks, power on VMs> ... Install Fuel Master Insert ISO ~/opnfv/fuel/fuel/deploy/fueltmp/deploy-opnfv-P0000.iso Inject our own astute.yaml settings Let the Fuel deployment continue Installing Fuel Plugins ...
MANUAL CLEANUP
May be useful if deployment script was stopped unexpectedly, and system ends in undefined state
sudo ~/opnfv/fuel/fuel/ci/deploy.sh -co \ -dha ../deploy/templates/virtual_environment/conf/dha.yaml
VERIFICATION
Deploy verification
Verify FUEL deployment on GUI: From Jumphost (Fuel Host):
$ firefox http://10.20.0.2:8000 & (admin/admin)
- Verify Networks
- Health Check
- * Note: Live-Migraition test will fail (Bug in ODL currently), you can skip this test in the list if you choose to not see the error message, simply uncheck it in the list
Check fuel version and plugins:
ssh root@10.20.0.2 (pwd: r00tme) [root@opnfv_virt ~]# fuel plugins fuel --versionid | name | version | package_version ---|--------------|---------|---------------- 1 | opendaylight | 0.6.2 | 2.0.0 [root@opnfv_virt ~]# fuel --version 6.1.0
Verify OpenStack GUI: From Jumphost (Fuel Host):
$ firefox http://172.16.0.3/horizon/ & (admin/admin)
Verify OpenDaylight:
Is GUI accessible - Point your browser to the following URL: 172.16.0.3:8181/index.html (admin/admin)
Verify services: SSH to Controller:
root@node-2:~# nova-manage service list Binary Host Zone Status State Updated_At nova-consoleauth node-2.opnfvericsson.ca internal enabled :-) 2015-10-22 06:08:20 nova-scheduler node-2.opnfvericsson.ca internal enabled :-) 2015-10-22 06:08:21 nova-conductor node-2.opnfvericsson.ca internal enabled :-) 2015-10-22 06:08:21 nova-cert node-2.opnfvericsson.ca internal enabled :-) 2015-10-22 06:07:43 nova-compute node-1.opnfvericsson.ca nova enabled :-) 2015-10-22 06:07:29 nova-compute node-3.opnfvericsson.ca nova enabled :-) 2015-10-22 06:07:36 nova-compute node-4.opnfvericsson.ca nova enabled :-) 2015-10-22 06:08:11 root@node-2:~# neutron agent-list +--------------------------------------+----------------+-------------------------+-------+----------------+------------------------+ | id | agent_type | host | alive | admin_state_up | binary | +--------------------------------------+----------------+-------------------------+-------+----------------+------------------------+ | 11709a29-24f8-46a2-b196-8b5545a6e0a5 | Metadata agent | node-2.opnfvericsson.ca | :-) | True | neutron-metadata-agent | | 2cb0a373-c08a-4ea2-a819-13d7bf892f2e | L3 agent | node-2.opnfvericsson.ca | :-) | True | neutron-l3-agent | | 42d08a62-83b5-4f12-a24c-8d375cde63f4 | DHCP agent | node-2.opnfvericsson.ca | :-) | True | neutron-dhcp-agent | +--------------------------------------+----------------+-------------------------+-------+----------------+------------------------+
CHALLENGES
CHALLENGE #1 - SOLVED
sudo ./deploy.sh ... error: Failed to start domain controller1 error: unsupported configuration: guest and host CPU are not compatible: Host CPU does not provide required features: erms, smep, fsgsbase, rdrand, f16c, avx, osxsave, xsave, tsc-deadline, x2apic
I am using Intel(R) Xeon(R) CPU E5620 @ 2.40GHz – Westmere architecture
I had to change arcthitecture in VM definition XMLs:
cd ../deploy/templates/virtual_environment/vms sed -i 's/SandyBridge/Westmere/g' compute.xml controller.xml fuel.xml cd -
I had to comment out flags not supported by my CPU in VM definition XMLs:
cd ../deploy/templates/virtual_environment/vms
for flag in erms smep fsgsbase rdrand f16c avx osxsave xsave tsc-deadline x2apic;do
sed -i "/'${flag}'/s/^/<\!-- /" compute.xml controller.xml fuel.xml &&
sed -i "/'${flag}'/s/$/ -->/" compute.xml controller.xml fuel.xml
done
cd –
CHALLENGE #2 - SOLVED
Deployment seems to be pretty error-prone, if you CTRL+C, and see error „OSError: [Errno 38] Function not implemented: '~/opnfv/fuel/fuel/deploy/fueltmp/origiso/boot.catalog‚”, iso is probably still mounted, check it with:
$ ~/opnfv/fuel/fuel/ci$ cat /etc/mtab fuseiso ~/opnfv/fuel/fuel/deploy/fueltmp/origiso fuse.fuseiso rw,nosuid,nodev 0 0
Unmount it with:
$ ~/opnfv/fuel/fuel/ci$ sudo fusermount -u ~/opnfv/fuel/fuel/deploy/fueltmp/origiso
CHALLENGE #3 - DISAPPEARED AFTER CLEANUP WITH MAKE CLEAN
BUILD fails with build.sh:
Installing Fuel Plugins
Fofakeroot /opt/fuel-createmirror-6.1/fuel-createmirror
* INFO: Started /opt/fuel-createmirror-6.1/deb-mirror /opt/fuel-createmirror-6.1/config/mos-ubuntu-updatesonly.cfg
* FATAL: Upstream mirror mirror.fuel-infra.org does not support rsync protocol, aborting
* INFO: Started /opt/fuel-createmirror-6.1/deb-mirror /opt/fuel-createmirror-6.1/config/ubuntu.cfg
* FATAL: Upstream mirror archive.ubuntu.com does not support rsync protocol, aborting
* FATAL: Creation of MOS mirror FAILED, check logs at /var/log/mirror-sync
* FATAL: Creation of Ubuntu mirror FAILED, check logs at /var/log/mirror-sync
make[2]: Leaving directory `~/opnfv/fuel/fuel/build/f_isoroot/f_repobuild'
make[2]: *** [.nailgun] Error 1
semop(1): encountered an error: Invalid argument
make[1]: *** [f_repobuild] Error 2
make: *** [f_isoroot] Error 2
make[1]: Leaving directory `~/opnfv/fuel/fuel/build/f_isoroot'
make: *** [all] Error 2
Build: make all failed, exiting ...
Exiting ...
und plugin /root/plugins/opendaylight-0.6-0.6.1-1.noarch.rpm, installing ...
Eject ISO file ~/opnfv/fuel/fuel/deploy/fueltmp/deploy-opnfv-P0000.iso
Remove ISO directory ~/opnfv/fuel/fuel/deploy/fueltmp
Deleting directory ~/opnfv/fuel/fuel/deploy/fueltmp
Installation of Fuel Plugin /root/plugins/opendaylight-0.6-0.6.1-1.noarch.rpm failed: Traceback (most recent call last):
...
plugin_tar = tarfile.open(plugin_path, 'r')
File "/usr/lib64/python2.6/tarfile.py", line 1657, in open
raise ReadError("file could not be opened successfully")
tarfile.ReadError: file could not be opened successfully
CHALLENGE #4 - WORKAROUND: REDEPLOY WITH "-nf" FLAG
Trying to SSH into Fuel VM 10.20.0.2 ... sleeping 10 seconds Wait until Fuel menu is up Inject our own astute.yaml settings Modifying Fuel astute Let the Fuel deployment continue Found FUEL menu as PID 1292, now killing it Wait until installation complete Waiting for one minute for Fuel to stabilize Delete deprecated fuel client config /etc/fuel/client/config.yaml Installing Fuel Plugins Found plugin /root/plugins/opendaylight-0.6-0.6.2-1.noarch.rpm, installing ... Eject ISO file ~/opnfv/fuel/fuel/deploy/fueltmp/deploy-opnfv-P0000.iso Remove ISO directory ~/opnfv/fuel/fuel/deploy/fueltmp Deleting directory ~/opnfv/fuel/fuel/deploy/fueltmp Installation of Fuel Plugin /root/plugins/opendaylight-0.6-0.6.2-1.noarch.rpm failed: http://mirror.fuel-infra.org/mos/centos-6/mos6.1/security/repodata/repomd.xml: [Errno 14] PYCURL ERROR 6 - "Couldn't resolve host 'mirror.fuel-infra.org'" Trying other mirror. http://mirror.fuel-infra.org/mos/centos-6/mos6.1/updates/repodata/repomd.xml: [Errno 14] PYCURL ERROR 6 - "Couldn't resolve host 'mirror.fuel-infra.org'" Trying other mirror.
TRY TO REDEPLOY WITH „-nf” FLAG (NO FUEL): Verification of fuel plugin install looks to be failed but in reality it is successfully installed, verify it:
ssh root@10.20.0.2 (pwd: r00tme) [root@opnfv_virt ~]# fuel plugins
Fuel will not de redeployed with this flag, so the deployment can be continued.
A fix for this is already under review.
CHALLENGE #5 - HANGS... WORKAROUND: REDEPLOY WITH FUEL (WITHOUT „-nf” FLAG)
~/opnfv/fuel/fuel/ci$ sudo ./deploy.sh -iso /home/egbohal/opnfv/fuel/fuel/build/release/opnfv-P0000.iso -dea /home/egbohal/opnfv/fuel/fuel/deploy/templates/virtual_environment/conf/dea.yaml -dha /home/egbohal/opnfv/fuel/fuel/deploy/templates/virtual_environment/conf/dha.yaml -s /home/egbohal/opnfv/fuel/fuel/deploy/images -nf Namespace(cleanup=False, cleanup_only=False, dea_file='/home/egbohal/opnfv/fuel/fuel/deploy/templates/virtual_environment/conf/dea.yaml', dha_file='/home/egbohal/opnfv/fuel/fuel/deploy/templates/virtual_environment/conf/dha.yaml', fuel_only=False, fuel_plugins_dir=None, iso_file='/home/egbohal/opnfv/fuel/fuel/build/release/opnfv-P0000.iso', no_fuel=True, no_health_check=False, pxe_bridge='pxebr', storage_dir='/home/egbohal/opnfv/fuel/fuel/deploy/images') Fuel Plugins Directory not specified! No external plugins will be installed! Power OFF Node controller1 Power OFF Node compute1 Power OFF Node compute2 Power OFF Node compute3 Set boot order ['network', 'hd'] on Node controller1 Set boot order ['network', 'hd'] on Node compute1 Set boot order ['network', 'hd'] on Node compute2 Set boot order ['network', 'hd'] on Node compute3 Power ON Node controller1 Power ON Node compute1 Power ON Node compute2 Power ON Node compute3 Check prerequisites Check supported release: Juno on Ubuntu 14.04.1 Check previous installation Deleting environment 1
CHALLENGE #6
If deployment hangs for a while, you can ssh to fuel and verify which state the deployment currently is to know where to dig in:
[root@opnfv_virt ~]# watch -n 1 fuel task id | status | name | cluster | progress | uuid ---|---------|----------------|---------|----------|------------------------------------- 1 | ready | check_networks | 1 | 100 | e0734875-e5b9-4b05-9eb5-0737001c0ad0 4 | running | deployment | 1 | 0 | 119ff982-48f1-40cd-91a5-63f5ce02ae7d 2 | running | deploy | 1 | 29 | 4a37d617-8752-41b2-b056-afc87dc8d022 3 | ready | provision | 1 | 100 | 5ec218aa-3e8d-4dae-aa46-de8fcaef5e90
MANUAL ISO BUILD
PREREQUISITES
SSH KEY
UPLOAD SSH KEYS TO:
- gerrit.opnfv.org
- github.com
FIREWALL RULES
ALLOW FIREWALL for HTTP, HTTPS, SSH, GIT(9418), RSYNC(873) protocols to:
- gerrit.opnfv.org, git.opnfv.org
- pypi.python.org
- get.docker.com
- p80.pool.sks-keyservers.net
- github.com
- mirror.fuel-infra.org
- archive.ubuntu.com
PKG DEPENDENCIES
sudo apt-get install -y libvirt-bin qemu-kvm python-pip fuseiso mkisofs sudo apt-get install -y python-dev libz-dev libxml2-dev libxslt-dev sudo pip install pyyaml netaddr paramiko lxml scp pycrypto ecdsa curl -sSL https://get.docker.com/gpg | sudo apt-key add - curl -sSL https://get.docker.com/ | sh sudo usermod -aG docker <user>
logout & login again
BUILD ARNO ISO
git clone https://gerrit.opnfv.org/gerrit/fuel cd fuel/fuel/build make all
It will generate a Centos FUEL image containing all the requirements for OPNFV deployments (e.g. OpenStack Juno, ODL Lithium, Fuel plugins):
- download OpenStack and other components
- configure docker containers
- Prepare bootable iso img
- Patch iso with fuel plugins
End of console output:
Making ISO... ... Product ISO is ~/opnfv/fuel/fuel/build/release/opnfv-P0000.iso
PROPOSED INVESTIGATION TOPICS
OVS PUPPET PLUGIN
https://review.openstack.org/#/c/222066/
From Guo Ruijing ruijing.guo@intel.com
Install OVS from specified URL in ODL plugin
Uses pre_build_hook and puppet
Uploaded Sep 10, 2015 10:17 AM
Not investigated, To Be Checked!
DEVELOPERS PLAYGROUND
Fuel@OPNFV devel pipeline/sandbox
Jonas Bjurel jonas.bjurel@ericsson.com
Devel pipeline homepage: http://jonasbjurel.github.io/OPNFV-Playground/
FUEL PLUGIN OVS
REFERENCES
genesis/fuel/ci/README: with examples for BM/VIRTUAL/FUEL_ONLY DEPLOYMENT
genesis/fuel/deploy/README: same as above, more verbose, many template configs
genesis/fuel/build/README: build container and .iso
https://gerrit.opnfv.org/gerrit/fuel
http://artifacts.opnfv.org/arno.2015.2.0/fuel/install-guide.arno.2015.2.0.pdf
Genesis doc: http://artifacts.opnfv.org/genesis/fuel/docs/build-instructions.html
Genesis doc: http://artifacts.opnfv.org/genesis/fuel/docs/installation-instructions.html
Docker doc: https://docs.docker.com/installation/ubuntulinux/
