Differences

This shows you the differences between two versions of the page.

--- deploy_opnfv_with_fuel_in_a_virtual_environment [2015/10/27 15:04]
Gabor Halasz created
+++ deploy_opnfv_with_fuel_in_a_virtual_environment [2015/11/10 15:07] (current)
Billy O'Mahony [CHALLENGES]
@@ Line 1: / Line 1: @@
-==== CONFIG FILES ====
+====CONFIG FILES====
+===CONFIG FILE TEMPLATES===
+==For Bare Metal and Virtual deployment, template files are available==
+fuel/deploy/templates/virtual_environment/conf/<dea.yaml,dha.yaml>
+fuel/deploy/templates/hardware_environment/conf/<ericsson_montreal_lab, linux_foundation_lab, opnfv_box>/<dea.yaml,dha.yaml>
+==VM definition files==
+fuel/deploy/templates/virtual_environment/vms/<controller.xml, compute.xml, fuel.xml>
+fuel/deploy/templates/hardware_environment/vms/fuel.xml
+==Network config for Virtual deployment==
+fuel/deploy/templates/virtual_environment/networks/fuel<x>.xml
+===DEA CONFIG FILES===
+There are 2 config files wired out from fuel: DEA, DHA
+==Deployment Environment Adapter: dea.yaml==
+<code>
+mode: ha
+net_segment_type: gre
+wanted_release: Juno on Ubuntu 14.04.1
+Node roles
+Fuel networks, ips, interfaces, bridge configs
+OS components settings
+...
+</code>
+===DHA CONFIG FILES===
+There are 2 config files wired out from fuel: DEA, DHA
+==Deployment Hardware Adapter: dha.yaml==
+<code>
+adapter: libvirt
+nodes:
+- id: 1
+  libvirtName: controller1
+  libvirtTemplate: templates/virtual_environment/vms/controller.xml
+- id: 5
+  libvirtName: fuel-master
+  libvirtTemplate: templates/virtual_environment/vms/fuel.xml
+  isFuel: yes
+  username: root
+  password: r00tme
+  virtNetConfDir: templates/virtual_environment/networks
+  disks:
+  fuel: 50G
+  controller: 50G
+  compute: 50G
+</code>
+====DEPLOYMENT====
+===AUTO DEPLOYER===
+fuel/ci/deploy.sh
+Typically:
+<code>
+sudo ./deploy.sh -iso opnfv.iso -dea dea.yaml -dha dha.yaml -s /mnt/images -p PLUGIN
+-> deploy OPNFV Cloud env with Fuel
+-iso	-> OPNFV iso downloaded or built from source
+-p	-> plugins for FUEL, like ODL puppet files to install ODL on nodes
+           * Not ODL itself
+           * Default ODL rpm for ARNO SR1 is integrated in OPNFV iso
+           * Can be downloaded or built from source if different version required
+-s	-> storage location for generated VMs, user defined
+-nf	-> no fuel (redeployment)
+-nh	-> no health check
+</code>
+===PREPARE ARNO ISO===
+DOWNLOAD ARNO ISO FILE:
+http://artifacts.opnfv.org/arno.2015.2.0/fuel/arno.2015.2.0.fuel.iso
+It contains a Centos FUEL image containing all the requirements for OPNFV deployments (e.g. OpenStack Juno, ODL Lithium, Fuel plugins
+===USE CORRECT NTP SETTINGS IN DEA.YAML===
+  * Section „fuel”:
+<code>
+NTP1: 10.118.34.219
+NTP2:
+NTP3:
+</code>
-==== DEPLOYMENT ====
+  * Section „ntp_list”:
+<code>
+value: 10.118.34.219
+</code>
+Unless using a correct value, deployment will fail - FUEL GUI ERROR:
+<code>
+Deployment has failed. Method granular_deploy. Failed to execute hook 'puppet' Puppet run failed. Check puppet logs for details
+---
+priority: 200
+fail_on_error: true
+type: puppet
+uids:
+- '3'
+parameters:
+  puppet_modules: "/etc/puppet/modules"
+  puppet_manifest: "/etc/puppet/modules/osnailyfacter/modular/ntp/ntp-check.pp"
+  timeout: 600
+  cwd: "/"
+.
+Inspect Astute logs for the details
+</code>
+===OPTIONAL ODL BUILD STEP===
+One should either use the ODL integrated in ARNO SR1 (iso downloaded or built manually in previour step already contains it, no manual step required) or build rpm from source, and add it to the deploy script as argument.
+BUILD: (readme: https://github.com/openstack/fuel-plugin-opendaylight)
+<code>
+git clone https://github.com/stackforge/fuel-plugin-opendaylight
+pip install fuel-plugin-builder
+gem install fpm
+fpb --build fuel-plugin-opendaylight/
+</code>
+INCLUDE ODL PLUGIN IN DEPLOYMENT (by using argument for deploy.sh on next slide):
+<code>
+-p <fuel-plugin-opendaylight-rmp-dir>
+</code>
+EXCLUDE ODL PLUGIN ENTIRELY
+If ODL is not required and the plugin deployment is causing the entire deployment to fail then the ODL plugin can be disabled entirely by editing /fuel/deploy/templates/virtual_environment/conf/plugins_conf/opendaylight.yaml
+<code>
+ opendaylight:
+   metadata:
+     enabled: false      <<<<< disable ODL
+     label: OpenDaylight plugin
+</code>
+===AUTOMATED DEPLOYMENT===
+Typical script parameters:
+<code>
+sudo ~/opnfv/fuel/fuel/ci/deploy.sh \
+-iso ~/opnfv/fuel/fuel/build/release/opnfv-P0000.iso \
+-dea ~/opnfv/fuel/fuel/deploy/templates/virtual_environment/conf/dea.yaml \
+-dha ~/opnfv/fuel/fuel/deploy/templates/virtual_environment/conf/dha.yaml \
+-s ~/opnfv/fuel/fuel/deploy/images \
+-nh # Would fail at Live-Migraition test phase because of a known ODL bug anyway, let’s test it afterwards
+</code>
+Console output:
+<code>
+Fuel Plugins Directory not specified! No external plugins will be installed!
+Using OPNFV ISO file: ~/opnfv/fuel/fuel/build/release/opnfv-P0000.iso
+Using image directory: ~/opnfv/fuel/fuel/deploy/images
+...
+<Cleanup, Create Networks, power on VMs>
+...
+Install Fuel Master
+Insert ISO ~/opnfv/fuel/fuel/deploy/fueltmp/deploy-opnfv-P0000.iso
+Inject our own astute.yaml settings
+Let the Fuel deployment continue
+Installing Fuel Plugins
+...
+</code>
+===MANUAL CLEANUP===
+May be useful if deployment script was stopped unexpectedly, and system ends in undefined state
+<code>
+sudo ~/opnfv/fuel/fuel/ci/deploy.sh -co \
+-dha ../deploy/templates/virtual_environment/conf/dha.yaml
+</code>
+====VERIFICATION====
+===Deploy verification===
+**Verify FUEL deployment on GUI:**
+From Jumphost (Fuel Host):
+<code>
+$ firefox http://10.20.0.2:8000 & (admin/admin)
+</code>
+  * Verify Networks
+  * Health Check
+  ** Note: Live-Migraition test will fail (Bug in ODL currently), you can skip this test in the list if you choose to not see the error message, simply uncheck it in the list
+**Check fuel version and plugins:**
+<code>
+ssh root@10.20.0.2 (pwd: r00tme)
+[root@opnfv_virt ~]# fuel plugins
+fuel --versionid | name         | version | package_version
+---|--------------|---------|----------------
+  | opendaylight | 0.6.2   | 2.0.0
+[root@opnfv_virt ~]# fuel --version
+.1.0
+</code>
+**Verify OpenStack GUI:**
+From Jumphost (Fuel Host):
+<code>
+$ firefox http://172.16.0.3/horizon/ & (admin/admin)
+</code>
+**Verify OpenDaylight:**
+Is GUI accessible - Point your browser to the following URL: 172.16.0.3:8181/index.html (admin/admin)
+**Verify services:**
+SSH to Controller:
+<code>
+root@node-2:~# nova-manage service list
+Binary           Host                                 Zone             Status     State Updated_At
+nova-consoleauth node-2.opnfvericsson.ca              internal         enabled    :-)   2015-10-22 06:08:20
+nova-scheduler   node-2.opnfvericsson.ca              internal         enabled    :-)   2015-10-22 06:08:21
+nova-conductor   node-2.opnfvericsson.ca              internal         enabled    :-)   2015-10-22 06:08:21
+nova-cert        node-2.opnfvericsson.ca              internal         enabled    :-)   2015-10-22 06:07:43
+nova-compute     node-1.opnfvericsson.ca              nova             enabled    :-)   2015-10-22 06:07:29
+nova-compute     node-3.opnfvericsson.ca              nova             enabled    :-)   2015-10-22 06:07:36
+nova-compute     node-4.opnfvericsson.ca              nova             enabled    :-)   2015-10-22 06:08:11
+root@node-2:~# neutron agent-list
++--------------------------------------+----------------+-------------------------+-------+----------------+------------------------+
+| id                                   | agent_type     | host                    | alive | admin_state_up | binary                 |
++--------------------------------------+----------------+-------------------------+-------+----------------+------------------------+
+| 11709a29-24f8-46a2-b196-8b5545a6e0a5 | Metadata agent | node-2.opnfvericsson.ca | :-)   | True           | neutron-metadata-agent |
+| 2cb0a373-c08a-4ea2-a819-13d7bf892f2e | L3 agent       | node-2.opnfvericsson.ca | :-)   | True           | neutron-l3-agent       |
+| 42d08a62-83b5-4f12-a24c-8d375cde63f4 | DHCP agent     | node-2.opnfvericsson.ca | :-)   | True           | neutron-dhcp-agent     |
++--------------------------------------+----------------+-------------------------+-------+----------------+------------------------+
+</code>
+====CHALLENGES====
+===CHALLENGE #1 - SOLVED===
+<code>
+sudo ./deploy.sh  ...
+error: Failed to start domain controller1
+error: unsupported configuration: guest and host CPU are not compatible: Host CPU does not provide required features: erms, smep, fsgsbase, rdrand, f16c, avx, osxsave, xsave, tsc-deadline, x2apic
+</code>
+I am using Intel(R) Xeon(R) CPU E5620 @ 2.40GHz – Westmere architecture
+**I had to change arcthitecture in VM definition XMLs:**
+<code>
+cd ../deploy/templates/virtual_environment/vms
+sed -i 's/SandyBridge/Westmere/g' compute.xml controller.xml fuel.xml
+cd -
+</code>
+**I had to comment out flags not supported by my CPU in VM definition XMLs:**
+<code>
+cd ../deploy/templates/virtual_environment/vms
+for flag in erms smep fsgsbase rdrand f16c avx osxsave xsave tsc-deadline x2apic;do
+sed -i "/'${flag}'/s/^/<\!-- /" compute.xml controller.xml fuel.xml &&
+sed -i "/'${flag}'/s/$/ -->/" compute.xml controller.xml fuel.xml
+done
+cd –
+</code>
+===CHALLENGE #2 - SOLVED===
+Deployment seems to be pretty error-prone, if you CTRL+C, and see error „OSError: [Errno 38] Function not implemented: '~/opnfv/fuel/fuel/deploy/fueltmp/origiso/boot.catalog‚”, iso is probably still mounted, check it with:
+<code>
+$ ~/opnfv/fuel/fuel/ci$ cat /etc/mtab
+fuseiso ~/opnfv/fuel/fuel/deploy/fueltmp/origiso fuse.fuseiso rw,nosuid,nodev 0 0
+</code>
+Unmount it with:
+<code>
+$ ~/opnfv/fuel/fuel/ci$ sudo fusermount -u ~/opnfv/fuel/fuel/deploy/fueltmp/origiso
+</code>
+===CHALLENGE #3 - DISAPPEARED AFTER CLEANUP WITH MAKE CLEAN===
+BUILD fails with build.sh:
+<code>
+Installing Fuel Plugins
+Fofakeroot /opt/fuel-createmirror-6.1/fuel-createmirror
+ * INFO: Started /opt/fuel-createmirror-6.1/deb-mirror /opt/fuel-createmirror-6.1/config/mos-ubuntu-updatesonly.cfg
+ * FATAL: Upstream mirror mirror.fuel-infra.org does not support rsync protocol, aborting
+ * INFO: Started /opt/fuel-createmirror-6.1/deb-mirror /opt/fuel-createmirror-6.1/config/ubuntu.cfg
+ * FATAL: Upstream mirror archive.ubuntu.com does not support rsync protocol, aborting
+ * FATAL: Creation of MOS mirror FAILED, check logs at /var/log/mirror-sync
+ * FATAL: Creation of Ubuntu mirror FAILED, check logs at /var/log/mirror-sync
+make[2]: Leaving directory `~/opnfv/fuel/fuel/build/f_isoroot/f_repobuild'
+make[2]: *** [.nailgun] Error 1
+semop(1): encountered an error: Invalid argument
+make[1]: *** [f_repobuild] Error 2
+make: *** [f_isoroot] Error 2
+make[1]: Leaving directory `~/opnfv/fuel/fuel/build/f_isoroot'
+make: *** [all] Error 2
+Build: make all failed, exiting ...
+Exiting ...
+und plugin /root/plugins/opendaylight-0.6-0.6.1-1.noarch.rpm, installing ...
+Eject ISO file ~/opnfv/fuel/fuel/deploy/fueltmp/deploy-opnfv-P0000.iso
+Remove ISO directory ~/opnfv/fuel/fuel/deploy/fueltmp
+Deleting directory ~/opnfv/fuel/fuel/deploy/fueltmp
+Installation of Fuel Plugin /root/plugins/opendaylight-0.6-0.6.1-1.noarch.rpm failed: Traceback (most recent call last):
+...
+    plugin_tar = tarfile.open(plugin_path, 'r')
+  File "/usr/lib64/python2.6/tarfile.py", line 1657, in open
+    raise ReadError("file could not be opened successfully")
+tarfile.ReadError: file could not be opened successfully
+</code>
+===CHALLENGE #4 - WORKAROUND: REDEPLOY WITH "-nf" FLAG===
+<code>
+Trying to SSH into Fuel VM 10.20.0.2 ... sleeping 10 seconds
+Wait until Fuel menu is up
+Inject our own astute.yaml settings
+Modifying Fuel astute
+Let the Fuel deployment continue
+Found FUEL menu as PID 1292, now killing it
+Wait until installation complete
+Waiting for one minute for Fuel to stabilize
+Delete deprecated fuel client config /etc/fuel/client/config.yaml
+Installing Fuel Plugins
+Found plugin /root/plugins/opendaylight-0.6-0.6.2-1.noarch.rpm, installing ...
+Eject ISO file ~/opnfv/fuel/fuel/deploy/fueltmp/deploy-opnfv-P0000.iso
+Remove ISO directory ~/opnfv/fuel/fuel/deploy/fueltmp
+Deleting directory ~/opnfv/fuel/fuel/deploy/fueltmp
+Installation of Fuel Plugin /root/plugins/opendaylight-0.6-0.6.2-1.noarch.rpm failed:
+http://mirror.fuel-infra.org/mos/centos-6/mos6.1/security/repodata/repomd.xml: [Errno 14] PYCURL ERROR 6 - "Couldn't resolve host 'mirror.fuel-infra.org'"
+Trying other mirror.
+http://mirror.fuel-infra.org/mos/centos-6/mos6.1/updates/repodata/repomd.xml: [Errno 14] PYCURL ERROR 6 - "Couldn't resolve host 'mirror.fuel-infra.org'"
+Trying other mirror.
+</code>
+**TRY TO REDEPLOY WITH „-nf” FLAG (NO FUEL):**
+Verification of fuel plugin install looks to be failed but in reality it is successfully installed, verify it:
+<code>
+ssh root@10.20.0.2 (pwd: r00tme)
+[root@opnfv_virt ~]# fuel plugins
+</code>
+Fuel will not de redeployed with this flag, so the deployment can be continued.
+A fix for this is already under review.
+===CHALLENGE #5 - HANGS... WORKAROUND: REDEPLOY WITH FUEL (WITHOUT „-nf” FLAG)===
+<code>
+~/opnfv/fuel/fuel/ci$ sudo ./deploy.sh -iso /home/egbohal/opnfv/fuel/fuel/build/release/opnfv-P0000.iso -dea /home/egbohal/opnfv/fuel/fuel/deploy/templates/virtual_environment/conf/dea.yaml -dha /home/egbohal/opnfv/fuel/fuel/deploy/templates/virtual_environment/conf/dha.yaml -s /home/egbohal/opnfv/fuel/fuel/deploy/images -nf
+Namespace(cleanup=False, cleanup_only=False, dea_file='/home/egbohal/opnfv/fuel/fuel/deploy/templates/virtual_environment/conf/dea.yaml', dha_file='/home/egbohal/opnfv/fuel/fuel/deploy/templates/virtual_environment/conf/dha.yaml', fuel_only=False, fuel_plugins_dir=None, iso_file='/home/egbohal/opnfv/fuel/fuel/build/release/opnfv-P0000.iso', no_fuel=True, no_health_check=False, pxe_bridge='pxebr', storage_dir='/home/egbohal/opnfv/fuel/fuel/deploy/images')
+Fuel Plugins Directory not specified! No external plugins will be installed!
+Power OFF Node controller1
+Power OFF Node compute1
+Power OFF Node compute2
+Power OFF Node compute3
+Set boot order ['network', 'hd'] on Node controller1
+Set boot order ['network', 'hd'] on Node compute1
+Set boot order ['network', 'hd'] on Node compute2
+Set boot order ['network', 'hd'] on Node compute3
+Power ON Node controller1
+Power ON Node compute1
+Power ON Node compute2
+Power ON Node compute3
+Check prerequisites
+Check supported release: Juno on Ubuntu 14.04.1
+Check previous installation
+Deleting environment 1
+</code>
+===CHALLENGE #6===
+If deployment hangs for a while, you can ssh to fuel and verify which state the deployment currently is to know where to dig in:
+<code>
+[root@opnfv_virt ~]# watch -n 1 fuel task
+id | status  | name           | cluster | progress | uuid
+---|---------|----------------|---------|----------|-------------------------------------
+  | ready   | check_networks | 1       | 100      | e0734875-e5b9-4b05-9eb5-0737001c0ad0
+  | running | deployment     | 1       | 0        | 119ff982-48f1-40cd-91a5-63f5ce02ae7d
+  | running | deploy         | 1       | 29       | 4a37d617-8752-41b2-b056-afc87dc8d022
+  | ready   | provision      | 1       | 100      | 5ec218aa-3e8d-4dae-aa46-de8fcaef5e90
+</code>
+===CHALLENGE #7 - SOLVED===
+If there is a timeout error relating to recreate-neutron-db in /var/log/docker-logs/astute/astute.log on fuel-master, this means the task for recreating neutron database during ODL plugin deployment is taking too much time and deployment will fail.
+Solutions:
+* Use a later version of the ODL plugin than that packaged with SR1 .iso (by cloning & building the ODL plugin and adding the -p flag to the deploy.sh command line).
+* Disable ODL plugin as described in a section above.
+====MANUAL ISO BUILD====
+===PREREQUISITES===
+==SSH KEY==
+UPLOAD SSH KEYS TO:
+  * gerrit.opnfv.org
+  * github.com
+==FIREWALL RULES==
+ALLOW FIREWALL for HTTP, HTTPS, SSH, GIT(9418), RSYNC(873) protocols to:
+  * gerrit.opnfv.org, git.opnfv.org
+  * pypi.python.org
+  * get.docker.com
+  * p80.pool.sks-keyservers.net
+  * github.com
+  * mirror.fuel-infra.org
+  * archive.ubuntu.com
+==PKG DEPENDENCIES==
+<code>
+sudo apt-get install -y libvirt-bin qemu-kvm python-pip fuseiso mkisofs
+sudo apt-get install -y python-dev libz-dev libxml2-dev libxslt-dev
+sudo pip install pyyaml netaddr paramiko lxml scp pycrypto ecdsa
+curl -sSL https://get.docker.com/gpg | sudo apt-key add -
+curl -sSL https://get.docker.com/ | sh
+sudo usermod -aG docker <user>
+</code>
+logout & login again
+===BUILD ARNO ISO===
+<code>
+git clone https://gerrit.opnfv.org/gerrit/fuel
+cd fuel/fuel/build
+make all
+</code>
+It will generate a Centos FUEL image containing all the requirements for OPNFV deployments (e.g. OpenStack Juno, ODL Lithium, Fuel plugins):
+  * download OpenStack and other components
+  * configure docker containers
+  * Prepare bootable iso img
+  * Patch iso with fuel plugins
+End of console output:
+<code>
+Making ISO...
+...
+Product ISO is ~/opnfv/fuel/fuel/build/release/opnfv-P0000.iso
+</code>
+====PROPOSED INVESTIGATION TOPICS====
+===OVS PUPPET PLUGIN===
+https://review.openstack.org/#/c/222066/
+From Guo Ruijing <ruijing.guo@intel.com>
+Install OVS from specified URL in ODL plugin
+Uses pre_build_hook and puppet
+Uploaded Sep 10, 2015 10:17 AM
+Not investigated, To Be Checked!
+===DEVELOPERS PLAYGROUND===
+Fuel@OPNFV devel pipeline/sandbox
+Jonas Bjurel jonas.bjurel@ericsson.com
+Devel pipeline homepage: http://jonasbjurel.github.io/OPNFV-Playground/
+Wiki: https://github.com/jonasbjurel/OPNFV-Playground/wiki
+===FUEL PLUGIN OVS===
+https://github.com/lingyu1/fuel-plugin-ovs
+https://github.com/lingyu1/fuel-rpm
+====REFERENCES====
+genesis/fuel/ci/README: with examples for BM/VIRTUAL/FUEL_ONLY DEPLOYMENT
+genesis/fuel/deploy/README: same as above, more verbose, many template configs
+genesis/fuel/build/README: build container and .iso
+https://gerrit.opnfv.org/gerrit/fuel
+http://artifacts.opnfv.org/arno.2015.2.0/fuel/install-guide.arno.2015.2.0.pdf
+Genesis doc: http://artifacts.opnfv.org/genesis/fuel/docs/build-instructions.html
+Genesis doc: http://artifacts.opnfv.org/genesis/fuel/docs/installation-instructions.html
-==== VERIFICATION ====
+Docker doc: https://docs.docker.com/installation/ubuntulinux/
-==== CHALLENGES ====
-==== MANUAL ISO BUILD ====
-==== INVESTIGATION TOPICS ====

Wiki

User Tools

Site Tools

Differences

Page Tools