Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====CONFIG FILES====
===CONFIG FILE TEMPLATES===
==For Bare Metal and Virtual deployment, template files are available==
fuel/deploy/templates/virtual_environment/conf/<dea.yaml,dha.yaml>
fuel/deploy/templates/hardware_environment/conf/<ericsson_montreal_lab, linux_foundation_lab, opnfv_box>/<dea.yaml,dha.yaml>

==VM definition files==
fuel/deploy/templates/virtual_environment/vms/<controller.xml, compute.xml, fuel.xml>
fuel/deploy/templates/hardware_environment/vms/fuel.xml

==Network config for Virtual deployment==
fuel/deploy/templates/virtual_environment/networks/fuel<x>.xml

===DEA CONFIG FILES===
There are 2 config files wired out from fuel: DEA, DHA

==Deployment Environment Adapter: dea.yaml==
<code>
mode: ha
net_segment_type: gre
wanted_release: Juno on Ubuntu 14.04.1
Node roles
Fuel networks, ips, interfaces, bridge configs
OS components settings
...
</code>

===DHA CONFIG FILES===
There are 2 config files wired out from fuel: DEA, DHA

==Deployment Hardware Adapter: dha.yaml==
<code>
adapter: libvirt
nodes:
- id: 1
  libvirtName: controller1
  libvirtTemplate: templates/virtual_environment/vms/controller.xml
- id: 5
  libvirtName: fuel-master
  libvirtTemplate: templates/virtual_environment/vms/fuel.xml
  isFuel: yes
  username: root
  password: r00tme

  virtNetConfDir: templates/virtual_environment/networks
  disks:
  fuel: 50G
  controller: 50G
  compute: 50G
</code>

====DEPLOYMENT====
===AUTO DEPLOYER===
fuel/ci/deploy.sh

Typically: 
<code>
sudo ./deploy.sh -iso opnfv.iso -dea dea.yaml -dha dha.yaml -s /mnt/images -p PLUGIN
-> deploy OPNFV Cloud env with Fuel

-iso	-> OPNFV iso downloaded or built from source
-p	-> plugins for FUEL, like ODL puppet files to install ODL on nodes
           * Not ODL itself
           * Default ODL rpm for ARNO SR1 is integrated in OPNFV iso
           * Can be downloaded or built from source if different version required
-s	-> storage location for generated VMs, user defined
-nf	-> no fuel (redeployment)
-nh	-> no health check
</code>

===PREPARE ARNO ISO===
DOWNLOAD ARNO ISO FILE:
http://artifacts.opnfv.org/arno.2015.2.0/fuel/arno.2015.2.0.fuel.iso

It contains a Centos FUEL image containing all the requirements for OPNFV deployments (e.g. OpenStack Juno, ODL Lithium, Fuel plugins

===USE CORRECT NTP SETTINGS IN DEA.YAML===
  * Section „fuel”:
<code>
NTP1: 10.118.34.219
NTP2:
NTP3:
</code>
 
  * Section „ntp_list”:
<code>
value: 10.118.34.219
</code>

Unless using a correct value, deployment will fail - FUEL GUI ERROR:
<code>
Deployment has failed. Method granular_deploy. Failed to execute hook 'puppet' Puppet run failed. Check puppet logs for details
---
priority: 200
fail_on_error: true
type: puppet
uids:
- '3'
parameters:
  puppet_modules: "/etc/puppet/modules"
  puppet_manifest: "/etc/puppet/modules/osnailyfacter/modular/ntp/ntp-check.pp"
  timeout: 600
  cwd: "/"
.
Inspect Astute logs for the details
</code>

===OPTIONAL ODL BUILD STEP===
One should either use the ODL integrated in ARNO SR1 (iso downloaded or built manually in previour step already contains it, no manual step required) or build rpm from source, and add it to the deploy script as argument.

BUILD: (readme: https://github.com/openstack/fuel-plugin-opendaylight)
<code>
git clone https://github.com/stackforge/fuel-plugin-opendaylight
pip install fuel-plugin-builder
gem install fpm
fpb --build fuel-plugin-opendaylight/
</code>

INCLUDE ODL PLUGIN IN DEPLOYMENT (by using argument for deploy.sh on next slide):
<code>
-p <fuel-plugin-opendaylight-rmp-dir>
</code>

EXCLUDE ODL PLUGIN ENTIRELY
If ODL is not required and the plugin deployment is causing the entire deployment to fail then the ODL plugin can be disabled entirely by editing /fuel/deploy/templates/virtual_environment/conf/plugins_conf/opendaylight.yaml
<code>
 opendaylight:
   metadata:
     enabled: false      <<<<< disable ODL
     label: OpenDaylight plugin
</code>

===AUTOMATED DEPLOYMENT===
Typical script parameters:
<code>
sudo ~/opnfv/fuel/fuel/ci/deploy.sh \
-iso ~/opnfv/fuel/fuel/build/release/opnfv-P0000.iso \
-dea ~/opnfv/fuel/fuel/deploy/templates/virtual_environment/conf/dea.yaml \
-dha ~/opnfv/fuel/fuel/deploy/templates/virtual_environment/conf/dha.yaml \
-s ~/opnfv/fuel/fuel/deploy/images \
-nh # Would fail at Live-Migraition test phase because of a known ODL bug anyway, let’s test it afterwards
</code>

Console output:
<code>
Fuel Plugins Directory not specified! No external plugins will be installed!
Using OPNFV ISO file: ~/opnfv/fuel/fuel/build/release/opnfv-P0000.iso
Using image directory: ~/opnfv/fuel/fuel/deploy/images
...
<Cleanup, Create Networks, power on VMs>
... 
Install Fuel Master
Insert ISO ~/opnfv/fuel/fuel/deploy/fueltmp/deploy-opnfv-P0000.iso
Inject our own astute.yaml settings
Let the Fuel deployment continue
Installing Fuel Plugins
...
</code>

===MANUAL CLEANUP===
May be useful if deployment script was stopped unexpectedly, and system ends in undefined state
<code>
sudo ~/opnfv/fuel/fuel/ci/deploy.sh -co \
-dha ../deploy/templates/virtual_environment/conf/dha.yaml
</code>

====VERIFICATION====
===Deploy verification===

**Verify FUEL deployment on GUI:**
From Jumphost (Fuel Host):
<code>
$ firefox http://10.20.0.2:8000 & (admin/admin)
</code>
  * Verify Networks
  * Health Check
  ** Note: Live-Migraition test will fail (Bug in ODL currently), you can skip this test in the list if you choose to not see the error message, simply uncheck it in the list

**Check fuel version and plugins:**
<code>
ssh root@10.20.0.2 (pwd: r00tme)
[root@opnfv_virt ~]# fuel plugins
fuel --versionid | name         | version | package_version
---|--------------|---------|----------------
1  | opendaylight | 0.6.2   | 2.0.0
[root@opnfv_virt ~]# fuel --version
6.1.0
</code>

**Verify OpenStack GUI:**
From Jumphost (Fuel Host):
<code>
$ firefox http://172.16.0.3/horizon/ & (admin/admin)
</code>

**Verify OpenDaylight:**

Is GUI accessible - Point your browser to the following URL: 172.16.0.3:8181/index.html (admin/admin)

**Verify services:**
SSH to Controller:
<code>
root@node-2:~# nova-manage service list
Binary           Host                                 Zone             Status     State Updated_At
nova-consoleauth node-2.opnfvericsson.ca              internal         enabled    :-)   2015-10-22 06:08:20
nova-scheduler   node-2.opnfvericsson.ca              internal         enabled    :-)   2015-10-22 06:08:21
nova-conductor   node-2.opnfvericsson.ca              internal         enabled    :-)   2015-10-22 06:08:21
nova-cert        node-2.opnfvericsson.ca              internal         enabled    :-)   2015-10-22 06:07:43
nova-compute     node-1.opnfvericsson.ca              nova             enabled    :-)   2015-10-22 06:07:29
nova-compute     node-3.opnfvericsson.ca              nova             enabled    :-)   2015-10-22 06:07:36
nova-compute     node-4.opnfvericsson.ca              nova             enabled    :-)   2015-10-22 06:08:11
root@node-2:~# neutron agent-list
+--------------------------------------+----------------+-------------------------+-------+----------------+------------------------+
| id                                   | agent_type     | host                    | alive | admin_state_up | binary                 |
+--------------------------------------+----------------+-------------------------+-------+----------------+------------------------+
| 11709a29-24f8-46a2-b196-8b5545a6e0a5 | Metadata agent | node-2.opnfvericsson.ca | :-)   | True           | neutron-metadata-agent |
| 2cb0a373-c08a-4ea2-a819-13d7bf892f2e | L3 agent       | node-2.opnfvericsson.ca | :-)   | True           | neutron-l3-agent       |
| 42d08a62-83b5-4f12-a24c-8d375cde63f4 | DHCP agent     | node-2.opnfvericsson.ca | :-)   | True           | neutron-dhcp-agent     |
+--------------------------------------+----------------+-------------------------+-------+----------------+------------------------+
</code>


====CHALLENGES====
===CHALLENGE #1 - SOLVED===
<code>
sudo ./deploy.sh  ...
error: Failed to start domain controller1
error: unsupported configuration: guest and host CPU are not compatible: Host CPU does not provide required features: erms, smep, fsgsbase, rdrand, f16c, avx, osxsave, xsave, tsc-deadline, x2apic
</code>

I am using Intel(R) Xeon(R) CPU E5620 @ 2.40GHz – Westmere architecture

**I had to change arcthitecture in VM definition XMLs:**
<code>
cd ../deploy/templates/virtual_environment/vms
sed -i 's/SandyBridge/Westmere/g' compute.xml controller.xml fuel.xml
cd -
</code>

**I had to comment out flags not supported by my CPU in VM definition XMLs:**
<code>
cd ../deploy/templates/virtual_environment/vms
for flag in erms smep fsgsbase rdrand f16c avx osxsave xsave tsc-deadline x2apic;do
sed -i "/'${flag}'/s/^/<\!-- /" compute.xml controller.xml fuel.xml &&
sed -i "/'${flag}'/s/$/ -->/" compute.xml controller.xml fuel.xml
done
cd –
</code>

===CHALLENGE #2 - SOLVED===
Deployment seems to be pretty error-prone, if you CTRL+C, and see error „OSError: [Errno 38] Function not implemented: '~/opnfv/fuel/fuel/deploy/fueltmp/origiso/boot.catalog‚”, iso is probably still mounted, check it with:
<code>
$ ~/opnfv/fuel/fuel/ci$ cat /etc/mtab
fuseiso ~/opnfv/fuel/fuel/deploy/fueltmp/origiso fuse.fuseiso rw,nosuid,nodev 0 0
</code>

Unmount it with:
<code>
$ ~/opnfv/fuel/fuel/ci$ sudo fusermount -u ~/opnfv/fuel/fuel/deploy/fueltmp/origiso
</code>

===CHALLENGE #3 - DISAPPEARED AFTER CLEANUP WITH MAKE CLEAN===
BUILD fails with build.sh:
<code>
Installing Fuel Plugins
Fofakeroot /opt/fuel-createmirror-6.1/fuel-createmirror
 * INFO: Started /opt/fuel-createmirror-6.1/deb-mirror /opt/fuel-createmirror-6.1/config/mos-ubuntu-updatesonly.cfg
 * FATAL: Upstream mirror mirror.fuel-infra.org does not support rsync protocol, aborting
 * INFO: Started /opt/fuel-createmirror-6.1/deb-mirror /opt/fuel-createmirror-6.1/config/ubuntu.cfg

 * FATAL: Upstream mirror archive.ubuntu.com does not support rsync protocol, aborting
 * FATAL: Creation of MOS mirror FAILED, check logs at /var/log/mirror-sync
 * FATAL: Creation of Ubuntu mirror FAILED, check logs at /var/log/mirror-sync
make[2]: Leaving directory `~/opnfv/fuel/fuel/build/f_isoroot/f_repobuild'
make[2]: *** [.nailgun] Error 1
semop(1): encountered an error: Invalid argument
make[1]: *** [f_repobuild] Error 2
make: *** [f_isoroot] Error 2
make[1]: Leaving directory `~/opnfv/fuel/fuel/build/f_isoroot'
make: *** [all] Error 2
Build: make all failed, exiting ...
Exiting ...
und plugin /root/plugins/opendaylight-0.6-0.6.1-1.noarch.rpm, installing ...
Eject ISO file ~/opnfv/fuel/fuel/deploy/fueltmp/deploy-opnfv-P0000.iso
Remove ISO directory ~/opnfv/fuel/fuel/deploy/fueltmp
Deleting directory ~/opnfv/fuel/fuel/deploy/fueltmp
Installation of Fuel Plugin /root/plugins/opendaylight-0.6-0.6.1-1.noarch.rpm failed: Traceback (most recent call last):
...
    plugin_tar = tarfile.open(plugin_path, 'r')
  File "/usr/lib64/python2.6/tarfile.py", line 1657, in open
    raise ReadError("file could not be opened successfully")
tarfile.ReadError: file could not be opened successfully
</code>

===CHALLENGE #4 - WORKAROUND: REDEPLOY WITH "-nf" FLAG===
<code>
Trying to SSH into Fuel VM 10.20.0.2 ... sleeping 10 seconds
Wait until Fuel menu is up
Inject our own astute.yaml settings
Modifying Fuel astute
Let the Fuel deployment continue
Found FUEL menu as PID 1292, now killing it
Wait until installation complete
Waiting for one minute for Fuel to stabilize
Delete deprecated fuel client config /etc/fuel/client/config.yaml
Installing Fuel Plugins
Found plugin /root/plugins/opendaylight-0.6-0.6.2-1.noarch.rpm, installing ...
Eject ISO file ~/opnfv/fuel/fuel/deploy/fueltmp/deploy-opnfv-P0000.iso
Remove ISO directory ~/opnfv/fuel/fuel/deploy/fueltmp
Deleting directory ~/opnfv/fuel/fuel/deploy/fueltmp

Installation of Fuel Plugin /root/plugins/opendaylight-0.6-0.6.2-1.noarch.rpm failed: 
http://mirror.fuel-infra.org/mos/centos-6/mos6.1/security/repodata/repomd.xml: [Errno 14] PYCURL ERROR 6 - "Couldn't resolve host 'mirror.fuel-infra.org'"
Trying other mirror.
http://mirror.fuel-infra.org/mos/centos-6/mos6.1/updates/repodata/repomd.xml: [Errno 14] PYCURL ERROR 6 - "Couldn't resolve host 'mirror.fuel-infra.org'"
Trying other mirror.
</code>

**TRY TO REDEPLOY WITH „-nf” FLAG (NO FUEL):**
Verification of fuel plugin install looks to be failed but in reality it is successfully installed, verify it:
<code>
ssh root@10.20.0.2 (pwd: r00tme)
[root@opnfv_virt ~]# fuel plugins
</code>

Fuel will not de redeployed with this flag, so the deployment can be continued.

A fix for this is already under review.

===CHALLENGE #5 - HANGS... WORKAROUND: REDEPLOY WITH FUEL (WITHOUT „-nf” FLAG)===
<code>
~/opnfv/fuel/fuel/ci$ sudo ./deploy.sh -iso /home/egbohal/opnfv/fuel/fuel/build/release/opnfv-P0000.iso -dea /home/egbohal/opnfv/fuel/fuel/deploy/templates/virtual_environment/conf/dea.yaml -dha /home/egbohal/opnfv/fuel/fuel/deploy/templates/virtual_environment/conf/dha.yaml -s /home/egbohal/opnfv/fuel/fuel/deploy/images -nf

Namespace(cleanup=False, cleanup_only=False, dea_file='/home/egbohal/opnfv/fuel/fuel/deploy/templates/virtual_environment/conf/dea.yaml', dha_file='/home/egbohal/opnfv/fuel/fuel/deploy/templates/virtual_environment/conf/dha.yaml', fuel_only=False, fuel_plugins_dir=None, iso_file='/home/egbohal/opnfv/fuel/fuel/build/release/opnfv-P0000.iso', no_fuel=True, no_health_check=False, pxe_bridge='pxebr', storage_dir='/home/egbohal/opnfv/fuel/fuel/deploy/images')

Fuel Plugins Directory not specified! No external plugins will be installed!
Power OFF Node controller1
Power OFF Node compute1
Power OFF Node compute2
Power OFF Node compute3
Set boot order ['network', 'hd'] on Node controller1
Set boot order ['network', 'hd'] on Node compute1
Set boot order ['network', 'hd'] on Node compute2
Set boot order ['network', 'hd'] on Node compute3
Power ON Node controller1
Power ON Node compute1
Power ON Node compute2
Power ON Node compute3
Check prerequisites
Check supported release: Juno on Ubuntu 14.04.1
Check previous installation
Deleting environment 1
</code>

===CHALLENGE #6===
If deployment hangs for a while, you can ssh to fuel and verify which state the deployment currently is to know where to dig in:
<code>
[root@opnfv_virt ~]# watch -n 1 fuel task
id | status  | name           | cluster | progress | uuid
---|---------|----------------|---------|----------|-------------------------------------
1  | ready   | check_networks | 1       | 100      | e0734875-e5b9-4b05-9eb5-0737001c0ad0
4  | running | deployment     | 1       | 0        | 119ff982-48f1-40cd-91a5-63f5ce02ae7d
2  | running | deploy         | 1       | 29       | 4a37d617-8752-41b2-b056-afc87dc8d022
3  | ready   | provision      | 1       | 100      | 5ec218aa-3e8d-4dae-aa46-de8fcaef5e90
</code>

===CHALLENGE #7===
If the task for recreating neutron database during ODL plugin deployment takes too much time then deployment will fail:

TODO example log file indicating this condition.

Solutions:
* Use a later version of the ODL plugin than that packaged with SR1 .iso (by cloning & building the ODL plugin and adding the -p flag to the deploy.sh command line).
* Disable ODL plugin as described in a section above.


====MANUAL ISO BUILD====
===PREREQUISITES===
==SSH KEY==
UPLOAD SSH KEYS TO:
  * gerrit.opnfv.org
  * github.com

==FIREWALL RULES==
ALLOW FIREWALL for HTTP, HTTPS, SSH, GIT(9418), RSYNC(873) protocols to:
  * gerrit.opnfv.org, git.opnfv.org
  * pypi.python.org
  * get.docker.com
  * p80.pool.sks-keyservers.net
  * github.com
  * mirror.fuel-infra.org
  * archive.ubuntu.com

==PKG DEPENDENCIES==
<code>
sudo apt-get install -y libvirt-bin qemu-kvm python-pip fuseiso mkisofs
sudo apt-get install -y python-dev libz-dev libxml2-dev libxslt-dev
sudo pip install pyyaml netaddr paramiko lxml scp pycrypto ecdsa

curl -sSL https://get.docker.com/gpg | sudo apt-key add -
curl -sSL https://get.docker.com/ | sh
sudo usermod -aG docker <user>
</code>
logout & login again

===BUILD ARNO ISO===
<code>
git clone https://gerrit.opnfv.org/gerrit/fuel
cd fuel/fuel/build
make all
</code>

It will generate a Centos FUEL image containing all the requirements for OPNFV deployments (e.g. OpenStack Juno, ODL Lithium, Fuel plugins):
  * download OpenStack and other components
  * configure docker containers
  * Prepare bootable iso img
  * Patch iso with fuel plugins

End of console output:
<code>
Making ISO...
...
Product ISO is ~/opnfv/fuel/fuel/build/release/opnfv-P0000.iso
</code>

====PROPOSED INVESTIGATION TOPICS====
===OVS PUPPET PLUGIN===
https://review.openstack.org/#/c/222066/

From Guo Ruijing <ruijing.guo@intel.com> 

Install OVS from specified URL in ODL plugin

Uses pre_build_hook and puppet

Uploaded Sep 10, 2015 10:17 AM

Not investigated, To Be Checked!

===DEVELOPERS PLAYGROUND===
Fuel@OPNFV devel pipeline/sandbox 

Jonas Bjurel jonas.bjurel@ericsson.com

Devel pipeline homepage: http://jonasbjurel.github.io/OPNFV-Playground/

Wiki: https://github.com/jonasbjurel/OPNFV-Playground/wiki

===FUEL PLUGIN OVS===
https://github.com/lingyu1/fuel-plugin-ovs
 
https://github.com/lingyu1/fuel-rpm


====REFERENCES====
genesis/fuel/ci/README: with examples for BM/VIRTUAL/FUEL_ONLY DEPLOYMENT

genesis/fuel/deploy/README: same as above, more verbose, many template configs

genesis/fuel/build/README: build container and .iso

https://gerrit.opnfv.org/gerrit/fuel

http://artifacts.opnfv.org/arno.2015.2.0/fuel/install-guide.arno.2015.2.0.pdf

Genesis doc: http://artifacts.opnfv.org/genesis/fuel/docs/build-instructions.html

Genesis doc: http://artifacts.opnfv.org/genesis/fuel/docs/installation-instructions.html

Docker doc: https://docs.docker.com/installation/ubuntulinux/
Wiki

User Tools

Site Tools

Page Tools
