=================== Integration Testing =================== Curtin includes an in-tree integration suite that runs hundreds of tests validating various forms of custom storage and network configurations across all of the supported Ubuntu LTS releases as well as some of the currently supported interim releases. Background ========== Curtin includes a mechanism called 'vmtest' that allows it to actually do installs and validate a number of configurations. The general flow of the vmtests is: #. each test has an associated yaml config file for curtin in examples/tests #. uses curtin-pack to create the user-data for cloud-init to trigger install #. create and install a system using 'tools/launch'. #. The install environment is booted from a maas ephemeral image. #. kernel & initrd used are from maas images (not part of the image) #. network by default is handled via user networking #. It creates all empty disks required #. cloud-init datasource is provided by launch #. like: ds=nocloud-net;seedfrom=http://10.7.0.41:41518/ provided by python webserver start_http #. via -drive file=/tmp/launch.8VOiOn/seed.img,if=virtio,media=cdrom as a seed disk (if booted without external kernel) #. dependencies and other preparations are installed at the beginning by curtin inside the ephemeral image prior to configuring the target #. power off the system. #. configure a 'NoCloud' datasource seed image that provides scripts that will run on first boot. #. this will contain all our code to gather health data on the install #. by cloud-init design this runs only once per instance, if you start the system again this won't be called again #. boot the installed system with 'tools/xkvm'. #. reuses the disks that were installed/configured in the former steps #. also adds an output disk #. additionally the seed image for the data gathering is added #. On this boot it will run the provided scripts, write their output to a "data" disk and then shut itself down. #. extract the data from the output disk #. vmtest python code now verifies if the output is as expected. Debugging ========= At 3.1 one can pull data out of the maas image the command ``mount-image-callback``. For example:: sudo mount-image-callback your.img -- sh -c 'cp $MOUNTPOINT/boot/* .' At step 3.6 through 4 ``tools/launch`` can be called in a way to give you console access. To do so just call tools/launch but drop the -serial=x parameter. One might want to change "'power_state': {'mode': 'poweroff'}" to avoid the auto reboot before getting control. Replace the directory usually seen in the launch calls with a clean fresh directory. In /curtin curtin and its config can be found. If the system gets that far cloud-init will create a user creds: ubuntu/passw0rd , otherwise one can use a cloud-image from https://cloud-images.ubuntu.com/ and add a backdoor user via:: bzr branch lp:~maas-maintainers/maas/backdoor-image backdoor-image sudo ./backdoor-image -v --user= --password-auth --password= IMG At step 6 -> 7 you might want to keep all the temporary images around. To do so you can set ``CURTIN_VMTEST_KEEP_DATA_PASS=all`` in your environment. :: export CURTIN_VMTEST_KEEP_DATA_PASS=all CURTIN_VMTEST_KEEP_DATA_FAIL=all That will keep the /tmp/tmpXXXXX directories and all files in there for further execution. At step 7 you might want to take a look at the output disk yourself. It is a normal qcow image, so one can use ``mount-image-callback`` as described above. To invoke xkvm on your own take the command you see in the output and remove the "-serial ..." but add ``-nographic`` instead For graphical console one can add ``--vnc 127.0.0.1:1`` Setup ===== In order to run vmtest you'll need some dependencies. To get them, you can run:: make vmtest-deps make sync-images # Uses the IMAGE_DIR environment variable mentioned below Running ======= Running tests is done most simply by:: make vmtest .. note:: By default, the vmtests for iSCSI will be skipped (see Environment Variable section for details). If you wish to all tests in test_network.py, do so with:: nosetests3 tests/vmtests/test_network.py Or run a single test with:: nosetests3 tests/vmtests/test_network.py:XenialTestNetworkBasic Environment Variables ===================== Some environment variables affect the running of vmtest - ``CURTIN_VMTEST_APT_PROXY``: test will set apt: { proxy } in the guests to the value of ``CURTIN_VMTEST_APT_PROXY`` environment variable. If that is not set it will look at the host's apt config and read ``Acquire::HTTP::Proxy``. This can be prevented by setting ``CURTIN_VMTEST_APT_PROXY`` to the empty string; in this case no proxy is used. .. note:: For compatibility, the ``apt_proxy`` environment variable is supported, with the same behaviour as described above. If both are present, ``CURTIN_VMTEST_APT_PROXY`` will be preferred. - ``CURTIN_VMTEST_CURTIN_EXE``: Defaults to '' This is the path to the curtin executable that should be used for testing. It will need to set any environment that is needed and correctly pack itself. The default 'curtin' command in bin/ or installed by ``apt-get install curtin`` should work. If the value is unset or empty, then curtin from /bin is used. So to run vmtest on an installed version of curtin with, you can simply set this variable to 'curtin' or '/usr/bin/curtin' - ``CURTIN_VMTEST_KEEP_DATA_PASS``: Defaults to none. - ``CURTIN_VMTEST_KEEP_DATA_FAIL``: Defaults to all. These 2 variables determine what portions of the temporary test data are kept. The variables contain a comma ',' delimited list of directories that should be kept in the case of pass or fail. Additionally, the values 'all' and 'none' are accepted. Each vmtest that runs has its own sub-directory under the top level ``CURTIN_VMTEST_TOPDIR``. In that directory are directories: - ``boot``: inputs to the system boot (after install) - ``install``: install phase related files - ``disks``: the disks used for installation and boot - ``logs``: install and boot logs - ``collect``: data collected by the boot phase - ``CURTIN_VMTEST_TAR_DISKS``: default 0 Vmtest writes out disk image files sparsely into a disks directory If this flag is set to a non-zero number, vmtest will tar all disks in the directory into a single disks.tar and remove the sparse disk files. - ``CURTIN_VMTEST_TOPDIR``: default $TMPDIR/vmtest- Vmtest puts all test data under this value. By default, it creates a directory in TMPDIR (/tmp) named with as ``vmtest-`` If you set this value, you must ensure that the directory is either non-existent or clean. - ``CURTIN_VMTEST_REUSE_TOPDIR``: default 0 If this variable is set to 1, then vmtest will detect if the test specified already exists in the configured ``CURTIN_VMTEST_TOPDIR`` location. If present, vmtest will skip executing the install and boot phase of vmtest, and install just execute the unittests specified. This allows developers to re-run unittests on existing data that's already been collected. - ``CURTIN_VMTEST_LOG``: default $TMPDIR/vmtest-.log Vmtest writes extended log information to this file. The default puts the log along side the TOPDIR. - ``CURTIN_VMTEST_IMAGE_SYNC``: default false (boolean) If set to true, each run will attempt a sync of images. If you want to make sure images are always up to date, then set to true. - ``CURTIN_VMTEST_BRIDGE``: ``user`` The network devices will be attached to this bridge. The default is ``user``, which means to use qemu user mode networking. Set it to ``virbr0`` or ``lxdbr0`` to use those bridges and then be able to ssh in directly. - ``CURTIN_VMTEST_BOOT_TIMEOUT``: default 300 timeout before giving up on the boot of the installed system. - ``CURTIN_VMTEST_INSTALL_TIMEOUT``: default 3000 timeout before giving up on installation. - ``CURTIN_VMTEST_PARALLEL``: default '' only supported through ./tools/jenkins-runner . - ``-1``: then run one per core. - ``0`` or ``''``: run with no parallel - ``>0``: run with N processes This modifies the invocation of nosetets to add '--processes' and other necessary nose arguments (--process-timeout) - ``CURTIN_VMTEST_NR_CPUS``: default '' Allow environment to override the number of virtual cpus to allocate in the target virtual machines. - ``CURTIN_VMTEST_IMAGE_DIR``: default /srv/images Vmtest keeps a mirror of maas ephemeral images in this directory. - ``IMAGES_TO_KEEP``: default 1 Controls the number of images of each release retained in the IMAGE_DIR. - ``CURTIN_VMTEST_EXTRA_CONFIG``: default '' This can be set to a valid path to a config yaml. That can be used to change behaviour of the tests however a current debugging session needs it. The following example shows how it can be used for tests against a ppa, but this can also be used to test proposed or actually any modification to ephemeral or target as needed:: # example ppa to test into install environment early_commands: 10_add_ppa: ['sh', '-xc', 'DEBIAN_FRONTEND=noninteractive add-apt-repository --yes '] # update & upgrade what is there already 97_update: ['apt-get', 'update'] 98_upgrade: ['sh', '-xc', 'DEBIAN_FRONTEND=noninteractive apt-get upgrade --yes'] # example ppa into target environment via apt feature apt: sources: ignored1: source: "" # example of any other modification early_commands: 01_something: ['sh', '-xc', ''] # in target late_commands: 02_something: ['sh', '-xc', 'curtin in-target -- '] - ``CURTIN_VMTEST_ISCSI_PORTAL``: default '' By default, iSCSI tests are skipped when running `make vmtest`, as iSCSI server configuration is necessary. ``tools/jenkins-runner`` will configure a ``tgt`` server if possible and set the necessary environment variables. If an accessible iSCSI server is available, it can be specified in this environment variable as ``HOST:PORT``. ``HOST`` can be a hostname, IPv4 address or IPv6 address. If an IPv6 address is used, it must be enclosed in ``[]``. Additionally, if a ``tgt`` server is running locally as the iSCSI server and is configured to listen on a non-default socket, it is necessary to specify ``TGT_IPC_SOCKET`` to indicate the path to the socket in use. As iSCSI server configuration by-hand can be difficult, there is a script in ``tools/find-tgt`` which can be used to run a local ``tgt`` server. It will find an available port and use the default route-able IPv4 address on the system. The script takes a directory as parameter, and will emit a ``info`` file in that directory which can be sourced as a shell script to set the relevant environment variables needed to run the iSCSI vmtests. For example:: mkdir output ./tools/find-tgt output . output/info nosetests3 tests/vmtests/test_iscsi.py Or, using ``jenkins-runner``: ./tools/jenkins-runner tests/vmtests/test_iscsi.py - ``CURTIN_VMTEST_SKIP_BY_DATE_BUGS``: default '' Curtin's vmtest code has a function 'skip_by_date' which is used to skip tests based until a certain date so developers can add a test for a bug that is not yet fixed in another package. Each skip_by_date call lists a bug by its bug number. This variable can be set to a comma separated list of bug numbers that should raise a SkipTest even if the 'fixby' date is already passed. The special value '*' will cause all skip_by_date calls to skip. This allows us to avoid failures when running curtin from an Ubuntu package or from some other "stale" source. - ``CURTIN_VMTEST_ADD_REPOS``: default '' This is a comma delimited list of apt repositories that will be added to the target environment. If there are repositories provided here, the and CURTIN_VMTEST_SYSTEM_UPGRADE is at its default setting (auto), then a upgrade will be done to make sure to include any new packages. The string 'proposed' is handled specially. It will enable the Ubuntu -proposed pocket for non-devel releases. If you wish to test the -proposed pocket for a devel release, use 'PROPOSED'. - ``CURTIN_VMTEST_SYSTEM_UPGRADE``: default 'auto' The default setting of 'auto' means to do a system upgrade if there are additional repos added. To enable this explicitly, set to any value other than case insensitive "false" or "0" or an empty string. - ``CURTIN_VMTEST_UPGRADE_PACKAGES``: default '' This is a comma delimited string listing packages that should have an 'apt-get install' done to them in curtin late commands. Environment 'boolean' values ============================ For boolean environment variables the value is considered True if it is any value other than case insensitive 'false', '' or "0". Test Class Variables ==================== The base VMBaseClass defines several variables that help creating a new test easily. Among those the common ones are: Generic: - ``arch_skip``: If a test is not supported on an architecture it can list the arch in this variable to auto-skip the test if executed on that arch. - ``conf_file``: The configuration that will be processed by this vmtest. - ``extra_kern_args``: Extra arguments to the guest kernel on boot. Data Collection: - ``collect_scripts``: The commands run when booting into the installed environment to collect the data for the test to verify a proper execution. - ``boot_cloudconf``: Extra cloud-init config content for the install phase. This allows to gather content of the install phase if needed for test verification. Disk Setup: - ``disk_block_size``: Default block size ``512`` bytes. - ``disk_driver``: Default block device driver is ``virtio-blk``. iSCSI Setup: - ``iscsi_disks``: