Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial LISAv3 via Pytest review #1065

Closed
wants to merge 84 commits into from

Conversation

andyleejordan
Copy link
Member

@andyleejordan andyleejordan commented Oct 21, 2020

I created the branch pytest/main and enabled branch protections on it so all code goes through review. This PR supersedes #1044 which was an initial demo. There are TODOs throughout, which I'd like to work with @lpereira to prioritize and work-on to finish this proof-of-concept, but I'd like to get what does exist reviewed before it goes any further. I have attempted to document design decisions and alternatives considered in pytest/README.md, and have setup some basic CI with a self-test and semantic analsysis.

The smoke test (when Azure is working, which it hasn't been today) demonstrates a runtime-configured matrix of images to test with the criteria given to me. The LIS test was my first test for the initial proof-of-concept, and I'd like to use @lpereira's XML->Pytest generator to lift-and-shift as many LISAv2 tests as possible. Anything that just runs existing Bash scripts can be made to run with this we believe practically automatically. The XDP test is a work-in-progress because it requires two NICs, which is a similar problem as deploying two nodes for the ntttcp test, but these are both obviously tractable problems that simply require small investment in scripting around az. I know @squirrelsc's big concern is enabling a YAML playbook-based test selection and environment creation, but as can be seen by the parameterization of the smoke test, this is also a tractable problem.

I borrowed the CoC and contributing guidelines from main branch, from where we can and should pull more in.

Please give this a thorough review. The implementation is still <200 lines of code, with another ~100 for the written tests, so it should be very straightforward to read. I would suggest using the file filter to look at just *.py.

This is a _working_ test.
Extend Fabric’s Connection class with a new (and simple) command “cat”
which return the value of a remote file as a string.

Setup a Config for the Connection when creating it that echoes every
command, disables the stdin forwarding (since we’re running under
Pytest), and fixes the PATH since the remote commands don’t run under a
login shell.
This could be extended to instead deploy a host of the specified distro.
Most likely we’ll want a command-line parameter that the fixture uses to
create a Node with the given requirements, and then tests will be
skipped if their requirements aren’t met.

Further more, the mark here is very simple. It can instead take keyword
arguments, which would map to our metadata.
As supplying types for these would be supremely annoying.
Accidentally eliminated Fabric’s default overrides of Invoke by
supplying my own config based on `invoke.Config` to Fabric. Oops.
Running with ‘-n 1’ takes 10 seconds, or 3 seconds for each feature in
sequence plus overhead. Running with ‘-n 8’ takes 4 seconds, 3 seconds
for each in parallel plus overhead.
Makefile Outdated Show resolved Hide resolved
pytest/conftest.py Outdated Show resolved Hide resolved
@andyleejordan andyleejordan force-pushed the andschwa/initial-pytest-review branch 2 times, most recently from e400dcb to b5eb4cc Compare November 6, 2020 23:05
Example from `make test`:

```
Created target: set() / {'platform': 'Azure', 'image': 'citrix:netscalervpx-130:netscalerbyol:latest', 'sku': 'Standard_DS1_v2'}
Created target: set() / {'platform': 'Azure', 'image': 'audiocodes:mediantsessionbordercontroller:mediantvirtualsbcazure:latest', 'sku': 'Standard_DS1_v2'}
Created target: set() / {'platform': 'Azure', 'image': 'credativ:Debian:9:9.0.201706190', 'sku': 'Standard_DS1_v2'}
Created target: set() / {'platform': 'Azure', 'image': 'github:github-enterprise:github-enterprise:latest', 'sku': 'Standard_DS1_v2'}
Created target: {'xdp'} / {'platform': 'Azure', 'image': 'citrix:netscalervpx-130:netscalerbyol:latest', 'sku': 'Standard_DS1_v2'}
Created target: {'xdp'} / {'platform': 'Azure', 'image': 'audiocodes:mediantsessionbordercontroller:mediantvirtualsbcazure:latest', 'sku': 'Standard_DS1_v2'}
Created target: {'xdp'} / {'platform': 'Azure', 'image': 'credativ:Debian:9:9.0.201706190', 'sku': 'Standard_DS1_v2'}
Created target: {'xdp'} / {'platform': 'Azure', 'image': 'github:github-enterprise:github-enterprise:latest', 'sku': 'Standard_DS1_v2'}
Created target: {'gpu'} / {'platform': 'Azure', 'image': 'citrix:netscalervpx-130:netscalerbyol:latest', 'sku': 'Standard_DS1_v2'}
Created target: {'gpu'} / {'platform': 'Azure', 'image': 'audiocodes:mediantsessionbordercontroller:mediantvirtualsbcazure:latest', 'sku': 'Standard_DS1_v2'}
Created target: {'gpu'} / {'platform': 'Azure', 'image': 'credativ:Debian:9:9.0.201706190', 'sku': 'Standard_DS1_v2'}
Created target: {'gpu'} / {'platform': 'Azure', 'image': 'github:github-enterprise:github-enterprise:latest', 'sku': 'Standard_DS1_v2'}
Created target: {'rdma'} / {'platform': 'Azure', 'image': 'citrix:netscalervpx-130:netscalerbyol:latest', 'sku': 'Standard_DS1_v2'}
Created target: {'rdma'} / {'platform': 'Azure', 'image': 'audiocodes:mediantsessionbordercontroller:mediantvirtualsbcazure:latest', 'sku': 'Standard_DS1_v2'}
Created target: {'rdma'} / {'platform': 'Azure', 'image': 'credativ:Debian:9:9.0.201706190', 'sku': 'Standard_DS1_v2'}
Created target: {'rdma'} / {'platform': 'Azure', 'image':
'github:github-enterprise:github-enterprise:latest', 'sku':
'Standard_DS1_v2'}
```
@andyleejordan
Copy link
Member Author

Closing this in preference of #1107!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🆕 LISAv3 Incubation work for the next version of LISA
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants