The "auto" documentation states:
That means they should run with every QEMU binary (also non-x86)
which is not the case as the check-system-fedora build which only
includes a rag tag group of rare and deprecated targets doesn't
support the virtio device required.
Fixes: ef9bba1484 ("quorum: Implement bdrv_co_block_status()")
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20210105100402.12350-1-alex.bennee@linaro.org>
- New block filter: preallocate (which, on writes beyond an image file's
end, allocates big chunks of data so that such post-EOF writes will
occur less frequently)
- write-zeroes and block-status support for Quorum
- Implementation of truncate for the nvme block driver similarly to the
existing implementations for host block devices and iscsi devices
- Block layer refactoring: Drop the tighten_restrictions concept in the
block permission functions
- iotest fixes
-----BEGIN PGP SIGNATURE-----
iQFGBAABCAAwFiEEkb62CjDbPohX0Rgp9AfbAGHVz0AFAl/cwIoSHG1yZWl0ekBy
ZWRoYXQuY29tAAoJEPQH2wBh1c9AnvMH+gOnZCwEUKWuBxGX3Wjb/kqV1OuhAhcP
IVrKLRnqdarCYMQ9M4SZL6pedfsujHA7vClTV7NTrenXBsEIradBQ59ztQ0oDirS
4ipIjVtNqj7m86l+IRZDq5HlwOYwwFnWogmLo2bcmNJGLpPQQfrhL2vRJ1wLgFYk
WjeAVlkkYcHnTIDvs4ne9WRSlxGVBWJ4X5nSlRdZqeyUcMY9v4wL4P9Wc4ZuORmq
/5HRcT5JKGaT2bAueaqAGEdtPFGbazEP5uU7MTTK/fueDKIRAXO2d0gqhANtOOJQ
7hMmKhwOPOOhrrpCVi9nxsVwdCOHfurV0km6cOs+Iprm/Wm2UtuS/A8=
=z+7k
-----END PGP SIGNATURE-----
Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2020-12-18' into staging
Block patches:
- New block filter: preallocate (which, on writes beyond an image file's
end, allocates big chunks of data so that such post-EOF writes will
occur less frequently)
- write-zeroes and block-status support for Quorum
- Implementation of truncate for the nvme block driver similarly to the
existing implementations for host block devices and iscsi devices
- Block layer refactoring: Drop the tighten_restrictions concept in the
block permission functions
- iotest fixes
# gpg: Signature made Fri 18 Dec 2020 14:45:30 GMT
# gpg: using RSA key 91BEB60A30DB3E8857D11829F407DB0061D5CF40
# gpg: issuer "mreitz@redhat.com"
# gpg: Good signature from "Max Reitz <mreitz@redhat.com>" [full]
# Primary key fingerprint: 91BE B60A 30DB 3E88 57D1 1829 F407 DB00 61D5 CF40
* remotes/maxreitz/tags/pull-block-2020-12-18: (30 commits)
iotests: Fix _send_qemu_cmd with bash 5.1
iotests/102: Pass $QEMU_HANDLE to _send_qemu_cmd
block/nvme: Implement fake truncate() coroutine
quorum: Implement bdrv_co_pwrite_zeroes()
quorum: Implement bdrv_co_block_status()
scripts/simplebench: add bench_prealloc.py
simplebench/results_to_text: make executable
simplebench/results_to_text: add difference line to the table
simplebench/results_to_text: improve view of the table
simplebench: move results_to_text() into separate file
simplebench: rename ascii() to results_to_text()
scripts/simplebench: use standard deviation for +- error
scripts/simplebench: support iops
scripts/simplebench: fix grammar: s/successed/succeeded/
iotests: add 298 to test new preallocate filter driver
iotests.py: execute_setup_common(): add required_fmts argument
iotests: qemu_io_silent: support --image-opts
qemu-io: add preallocate mode parameter for truncate command
block: introduce preallocate filter
block: bdrv_check_perm(): process children anyway
...
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
With bash 5.1, the output of the following script changes:
a=("double space")
a=${a[@]:0:1}
echo "$a"
from "double space" to "double space", i.e. all white space is
preserved as-is. This is probably what we actually want here (judging
from the "...to accommodate pathnames with spaces" comment), but before
5.1, we would have to quote the ${} slice to get the same behavior.
In any case, without quoting, the reference output of many iotests is
different between bash 5.1 and pre-5.1, which is not very good. The
output of 5.1 is what we want, so whatever we do to get pre-5.1 to the
same result, it means we have to fix the reference output of basically
all tests that invoke _send_qemu_cmd (except the ones that only use
single spaces in the commands they invoke).
Instead of quoting the ${} slice (cmd="${$@: 1:...}"), we can also just
not use array slicing and replace the whole thing with a simple "cmd=$1;
shift", which works because all callers quote the whole $cmd argument
anyway.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20201217153803.101231-3-mreitz@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
The first parameter passed to _send_qemu_cmd is supposed to be the
$QEMU_HANDLE. 102 does not do so here, fix it.
As a result, the output changes: Now we see the prompt this command is
supposedly waiting for before the resize message - as it should be.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20201217153803.101231-2-mreitz@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
This simply calls bdrv_co_pwrite_zeroes() in all children.
bs->supported_zero_flags is also set to the flags that are supported
by all children.
Signed-off-by: Alberto Garcia <berto@igalia.com>
Message-Id: <2f09c842781fe336b4c2e40036bba577b7430190.1605286097.git.berto@igalia.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
The quorum driver does not implement bdrv_co_block_status() and
because of that it always reports to contain data even if all its
children are known to be empty.
One consequence of this is that if we for example create a quorum with
a size of 10GB and we mirror it to a new image the operation will
write 10GB of actual zeroes to the destination image wasting a lot of
time and disk space.
Since a quorum has an arbitrary number of children of potentially
different formats there is no way to report all possible allocation
status flags in a way that makes sense, so this implementation only
reports when a given region is known to contain zeroes
(BDRV_BLOCK_ZERO) or not (BDRV_BLOCK_DATA).
If all children agree that a region contains zeroes then we can return
BDRV_BLOCK_ZERO using the smallest size reported by the children
(because all agree that a region of at least that size contains
zeroes).
If at least one child disagrees we have to return BDRV_BLOCK_DATA.
In this case we use the largest of the sizes reported by the children
that didn't return BDRV_BLOCK_ZERO (because we know that there won't
be an agreement for at least that size).
Signed-off-by: Alberto Garcia <berto@igalia.com>
Tested-by: Tao Xu <tao3.xu@intel.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: <db83149afcf0f793effc8878089d29af4c46ffe1.1605286097.git.berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20201021145859.11201-13-vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
Add a parameter to skip test if some needed additional formats are not
supported (for example filter drivers).
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20201021145859.11201-12-vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20201021145859.11201-11-vsementsov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
device[NUMBER] thing in QOM path is not stable and tracking it during
code modifications is not fun. Let's filter it like it's already done
in iotest 186.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20201216095205.526235-3-vsementsov@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
According to original commit, that added this filter (627f607e3d),
the problematic thing in qom path is device[NUMBER], not the whole
path. Seems that tracking the other parts of the path in iotest output
is not bad. Let's make _filter_qom_path stricter.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20201216095205.526235-2-vsementsov@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Commit 8b1170012b has added a global maximum disk length for the block
layer, so the error message when creating an overly large disk has
changed.
Fixes: 8b1170012b
("block: introduce BDRV_MAX_LENGTH")
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20201214175158.299919-1-mreitz@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
We are going to modify block layer to work with 64bit requests. And
first step is moving to int64_t type for both offset and bytes
arguments in all block request related functions.
It's mostly safe (when widening signed or unsigned int to int64_t), but
switching from uint64_t is questionable.
So, let's first establish the set of requests we want to work with.
First signed int64_t should be enough, as off_t is signed anyway. Then,
obviously offset + bytes should not overflow.
And most interesting: (offset + bytes) being aligned up should not
overflow as well. Aligned to what alignment? First thing that comes in
mind is bs->bl.request_alignment, as we align up request to this
alignment. But there is another thing: look at
bdrv_mark_request_serialising(). It aligns request up to some given
alignment. And this parameter may be bdrv_get_cluster_size(), which is
often a lot greater than bs->bl.request_alignment.
Note also, that bdrv_mark_request_serialising() uses signed int64_t for
calculations. So, actually, we already depend on some restrictions.
Happily, bdrv_get_cluster_size() returns int and
bs->bl.request_alignment has 32bit unsigned type, but defined to be a
power of 2 less than INT_MAX. So, we may establish, that INT_MAX is
absolute maximum for any kind of alignment that may occur with the
request.
Note, that bdrv_get_cluster_size() is not documented to return power
of 2, still bdrv_mark_request_serialising() behaves like it is.
Also, backup uses bdi.cluster_size and is not prepared to it not being
power of 2.
So, let's establish that Qemu supports only power-of-2 clusters and
alignments.
So, alignment can't be greater than 2^30.
Finally to be safe with calculations, to not calculate different
maximums for different nodes (depending on cluster size and
request_alignment), let's simply set QEMU_ALIGN_DOWN(INT64_MAX, 2^30)
as absolute maximum bytes length for Qemu. Actually, it's not much less
than INT64_MAX.
OK, then, let's apply it to block/io.
Let's consider all block/io entry points of offset/bytes:
4 bytes/offset interface functions: bdrv_co_preadv_part(),
bdrv_co_pwritev_part(), bdrv_co_copy_range_internal() and
bdrv_co_pdiscard() and we check them all with bdrv_check_request().
We also have one entry point with only offset: bdrv_co_truncate().
Check the offset.
And one public structure: BdrvTrackedRequest. Happily, it has only
three external users:
file-posix.c: adopted by this patch
write-threshold.c: only read fields
test-write-threshold.c: sets obviously small constant values
Better is to make the structure private and add corresponding
interfaces.. Still it's not obvious what kind of interface is needed
for file-posix.c. Let's keep it public but add corresponding
assertions.
After this patch we'll convert functions in block/io.c to int64_t bytes
and offset parameters. We can assume that offset/bytes pair always
satisfy new restrictions, and make
corresponding assertions where needed. If we reach some offset/bytes
point in block/io.c missing bdrv_check_request() it is considered a
bug. As well, if block/io.c modifies a offset/bytes request, expanding
it more then aligning up to request_alignment, it's a bug too.
For all io requests except for discard we keep for now old restriction
of 32bit request length.
iotest 206 output error message changed, as now test disk size is
larger than new limit. Add one more test case with new maximum disk
size to cover too-big-L1 case.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20201203222713.13507-5-vsementsov@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
See the new comment for why this should be done.
I do not have a reproducer on master, but when using FUSE block exports,
this test breaks depending on the underlying filesystem (for me, it
works on tmpfs, but fails on xfs, because the block allocated by
file-posix has 16 kB there instead of 4 kB).
Suggested-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20201207152245.66987-1-mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
We have good coverage of the normal I/O paths now, but what remains is a
test that tests some more special cases: Exporting an image on itself
(thus turning a formatted image into a raw one), some error cases, and
non-writable and non-growable exports.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20201027190600.192171-21-mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Many tests (that do not support generic protocols) can run just fine
with FUSE-exported images, so allow them to. Note that this is no
attempt at being definitely complete. There are some tests that might
be modified to run on FUSE, but this patch still skips them. This patch
only tries to pick the rather low-hanging fruits.
Note that 221 and 250 only pass when .lseek is correctly implemented,
which is only possible with a libfuse that is 3.8 or newer.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20201027190600.192171-20-mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
This pretends FUSE exports are a kind of protocol. As such, they are
always tested under the format node. This is probably the best way to
test them, actually, because this will generate more I/O load and more
varied patterns.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20201027190600.192171-19-mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
287 creates an image in a subshell (thanks to the pipe) to see whether
that is possible with compression_type=zstd. If _make_test_img were to
modify any global state, this global state would then be lost before we
could cleanup the image.
When using FUSE as the test protocol, this global state is important, so
clean up the image before the state is lost.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20201027190600.192171-16-mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
When most iotests want to create a test image that is named differently
from the default $TEST_IMG, they do something like this:
TEST_IMG="$TEST_IMG.base" _make_test_img $options
This works fine with the "file" protocol, but not so much for anything
else: _make_test_img tries to create an image under $TEST_IMG_FILE
first, and only under $TEST_IMG if the former is not set; and on
everything but "file", $TEST_IMG_FILE is set.
There are two ways we can fix this: First, we could make all tests
adjust not only TEST_IMG, but also TEST_IMG_FILE if that is present
(e.g. with something like _set_test_img_suffix $suffix that would affect
not only TEST_IMG but also TEST_IMG_FILE, if necessary). This is a
pretty clean solution, and this is maybe what we should have done from
the start.
But it would also require changes to most existing bash tests. So the
alternative is this: Let _make_test_img see whether $TEST_IMG_FILE still
points to the original value. If so, it is possible that the caller has
adjusted $TEST_IMG but not $TEST_IMG_FILE. In such a case, we can (for
most protocols) derive the corresponding $TEST_IMG_FILE value from
$TEST_IMG value and thus work around what technically is the caller
misbehaving.
This second solution is less clean, but it is robust against people
keeping their old habit of adjusting TEST_IMG only, and requires much
less changes. So this patch implements it.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20201027190600.192171-15-mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Most Python tests are restricted to the file protocol (without
explicitly saying so), but these are the ones that would break
./check -fuse -qcow2.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20201027190600.192171-14-mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
If the test environment has some other child processes running (like a
storage daemon that provides a FUSE export), then "wait" will never
finish. Use wait=yes _cleanup_qemu instead.
(We need to discard the output so there is no change to the reference
output.)
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20201027190600.192171-13-mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Avoid creating images with custom filenames in $TEST_DIR, because
non-file protocols may want to keep $TEST_IMG (and all other test
images) in some other directory.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20201027190600.192171-12-mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
This generally does not work on non-file protocols. It is better to
create the image with the final name from the start, and most tests do
this already. Let 046 follow suit.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20201027190600.192171-11-mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
qemu-img convert (without -n) can often be replaced by a combination of
_make_test_img + qemu-img convert -n. Doing so allows converting to
protocols that do not allow direct file creation, such as FUSE exports.
The only problem is that for formats other than qcow2 and qed (qcow1 at
least), this may lead to high disk usage for some reason, so we cannot
do it everywhere.
But we can do it in 028 and 089, so let us do that so they can run on
FUSE exports. Also, in 028 this allows us to remove a 9-line comment
that used to explain why we cannot safely filter drive-backup's image
creation output.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20201027190600.192171-10-mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Executing _make_test_img as part of a pipe will undo all variable
changes it has done. As such, this could not work with FUSE (because
we want to remember all of our exports and their qemu instances).
Replace the pipe by a temporary file in 071 and 174 (the two tests that
can run on FUSE).
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20201027190600.192171-9-mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
In most cases, _make_test_img does not need a _filter_imgfmt on top. It
does that by itself.
(The exception is when IMGFMT has been overwritten but TEST_IMG has not.
In such cases, we do need a _filter_imgfmt on top to filter the test's
original IMGFMT from TEST_IMG.)
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20201027190600.192171-8-mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
block-commit defaults @base-node to the deepest backing image. When
there is none, it fails with "Base 'NULL' not found". Improve to
"There is no backing image".
block-commit and block-stream reject a @base argument that doesn't
resolve with "Base 'BASE' not found". Commit 6b33f3ae8b "qemu-img:
Improve commit invalid base message" improved this message in
qemu-img. Improve it here, too: "Can't find '%s' in the backing
chain".
QERR_BASE_NOT_FOUND is now unused. Drop.
Cc: Kevin Wolf <kwolf@redhat.com>
Cc: Max Reitz <mreitz@redhat.com>
Cc: qemu-block@nongnu.org
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20201113082626.2725812-4-armbru@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Test what happens when a rewrite-corrupted quorum node performs such a
rewrite, while there is no parent that has taken the WRITE permission.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20201113211718.261671-4-mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Otherwise, this breaks whenever the test directory contains the image
format (e.g. "/tmp/test-raw-file" is filtered to "/tmp/test-IMGFMT-file"
instead of "TEST_DIR").
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20201113211718.261671-3-mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
iotest 277 fails on Fedora 33 (Python 3.9) because a deprecation warning
changes the output:
nbd-fault-injector.py:230: DeprecationWarning: This method will be
removed in future versions. Use 'parser.read_file()' instead.
In fact, readfp() has already been deprecated in Python 3.2 and the
replacement has existed since the same version, so we can now
unconditionally switch to read_file().
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20201113100602.15936-1-kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
The recent changes that brought RCU delayed device deletion,
broke few tests and this test breakage went unnoticed.
Fix this test by rewriting it in python
(which allows to wait for DEVICE_DELETED events before continuing).
Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>
Tested-by: Christian Borntraeger <borntraeger@de.ibm.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20201104185025.434703-3-mlevitsk@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
filter_qmp_virtio_scsi can be used to filter virtio-scsi-pci/ccw differences.
Note that this patch was only tested on x86.
Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>
Tested-by: Christian Borntraeger <borntraeger@de.ibm.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20201104185025.434703-2-mlevitsk@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
pylint complains about the use of super with the current class and
instance as arguments in VM.__init__():
iotests.py:546:8: R1725: Consider using Python 3 style super() without arguments (super-with-arguments)
No reason not to follow the advice and make it happy, so let's do this.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20201027163806.290960-4-kwolf@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
When run with Python 3.9, pylint incorrectly warns about things like
Optional[foo] because it doesn't recognise Optional as unsubscriptable.
This is a known pylint bug:
https://github.com/PyCQA/pylint/issues/3882
Just disable this check to get rid of the warnings.
Disabling this shouldn't make us miss any real bug because mypy also
has a similar check ("... is not indexable").
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20201027163806.290960-3-kwolf@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Commit 1847a4a8c2 clarified that event_wait() can return None (though
only with timeout=0) and commit f12a282ff4 annotated it as returning
Optional[QMPMessage].
Type checks in wait_migration() fail because of the unexpected optional
return type:
iotests.py:750: error: Value of type variable "Msg" of "log" cannot be "Optional[Dict[str, Any]]"
iotests.py:751: error: Value of type "Optional[Dict[str, Any]]" is not indexable
iotests.py:754: error: Value of type "Optional[Dict[str, Any]]" is not indexable
Fortunately, the non-zero default timeout is used in the event_wait()
call, so we can make mypy happy by just asserting this.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20201027163806.290960-2-kwolf@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Allow the server to expose an additional metacontext to be requested
by savvy clients. qemu-nbd adds a new option -A to expose the
qemu:allocation-depth metacontext through NBD_CMD_BLOCK_STATUS; this
can also be set via QMP when using block-export-add.
qemu as client is hacked into viewing the key aspects of this new
context by abusing the already-experimental x-dirty-bitmap option to
collapse all depths greater than 2, which results in a tri-state value
visible in the output of 'qemu-img map --output=json' (yes, that means
x-dirty-bitmap is now a bit of a misnomer, but I didn't feel like
renaming it as it would introduce a needless break of back-compat,
even though we make no compat guarantees with x- members):
unallocated (depth 0) => "zero":false, "data":true
local (depth 1) => "zero":false, "data":false
backing (depth 2+) => "zero":true, "data":true
libnbd as client is probably a nicer way to get at the information
without having to decipher such hacks in qemu as client. ;)
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20201027050556.269064-11-eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
With this, 'qemu-nbd -B b0 -B b1 -f qcow2 img.qcow2' can let you sniff
out multiple bitmaps from one server. qemu-img as client can still
only read one bitmap per client connection, but other NBD clients
(hello libnbd) can now read multiple bitmaps in a single pass.
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20201027050556.269064-8-eblake@redhat.com>
nbd_server_start_unix_socket() includes an implicit nbd_server_stop(),
but we still need an explicit one at the end of the test (where there
follows no next nbd_server_start_unix_socket()), or qemu-nbd will linger
until the test exits.
This will become important when enabling this test to run on FUSE
exports, because then the export (which is the image used by qemu-nbd)
will go away before qemu-nbd exits, which will lead to qemu-nbd
complaining that it cannot flush the bitmaps in the image.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20201027164416.144115-3-mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
We need to let _img_info emit the format-specific information so we get
the list of bitmaps we want, but we do not need anything but the
bitmaps. So filter out everything that is irrelevant to us. (Ideally,
this would be a generalized function in common.filters that takes a list
of things to keep, but that would require implementing an anti-bitmap
filter, which would be hard, and which we do not need here. So that is
why this function is just a local hack.)
This lets 291 pass with qcow2 options like refcount_bits or data_file
again.
Fixes: 14f16bf947
("qemu-img: Support bitmap --merge into backing image")
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20201027164416.144115-2-mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
Reporting "Failed to connect socket" is essentially useless for a user
attempting to diagnose failure. It needs to include the target address
details. Similarly when failing to create a socket we should include the
socket family info, so the user understands what particular feature was
missing in their kernel build (IPv6, VSock in particular).
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
These cases are fixed by previous patches around block_status and
is_allocated.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Message-id: 20200924194003.22080-6-vsementsov@virtuozzo.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
mypy wants to ensure there's consistency between the kwargs arguments
types and any unspecified keyword arguments. In this case, conv_keys is
a bool, but the remaining keys are Any type. Mypy (correctly) infers the
**kwargs type to be **Dict[str, str], which is not compatible with
conv_keys: bool.
Because QMP typing is a little fraught right now anyway, re-type kwargs
to Dict[str, Any] which has the benefit of silencing this check right
now.
A future re-design might type these more aggressively, but this will
give us a baseline to work from with minimal disruption.
(Thanks Kevin Wolf for the debugging assist here)
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 20201006235817.3280413-11-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
downtime in migration test, less verbose output when running w/o KVM)
* Improve handling of acceptance tests in the Gitlab-CI
* Run checkpatch.pl in the Gitlab-CI
* Improve the gitlab-pipeline-status script
* Misc patches (mark 'moxie' as deprecated, remove stale .gitignore files, ...)
-----BEGIN PGP SIGNATURE-----
iQJFBAABCAAvFiEEJ7iIR+7gJQEY8+q5LtnXdP5wLbUFAl+FhiIRHHRodXRoQHJl
ZGhhdC5jb20ACgkQLtnXdP5wLbXfcBAAkMc4eUbZ0Wkm7M7TdIRkn1vQEstgvyJN
6t02MuqY0R01rdbIBAnCLSw9okxfCTf7Q33VmC7snLtPo6WmvYIPAXZAnUiz13K1
hGhMJfEY0JSyPEXlENMC/SWcRfNuHud6OPp6KePvn6EQsVZ5CR9SeO5zMsCVj2SP
bMaBYIAJsVCEHkR2lq9UXbjckjyO0GQnQ/oR3mNiqDLYBmrXUOxIFMBctgfbuUtm
uPuvvknHVQa8foD18qVJ8QYZrpwrqN4edFjcoW3yvwfX6OOhTnx+pY43BG/of9YB
OoRY7V4VN8aYmVR08sqyn6PRNpXW9WcSUn8D3JNeiAhLzO/8H197JhHwFVvbZc7t
puLECIINy91wH2i3Onx7HWhss3XLUK3HsvWNLrvLui6vdbFHEtiW2/0GbwJzrcA0
a9inH7bvI7BlPiIau/J7goaDv0fzZ7xVXlQcrM8hC9oCWH5gvmvcgTBWJn/5OxUZ
fov3iFxcRWslFSQe+D66gBceIl/fScF+TUmPoWyeSlD/f1OR2WW+q8N1FvnbLflz
oPutIoja8b6CobzAzp8Igc6/9uQvzCAFB92Y8q1Og7eguQybw7dDtbArjBmjUBVi
slFWoY8/ri2+uyiPsyU13Yfu9N5myqdwIQeM7H8sQ7qS40QHp0z2tj18o951xH2w
WJv3PlGcez4=
=lCRK
-----END PGP SIGNATURE-----
Merge remote-tracking branch 'remotes/huth-gitlab/tags/pull-request-2020-10-13' into staging
* qtest improvements (test for crash found with the fuzzer, increase
downtime in migration test, less verbose output when running w/o KVM)
* Improve handling of acceptance tests in the Gitlab-CI
* Run checkpatch.pl in the Gitlab-CI
* Improve the gitlab-pipeline-status script
* Misc patches (mark 'moxie' as deprecated, remove stale .gitignore files, ...)
# gpg: Signature made Tue 13 Oct 2020 11:49:06 BST
# gpg: using RSA key 27B88847EEE0250118F3EAB92ED9D774FE702DB5
# gpg: issuer "thuth@redhat.com"
# gpg: Good signature from "Thomas Huth <th.huth@gmx.de>" [full]
# gpg: aka "Thomas Huth <thuth@redhat.com>" [full]
# gpg: aka "Thomas Huth <huth@tuxfamily.org>" [full]
# gpg: aka "Thomas Huth <th.huth@posteo.de>" [unknown]
# Primary key fingerprint: 27B8 8847 EEE0 2501 18F3 EAB9 2ED9 D774 FE70 2DB5
* remotes/huth-gitlab/tags/pull-request-2020-10-13: (23 commits)
scripts/ci/gitlab-pipeline-status: wait for pipeline creation
scripts/ci/gitlab-pipeline-status: use more descriptive exceptions
scripts/ci/gitlab-pipeline-status: handle keyboard interrupts
scripts/ci/gitlab-pipeline-status: refactor parser creation
scripts/ci/gitlab-pipeline-status: give early feedback on running pipelines
scripts/ci/gitlab-pipeline-status: improve message regarding timeout
scripts/ci/gitlab-pipeline-status: make branch name configurable
gitlab: assign python helper files to GitLab maintainers section
gitlab: add a CI job to validate the DCO sign off
gitlab: add a CI job for running checkpatch.pl
configure: fixes indent of $meson setup
docs/system/deprecated: Mark the 'moxie' CPU as deprecated
Remove superfluous .gitignore files
MAINTAINERS: Ignore bios-tables-test in the qtest section
Add a comment in bios-tables-test.c to clarify the reason behind approach
softmmu/vl: Be less verbose about missing KVM when running the qtests
tests/migration: Allow longer timeouts
qtest: add fuzz test case
Acceptance tests: show test report on GitLab CI
Acceptance tests: do not show canceled test logs on GitLab CI
...
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Since we are now always doing out-of-tree builds, these gitignore
files should not be necessary anymore.
Message-Id: <20200919133637.72744-1-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Test 067 from qemu-iotests is executing QMP commands to hotplug
and hot-unplug disks, devices and blockdevs. Because the power
of the text-based test harness is limited, it is actually limiting
the checks that it does, for example by skipping DEVICE_DELETED
events.
tests/qtest already has a similar test, drive_del-test.c.
We can merge them, and even reuse some of the existing code in
drive_del-test.c. This will improve the quality of the test by
covering DEVICE_DELETED events and testing multiple architectures
(therefore covering multiple PCI hotplug mechanisms as well as s390x
virtio-ccw).
The only difference is that the new test will always use null-co:// for
the medium rather than qcow2 or raw, but this should be irrelevant for
what the test is covering. For example there are no "qemu-img check"
runs in 067 that would check that the file is properly closed.
The new tests requires PCI hot-plug support, so drive_del-test
is moved from qemu-system-ppc to qemu-system-ppc64.
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Saving icount as a parameters of the snapshot allows navigation between
them in the execution replay scenario.
This information can be used for finding a specific snapshot for proceeding
the recorded execution to the specific moment of the time.
E.g., 'reverse step' action (introduced in one of the following patches)
needs to load the nearest snapshot which is prior to the current moment
of time.
This patch also updates snapshot test which verifies qemu monitor output.
Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
Acked-by: Markus Armbruster <armbru@redhat.com>
Acked-by: Kevin Wolf <kwolf@redhat.com>
--
v4 changes:
- squashed format update with test output update
v7 changes:
- introduced the spaces between the fields in snapshot info output
- updated the test to match new field widths
Message-Id: <160174518865.12451.14327573383978752463.stgit@pasha-ThinkPad-X280>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
This patch introduces the icount field for saving within the snapshot.
It is required for navigation between the snapshots in record/replay mode.
Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
Acked-by: Kevin Wolf <kwolf@redhat.com>
--
v7 changes:
- also fix the test which checks qcow2 snapshot extra data
Message-Id: <160174518284.12451.2301137308458777398.stgit@pasha-ThinkPad-X280>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
