Document skips.txt

Author: honno

.github/workflows/numpy.yml

@@ -35,7 +35,7 @@ jobs:
         # https://github.com/numpy/numpy/issues/20870
         array_api_tests/test_data_type_functions.py::test_can_cast
         # The return dtype for trace is not consistent in the spec
-        # (https://github.com/data-apis/array-api/issues/202#issuecomment-952529197)
+        # https://github.com/data-apis/array-api/issues/202#issuecomment-952529197
         array_api_tests/test_linalg.py::test_trace
         # waiting on NumPy to allow/revert distinct NaNs for np.unique
         # https://github.com/numpy/numpy/issues/20326#issuecomment-1012380448

README.md

@@ -151,15 +151,74 @@ library to fail.
 
 ### Configuration
 
+#### CI flag
+
 Use the `--ci` flag to run only the primary and special cases tests. You can
 ignore the other test cases as they are redundant for the purposes of checking
 compliance.
 
+#### Extensions
+
 By default, tests for the optional Array API extensions such as
 [`linalg`](https://data-apis.org/array-api/latest/extensions/linear_algebra_functions.html)
 will be skipped if not present in the specified array module. You can purposely
 skip testing extension(s) via the `--disable-extension` option.
 
+#### Skip test cases
+
+Test cases you want to skip can be specified in a `skips.txt` file in the root
+of this repository, e.g.:
+
+```
+# ./skips.txt
+# Line comments can be denoted with the hash symbol (#)
+
+# Skip specific test case, e.g. when argsort() does not respect relative order
+# https://github.com/numpy/numpy/issues/20778
+array_api_tests/test_sorting_functions.py::test_argsort
+
+# Skip specific test case parameter, e.g. you forgot to implement in-place adds
+array_api_tests/test_add[__iadd__(x1, x2)]
+array_api_tests/test_add[__iadd__(x, s)]
+
+# Skip module, e.g. when your set functions treat NaNs as non-distinct
+# https://github.com/numpy/numpy/issues/20326
+array_api_tests/test_set_functions.py
+```
+
+For GitHub Actions, you might like to keep everything in the workflow config
+instead of having a seperate `skips.txt` file, e.g.:
+
+```yaml
+# ./.github/workflows/array_api.yml
+...
+    ...
+    - name: Run the test suite
+      env:
+        ARRAY_API_TESTS_MODULE: your.array.api.namespace
+      run: |
+        # Skip test cases with known issues
+        cat << EOF >> skips.txt
+
+        # Skip specific test case, e.g. when argsort() does not respect relative order
+        # https://github.com/numpy/numpy/issues/20778
+        array_api_tests/test_sorting_functions.py::test_argsort
+
+        # Skip specific test case parameter, e.g. you forgot to implement in-place adds
+        array_api_tests/test_add[__iadd__(x1, x2)]
+        array_api_tests/test_add[__iadd__(x, s)]
+
+        # Skip module, e.g. when your set functions treat NaNs as non-distinct
+        # https://github.com/numpy/numpy/issues/20326
+        array_api_tests/test_set_functions.py
+
+        EOF
+
+        pytest -v -rxXfE --ci
+```
+
+#### Max examples
+
 The tests make heavy use
 [Hypothesis](https://hypothesis.readthedocs.io/en/latest/). You can configure
 how many examples are generated using the `--max-examples` flag, which defaults