Tutorial#

Basic CLI#

scim2-tester is integrated in scim2-cli:

pip install scim2-cli
export SCIM_CLI_URL="https://auth.example"
scim test

You can check the scim2-cli test command reference for more details.

Code integration#

Warning

The checks create, modify and delete resources on the server. Never run them against a production server.

Note

Some checks list all the resources of a type, which can be slow on a server with many resources.

If you need to integrate the tester in your code, you can initialize a scim2-client engine and pass it to iter_check_server(). It yields each result as soon as its check ends, so you can display the progress of a run:

from httpx2 import Client
from scim2_client.engines.httpx2 import SyncSCIMClient
from scim2_tester import iter_check_server

httpx_client = Client(
    base_url="https://auth.example/scim/v2",
    headers={"Authorization": "Bearer foobar"},
)
scim_client = SyncSCIMClient(httpx_client)
for result in iter_check_server(scim_client):
    print(result.status.name, result.title)

check_server() takes the same parameters, and returns all the results at once. The outcome of a status tells whether the result passes, warns, fails or is skipped. Use it to decide whether the server passes the checks:

from scim2_tester import Outcome
from scim2_tester import check_server

results = check_server(scim_client)
failed = [r for r in results if r.status.outcome is Outcome.FAILED]

scim2-tester supports filtering tests using tags and resource types to run only specific subsets of checks. This is useful for targeted testing or when developing specific features.

Tag-based filtering#

Tests are organized using hierarchical tags that allow fine-grained control over which checks to execute. Use the include_tags and exclude_tags parameters:

from scim2_tester import check_server

# Run only discovery tests
results = check_server(client, include_tags={"discovery"})

# Run all CRUD operations except delete
results = check_server(
    client,
    include_tags={"crud"},
    exclude_tags={"crud:delete"}
)

# Run only resource creation tests
results = check_server(client, include_tags={"crud:create"})

Available tags include:

  • discovery: Configuration endpoint tests (ServiceProviderConfig, ResourceType, Schema)

  • service-provider-config: ServiceProviderConfig endpoint tests

  • resource-types: ResourceType endpoint tests

  • schemas: Schema endpoint tests

  • crud: All CRUD operation tests

  • crud:create: Resource creation tests

  • crud:read: Resource reading tests

  • crud:update: Resource update tests

  • crud:delete: Resource deletion tests

  • misc: Miscellaneous tests

The tag system is hierarchical, so crud will match crud:create, crud:read, etc.

Resource type filtering#

You can also filter tests by resource type using the resource_types parameter:

# Test only User resources
results = check_server(client, resource_types=["User"])

# Test both User and Group resources
results = check_server(client, resource_types=["User", "Group"])

Combining filters#

Filters can be combined for precise control using both include_tags and resource_types parameters:

# Test only User creation and reading
results = check_server(
    client,
    include_tags={"crud:create", "crud:read"},
    resource_types=["User"]
)

Unit test suite integration#

If you build a Python SCIM server application, you can run the checks of scim2-tester in its test suite. With the scim2-client WSGISCIMClient engine, no real HTTP request is made: the server code runs directly. This engine only needs the standard library, and works with any WSGI framework.

pytest suite#

ScimServerContract runs the checks as a pytest test. It needs the testing extra, which installs pytest:

pip install scim2-tester[testing]

Subclass it in your tests, and give it a scim_client fixture:

import pytest
from scim2_client.engines.wsgi import WSGISCIMClient
from scim2_tester.testing import ScimServerContract
from myapp import create_app


class TestMyServer(ScimServerContract):
    @pytest.fixture
    def scim_client(self):
        app = create_app(...)
        return WSGISCIMClient(app, base_url="http://localhost/scim/v2")

The test runs the checks once, and reports each result as a subtest. Successful, compliant and acceptable results pass. Skipped results are skipped. Errors and critical results fail. Run pytest with -v to see each subtest.

Deviations pass with a ScimDeviationWarning. To make them fail, turn the warning into an error:

[tool.pytest.ini_options]
filterwarnings = ["error::scim2_tester.testing.ScimDeviationWarning"]

The class attributes select the checks, as the parameters of check_server() do:

class TestMyServer(ScimServerContract):
    include_tags = {"crud", "patch"}
    exclude_tags = {"patch:remove"}
    resource_types = ["User", "Group"]

Known failures#

When you check a server you do not maintain, some checks may fail until the server is fixed. List them in expected_results, with their status. You can add a reason to the status. The reason is free text, printed in the report, such as a ticket URL or an explanation:

from scim2_tester import Status


class TestThirdPartyServer(ScimServerContract):
    expected_results = {
        ("User", "check_add_attribute", "nickName"): (
            Status.ERROR,
            "https://github.com/example/scim-server/issues/123",
        ),
        (None, "schemas_endpoint_methods", None): Status.DEVIATION,
    }

The key of a result is its resource type, its title and its attribute. A result with the expected status is reported as an expected failure. A result with another status fails, including a result that now passes: remove it from the list. An expected result that the checks no longer return also fails the test. A key applies to all the results that have it. When these results have different statuses and one of them fails, the key cannot be expected, and the test fails.

When the test fails, its output prints the current failures in the format of expected_results. Copy them into your test class, and add a reason where it helps.

Replaying a run#

The checks send random values. When the test fails, its output prints the seed of these values:

scim2-tester seed: 2826105314

Set seed to send the same values again:

class TestMyServer(ScimServerContract):
    seed = 2826105314

Debugging a check#

When a check fails on an exception raised by the server code, the failed subtest shows its traceback. The server must let its exceptions through. For instance, a Flask application does it in testing mode only.

With raise_exceptions, check_server() raises the first error instead of returning it. The traceback leads to the server code that failed:

from scim2_client.engines.wsgi import WSGISCIMClient
from scim2_tester import check_server
from myapp import create_app

def test_scim_tester():
    app = create_app(...)
    client = WSGISCIMClient(app, base_url="http://localhost/scim/v2")
    check_server(client, raise_exceptions=True)