Skip to content

Testing Documentation

Overview

The Digital Landscape application includes a comprehensive testing framework to ensure API endpoints function correctly and reliably. The testing suite focuses primarily on backend API validation, verifying that data is correctly retrieved, filtered, and processed according to specifications.

Testing Architecture

The testing framework is built using Python with pytest and follows these key principles:

  • Isolated Tests: Each test function validates a specific endpoint or functionality
  • Comprehensive Coverage: Tests cover all API endpoints and their various parameters
  • Clear Documentation: Each test includes detailed docstrings explaining purpose and expectations
  • Error Handling Validation: Tests verify proper error responses for invalid inputs
  • Organized Structure: Tests are grouped by API type across multiple files

Test Structure

The tests are organized into three main files:

Test File Endpoint Group Description
test_main.py /api/* Core API endpoints (health, CSV, JSON, repository)
test_admin.py /admin/api/* Admin API endpoints for banner management
test_review.py /review/api/* Review API endpoints for tech radar updates

Test Setup

Prerequisites

  • Python 3.8 or higher
  • Make (for using Makefile commands)
  • Backend server running on localhost:5001

Installation

# Navigate to the testing directory
cd testing

# Create a virtual environment (recommended)
python3 -m venv venv
source venv/bin/activate

# Install dependencies
make setup

Running Tests

The testing framework provides several commands for running tests:

# Run all tests
make test

# Run only core API tests
make test-main

# Run only admin API tests
make test-admin

# Run only review API tests
make test-review

# Run a specific test
python3 -m pytest backend/test_main.py::test_name -v

# Example: Run only the health check test
python3 -m pytest backend/test_main.py::test_health_check -v

Test Categories

The test suite covers the following API endpoints:

Health Check Endpoint

Tests the /api/health endpoint to verify server status and health metrics.

Test the health check endpoint functionality.

This test verifies that the health check endpoint is operational and returns the expected health status information about the server. It checks for the presence of essential health metrics and status indicators.

Endpoint

GET /api/health

Expects
  • 200 status code
  • JSON response containing:
    • "healthy" status indicator
    • Current timestamp
    • Server uptime in seconds
    • Memory usage statistics
    • Process ID
Source code in testing/backend/test_main.py 
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
def test_health_check():
    """Test the health check endpoint functionality.

    This test verifies that the health check endpoint is operational and returns
    the expected health status information about the server. It checks for the
    presence of essential health metrics and status indicators.

    Endpoint:
        GET /api/health

    Expects:
        - 200 status code
        - JSON response containing:
            - "healthy" status indicator
            - Current timestamp
            - Server uptime in seconds
            - Memory usage statistics
            - Process ID
    """
    response = requests.get(f"{BASE_URL}/api/health", timeout=10)
    assert response.status_code == 200
    data = response.json()
    assert data["status"] == "healthy"
    assert "timestamp" in data
    assert "uptime" in data
    assert "memory" in data
    assert "pid" in data

Project Data Endpoint

Tests the /api/csv endpoint that provides project data from CSV sources.

Test the CSV data endpoint functionality.

This test verifies that the CSV endpoint correctly returns parsed CSV data from the S3 bucket. It checks that the data is properly formatted and contains the expected structure.

Endpoint

GET /api/csv

Expects
  • 200 status code
  • JSON array response
  • Non-empty data entries
  • Each entry should be a dictionary with multiple fields
  • No empty or malformed entries
Source code in testing/backend/test_main.py 
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
def test_csv_endpoint():
    """Test the CSV data endpoint functionality.

    This test verifies that the CSV endpoint correctly returns parsed CSV data
    from the S3 bucket. It checks that the data is properly formatted and
    contains the expected structure.

    Endpoint:
        GET /api/csv

    Expects:
        - 200 status code
        - JSON array response
        - Non-empty data entries
        - Each entry should be a dictionary with multiple fields
        - No empty or malformed entries
    """
    response = requests.get(f"{BASE_URL}/api/csv", timeout=10)
    assert response.status_code == 200
    data = response.json()
    assert isinstance(data, list)
    if len(data) > 0:
        first_item = data[0]
        assert isinstance(first_item, dict)
        assert len(first_item.keys()) > 1  # Verify it's not empty

Tech Radar Data Endpoint

Tests the /api/tech-radar/json endpoint that provides Tech Radar configuration data.

Test the tech radar JSON endpoint functionality.

This test verifies that the tech radar endpoint correctly returns the radar configuration data from the S3 bucket. The data defines the structure and content of the technology radar visualization.

Endpoint

GET /api/tech-radar/json

Expects
  • 200 status code
  • JSON object response
  • Non-empty configuration data
  • Multiple configuration keys present
Source code in testing/backend/test_main.py 
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
def test_tech_radar_json_endpoint():
    """Test the tech radar JSON endpoint functionality.

    This test verifies that the tech radar endpoint correctly returns the
    radar configuration data from the S3 bucket. The data defines the structure
    and content of the technology radar visualization.

    Endpoint:
        GET /api/tech-radar/json

    Expects:
        - 200 status code
        - JSON object response
        - Non-empty configuration data
        - Multiple configuration keys present
    """
    response = requests.get(f"{BASE_URL}/api/tech-radar/json", timeout=10)
    assert response.status_code == 200
    data = response.json()
    assert isinstance(data, dict)
    assert len(data.keys()) > 1  # Verify it's not empty

Repository Statistics Endpoints

Tests the /api/json endpoint with various filtering parameters:

  • No parameters (default behavior)
  • Date filtering
  • Archived status filtering
  • Combined parameter filtering
  • Invalid parameter handling

Test the JSON endpoint without query parameters.

This test verifies the default behavior of the JSON endpoint when no filters are applied. It checks that the endpoint returns complete repository statistics and metadata.

Endpoint

GET /api/json

Expects
  • 200 status code
  • JSON response containing:
    • Repository statistics
    • Language usage statistics
    • Metadata information
  • Complete stats structure with:
    • Total repository count
    • Private repository count
    • Public repository count
    • Internal repository count
Source code in testing/backend/test_main.py 
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
def test_json_endpoint_no_params():
    """Test the JSON endpoint without query parameters.

    This test verifies the default behavior of the JSON endpoint when no
    filters are applied. It checks that the endpoint returns complete
    repository statistics and metadata.

    Endpoint:
        GET /api/json

    Expects:
        - 200 status code
        - JSON response containing:
            - Repository statistics
            - Language usage statistics
            - Metadata information
        - Complete stats structure with:
            - Total repository count
            - Private repository count
            - Public repository count
            - Internal repository count
    """
    response = requests.get(f"{BASE_URL}/api/json", timeout=10)
    assert response.status_code == 200
    data = response.json()
    assert "stats" in data
    assert "language_statistics" in data
    assert "metadata" in data

    stats = data["stats"]
    assert "total_repos" in stats
    assert "total_private_repos" in stats
    assert "total_public_repos" in stats
    assert "total_internal_repos" in stats

Test the JSON endpoint with datetime filtering.

This test verifies that the endpoint correctly filters repository data based on a specified datetime parameter. It checks repositories modified within the last 7 days.

Parameters:

Name Type Description Default
datetime str

ISO formatted datetime string for filtering

 required
Example

GET /api/json?datetime=2024-03-20T00:00:00Z

Expects
  • 200 status code
  • Filtered repository data
  • Metadata containing the applied datetime filter
Source code in testing/backend/test_main.py 
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
def test_json_endpoint_with_datetime():
    """Test the JSON endpoint with datetime filtering.

    This test verifies that the endpoint correctly filters repository data
    based on a specified datetime parameter. It checks repositories modified
    within the last 7 days.

    Parameters:
        datetime (str): ISO formatted datetime string for filtering

    Example:
        GET /api/json?datetime=2024-03-20T00:00:00Z

    Expects:
        - 200 status code
        - Filtered repository data
        - Metadata containing the applied datetime filter
    """
    seven_days_ago = (datetime.now() - timedelta(days=7)).isoformat()
    response = requests.get(f"{BASE_URL}/api/json",
                            params={"datetime": seven_days_ago}, timeout=10)
    assert response.status_code == 200
    data = response.json()
    assert data["metadata"]["filter_date"] == seven_days_ago

Repository Project Endpoints

Tests the /api/repository/project/json endpoint with various parameters:

  • Missing parameters (error handling)
  • Single repository filtering
  • Multiple repository filtering
  • Date filtering
  • Archived status filtering
  • Combined parameter filtering
  • Language statistics validation

Test the repository project JSON endpoint with a valid repository parameter.

This test verifies the endpoint's basic functionality when requesting data for a single repository. It checks the complete response structure including repository data, statistics, and metadata.

Parameters:

Name Type Description Default
repositories str

Name of the repository to query (e.g., "tech-radar")

 required
Expects
  • 200 status code
  • JSON response with complete repository data
  • Valid statistics for the repository
  • Correct metadata including requested repository names
  • Language statistics if available
Source code in testing/backend/test_main.py 
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
def test_repository_project_json_with_repos():
    """Test the repository project JSON endpoint with a valid repository parameter.

    This test verifies the endpoint's basic functionality when requesting data
    for a single repository. It checks the complete response structure including
    repository data, statistics, and metadata.

    Parameters:
        repositories (str): Name of the repository to query (e.g., "tech-radar")

    Expects:
        - 200 status code
        - JSON response with complete repository data
        - Valid statistics for the repository
        - Correct metadata including requested repository names
        - Language statistics if available
    """
    response = requests.get(f"{BASE_URL}/api/repository/project/json",
                            params={"repositories": "tech-radar"}, timeout=10)
    assert response.status_code == 200
    data = response.json()

    # Verify response structure
    assert "repositories" in data
    assert "stats" in data
    assert "language_statistics" in data
    assert "metadata" in data

    # Verify stats structure
    stats = data["stats"]
    assert "total_repos" in stats
    assert "total_private_repos" in stats
    assert "total_public_repos" in stats
    assert "total_internal_repos" in stats

    # Verify metadata
    metadata = data["metadata"]
    assert "requested_repos" in metadata
    assert "found_repos" in metadata
    assert metadata["requested_repos"] == ["tech-radar"]

Test the repository project JSON endpoint with multiple repositories.

This test verifies that the endpoint correctly handles requests for multiple repositories in a single call. It checks that all requested repositories are processed and included in the response.

Parameters:

Name Type Description Default
repositories str

Comma-separated list of repository names

 required
Example

GET /api/repository/project/json?repositories=tech-radar,another-repo

Expects
  • 200 status code
  • Data for all requested repositories
  • Metadata containing all requested repository names
  • Aggregated statistics across all repositories
Source code in testing/backend/test_main.py 
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
def test_repository_project_json_multiple_repos():
    """Test the repository project JSON endpoint with multiple repositories.

    This test verifies that the endpoint correctly handles requests for
    multiple repositories in a single call. It checks that all requested
    repositories are processed and included in the response.

    Parameters:
        repositories (str): Comma-separated list of repository names

    Example:
        GET /api/repository/project/json?repositories=tech-radar,another-repo

    Expects:
        - 200 status code
        - Data for all requested repositories
        - Metadata containing all requested repository names
        - Aggregated statistics across all repositories
    """
    params = {
        "repositories": "tech-radar,another-repo"
    }
    response = requests.get(
        f"{BASE_URL}/api/repository/project/json", params=params, timeout=10)
    assert response.status_code == 200
    data = response.json()

    # Verify the requested repos are in metadata
    assert len(data["metadata"]["requested_repos"]) == 2
    assert "tech-radar" in data["metadata"]["requested_repos"]
    assert "another-repo" in data["metadata"]["requested_repos"]

Tech Radar Update Endpoints

Tests the endpoints for updating Tech Radar data:

  • Empty update handling
  • Partial updates
  • Invalid entry handling
  • Structure validation
  • Reference validation

Test the tech radar update endpoint with valid complete structure.

This test verifies that the endpoint correctly processes a complete tech radar update with valid structure for all components.

Endpoint

POST /review/api/tech-radar/update

Test Data
  • Valid title
  • Valid quadrants with required fields
  • Valid rings with required fields
  • Valid entries with required fields
Expects
  • 200 status code
  • Successful update confirmation
  • Correct structure in stored data
Source code in testing/backend/test_review.py 
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
def test_tech_radar_update_valid_structure():
    """Test the tech radar update endpoint with valid complete structure.

    This test verifies that the endpoint correctly processes a complete
    tech radar update with valid structure for all components.

    Endpoint:
        POST /review/api/tech-radar/update

    Test Data:
        - Valid title
        - Valid quadrants with required fields
        - Valid rings with required fields
        - Valid entries with required fields

    Expects:
        - 200 status code
        - Successful update confirmation
        - Correct structure in stored data
    """
    random_number = random.randint(100, 1000)
    test_data = {
        "entries": [
            {
                "id": "test-entry-1",
                "title": "Test Entry 1",
                "description": "Languages",
                "key": "test1",
                "url": "#",
                "quadrant": "1",
                "timeline": [
                    {
                        "moved": 0,
                        "ringId": "ignore",
                        "date": "2000-01-01",
                        "description": f"For testing purposes [CASE:{random_number}:2]"
                    }
                ],
                "links": []
            }
        ]
    }

    response = requests.post(
        f"{BASE_URL}/review/api/tech-radar/update",
        json=test_data,
        timeout=10
    )
    assert response.status_code == 200
    assert response.json()["message"] == "Tech radar updated successfully"

    # Verify the update
    get_response = requests.get(f"{BASE_URL}/api/tech-radar/json", timeout=10)
    assert get_response.status_code == 200
    updated_data = get_response.json()

    # Verify entry structure
    entries = updated_data["entries"]
    test_entry = next(
        (entry for entry in entries if entry["id"] == "test-entry-1"), None)
    assert test_entry is not None, "No entry with id 'test-entry-1' found"
    assert str(
        random_number) in test_entry["timeline"][0]["description"], "Entry with id 'test-entry-1' does not have the expected description"

Test the tech radar update endpoint with invalid structure.

This test verifies that the endpoint correctly validates the complete structure of the tech radar data, including title, quadrants, rings, and entries.

Endpoint

POST /review/api/tech-radar/update

Test Data
  • Missing title
  • Invalid quadrants structure
  • Invalid rings structure
  • Invalid entries structure
Expects
  • 400 status code for each invalid case
  • Appropriate error messages
  • No changes to existing data
Source code in testing/backend/test_review.py 
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
def test_tech_radar_update_invalid_structure():
    """Test the tech radar update endpoint with invalid structure.

    This test verifies that the endpoint correctly validates the complete
    structure of the tech radar data, including title, quadrants, rings,
    and entries.

    Endpoint:
        POST /review/api/tech-radar/update

    Test Data:
        - Missing title
        - Invalid quadrants structure
        - Invalid rings structure
        - Invalid entries structure

    Expects:
        - 400 status code for each invalid case
        - Appropriate error messages
        - No changes to existing data
    """
    # Test missing title
    response = requests.post(
        f"{BASE_URL}/review/api/tech-radar/update",
        json={
            "quadrants": [],
            "rings": [],
            "entries": []
        },
        timeout=10
    )
    assert response.status_code == 400
    assert response.json()["error"] == "Invalid or empty entries data"

    # Test invalid quadrants
    response = requests.post(
        f"{BASE_URL}/review/api/tech-radar/update",
        json={
            "title": "Test Radar",
            "quadrants": [{"invalid": "structure"}],
            "rings": [],
            "entries": []
        },
        timeout=10
    )
    assert response.status_code == 400
    assert response.json()["error"] == "Invalid or empty entries data"

    # Test invalid rings
    response = requests.post(
        f"{BASE_URL}/review/api/tech-radar/update",
        json={
            "title": "Test Radar",
            "quadrants": [{"id": "1", "name": "Test"}],
            "rings": [{"invalid": "structure"}],
            "entries": []
        },
        timeout=10
    )
    assert response.status_code == 400
    assert response.json()["error"] == "Invalid or empty entries data"

Admin Banner Management Endpoints

Tests the endpoints for managing banner messages:

  • Banner retrieval
  • Banner creation
  • Banner visibility toggling
  • Banner deletion
  • Validation of requests

Test the admin banners endpoint for retrieving banner messages.

This test verifies that the endpoint correctly returns banner messages from the S3 bucket. It checks the structure of the response and ensures the messages array is present.

Endpoint

GET /admin/api/banners

Expects
  • 200 status code
  • JSON response containing a messages array
  • Valid structure that can be parsed by the admin UI
Source code in testing/backend/test_admin.py 
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
def test_admin_banner_get():
    """Test the admin banners endpoint for retrieving banner messages.

    This test verifies that the endpoint correctly returns banner messages
    from the S3 bucket. It checks the structure of the response and ensures
    the messages array is present.

    Endpoint:
        GET /admin/api/banners

    Expects:
        - 200 status code
        - JSON response containing a messages array
        - Valid structure that can be parsed by the admin UI
    """
    response = requests.get(f"{BASE_URL}/admin/api/banners", timeout=10)
    assert response.status_code == 200
    data = response.json()

    # Verify the response structure
    assert "messages" in data
    assert isinstance(data["messages"], list)

Test the admin banners update endpoint.

This test verifies that the endpoint correctly processes banner updates with valid data and saves them to the S3 bucket.

Endpoint

POST /admin/api/banners/update

Test Data
  • Valid banner message
  • Array of pages where the banner should appear
  • Optional banner title and type
Expects
  • 200 status code
  • Success message confirming the banner was added
  • Banner should be retrievable in subsequent GET requests
Source code in testing/backend/test_admin.py 
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
def test_admin_banner_update():
    """Test the admin banners update endpoint.

    This test verifies that the endpoint correctly processes banner 
    updates with valid data and saves them to the S3 bucket.

    Endpoint:
        POST /admin/api/banners/update

    Test Data:
        - Valid banner message
        - Array of pages where the banner should appear
        - Optional banner title and type

    Expects:
        - 200 status code
        - Success message confirming the banner was added
        - Banner should be retrievable in subsequent GET requests
    """
    test_data = {
        "banner": {
            "message": "Test Banner Message",
            "title": "Test Banner",
            "type": "info",
            "pages": ["radar", "statistics"],
            "show": True
        }
    }

    response = requests.post(
        f"{BASE_URL}/admin/api/banners/update",
        json=test_data,
        timeout=10
    )
    assert response.status_code == 200
    data = response.json()
    assert "message" in data
    assert data["message"] == "Banner added successfully"

    # Verify the banner was added
    get_response = requests.get(f"{BASE_URL}/admin/api/banners", timeout=10)
    assert get_response.status_code == 200
    get_data = get_response.json()
    assert "messages" in get_data

    # Find the added banner by message
    added_banner = next((banner for banner in get_data["messages"]
                         if banner["message"] == "Test Banner Message"), None)
    assert added_banner is not None
    assert added_banner["title"] == "Test Banner"
    assert added_banner["type"] == "info"
    assert "radar" in added_banner["pages"]
    assert "statistics" in added_banner["pages"]
    assert added_banner["show"] is True

Error Handling Tests

The test suite includes specific tests for error conditions:

  • Invalid endpoints
  • Invalid date parameters
  • Missing required parameters
  • Invalid data structures

Test error handling for invalid endpoints.

This test verifies that the server properly handles requests to non-existent endpoints by returning appropriate error status codes.

Example

GET /api/nonexistent

Expects
  • Either 404 (Not Found) or 500 (Internal Server Error) status code
  • Proper error handling for invalid routes
Source code in testing/backend/test_main.py 
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
def test_invalid_endpoint():
    """Test error handling for invalid endpoints.

    This test verifies that the server properly handles requests to
    non-existent endpoints by returning appropriate error status codes.

    Example:
        GET /api/nonexistent

    Expects:
        - Either 404 (Not Found) or 500 (Internal Server Error) status code
        - Proper error handling for invalid routes
    """
    response = requests.get(f"{BASE_URL}/api/nonexistent", timeout=10)
    assert response.status_code in [404, 500]  # Either is acceptable

Test the JSON endpoint's handling of invalid date parameters.

This test verifies that the endpoint gracefully handles invalid datetime parameters without failing. It should ignore the invalid date and return unfiltered results.

Parameters:

Name Type Description Default
datetime str

An invalid datetime string

 required
Example

GET /api/json?datetime=invalid-date

Expects
  • 200 status code (graceful handling)
  • Null filter_date in metadata
  • Valid response with unfiltered stats
  • Complete language statistics
Source code in testing/backend/test_main.py 
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
def test_json_endpoint_invalid_date():
    """Test the JSON endpoint's handling of invalid date parameters.

    This test verifies that the endpoint gracefully handles invalid datetime
    parameters without failing. It should ignore the invalid date and return
    unfiltered results.

    Parameters:
        datetime (str): An invalid datetime string

    Example:
        GET /api/json?datetime=invalid-date

    Expects:
        - 200 status code (graceful handling)
        - Null filter_date in metadata
        - Valid response with unfiltered stats
        - Complete language statistics
    """
    response = requests.get(f"{BASE_URL}/api/json",
                            params={"datetime": "invalid-date"}, timeout=10)
    assert response.status_code == 200  # Backend handles invalid dates gracefully
    data = response.json()
    assert data["metadata"]["filter_date"] is None
    assert "stats" in data
    assert "language_statistics" in data

Code Quality

The testing framework includes tools for maintaining code quality:

# Run linting checks
make lint

# Run specific linters
make ruff
make pylint

# Clean up cache files
make clean

Integration with Utilities

The tests validate the same endpoints used by the frontend utilities:

  • Project Data Utility: Tests the /api/csv endpoint used by fetchCSVFromS3()
  • Repository Data Utility: Tests the /api/repository/project/json endpoint used by fetchRepositoryData()
  • Tech Radar Data Utility: Tests the /api/tech-radar/json endpoint used by fetchTechRadarJSONFromS3()
  • Admin Utilities: Tests the /admin/api/banners* endpoints used by the admin interface for banner management

This ensures that the data providers for the DataContext are functioning correctly and returning the expected data structures.