Skip to content

Latest commit

 

History

History
512 lines (369 loc) · 11.4 KB

testing.md

File metadata and controls

512 lines (369 loc) · 11.4 KB

Concourse Testing Guide

This guide covers the testing frameworks, conventions, and best practices for testing Concourse.

Table of Contents

Overview

Concourse uses multiple testing frameworks to ensure quality at different levels:

Framework Purpose Location
concourse-unit-test-core Base classes for unit tests Unit tests in each module
concourse-integration-tests Tests against in-process server concourse-integration-tests/
concourse-ete-test-core End-to-end testing framework ETE tests
concourse-ete-tests Cross-version and performance tests concourse-ete-tests/

Test Frameworks

concourse-unit-test-core

Provides base classes and utilities for unit testing:

  • ConcourseBaseTest - Base class for all unit tests
  • Variables - Utility for tracking test state
  • Test data generators

concourse-integration-tests

Integration tests that run against the actual storage engine:

  • ConcourseIntegrationTest - Base class with client/server setup
  • Tests run against an in-process Concourse instance
  • Full API testing including transactions

concourse-ete-test-core

End-to-end testing framework for realistic scenarios:

  • ClientServerTest - Tests against external server process
  • CrossVersionTest - Tests across multiple Concourse versions
  • Automatic server lifecycle management

concourse-ete-tests

Actual end-to-end tests including:

  • Cross-version compatibility tests
  • Performance benchmarks
  • Upgrade scenario tests

Writing Unit Tests

Base Class

All unit tests should extend ConcourseBaseTest:

import com.cinchapi.concourse.test.ConcourseBaseTest;
import org.junit.Test;

public class MyFeatureTest extends ConcourseBaseTest {

    @Test
    public void testMyFeature() {
        // Test implementation
    }
}

Using Variables for Debugging

The Variables class tracks test state and dumps it on failure:

import com.cinchapi.concourse.test.Variables;

@Test
public void testAddWithRandomData() {
    // Register variables for debugging
    String key = Variables.register("key", TestData.getString());
    Object value = Variables.register("value", TestData.getObject());
    long record = Variables.register("record", TestData.getLong());

    // Perform test
    boolean result = store.add(key, value, record);

    // Assert
    assertTrue(result);
}

On test failure, output looks like:

TEST FAILURE in testAddWithRandomData: expected:<true> but was:<false>
---
key = myTestKey (java.lang.String)
value = 42 (java.lang.Integer)
record = 12345 (java.lang.Long)

Test Method Naming

Use descriptive names that explain what is being tested:

// Good: Clear what is being tested and expected outcome
@Test
public void testAddReturnsFalseWhenValueAlreadyExists() { }

@Test
public void testFindReturnsEmptySetWhenNoCriteriaMatch() { }

@Test
public void testTransactionAbortsOnConflict() { }

// Avoid: Vague or unclear names
@Test
public void testAdd() { }

@Test
public void test1() { }

Setup and Teardown

Override the provided methods for setup/teardown:

public class MyTest extends ConcourseBaseTest {

    private MyComponent component;

    @Override
    protected void beforeEachTest() {
        component = new MyComponent();
        component.initialize();
    }

    @Override
    protected void afterEachTest() {
        component.cleanup();
    }

    @Test
    public void testSomething() {
        // component is ready to use
    }
}

Writing Integration Tests

Base Class

Integration tests extend ConcourseIntegrationTest:

import com.cinchapi.concourse.test.ConcourseIntegrationTest;
import org.junit.Test;

public class MyIntegrationTest extends ConcourseIntegrationTest {

    @Test
    public void testFeatureEndToEnd() {
        // 'client' variable is automatically available
        client.add("key", "value", 1);

        Object result = client.get("key", 1);
        assertEquals("value", result);
    }
}

Available Variables

Integration tests have access to:

  • client - Connected Concourse client instance
  • server - ManagedConcourseServer for server control

Server Control

@Test
public void testServerRestart() {
    // Add data
    client.add("key", "value", 1);

    // Restart server
    server.stop();
    server.start();

    // Reconnect
    client = server.connect();

    // Verify data persisted
    assertEquals("value", client.get("key", 1));
}

Transaction Testing

@Test
public void testTransactionIsolation() {
    // Start transaction
    client.stage();

    try {
        client.add("key", "value", 1);
        client.set("status", "pending", 1);

        // Commit
        assertTrue(client.commit());
    }
    catch (TransactionException e) {
        client.abort();
        fail("Transaction should not fail");
    }
}

End-to-End Tests

Single Version Tests

Extend ClientServerTest for tests against a specific version:

import com.cinchapi.concourse.test.ClientServerTest;

public class MyEteTest extends ClientServerTest {

    @Override
    protected String getServerVersion() {
        // Return version number or path to installer
        return "0.12.0";
        // Or: return "/path/to/concourse-server-0.12.0.bin";
    }

    @Test
    public void testFeature() {
        // 'client' is connected to external server
        client.add("key", "value", 1);

        // Server runs in separate JVM
        assertNotNull(client.get("key", 1));
    }
}

Cross-Version Tests

Test across multiple versions using CrossVersionTest:

import com.cinchapi.concourse.test.CrossVersionTest;
import com.cinchapi.concourse.test.runners.CrossVersionTestRunner.Versions;

@Versions({"0.10.0", "0.11.0", "0.12.0"})
public class MyCrossVersionTest extends CrossVersionTest {

    @Test
    public void testBackwardCompatibility() {
        // This test runs against each specified version
        client.add("key", "value", 1);
        assertEquals("value", client.get("key", 1));
    }
}

Performance Tests

public class MyPerformanceTest extends ClientServerTest {

    @Override
    protected String getServerVersion() {
        return "0.12.0";
    }

    @Test
    public void testWriteThroughput() {
        long start = System.currentTimeMillis();

        for (int i = 0; i < 10000; i++) {
            client.add("key" + i, "value", i);
        }

        long duration = System.currentTimeMillis() - start;
        System.out.println("Throughput: " + (10000.0 / duration * 1000) + " ops/sec");
    }
}

Running Tests

All Tests

./gradlew test

Specific Module

./gradlew :concourse-server:test
./gradlew :concourse-integration-tests:test
./gradlew :concourse-ete-tests:test

Specific Test Class

./gradlew :concourse-server:test --tests "*.EngineTest"
./gradlew :concourse-integration-tests:test --tests "*.AddTest"

Specific Test Method

./gradlew :concourse-server:test --tests "*.EngineTest.testAddBasic"

With Detailed Output

./gradlew test --info
./gradlew test --debug

Test Suites

Run predefined test suites:

# Integration test suite
./gradlew :concourse-integration-tests:test --tests "*.IntegrationTestSuite"

# Performance test suite
./gradlew :concourse-integration-tests:test --tests "*.PerformanceTestSuite"

# Bug reproduction suite
./gradlew :concourse-integration-tests:test --tests "*.BugReproSuite"

Parallel Test Execution

# Run tests in parallel
./gradlew test --parallel

# Control max workers
./gradlew test --max-workers=4

Bug Reproduction Tests

Bug reproduction tests live in the bugrepro/ package and follow a naming convention:

Naming Convention

  • GitHub issues: GH{number}.java (e.g., GH123.java)
  • JIRA issues: CON{number}.java (e.g., CON456.java)

Structure

package com.cinchapi.concourse.bugrepro;

import com.cinchapi.concourse.test.ConcourseIntegrationTest;
import org.junit.Test;

/**
 * Reproduction test for GitHub issue #123.
 *
 * @see <a href="https://github.com/cinchapi/concourse/issues/123">GH-123</a>
 */
public class GH123 extends ConcourseIntegrationTest {

    @Test
    public void testReproduction() {
        // Steps to reproduce the bug
        // After fix, this test should pass
    }
}

When to Add Bug Repro Tests

  1. When fixing a reported bug
  2. When a test reveals unexpected behavior
  3. When regression testing is needed

Test Utilities

TestData

Generate random test data:

import com.cinchapi.concourse.util.TestData;

String key = TestData.getString();
long record = TestData.getLong();
int number = TestData.getInt();
Object value = TestData.getObject();

Time Utilities

import com.cinchapi.concourse.Timestamp;

// Create timestamps for testing
Timestamp now = Timestamp.now();
Timestamp past = Timestamp.fromMicros(now.getMicros() - 1000000);

Assertions

Use JUnit assertions with meaningful messages:

import static org.junit.Assert.*;

// Good: Descriptive message on failure
assertEquals("Value should be stored", expected, actual);
assertTrue("Add should succeed for new value", result);
assertFalse("Add should fail for duplicate", result);

// Checking collections
assertNotNull("Result should not be null", result);
assertFalse("Result should not be empty", result.isEmpty());
assertEquals("Should return exactly 3 records", 3, result.size());

Concurrency Testing

import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

@Test
public void testConcurrentWrites() throws Exception {
    int numThreads = 10;
    int writesPerThread = 100;
    CountDownLatch latch = new CountDownLatch(numThreads);
    ExecutorService executor = Executors.newFixedThreadPool(numThreads);

    for (int t = 0; t < numThreads; t++) {
        final int threadId = t;
        executor.submit(() -> {
            try {
                for (int i = 0; i < writesPerThread; i++) {
                    client.add("key", threadId * 1000 + i, 1);
                }
            }
            finally {
                latch.countDown();
            }
        });
    }

    latch.await();
    executor.shutdown();

    // Verify all writes succeeded
    Set<Object> values = client.select("key", 1).get("key");
    assertEquals(numThreads * writesPerThread, values.size());
}

Best Practices

Do

  • Write tests before or alongside code changes
  • Use descriptive test method names
  • Register variables with Variables.register() for debugging
  • Clean up resources in afterEachTest()
  • Test edge cases and error conditions

Don't

  • Skip tests to speed up development
  • Use @Ignore without a tracking issue
  • Write tests that depend on execution order
  • Leave debugging output in committed tests
  • Test implementation details instead of behavior

Related Documentation