Frequently Asked Questions

How does redefine work?

Redefine is the first of its kind AI-based test selection platform, built to speed CI time and dramatically reduce cloud costs. Redefine’s algorithm identifies and runs tests that matter for each specific code change, offering real-time pass/fail input as each test is completed. To do this, Redefine studies your historic code changes and test suite to form a test selection model that helps your team predict failure sooner. This speeds up the inner development loop, improves code quality, and helps your team ship changes faster.In less than 15 minutes, Redefine integrates with your existing CI environment and testing framework. Our AI analyzes your code change history and test results to understand your team’s coding patterns, find dependencies and correlations, and recognize common failures.Every time a new code change is pushed, the test selection model automatically selects only the most relevant tests from your test suite. These tests will rapidly determine whether your code build will fail or succeed. This way developers can move fast making changes during their test run and avoid breaking things.

What data Redefine’s AI is based on?

Redefine basically studies your historic code changes and test suite to form a unique test selection model that helps your team predict failure sooner.

Will Redefine run all my tests?

Redefine tests only what is relevant for every code-change / run, so you can enjoy a much faster dev cycle and during your CI. The AI-based test selection model automatically selects only the most relevant tests from your test suite. These tests will rapidly determine whether your code build will fail or succeed. Developers get real-time fail responses as each test completes, so they can make changes during their test run.

How does redefine treat new tests?

Redefine learns new tests, with exploration mode, by running new tests a few times to collect information about their dependencies.

How does redefine treat flaky tests?

Redefine automatically detects flaky tests and provides a configuration to include/exclude them from redefine builds.

What is Redefine Time Budget?

Redefine time budget (measured in seconds) is selected by the user and determines how long the longest test suite may take. Redefine uses this parameter to make sure build times are consistent across different code changes, while only running relevant tests.

What is Redefine Coverage (Model Accuracy)?

Redefine coverage is the number of the CI builds where redefine selected all of the failed tests and surfaced all of the bugs out of all of the CI builds.

What is the Time budget and coverage relationship?

Note that increasing the time budget will always increase the coverage, meanwhile, lowering the budget will inherently decrease the coverage.

How does redefine measure the accuracy of the AI model?

As many relevant tests are selected for the diff the accuracy metric is defined by Redefine Coverage.

Given your accuracy metric, what would you say the accuracy of the model is?

The coverage can be adjusted according to your desired build time.

When would the model trigger a full run of tests?

Recent result was 98% coverage with a 3 minutes time budget (out of 11 minutes average build time).

How many tests are running on average?

It depends on your desired time budget, so in case you have 100 tests, and we used about 30% of total build time it would be around 30 tests.

Do I need a CI in place in order to use Redefine?

Yes, you do :). Having a Continuous Integration (CI) in place is inevitable, it helps ensure that the codebase is always in a deployable state. It also helps to identify any errors and problems in the codebase quickly, so that they can be addressed before they become major issues. Additionally, CI allows developers to focus on their own tasks, as the integration process is automated. This helps them to produce better quality code and deliver it faster, which is beneficial for the entire project.

Can I use any CI framework in order to use Redefine?

Yes, Redefine supports all CI platforms.

Which testing framework do you already support?

We currently support Pytest and Selenium. Mocha, Jest and Cypress are coming soon.

Do I need a dedicated engineer to operate Redefine?

You don't need a dedicated engineer to operate Redefine. Redefine is an AI-based tool and as such it is designed to self discover and learn the data it needs in order to operate and automatically train itself for self-improvement (without any human touch or maintenance).

How long does it take to install Redefine?

It takes 5-6 minutes to install Redefine. The first step is to add the Redefine CLI tool to your existing CI, which takes about 3 minutes. After installation, Redefine enters its discovery stage, collecting all build histories and coverage (this stage runs in the background and you are not required to do anything during it). The final step is to activate the Redefine tool on your CI production environment and start saving cloud costs and accelerating development.

Does Redefine SOC2 compliance?

Redefine is SOC2 compliant, as we have a strong background in cybersecurity and understand the requirement to ensure customer security, reliability, and compliance. Generally, SOC2 standards are designed to ensure that tools are secure and in compliance with industry-specific regulations.

Is your infra hosted on-prem or on a cloud provider?

Redefine platform is hosted on AWS. There is a great value in having your CI optimization operated on the cloud: the first one is cost savings: Redefine, like other cloud-based tools, is more cost-effective than on-premise tools because it requires no upfront capital expenses and is billed on a pay-as-you-go basis. The second benefit is scalability: Redefine is easily scalable, allowing businesses to quickly add or remove users and resources as needed without any additional hardware or software investments. The third and last is flexibility: Redefine is highly flexible, allowing businesses to access their data and applications at all times without any limitations.

If the accuracy is less than 100%, what measures do you recommend to protect bad code from reaching production?

In order to increase velocity while protecting production, we recommend running redefine on every push to the feature branch, while running the full suite on merge to the main branch. If the organization is running scheduled builds (i.e. nightly), we recommend running the full suite there.

How strong is your security posture?

At Redefine, we take security very seriously; redefine is certified COC2 and the founders' combined vulnerability research experience exceeds seven years.

Does Redefine have a free plan?

Yes, Redefine offers a free plan. A free plan is available to developers with fewer than 50 builds per month. You can use Redefine for free if you meet our application requirements. Our Professional plan for larger organizations offers a free trial of one month.

How long does it take to see developers’ time saved?

Most of our customers experienced saved dev time from the first code change run they shipped with Redefine, the best result so far was 91% saved time equivalent to 4 years, 345 days, 18 hrs and 13 min.

How long does it take to see test time decrease?

Most of our customers started experiencing 90% saved time once the discovery stage ended and the dev-teams are running their CI with the Redefine tool.

Do you recommend running the full suite of tests at some cadence?

We recommend running the full suite as a last-stop before merge, or as a scheduled (i.e. nightly) build if one is used.

What is the pricing model?

The pricing model is based on the number of CI runs per month, we offer 2 packages: one for professionals: priced per build and the second package is for enterprises, big engineering organizations who are looking for an enhanced solution to suit their needs.