Frequently Asked Question

Got Questions? Click a Category Below to Get Answers!

1. Deployment & Architecture

1.1 Where is our code processed?  

+

BaseRock is deployed in the client environment in their Kubernetes namespace - so it is all processed locally.

1.2 Does it support on-premise or private cloud deployment?

+

Yes, we support both on-premise and private cloud deployment options.

1.3 Where would the Agent be deployed, can that be deployed in azure too

+

The agent is deployed as part of BaseRock.ai control plane. Yes, it can be deployed on Azure or any other cloud service provider like AWS, GCP, etc.

2. Security, Privacy & Compliance

2.1 What compliance certifications does baserock.ai hold?

+

We are in the process of getting our SOC 2 compliance.

2.2 What data is stored and for how long?

+

 In OTEL based approach it’s the metadata and in github based approach it’s source code (read access only) and meta data.
Every data is stored in the client environment and duration can be configured.

2.3 How can we be assured of Data privacy - ai may analyze sensitive data

+

 The prompts are developed in a way where it only utilises the meta data just to create test cases and synthetic data.
On top of that our control plane is inside your VPC or on-premise and we can also deploy our fine tuned model in your environment or leverage AWS Bedrock or Azure OpenAI where you have agreements in place.

2.4 How is Source code leakage prevented?

+

 Since everything is processed locally, you can setup appropriate firewall rules to control access and prevent leakage.

2.5 Is data transferred to third-party providers, and if so, is there a Data Processing Agreement (DPA)?

+

 If third-party LLMs are enabled for test generation, data may be shared with providers like Claude (Anthropic), Sonet, and Voyage. In such cases, DPAs are in place. No data is ever used for model training.

2.6 What technical and organisational measures are taken to protect the data?

+

 We are currently undergoing SOC 2 compliance. Measures include:

  • Data encryption at rest and in transit
  • Role-based access control
  • Customer environment isolation
  • Secure CI/CD pipelines

Comprehensive audit logging We are happy to share our internal data protection concept or SOC 2 summary upon request.

3. Functionality & Test Coverage

3.1 Is frontend test generation (e.g., React, Angular) supported?

+

 Not right now - it’s in future roadmap.

3.2 Are UI test scripts for Selenium/playwright supported?

+

 Not right now, this is an integration testing product. We will be releasing a new product for UI tests later this year.

3.3 Is it possible to automate user interface flows and Real-time systems or hardware interactions?

+

 Yes, we utilize test-profiles associated with respective environments for validations.

3.4 Is it possible to automate cross environment validations?

+

 UI flows and real-time systems or hardware cannot be automated currently, however the underlying API calls and system interactions can be automated.

3.5 Is there a chance that baserock.ai could do some manipulations to database, like re-writing
database values.

+

 UI flows and real-time systems or hardware cannot be automated currently, however the underlying API calls and system interactions can be automated.

4. AI Model & Learning Process

4.1 How long would it take to complete the Learn phase?

+

 If it's source code - Typically it takes less than a day once the setup is completed.
If it's Open Telemetry (OT) - we want the client to run their regression test suite once so we discover all end points and scenarios at a minimum. This can be 2-3 days.

4.2 Since Telemetry can't be used on staging, how effective would it really be?

+

 Even without OT, the tests are of very high quality. OT does gives us more realistic synthetic data, however our system can also learn from existing test suites to create appropriate synthetic data.

4.3 How often is the AI model updated? Can it adapt without complete retraining?

+

 We have releases every 2 weeks, but you can expect pushes to customer environments maybe once every month or two.
Yes, it can adapt without complete training. QA team can use the Playbook feature to guide the system via simple natural language.

5. Maintenance, Debugging & Integration

5.1 How are tests maintained over time as code evolves?

+

 We store different versions of the test as we detect more code changes. The tests are automatically updated without human intervention. However, the QA team can audit and make adjustments as needed.

5.2 What CI/CD tools does baserock.ai integrate with?

+

 We integrated with all the CI/CD tools and can accommodate any specific requests that you may have.

5.3 Are any analytics or dashboards provided?

+

 Yes, it’s provided via our user interface.

5.4 How to debug baserock.ai failures and where are diagnosis reports sent to?

+

 UI flows and real-time systems or hardware cannot be automated currently, however the underlying API calls and system interactions can be automated.

1. Test Coverage Strategy & Quality

1.1 How do you ensure that generated tests provide high logical coverage, including complex code paths measured by cyclomatic complexity?  

+

 Our fine-tuned large language model (LLM) is designed to generate optimal test cases that ensure high path coverage while avoiding unnecessary complexity in the codebase. In addition, we leverage static code analysis and Abstract Syntax Tree (AST) parsing to detect complex branches and conditionals, helping ensure comprehensive test coverage aligned with cyclomatic complexity.

1.2 How do you avoid bloated or excessive test generation when working with large codebases?

+

Yes, we support both on-premise and private cloud deployment options.

1.3 How do you evaluate test quality beyond standard code coverage metrics?

+

 Beyond code coverage, we assess logical depth and edge-case handling. The model is tuned to produce scenario-based test cases that reflect real-world behavior, not just surface-level paths. Test quality is also auditable through our dashboard, and generated tests can be benchmarked against historical regression failures or QA review inputs.

2. Framework & Repository Support

2.1 Does the test generation tool support both mono-repo and multi-repo structures? Which frameworks are supported?

+

Yes, it supports both mono-repo and multi-repo setups. Frameworks supported include NestJS, Next.js, Vue.js, React.js, Django, Spring Boot, Go, and TypeScript-based stacks. Baserock’s system auto-detects the project structure and dependencies to intelligently select the appropriate framework and generate accurate tests.

2.2 Is it possible to override or customize the generated test cases?

+

Yes, tests can be regenerated and customized to suit your specific requirements. They are editable directly in the portal, and any test can be locked to prevent it from being overwritten during future test generation cycles.

3. Pricing

3.1 What is a Class? How do you limit per Class?

+

A Java class that contains methods in your source code repository that you're testing against. We only count classes, regardless of the amount of methods it contains. Whether if your class contains 500 methods or 5 methods, it's still counted as one class.

3.2 Awesome, I'm hooked. How do I signup for an account? 

+

You can shoot an email to help@baserock.ai and we'll send over a signup link!

3.3 Is there a trial period for the Developer and Enterprise tiers?

+

Yes, we offer a trial period for both the Developer and Enterprise tiers to ensure the solution meets your needs. For more details, contact help@baserock.ai.

3.4 How does BaseRock.ai handle data privacy and security across different plans?

+

BaseRock.ai prioritizes data privacy and security across all plans. We adhere to industry best practices and compliance standards to ensure your data is secure. For specific security protocols related to each plan, please reach out to help@baserock.ai.

3.5 Do you offer discounts?

+

Yes, we offer discounts on a case-by-case basis and strive to accommodate your needs. For more details, contact us at help@baserock.ai.

3.6 What's the process for upgrading or downgrading my plan?

+

Reach out to help@baserock.ai for seamless plan transitions. We'll guide you through the upgrade or downgrade process.

1. What is a Class? How do you limit per Class?

+

A Java class that contains methods in your source code repository that you're testing against. We only count classes, regardless of the amount of methods it contains. Whether if your class contains 500 methods or 5 methods, it's still counted as one class.

2. Awesome, I'm hooked. How do I signup for an account? 

+

You can shoot an email to help@baserock.ai and we'll send over a signup link!

3. Is there a trial period for the Developer and Enterprise tiers?

+

Yes, we offer a trial period for both the Developer and Enterprise tiers to ensure the solution meets your needs. For more details, contact help@baserock.ai.

4. How does BaseRock.ai handle data privacy and security across different plans?

+

BaseRock.ai prioritizes data privacy and security across all plans. We adhere to industry best practices and compliance standards to ensure your data is secure. For specific security protocols related to each plan, please reach out to help@baserock.ai.

5. Do you offer discounts?

+

Yes, we offer discounts on a case-by-case basis and strive to accommodate your needs. For more details, contact us at help@baserock.ai.

6. What's the process for upgrading or downgrading my plan?

+

Reach out to help@baserock.ai for seamless plan transitions. We'll guide you through the upgrade or downgrade process.

Sometimes you may forget the best practices and hard code things, making the code untestable. We are all human - we got you!
We give you the feedback to ensure 100% test coverage.