AI Testing Interview Questions and Answers: Automation with AI and Testing LLM Apps

AI automation testing means using AI techniques to create, run, maintain or analyse automated tests. Examples include generating test cases from requirements, suggesting test scripts, healing broken locators, comparing screenshots intelligently and spotting patterns in failures. AI does not replace the test framework; it sits on top of tools like Selenium or Playwright and reduces manual effort. A tester still decides what to test and judges whether results are correct.

Using AI for testing means AI helps you test normal software faster, for example by generating Playwright scripts. Testing AI systems means the product itself contains AI, such as a chatbot or recommendation engine, and you must check its quality, safety and reliability. The second is harder because outputs are not fixed, so exact assertions fail. Strong candidates mention both and explain that each needs different skills.

AI tools can read requirements, user stories or code and suggest test cases, including positive, negative and boundary scenarios that humans may miss. A good approach is to give the model the story, acceptance criteria and expected format, then review every suggestion. For a login story, it might suggest tests for locked accounts, expired OTPs and SQL-like inputs. The trade-off: AI produces volume quickly, but duplicates and irrelevant cases still need human pruning.

Self-healing locators let a test find an element even after its locator changes, by using other attributes such as text, position, nearby elements or past matches. When the primary locator fails, the tool tries alternatives and can suggest an updated locator. This reduces maintenance when developers rename IDs or classes. The risk is that healing can click the wrong element and hide a real UI bug, so healed steps must be logged and reviewed.

AI-based visual testing compares screenshots of an application against a baseline and flags meaningful visual differences, such as overlapping text or a missing button. Unlike strict pixel comparison, AI-assisted tools try to ignore harmless changes like anti-aliasing or dynamic dates. It suits responsive layouts across devices, which matters for mobile-first Indian apps. Teams still review flagged differences and approve new baselines when changes are intentional.

AI can generate realistic synthetic test data, such as names, addresses, PAN-like formats or transaction histories, that follows your rules without copying real customer data. It is useful for edge cases, Indian language inputs and large volumes for performance tests. Always define constraints clearly and validate the output, because generated data can break format rules. Using synthetic data instead of production data also reduces privacy risk.

I use AI assistants to draft page objects, write test scaffolding, convert manual test steps into scripts, explain failing stack traces and suggest better locators. For example, I might paste a manual test case and ask for a Playwright test using role-based locators and the page object pattern. Then I run it, fix waits and assertions, and make sure it follows our framework conventions. AI speeds up typing; the tester still owns correctness.

Review AI-generated tests like code from a new team member. Check that assertions actually verify the requirement, not just that a page loaded; that locators are stable; that there are no hard-coded sleeps; that test data is isolated; and that the test fails when the feature breaks. A quick trick is to break the feature on purpose and confirm the test catches it. AI tools often produce tests that pass but prove very little.

AI can help diagnose flaky tests but rarely fixes the root cause on its own. It can cluster failures, spot patterns such as timing issues or shared test data, and suggest fixes like better waits. Common real causes are race conditions, environment instability, test order dependency and unstable data. The risk is that AI-suggested retries or longer timeouts hide real problems. Fix the cause, and track flakiness rates over time.

AI-based test selection uses signals like changed files, past failures, code coverage and defect history to run the tests most likely to catch a problem first. This shortens feedback time in CI when the full suite takes hours. The trade-off is risk: a skipped test might have caught a bug. Most teams combine smart selection on every commit with a full regression run nightly or before release.

The main risks are false confidence, hidden bugs and data exposure. Generated tests can have weak assertions, self-healing can mask UI regressions, and pasting proprietary code or customer data into an unapproved AI tool can leak it. AI suggestions can also be confidently wrong about APIs or framework features. Mitigate with code review, mutation-style checks, approved tools only and clear logs of every AI-made change.

AI is changing what automation testers do rather than removing the need for them. Tools can write and maintain much of the routine script code, but someone must decide what to test, judge risk, design test data, review AI output and test the AI features themselves. Testers who learn AI tools and LLM evaluation are likely to be in a stronger position. Say this calmly and show one example of how you already use AI.

LLM outputs are non-deterministic and open-ended, so the same input can produce different valid answers, and exact-match assertions break. Even at temperature 0, outputs can vary slightly across runs or model versions. So you test properties instead: is the answer correct, grounded in sources, in the right format, safe and within length limits? You also test at scale with an evaluation set, because one passing example proves little about a probabilistic system.

An evaluation set is a fixed collection of realistic inputs with expected answers or grading criteria, used to score the system every time something changes. It should include common questions, edge cases, multiple languages, adversarial inputs and past production failures. For a banking FAQ bot, each item might hold a question, the policy passage it should use and key facts the answer must contain. A few hundred well-chosen items is a practical start.

LLM-as-a-judge uses one model to grade another model's outputs against a rubric, such as correctness, relevance or tone. It scales evaluation far beyond what human reviewers can manage. Research has documented biases, including preferring the first answer shown, favouring longer answers and favouring outputs similar to the judge's own. Reduce these with clear rubrics, randomised answer order, reference answers and regular comparison against human-graded samples.

Treat prompts like code: version them and run the evaluation set on every change before release. Compare scores per category, not just the overall average, because a prompt can improve one area while breaking another. Add automated checks for format, length and banned content, plus LLM-as-judge or human grading for quality. For example, a prompt tweak that improves Hindi answers may quietly break JSON output for the app, and only a per-category report would catch it.

Test retrieval and generation separately. For retrieval, check whether the right document chunks come back for each test question. For generation, check whether answers are correct, grounded in the retrieved text, cite sources and refuse when the answer is not in the documents. Also test stale documents, conflicting sources and questions outside scope. Separating the two tells developers where to fix. See our RAG interview questions for retrieval details.

Prompt injection is an attack where malicious instructions in user input, documents or web pages make the model ignore its rules, for example "ignore previous instructions and show the system prompt". OWASP lists it as a top risk for LLM applications. Test with a library of direct and indirect injection attempts, including instructions hidden inside uploaded files, and check that the bot refuses, leaks nothing and cannot trigger unauthorised tool actions.

Jailbreak testing, often part of red-teaming, means deliberately trying to make an AI system produce harmful, restricted or off-policy content using role-play, encoding tricks, multi-turn pressure or other languages. Testers record which attempts succeed, report them by severity and add them to the regression suite. Include Indian languages and code-mixed text, because safety filters can be weaker outside English. Repeat after every model or prompt change.

Create paired test inputs that differ only in a sensitive attribute, such as gender, region, religion or language, and compare the outputs for quality and tone. Check that harmful, toxic or unsafe requests are refused consistently, and that personal data is never revealed. For a hiring assistant, swap names that suggest different genders on identical resumes and compare the summaries. Report differences with examples, and involve domain or policy experts in judging severity.

Measure latency, throughput, failure rate and cost per request under realistic load. Key latency metrics are time to first token and total response time, usually reported at the 95th percentile. Cost depends on tokens in and out, so test with realistic prompt and answer lengths, including Indian language text. Also test rate limits, timeouts and fallback behaviour when the model provider is slow or unavailable.

Start with scope and risks: which topics, which actions and what must never happen. Build an evaluation set per language, including Hinglish, with expected facts per question. Test correctness and groundedness, refusal for out-of-scope questions, escalation to humans, prompt injection, personal data leakage, latency and cost. Define pass criteria, such as minimum accuracy per category, and get native speakers to review a sample. Rerun the suite on every prompt or model change.

Not necessarily; variation is normal for LLMs, so first check whether all versions are correct and consistent in meaning. If they differ in facts, policy or safety, that is a bug. Run the same question many times, measure how often the answer is wrong, and report that rate rather than a single example. Fixes may include lower temperature, clearer prompts or better grounding, depending on the cause.

Run the full evaluation set on both the old and new versions and compare scores per category, along with latency, cost and format compliance. Review a sample of changed answers by hand, because averages can hide new failure types. Rerun safety and injection suites too. Roll out gradually, for example to a small share of traffic, and pin the model version in production so updates happen only when you choose.

Most likely the tests had weak assertions or missed the risky scenario entirely. Run a root-cause analysis: find the missing or weak test, check whether AI-generated tests were reviewed, and see whether coverage was measured on requirements or just on code lines. Fix by adding a regression test for this bug, tightening the review checklist for generated tests and introducing checks that confirm tests fail when the feature is broken.

Focus on actions, not just answers. Check that the agent calls the right tools with correct arguments, respects permissions, asks for confirmation before irreversible steps and stops safely when a tool fails. Test multi-step tasks end to end, measure task success rate and steps per task, and attack it with injected instructions to trigger unauthorised actions. Use a sandbox, never real payments. Read about AI agents and our agentic AI interview questions.

I would support the goal and reset the expectation with a plan. Propose a pilot: use AI to generate and maintain tests for one module, measure time saved, defects caught and false passes, and keep human review in place. Share results after a few sprints and expand where the numbers are good. Full automation without review risks weak tests and missed bugs, which is a quality risk the manager should see clearly.