#1 Job Board for tech industry in Europe

  • Job offers
  • Test Automation Engineer (Python+genAI)
    New

    Test Automation Engineer (Python+genAI)

    Type of work
    Full-time
    Experience
    Senior
    Employment Type
    Permanent, B2B
    Operating mode
    Remote
    Transition Technologies MS

    Transition Technologies MS

    We are a rapidly growing IT company with global reach. We deal with IT outsourcing and implementation projects in flexible cooperation models, providing access to competence and experts in technologies from mainstream to cloud. TTMS' greatest strength is its skilled professionals, so people are at the heart of our organisational culture.

    Company profile

    Tech stack

      English

      B2

      Python

      advanced

      AI

      advanced

      Automated Testing

      advanced

      ISTQB

      regular

    Job description

    Online interview

    We're searching for a Test Automation Engineer.


    Your responsibilities:

    • Generative AI Testing Strategy: Design and implement comprehensive testing strategies specifically tailored for LLM-based applications, including evaluation of model responses, RAG pipelines accuracy, and overall system reliability
    • Quality Assurance Framework Development: Utilize Client's testing frameworks that address both traditional software quality aspects and AI-specific concerns such as output consistency, contextual accuracy, and ethical compliance; co-create and maintain such frameworks when required
    • Test Automation Development: Design and implement automated testing solutions for continuous evaluation of LLM applications, including integration tests, performance tests, and specialized AI behavior tests
    • Edge Case Analysis: Identify and develop test scenarios for edge cases in LLM behavior, including handling of ambiguous inputs, potential biases, and unexpected response patterns
    • Bias and Fairness Testing: Design and execute tests to identify potential biases in model outputs and ensure fair treatment across different user groups and use cases
    • Security Testing: Collaborate within development teams to test for potential vulnerabilities specific to LLM applications, including prompt injection, data leakage, and other AI-specific security concerns
    • Test Documentation: Create and maintain comprehensive test documentation testing strategy, test cases, and testing guidelines specific to AI applications and compliant with Client practices; document the analysis of requirements and risks and develop tests accordingly
    • Performance Testing: Collaborate with other engineers to conduct thorough performance testing of GenAI applications, including response time analysis, load testing, and resource utilization monitoring


    We are looking for you, if you have:

    • Experience: 3+ years of experience in software testing, with at least 1 year focused on genAI applications
    • Technical skills: Strong proficiency in Python and testing frameworks (e.g., pytest, unittest); experience with AI testing tools and frameworks
    • AI/ML knowledge: Understanding of LLM architectures, RAG systems, and common failure modes in AI applications; familiarity with evaluation metrics for generative AI
    • Test Automation: Experience with test automation frameworks and CI/CD pipelines; ability to design and implement automated testing solutions for AI applications
    • Ability to analyze requirements, identify test scenarios, and prioritize testing activities based on risk and impact
    • Cloud Platforms: Practical experience with testing applications on cloud platforms (AWS, Azure) and working with cloud-based AI services
    • Monitoring tools: Experience with monitoring and observability tools, log analysis, and performance metrics tracking (e.g. DataDog, Grafana)
    • ALM tools: Experienced in ALM tools (e.g. Jira, GitLab, ServiceNow)
    • Security Testing: Understanding of security testing methodologies, particularly in the context of AI applications
    • Very good practical and theoretical knowledge of testing processes (ISTQB Advanced, etc.)
    • The successful candidate should also hold B.Sc., B.Eng., M.Sc., M.Eng. in Computer Science, Software Engineering, or equivalent degree with strong background in testing methodologies
    • Be able to communicate in English at the level of C1+


    We offer:

    • Participation in interesting and demanding projects
    • Flexible working hours
    • A great, non-corporate atmosphere
    • Stable employment conditions (contract of employment or B2B contract)
    • Opportunity to develop your career

    We reserve the right to contact the selected candidates.

    Undisclosed Salary

    Permanent, B2B

    Check similar offers

    Lider Testów Automatycznych

    New
    4IT Solutions
    0.11K - 0.13K PLN/h
    Warszawa
    , Fully remote
    Fully remote
    TypeScript
    JMeter
    Playwright

    iOS Test Automation Engineer

    New
    7N
    23.5K - 30.2K PLN/month
    Warszawa
    , Fully remote
    Fully remote
    iOS
    Selenium
    API Testing

    Tester Manualny

    New
    Alter Solutions Polska
    0.07K - 0.126K PLN/h
    Warszawa
    , Fully remote
    Fully remote
    Jira
    Postman
    Xray

    Engineer in Test / QA Automation Specialist (Python)

    New
    1dea
    20K - 40K PLN/month
    Kraków
    , Fully remote
    Fully remote
    Python
    Linux
    Embedded

    Senior Automation Software Tester

    New
    Aspire Systems Poland
    14K - 18.5K PLN/month
    Wrocław
    , Fully remote
    Fully remote
    Playwright
    Automated Testing
    C#