AI Quality & Evaluation
Review AI responses for factual accuracy, instruction-following, consistency, usefulness, and failure points.
LISA MARIE KOCIAN
AI Quality
Assurance Analyst
Domain SME: E-Commerce & Digital Product Operations
I evaluate AI responses and digital products for accuracy, reliability, and usability. My work combines quality assurance, e-commerce operations, and human-centered evaluation to identify problems, understand why they happen, and help make technology more dependable.
Open to opportunities in AI Quality Assurance, LLM Evaluation, Content Quality, and AI Product Testing.
QUALITY REVIEW · WORKFLOW
DefineSet scope and quality criteria.
ReviewCheck accuracy, clarity, and fit.
DocumentRecord clear, reproducible findings.
RecheckVerify changes against the brief.
PROFESSIONAL FOCUS
What I Bring to the Table
Where technical review, digital operations, and practical judgment meet.
Programming & Technical Fluency
Experience with Python, Java, PHP, C++, and Ruby, combining programming knowledge with technical troubleshooting, software QA, and practical problem-solving.
Software & E-Commerce QA
Test website workflows, navigation, product data, responsive layouts, and customer-facing experiences.
Digital Product Operations
Combine web development, business operations, troubleshooting, content management, and documented reviews.
FEATURED WORK
Selected work,
examined closely.
Examples of what I built, reviewed, and learned along the way.
TORCH REPUBLIC / LIVE STOREFRONTCommerce.
Craft.
Quality.React development, Shopify integration & workflow QA
Craft.
Quality.React development, Shopify integration & workflow QA
01REACT DEVELOPMENT · SHOPIFY INTEGRATION · WORKFLOW QA
Torch Republic — AI-Assisted Commerce & Quality Engineering
An independently developed React-based commerce experience connecting boutique coffee merchandising, Shopify product and cart flows, and ongoing quality review across mobile journeys, catalog accuracy, and customer-facing interactions.
Evidence & scope: Working storefront, component and integration work, documented UI and catalog investigations, and iterative build checks. A complete live checkout regression suite and measured conversion outcomes are not claimed.
React & Base44Shopify IntegrationFunctional QAProduct Data AccuracyAI-Assisted DevelopmentMobile UX
LUKE’S LIGHT / LIVE WEBSITEGuidance.
Clarity.
Care.Human-centered digital safety platform
Clarity.
Care.Human-centered digital safety platform
02PRODUCT DESIGN · REACT DEVELOPMENT · SAFETY-FOCUSED QA
Luke’s Light — Child Safety & Digital Resource Platform
An independently developed digital resource platform created in memory of Luke Andrew Rogers, helping children, teens, and concerned adults navigate difficult situations and connect with established support services. Designed with a focus on accessibility, privacy awareness, responsible communication, and reliable user experiences.
Evidence & scope: React-based implementation, accessible interaction considerations, and safety-focused manual QA; no independent accessibility or clinical certification claimed.
Product Design & UXFront-End DevelopmentTechnical QAAccessibilityAI-Assisted DevelopmentDigital Safety
HUMAN-CENTERED EVALUATION
The Human Side of Technology
I'm drawn to the ways business and technology intersect with psychology and human behavior—particularly how large language models learn, respond, and improve through human evaluation. Behind every AI interaction is a person who deserves technology they can trust. I believe meaningful innovation requires more than technical progress. It requires curiosity, critical thinking, and a commitment to truth, empathy, and accountability.
Truth & grounding
Check claims against authoritative sources; flag hallucinations and disclose uncertainty rather than filling gaps with confident guesses.
Fairness & context
Look for exclusions, uneven treatment, and assumptions that may disadvantage users or misrepresent what they need.
Privacy & oversight
Avoid exposing sensitive information and keep human review central when an answer is ambiguous, consequential, or potentially harmful.
This describes an independent quality-review philosophy, not a formal safety audit, regulatory assessment, or certification.
EVALUATION METHODS
How I evaluate
AI outputs.
Practical checks for model responses, training examples, and the workflows that deliver them.
Evaluation rubrics & scoring
Score outputs against factual accuracy, instruction following, tone and brand consistency, and schema adherence for JSON or other structured data.
Adversarial tests & edge cases
Stress-test multi-step agent behavior with edge cases; classify hallucinations, prompt leakage, logic breaks, and behavioral drift, then document reproducible observations.
Training-data review (SFT)
Review sample prompt–response pairs for source accuracy, consistent labels, and clear evaluation criteria. Extending this into a fine-tuning workflow is a development goal.
End-to-end product QA & integration testing
Compare expected and actual behavior across AI workflows, e-commerce journeys, and app integrations within the available test environment.
DEFINE → REVIEW → DOCUMENT → RECHECK
Standards make review repeatable.
The same four-stage process applies to a product page, an AI-generated answer, or an entire customer journey. This is my working review framework—not an employer's proprietary standard.
Define
Identify users, requirements, approved sources, important risks, edge cases, and acceptance criteria.
Review
Compare actual behavior with expectations; check accuracy, instruction fit, consistency, usability, and unsupported claims.
Document
Record steps, conditions, source evidence, expected versus actual behavior, category, impact, and severity.
Recheck
Verify a change against the original issue and assess affected routes for regressions or new inconsistencies.
Defect severity reference
| Severity | Illustrative meaning |
|---|---|
| Critical | Severe safety, security, privacy, or system-integrity failure. |
| High | Material misinformation or a major workflow failure. |
| Medium | Significant functionality or content-quality issue. |
| Low | Minor formatting, style, or presentation issue. |
Illustrative criteria only. Severity depends on context, actual exposure, and the system's own risk model.
MY APPROACH
A repeatable process.
A clearer result.
I start with the expected behavior, test what actually happens, and document any differences. I turn those observations into clear, actionable findings that teams can investigate, resolve, and retest.
Learn more about my work01
Define
Set requirements, intended users, ground-truth sources, risks, and acceptance criteria.
02
Review
Compare output and behavior with sources, instructions, edge cases, and real user needs.
03
Document
Record conditions, expected versus actual results, impact, severity, and reproducible evidence.
04
Recheck
Retest the original issue after changes, then check nearby behaviors for regressions.
EXPERIENCE
Where I’ve put this
work into practice.
Ten years of hands-on web design and digital operations underpin my work in e-commerce QA, AI evaluation, publishing, and document quality.
AI Quality Review Contractor
Alignerr by Labelbox
Through qualification assessments and hands-on practice, I’ve developed familiarity with AI data annotation, labeling, and evaluation guidelines. I focus on accuracy, instruction-following, consistency, and the quality of human feedback used to improve AI outputs.
Published Author & Editorial Project Lead
The Stop Trying Method
I manage editorial quality from manuscript through publication, performing manual checks for clarity, accuracy, consistency, formatting, and reader usability. I review print and digital proofs, correct defects, and refine content through careful revision.
Founder / AI E-Commerce Product & Quality Lead
Torch Republic Coffee Company
I oversee the storefront and test its customer experience, using manual QA to identify issues with product information, mobile layouts, navigation, carts, and Shopify integrations. I also evaluate AI-assisted content and workflows for accuracy, clarity, and consistency.
Remote Freelance Paralegal / Legal Document QA
Self-employed contract support
I reviewed legal and supporting documents for accuracy, completeness, organization, and formatting. This detail-driven work strengthened my manual QA approach: checking facts against source records, catching inconsistencies, and documenting corrections with care.
Web Design & Digital Operations
Multiple Businesses & Organizations
For more than a decade, I’ve designed and maintained websites for multiple businesses and organizations. My hands-on QA work includes testing responsive layouts, navigation, usability, forms, and content accuracy, then troubleshooting defects and refining the experience.
CREDENTIALS
Certifications &
professional development
01
AWS Certified AI Practitioner
Amazon Web Services
02
Career Essentials in Generative AI
Microsoft & LinkedIn
03
Intermediate Machine Learning
Kaggle
04
Agentic AI
DeepLearning.AI
05
Introduction to Automation Diploma
UiPath Academy
06
Jira Fundamentals Badge
Atlassian University
EDUCATION
B.A. Psychology
Minor in Business Administration
Texas A&M University–Corpus Christi
TECHNICAL SKILLS
Software QA Fundamentals
- SDLC and STLC
- Risk-based testing and shift-left practices
- REST API, webhook, and microservice testing concepts; Postman / REST Assured learning
- Foundational SQL for data integrity checks; advanced commerce validation is a learning goal
- Jira issue tracking
- CI/CD and GitHub Actions concepts; automated QA pipelines are a development goal
Automation & Scripting
- Programming experience in Python, Java, PHP, C++, and Ruby; JavaScript fixture-validation demonstration
- Playwright / Cypress: browser automation methods under study; no completed production suite claimed
- AI-assisted visual review for layout, image, and responsive display issues across mobile and desktop
E-Commerce QA
- End-to-end review of product search, cart state, discounts, payment gateways, and checkout flows
Web Design & Digital Operations
- Web design and development for multiple businesses and organizations
- Responsive design and mobile optimization
- Website QA and UI/UX evaluation
- Content management systems, website troubleshooting, and maintenance
- E-commerce website development and digital business operations
- Hands-on tools: Shopify, Wix, Canva, and Base44
TOOLS IN CONTEXT
What I apply. What I'm learning.
Here’s where I’ve put my skills to work, along with the demonstration projects I’m building next.
Manual & software QA
Applied in independent projectsFollow product discovery, navigation, cart entry points, content accuracy, responsive layouts, and the behavior users actually encounter.
LLM evaluation & fixture testing
Demonstrated in portfolio projectWrite expected behavior, compare synthetic responses against a source-of-truth catalog, classify failures, and keep evidence beside each result.
Shopify, Base44 & website operations
Applied in independent projectsMaintain catalog content and site experiences, investigate broken routes or mobile controls, and recheck changes after updates.
JavaScript & structured data
Demonstrated in portfolio project; continuing developmentReview synthetic fixtures and deterministic criteria, validate JSON request/response shapes, and organize result summaries.
Python & SQL
Upcoming project: E-Commerce Data Quality AuditRelevant next steps include repeatable data comparisons, integrity checks, and concise reporting scripts; not claimed as production deployments.
Playwright / Cypress
Upcoming project: Automated Storefront Quality TestsTarget uses include repeatable browser flows and mobile viewport regressions. No finished automation suite is claimed here.
Postman & REST APIs
Upcoming project: Product API ValidationTarget uses include request/response checks, invalid input, status codes, and API contracts. The current harness validates local contracts, not a live API.
GitHub, Jira & CI/CD
Upcoming project: QA Workflow & Automated Test PipelineVersion history and clear defects support controlled changes; Jira triage and CI test pipelines are learning goals unless demonstrated by a linked artifact.
Selected QA terminology in practice
- Red-teaming
- Testing unusual or challenging situations to find ways AI may fail.
- Evaluation rubrics
- Structured criteria for scoring responses consistently against expectations.
- Regression testing
- Checking whether features still work after a change or fix.
- RLHF
- Reinforcement learning from human feedback; human preferences can be used to guide model training.
- SDLC / STLC
- Development and software-testing life cycles, from requirements through validation.
LET’S CONNECT
Good technology needs
careful eyes.
I’m interested in teams that value sound judgment, thoughtful testing, and clear documentation. If my work fits what you’re building, I’d be glad to talk.
Email Lisa Résumé and additional work samples available upon request by email.