Projects
Portfolio build 0.00%

QA case study · 01

Analytics-driven QA

Reading guideQuestionMethodEvidenceDecision

The question

How do you test what the interface cannot show?

In a mobile game, a tap can trigger analytics, experiment assignment, advertising callbacks and cloud-save transitions. The UI shows the outcome; ClickHouse shows the evidence.

Synthetic funnel7 days

Start 4,293
Complete 4,064
Fail 3
A failed ad event is an outcome signal—not automatically an ANR or crash.

The method

01ActionReproduce a controlled test step 02EventLocate the emitted telemetry 03QueryCheck sequence and parameters 04DecisionConfirm, investigate or prioritize

01 · Event trail

Rebuild a user session

Read only

Trace ordered analytics events and inspect experiment context without touching production state.

How to read this

This baseline trace reconstructs the session. Device-owned install and session rows can leave product fields empty, so they are filtered before event order is assessed.
Context ClickHouse · OLAP · columnar
SELECT
    created_at,
    activity_kind,
    event_name,
    JSONExtractString(base_parameters, 'ab_group') AS ab_group
FROM analytics.realtime_events
WHERE user_id = {user_id:String}
  AND toDate(created_at) = today()
  AND activity_kind NOT IN ('install', 'session')
ORDER BY created_at DESC;
Signal ordered event trailUse reproduce & verifyGuard synthetic identifiers

The evidence

Eight recurring QA questions

Choose a scenario to update the explanation, query and interpretation.

Conclusion · decision

01

The event sequence is the evidence

The logging contract is healthy when the emitted order and parameters match the controlled actions performed on the device.

Decision

Confirm the scenario, or investigate the first missing, duplicated or malformed event.

Article conclusions

  • Use telemetry to verify what the interface cannot expose.
  • Interpret counts against an expected result—not in isolation.
  • Keep diagnostics read-only; perform recovery only through approved tooling.

Graphite portrait

Albert Einstein

Graphite portrait of Albert Einstein
Original artwork · graphite on paper 2015