Your new signup button attracts more clicks, but fewer visitors finish the form. Did the redesign work? The answer depends on whether the experiment was meant to generate curiosity or completed signups. Deciding that after seeing the numbers invites an attractive result to become the goal.
Start with one primary success measure tied to the customer action that matters. For a signup-page experiment, that might be completed eligible signups per assigned visitor over a defined observation window. Button clicks can remain useful diagnostic information without becoming the deciding metric.
Write the decision before running the comparison
The UK government’s guide to A/B testing explains the role of a controlled comparison. Microsoft’s pre-experiment guidance also emphasises decisions made before results arrive. The practical lesson is to agree what you will measure, for whom and over what period.
An illustrative test brief could read: “Compare the existing explanation with a shorter version. Primary outcome: completed signup per visitor assigned to the test. Monitor support requests and form errors as guardrails. Keep the offer and eligibility rules unchanged.” This is a planning example, not a statistically complete design.
Define the denominator carefully. Comparing clicks per page view with signups per unique visitor mixes different units. Decide how repeat visits, bots, failed event delivery and visitors who switch devices are handled before calling the result a conversion rate.
Let supporting measures explain, not replace
If the primary outcome falls while clicks rise, investigate the journey. The new wording may create an expectation the form does not meet, or a technical issue may interrupt completion. Those are hypotheses to examine; the click increase does not by itself prove either one.
Use an appropriate sample-size plan, analysis method and stopping rule. Repeatedly checking results and stopping at the first favourable number can make conclusions unreliable. A small business without enough traffic may learn more from usability observation than from a comparison too small to resolve a meaningful difference.
When reporting the test, keep the original primary measure visible even if it disappoints. Include uncertainty, guardrail outcomes and known tracking limitations. The strongest report is one that helps the next decision, including a decision to retain the existing page or gather better evidence.