Let Claude fix the lies in your homegrown analytics

I have an admin dashboard wired up with Ahoy. I picked Ahoy originally because I wanted everything in Rails and didn't want a third-party dependency. Later I bolted a third-party tool alongside it (Simple Analytics, Plausible, pick whichever — the point was to have a comparator).

One day I stared at the two side by side and noticed unique visitors differed by 5 to 10×. My first instinct: the third-party tool must be missing data. I asked Claude to glance at it. Turns out my own dashboard wasn't lying for one reason — it was three different bugs stacked on top of each other. This post is those three traps.

Trap 1: server_side_visits = true

Ahoy defaults to browser-side JS tracking. I had flipped on server_side_visits = true because I wanted to capture requests that didn't run JS — bots, API calls, cURL.

What I hadn't priced in: with that flag on, every controller action creates a Visit row.

• A user comes in, reads two articles (2 pageviews), hits a like API (JSON), gets a redirect to sign in and back, plus a Turbo Frame sub-request that lazy-loads the next page of content. Real Visit row count: comfortably 6–10.
• "Unique visitors" used to be Visit.distinct.count(:visitor_token). That number was inflated 5–10× from the truth.

The third-party tool only counts visits that fired a real $view event, so it never inflated.

The fix isn't complicated, but the framing has to change: the headline number can't use total Visits, it has to use "Visits that produced at least one pageview". The remaining API / redirect / Turbo Frame sub-requests get bucketed into a separate noise_visits_count — they aren't visitors, but they are real traffic, and surfacing them is useful.

visits_with_any_pageview = Ahoy::Event
  .where(name: "$view", time: range)
  .joins(:visit)
  .distinct
  .pluck("ahoy_visits.id")

# headline only counts these
unique_visitors = Ahoy::Visit.where(id: visits_with_any_pageview).distinct.count(:visitor_token)
noise_visits_count = total_visits - visits_with_any_pageview.size

Trap 2: bounce_rate and engaged_rate never sum to 100%

This bug looks subtler, but it's the one that makes my back cold.

bounce_rate: visits that viewed only one page / visits that viewed at least one page.
engaged_rate: visits with ≥2 pageviews or stay >30s / total visits.

Two different denominators.

Intuitively the two should be complements — a visit either bounced or engaged, so they ought to sum to ~100%. The dashboard showed bounce 47% + engaged 31% = 78%. Where did the other 22% go?

Into the bucket of "Visits that didn't produce any pageview" from Trap 1. They landed in the engaged_rate denominator (total visits) but were excluded from the bounce_rate denominator (pageview-visits).

Fix: align both denominators to visits_with_any_pageview, then both rates sum to ~100%, and lock that down with a spec.

denominator = visits_with_any_pageview.size
bounce_rate  = bounced_visits.to_f  / denominator * 100
engaged_rate = engaged_visits.to_f  / denominator * 100
# spec: expect(bounce_rate + engaged_rate).to be_within(1.0).of(100)

I'd been looking at the dashboard for days and never noticed those two numbers didn't sum to 100% — because I was always looking at them separately. The person who put them next to each other was Claude, while wiring up an admin overview component, who asked me: "why aren't these two complementary?"

Trap 3: start_with?("/up") silently drops entire users

This is the most absurd one.

config/initializers/ahoy.rb has an Ahoy.exclude_method that filters out paths I don't care about — health checks, assets, admin, internal jobs. The original:

Ahoy.exclude_method = lambda do |controller, request|
  path = request.path
  path.start_with?("/rails/", "/cable", "/jobs", "/up", "/assets", "/admin", "/ahoy") ||
    path.end_with?(".json", ".xml", ".gz") ||
    path.start_with?("/sitemap")
end

"/up" is the Rails 8 health check endpoint. start_with?("/up") looks fine — until you remember the site has vanity username profiles: every user's home is /<username>.

Which means:

• /update (the user) is excluded — starts with /up
• /administrator is excluded
• /jobsworth is excluded
• /cabletv is excluded
• /ahoyo would be excluded too if that user existed

These users' pageviews never enter Ahoy. They view their own profile, view others', refresh, leave comments, and every page hit is silently dropped. From the dashboard's perspective, those users have never used the site.

The third-party tool didn't share that exclude list, so it could see those users. That accounts for some of the 5–10× delta.

Fix: exact match for /up, segment-anchored regex for the namespace prefixes — /admin/ excluded, /administrator not.

NAMESPACE_PREFIX_PATTERN = %r{\A/(cable|jobs|assets|admin|ahoy)(/|\z)}.freeze

Ahoy.exclude_method = lambda do |controller, request|
  path = request.path
  path == "/up" ||
    path.start_with?("/rails/", "/sitemap") ||
    NAMESPACE_PREFIX_PATTERN.match?(path) ||
    path.end_with?(".json", ".xml", ".gz")
end

The spec gets path-boundary cases: /up excluded but /update not, /admin/foo excluded but /administrator not. Without those specs you'll trip on the same class of bug the next time someone refactors the exclude list.

Bonus cleanup: backfill rake's second-boundary dedup

Not the main course, but the same PR cleaned this up.

A rake task backfills historical sign-in events as visits. Its dedup logic put timestamps into a per-second bucket via Time#to_i:

existing_buckets = existing_visits.map { |v| v.started_at.to_i }
return if existing_buckets.include?(time.to_i)

Problem: the user double-clicks sign-in. The two timestamps land at 12:34:56.9 and 12:34:57.1. After to_i they become 56 and 57, don't match, dedup fails, two visits get backfilled.

Fix: switch to a 1-second time window:

return if existing_visits.any? { |v| (v.started_at - time).abs < 1.0 }

This also fixed a flaky spec that depended on the random sub-second offset of 1.hour.ago.

The actual lesson: you need a second clock

If you only look at one clock, the wrong number can stay wrong forever.

I had been looking at my admin dashboard for days without noticing that "unique visitors" was 5–10× the real number — because I had no reference. Once I bolted the third-party tool alongside, a new question popped up out of nowhere: "why is the gap this large?" The brand of the third-party tool doesn't matter — Simple Analytics, Plausible, anything that gives you a unique-visitor and pageview count will do. Its purpose is calibration, not replacement.

For Ahoy specifically, where you can do both browser-side and server-side tracking, the three traps that bit me are all here:

1. With server_side_visits = true, the headline number must count "Visits that produced a pageview", not total Visits.
2. Ratios like bounce / engaged must explicitly align denominators — write the spec: expect(bounce + engaged).to be_within(1.0).of(100).
3. start_with? in exclude_method assumes you have no vanity URLs. If you do, switch to exact match plus segment-anchored regex.

Self-hosted analytics is unusually vulnerable to confirmation bias — you wrote the tracker, you trust the tracker. Letting Claude pull in the third-party numbers and compare made all three errors surface together.

If your admin dashboard hasn't been audited in a while, bolt a second clock alongside it. The bigger the gap, the more you'll find.