TubeMate

People use YouTube to learn, but long-form content is a poor reference format. Finding the one section of a two-hour talk that matters means scrubbing through it, notes taken while watching are fragmented, and there is no way to interrogate a video after the fact.

That was my starting hypothesis, not a validated market finding. I built the first version to test it. What the usage data eventually told me was more complicated than the hypothesis, and that is the more interesting part of this case study.

I own product and technical direction, and I wrote the backend, the web frontend and the Chrome extension. That covers the transcript pipeline, the summarisation chain, the chat layer, authentication, rate limiting and abuse controls, Stripe billing, the analytics and error-tracking setup, and deployment.

Two collaborators work on distribution, content and user acquisition. The engineering is mine; the growth work is not.

The obvious architecture for “ask questions about a document” is retrieval-augmented generation: chunk the transcript, embed the chunks, store them in a vector index, and retrieve the relevant ones per question. I decided against it, and it is the decision I get asked about most.

The reasoning was that the premise for RAG did not hold here. A typical target video runs 15–90 minutes, which is roughly 3,000–20,000 tokens of transcript. That fits inside Gemini 2.5 Flash Lite’s context window with room to spare. Retrieval solves the problem of content that cannot fit in context. I did not have that problem.

Adopting it anyway would have meant an embedding call on every video, a vector index to provision and keep consistent with Postgres, a retrieval step adding latency to every question, and a new failure mode where the right chunk simply is not retrieved and the model answers confidently from the wrong context. In exchange for that, I would have gained nothing a user could perceive, since the model can already see the whole transcript.

So summarisation stuffs the full transcript into one prompt, and chat passes the stored summary or transcript back as context. The free and premium tiers cap videos at 30 and 120 minutes respectively — a product decision that is really a technical one, since the cap is what keeps transcripts inside the context window.

The first version used one summarisation prompt for everything. It worked acceptably for lectures and tutorials and produced poor structure for podcasts and interviews, where there is no thesis to extract and the value is in the exchange.

I added a detection pass that runs a keyword heuristic over the transcript and selects one of five prompt templates: default, educational, tutorial, entertainment, or conversation. A video opening with “welcome back to the show” is summarised differently from one opening with “step one”.

I chose keywords over asking the model to classify the content first. Classification would have meant a second model call on every summary, roughly doubling cost and adding latency, to solve a problem that a list of phrases solves most of the time. The heuristic is crude and misroutes edge cases, and that is a trade I am comfortable defending.

Output structure is enforced by prompt instruction alone — the prompt asks for specific markdown headings and the response passes through a plain string parser. There is no schema and no function calling. If I were hardening this, structured output would be the first change, because prompt-instructed formatting fails occasionally and nothing downstream catches it.

After launch, registrations arrived faster than any distribution work justified. The accounts were real enough to pass validation and never verified their email addresses.

I treated it as a funnel problem rather than a security one. Rather than block registration harder, I made unverified accounts worthless: a require_verified_email dependency gates product use, so an unverified account can exist but cannot generate anything. That removed the incentive without adding friction for real users.

Layered around that, what runs today:

• Per-IP registration limit: 5 attempts per hour
• Per-IP login lockout: 10 failures per hour
• Browser fingerprint plus IP limit on the public summary endpoint: 3 per day
• Verification email resend limit: 1 per 60 seconds

The fingerprint limit exists because the public endpoint is the one path that works without an account, which made it the obvious target once the verification gate closed the other one.

Weekly signups after launch ran between 1 and 12. In mid-May they climbed sharply — 30, then 46, then 28 across three weeks. Then the following week they collapsed to 2, and have run between 2 and 9 since.

The 46-signup week looked like the product working. Set against the full funnel, it was not: 224 people have registered, 118 verified an email address, and 63 have ever generated a summary. Nearly half of everyone who signed up never confirmed they were a real person, and only a quarter did the one thing the product exists to do.

The number I had been watching was the one least connected to whether anyone found the product useful. Signups measure curiosity and are trivially inflatable by bots and disposable addresses. The verified-to-activated conversion is the honest measure, and at 53% it is a genuinely reasonable rate — the problem was never activation, it was that I was reporting the wrong denominator to myself.

I now treat activated users as the headline number. It is smaller and considerably more useful.

YouTube rate-limiting cascade. Between June and August, roughly 70 Sentry events traced to the same cause: once transcript volume passed a threshold, YouTube served the server’s IP a 429 behind a CAPTCHA gate. The transcript library retried immediately against the same 429, which sustained the block and generated the error cascade. I added a Redis-backed five-minute cooldown so a 429 stops the retry loop instead of feeding it. Fixed.

Retention data was never being recorded. While pulling the numbers above, I found last_login_at is null for every user in the database. Neither the password path nor Google OAuth writes it on successful authentication. The consequence is that I have no returning-user data for the entire life of the product, and no way to reconstruct it. The metric grid above says “not measured” for that row, and that is why.

A paying customer was not getting what they paid for. Cross-referencing Stripe against the database showed the one active subscriber carrying plan_type = 'free'. The webhook receives the subscription event but the handler does not sync the plan field, so the account was billed monthly while limited to free-tier caps.

Honestly: manually. I spot-check summaries during development and after changing a prompt template. The automated tests cover the pipeline up to the cache layer, and the model-invoking chains themselves are exercised by hand against the live model. There is no golden dataset, no regression suite for output quality, and no automated hallucination check.

That was survivable while I was the only person changing prompts, and it is the weakest part of the system. The content-type routing exists because I noticed bad podcast summaries by eye — which means the discovery path for quality regressions is currently “Mayowa happens to look”.

What it needs is a fixed set of test videos with expected section coverage, scored by rubric on each prompt change, plus a grounding check on chat answers against the transcript. That is the next engineering investment I would make, ahead of any new feature.

Pick the architecture the problem needs, not the one the category expects. RAG was the default answer for this product shape and it would have been the wrong one. Being able to explain why I did not build something has been more useful in conversation than the features I did build.

An unwatched metric is worse than no metric. I tracked signups because they were easy to see, and the number moved in a way that felt like progress while telling me nothing. Two of the three bugs above were found by finally interrogating my own instrumentation.

After launch, reliability beats features. The work that mattered most in the last few months was a cooldown, a verification gate and a rate limit — none of which are visible to a user who is having a good time, and all of which are the reason they are.

Build the evaluation harness before the second prompt template. The moment there was more than one prompt path, I needed a way to tell whether a change to one regressed another. I still do not have it.

Instrument the funnel, not the front door. Verified and activated counts should have been the dashboard from day one. Writing last_login_at is a two-line change I never made because nothing forced me to look.

Test the billing path like it has someone’s money in it. The Stripe webhook was verified as “receives events” and never as “produces the correct account state”. An integration test against a test-mode subscription would have caught it before a customer did.

Use structured output. Prompt-instructed markdown headings work until they do not, and nothing downstream validates the shape.