Found an old strategy document I wrote for something I called Tulipr, a scraper meant to find and grade suppliers automatically instead of researching them one by one. Never built it, but the thinking still holds up.

Two approaches, sketched side by side. The first: a general AI webscraper, search for a company, crawl every resulting page, and run the aggregated content through an LLM to pull out whatever information actually matters. The second, more conservative: skip the general crawl entirely and just target a curated list of known-good sources.

What's interesting rereading it now is that I'd already landed on "use the ChatGPT API to process text and extract valuable information" as the core mechanism, before RAG pipelines and structured extraction were the default answer to that problem. I also flagged the two things that actually make this hard in practice: sites actively blocking bots (captchas, proxies), and the fact that if you want a model that filters relevant links instead of just summarizing everything, you need labeled training data you probably don't have yet.

Estimated it would take "several months" to build properly. That estimate was right, which is probably why it stayed a design doc instead of a shipped project, the sensible move was designing it and moving on rather than sinking that time into a v1 that might not have been the right approach anyway.