Part 1 - The Make-or-Buy Decision: When to Build Your Own Automation (And When to Just Pay For It)

I asked ChatGPT to write me a web scraper. It worked on exactly one website. I could spend 300 hours learning to fix it, or $100/month to make the problem go away. Here's the math that made the decision obvious.

The Manual Research Trap

If you've ever researched private equity portfolios, you know this nightmare: every firm publishes their investments publicly, but each structures the data differently. Some use elegant filterable grids. Others have basic lists. Many bury critical details—investment dates, ownership stakes, exit status—in paragraph descriptions requiring careful reading.

The result? Hours per firm. Days for comprehensive analysis. Error-prone manual copying into spreadsheets.

For a one-off project, you suffer through it. But for ongoing competitive intelligence, market mapping, or sales prospecting? The inefficiency becomes unsustainable.

I initially thought: "How hard can web scraping be? I'll just learn Python and extract this data myself."

Spoiler: Much harder than it looks.

When the "Quick Script" Becomes a Full-Time Job

I started where any finance person would—I asked ChatGPT to write me a scraper.

Me: "Write a Python script to scrape portfolio companies from a private equity website"

ChatGPT: [outputs 50 lines of BeautifulSoup code]

I ran it on a test website. It worked. Company names extracted perfectly. I felt like a coding wizard.

Then I tried it on a second PE firm's website. Error messages everywhere. The CSS selectors didn't match. The HTML structure was different. Nothing worked.

Back to ChatGPT: "Fix this error..."

Then: "Now it's not finding the company names..."

Then: "How do I handle pagination..."

Three hours later, I had five different versions of the script, each customized for a different website. I wasn't building a scraper—I was becoming a full-time maintenance programmer for a tool that broke constantly.

The problems kept stacking up:

• Structural inconsistency: No two portfolio pages are alike. CSS selectors that work for one site break instantly on another.
• JavaScript rendering: Modern websites don't put data directly in HTML—they render it client-side with React or Vue. Traditional scrapers only see empty divs.
• Data location variability: Is "Series B, 2022" an investment date or a company description? Is "Austin" a location or part of the company name?

I realized I had two paths forward: spend months becoming a web scraping expert, or find someone who already solved this problem and pay them.

The Make-or-Buy Framework for Automation

In finance, we have a framework for these decisions: make-or-buy analysis. You calculate the total cost of building something in-house versus purchasing it from a vendor, including opportunity costs.

I applied the same logic here:

Option A: Build It Myself

• Watch Python tutorials: 20 hours
• Learn web scraping fundamentals: 30 hours
• Debug JavaScript rendering issues: 15 hours
• Figure out why sites are blocking me: 10 hours
• Customize for each of 50 different websites: 250 hours (5 hours × 50)
• Total: ~325 hours

At my consulting rate of $200/hour, that's $65,000 worth of my time. And that assumes I even figure it out—realistically, I'd get stuck multiple times and waste even more hours.

Option B: Pay for Tools That Do It

• Research scraping APIs: 3 hours
• Learn to use the API: 5 hours
• Minor adjustments per site: 50 hours (1 hour × 50)
• Monthly API costs: ~$100-500
• Total: ~58 hours + $500/month

Even if I used the API for two years, I'd spend $12,000 in fees versus $65,000 in opportunity cost. The ROI was obvious.

Key Framework: The Make-or-Buy Decision Matrix

Calculate:Your time value (hourly consulting rate or opportunity cost)Realistic time to build from scratch (multiply your estimate by 3x)Cost of existing tools/APIs that solve 80% of the problemOngoing maintenance time (usually 20% of build time annually)

Decision Logic:If (your time × hours) > (tool cost × 24 months) → BUYIf you need customization beyond what tools offer → BUILDIf neither math works out → DON'T AUTOMATE

More importantly: I'm not trying to become a web scraping expert. I'm trying to solve a data aggregation problem. Big difference.

Choosing the Right Tools: My Three-Hour Research Sprint

I researched three major scraping APIs:

Apify: Mature platform with huge marketplace. Verdict: Powerful but overkill—like buying enterprise software when you need a simple tool.

Bright Data: Enterprise-grade with massive proxy network. Verdict: Like buying a semi-truck when you need a pickup. Minimum $500/month, overwhelming feature set.

Firecrawl: Dead simple API with built-in AI extraction. Verdict: Purpose-built for exactly my problem.

Why Firecrawl Won (In Plain English)

1. "Just Tell Me What You Want" Extraction

Instead of writing code to find data, I could just describe what I wanted: "Find me the company name, industry, CEO, and investment date."

Firecrawl's AI figures out where that data lives—whether it's in a table, a card, or buried in paragraph text. No CSS selectors, no HTML parsing, no technical knowledge required.

2. It Handles Modern Websites Automatically

Modern websites that load data with JavaScript? Firecrawl handles that automatically. Portfolio pages with "Load More" buttons, infinite scroll, paginated results? All handled transparently.

3. Clean Output When Things Go Wrong

When AI extraction fails (and it does sometimes), Firecrawl gives me clean markdown text. This makes it much easier to write simple fallback rules.

Cost reality check: Firecrawl charges about $0.10 per page scraped. Scraping a typical PE portfolio (20-30 pages) costs $2-3. That's cheap enough to test freely, expensive enough to optimize for—the perfect sweet spot.

Building a Product with AI Coding Tools

With Firecrawl solving the scraping problem, I still needed a web application. I'm not a developer, but I'd heard about AI coding tools that let you "vibe code" applications just by describing what you want.

I tested that claim.

The Complete Stack (In Human Terms)

Frontend (what users see): Lovable-generated React app—I described what I wanted in plain English, and it generated working code in five minutes.

Backend (behind-the-scenes logic): Supabase Edge Functions—no server management, just copy-paste code and deploy.

Database (storing results): Supabase PostgreSQL with a nice UI

Scraping engine (the hard work): Firecrawl API

Total setup time: One afternoon

Monthly costs: ~$50 (Supabase free tier + Firecrawl starter plan)

Lines of code I personally wrote: Maybe 100, mostly with AI assistance

Real developers would build this differently—better error handling, proper tests, cleaner architecture. They're right. But I shipped a working product in one weekend instead of spending six months in tutorial hell.

The 80/20 Rule for Non-Developers Building Software

Here's what I learned as a finance person building software:

Developer mindset: "I should learn to do this properly from scratch so I really understand it."

Finance mindset: "What's the fastest path to a working product that solves the problem?"

I could've spent six months learning React, mastering web scraping, becoming proficient in backend development. I'd be a much better programmer. I'd also have nothing to show for it except tutorial completion certificates.

Instead, I spent one weekend:

• Using Firecrawl to handle scraping complexity
• Using Lovable to handle frontend complexity
• Using Supabase to handle backend complexity
• Focusing my time on the unique problem: extracting investment data accurately

The 80/20 Rule for Automation:Use AI and low-code tools for 80% of generic functionalitySpend your time on the 20% that's unique to your problemShip something that works, then improve based on real feedback

Key Takeaway: Focus on Solving Real Problems

You're not trying to become a developer. You're trying to solve a problem. If someone already solved 80% of it, pay them and move on. Your time is worth more than learning every technical detail from scratch.

The question isn't "Can I build this myself?" It's "Should I?" Run the numbers. Calculate opportunity cost. And if the math says buy—buy without guilt.

I'm not a software engineer. I'm someone who solved a data problem that was costing finance professionals hundreds of hours per year. Firecrawl does scraping. Supabase does infrastructure. Lovable does UI. I do investment data extraction logic.

That's enough.

Next time: How I built a 3-phase scraping strategy with AI extraction, intelligent fallbacks, and zero per-site configuration—the technical architecture that turns URLs into structured data.

Quick exercise: Think of a manual task you do weekly. Estimate hours to automate it. Multiply by 3. Now check if a tool exists for $50/month. What's the answer?