As a Sales Professional, know who your Ideal Customer Profile, and where to find those prospects and their data is essential when prospecting for clients that you can help. See, as a Seller, I see my efforts as an opportunity to help solve people's problems. One of the biggest perks of being in IT Solution Sales is the fact that you get insights into so many Verticals and Industries. As someone who tends to get bored doing the same thing repeatedly, exposure to so many different businesses is the best part of what I do.
Australia has many public registers of professions. Anyone can download it. Almost nobody does anything useful with it, because a spreadsheet with hundreds to thousands of rows is not information, it is raw material.
This is the story of turning that raw material into a system I actually use.
What the system holds today
The current database is PostgreSQL, running in Docker. As of writing it contains:
| Table | Records | What it holds |
|---|---|---|
| prospects | 65,153 | Registered target companies |
| members | 114,396 | Responsible people/stakeholders extracted |
| enrichment_data | 5,901 | Revenue, funding sources, budget signals |
| prospect_tags / tags | 1,880 | Qualification categories |
| research_notes | 0 | Collaborative notes during research |
| scrape_progress | 0 | Checkpointing so collection can resume |
Seven tables. Two of them carry the weight. Everything else exists to make the first two usable.
The architecture
The application around the data is a conventional stack, chosen because it is boring and works and after all, my AI Agent chose it, not me, but I knew enough to say go for it:
- Backend: Node.js with Express and TypeScript
- Frontend: React with Vite and TypeScript
- Database: PostgreSQL in Docker
- Orchestration: Docker Compose for local development
- Collection: a set of Node scrapers, checkpointed so a run can resume where it stopped
The scraping layer is the part that required the most care, and not for technical reasons.
The part nobody writes about: collecting data respectfully
It is trivially easy to write a scraper that hammers a public register at full speed. It is also the wrong thing to do, and it gets you blocked within the hour.
The rules I settled on:
- Rate limit deliberately. The scraper runs every 10 minutes on a schedule and collects a bounded batch, rather than trying to pull everything in one aggressive pass.
- Checkpoint everything. A scrape_progress table means an interrupted run resumes instead of restarting. This also means you never re-request data you already have.
- Only collect what is already public and already published for this purpose. Responsible-person records are published by the regulator specifically so the public can see who governs a charity.
- Back up before every enrichment pass. A nightly backup runs at 11pm. Data you cannot restore is data you will eventually lose.
- Report on the collection, not just the data. Status reports run twice daily so I know whether the pipeline is healthy without going and looking.
The technical challenge in data collection is throughput. The actual challenge is restraint.
Where AI made the difference
Three places, specifically.
Structure extraction
Public registers are built for human reading, not machine consumption. Names, roles and organisation relationships arrive in inconsistent shapes. Getting from a rendered page to clean, normalised rows is the sort of fiddly transformation work that AI is genuinely good at and humans genuinely hate.
Qualification logic
Having 65,000 organisations is not useful. Knowing which 300 of them look like real prospects is. The tagging system encodes qualification rules: organisation size, board composition, funding signals, the presence of governance needs that map to what I would actually be selling. That logic was built and refined conversationally, which is much faster than specifying it up front.
Automation of the boring loop
Scrape, enrich, report, back up, repeat. None of that needs me. All of it needed to be built once and then left alone.
What I got wrong
I over-engineered the first version of the proposal layer. When I showed early output, the honest feedback was that it was over the top for what the situation needed. That was correct. A deliverable should be proportionate to the decision it supports, and an agent that can produce forty pages will happily produce forty pages if you do not tell it not to.
The second mistake was assuming the data was the product. It is not. The data is the input. The product is the answer to a specific question: which organisations have the governance characteristics that make them worth a conversation, and who is on their board.
The transferable lesson
Every industry has a public dataset like this sitting in plain sight. Registers, filings, licences, planning approvals, procurement notices. Most of them are ignored because the gap between 'downloadable' and 'usable' is large enough to stop people.
That gap is exactly what AI tooling closes now, and it closes it for individuals, not just for organisations with a data team. The hard part was never getting the data. The hard part is deciding what question you are trying to answer with it, and then having the discipline to only build what answers that question.