Two years ago, I worked as a director at Genmap, in a kind of interim position that combined the roles of company director and project director. It was one of the best experiences I've ever had. As someone who programs as a hobby, I learned a lot from the company's professionals.

And I learned. I learned a lot about crawlers.

After all, Genmap is a Brazilian company that specializes in sitemaps.

That time at the company was truly incredible. We built so many engines and bots that I lost count, with new features every day, all designed to find as many links as possible on a website (after all, that's what a sitemap does).

I learned about crawl depth, crawling, concurrency, and other incredible related topics that I'll talk about another time.

The Idea

Last night, I called a former coworker. We stayed on a video call for almost an hour because I couldn't get an idea out of my head.

The idea was to build a crawler for the IndieWeb.

No admin panel, dashboard, or anything else. I would use only one or two seed URLs as starting points, then let it crawl as much as it could handle.

My questions for my former coworker were things like:

•     What's the logic behind search engines?

•     How much hardware would a given task require?

•     What about bot blocking?

•     What about robots.txt?

And many others.

There really were many more questions. I needed to cut the conversation short before it ran all night.

Reverse Scope

Well, as with any other project, I start by writing down what I don't want it to be. I write down everything my project should stay away from. What remains is what it should be, what it could be, or what it could grow into.

It's more or less like saying, "This water won't be coffee..." You can understand that as, "Then it could be any kind of juice."

This way of thinking has always kept me optimistic about everything I've done in life, because if I knew where not to go, every other path was an adventure and any result was satisfying.

I had made a decision. I needed to find experienced developers to build an engine for the IndieWeb. That would take money, time, and attention. And, of course, every bit of help I could get.

From 10:00 p.m. until a little after 11:00, I thought about and wrote down everything my project shouldn't be. And look, I had the perfect reverse scope, hehe.

Finding the People

I joined my friends in the Portal IDEA channel on our private IRC network and dropped the bomb. In less than an hour, I had about four people interested. They seemed genuinely interested because we were doing it for the love of it and to learn.

I offered to cover the infrastructure. They would take care of the code.

After a few very complicated weeks in my life, I was actually pretty excited to help make this project happen.

Now We Wait

Now we'll wait a few months and see what happens...