Hey, what are you doing this weekend? Want to crawl a site? Me neither, but in this industry, sometimes we have to.
“Our IT team says it’s fine if you crawl our site. Go ahead and start at 4:30 AM on Sunday.”
Here at Greenlane, we’ve even got a dedicated crawling computer (named R2D2!) with a super-pumped-up install of Screaming Frog that can go as fast and hard as an IT team lets us. And they often tell us to have at it… when the site isn’t busy. Which is usually like 4:30 in the morning on Sunday. Which really interferes with my long-standing whiskey and sleeping plans. So in order to preserve our sanity, we built a way to schedule crawls automatically: Punctual Frog.
Introducing Punctual Frog
Screaming Frog can bring a some site to its knees. Chances are, whoever is in charge of IT heard the word “crawler” and literally cringed. Although tools like Screaming Frog are essential to understanding the structure and function of a website, they can push a web server to its limits. Unless you set some limits in Screaming Frog’s settings, it can get pretty heavy when added to a website’s normal daily traffic.
We built Punctual Frog after fighting a losing battle with the often finicky Windows Task Scheduler. Using Task Scheduler works, of course, and it’s even the way Screaming Frog originally recommended you schedule a crawl. But it’s just so hard to use. You set it up, test it, and then change a setting and it decides to not trigger.
Using the same Screaming Frog feature that lets Task Scheduler work – command line Screaming Frog – Punctual Frog is a quick and reliable downloadable/desktop app that lets you set a website and schedule a crawl. It’s simple, and it just works. And you can download it totally free!
How to make your frogs punctual
Using Punctual Frog is totally easy. No installation is necessary – just unzip the folder, and keep all the files together. To launch the app, just double-click!
Then, set and forget. Enter the website you’d like to crawl in the first box, and then use the “Browse” button to locate your Screaming Frog application. When you’ve got that, enter a date and time and hit “Schedule.”
Once the time hits, Screaming Frog will launch, and start crawling the site you specified! Screaming Frog will use your default settings, so if you’d like to change how the crawl works, you’ll need to change up your defaults.
Important note: Make sure you set you computer not to sleep. Punctual Frog is punctual, but he’s not going to wake up once he’s sleeping!
Getting some weird Windows warning?
Good. That means your Windows security features are working really well, stopping all software from unknown developers from launching. All you have to do is give the app permission to run – click “More Info”, and then “Run Anyway”.
What do you think? Does this make your frogs more punctual?