Hey, what are you doing this weekend? Want to crawl a site? Me neither, but in this industry, sometimes we have to.

“Our IT team says it’s fine if you crawl our site. Go ahead and start at 4:30 AM on Sunday.”

Here at Greenlane, we’ve even got a dedicated crawling computer (named R2D2!) with a super-pumped-up install of Screaming Frog that can go as fast and hard as an IT team lets us. And they often tell us to have at it… when the site isn’t busy. Which is usually like 4:30 in the morning on Sunday. Which really interferes with my long-standing whiskey and sleeping plans. So in order to preserve our sanity, we built a way to schedule crawls automatically: Punctual Frog.

Introducing Punctual Frog

punctual-frog-launched

Screaming Frog can bring a some site to its knees. Chances are, whoever is in charge of IT heard the word “crawler” and literally cringed. Although tools like Screaming Frog are essential to understanding the structure and function of a website, they can push a web server to its limits. Unless you set some limits in Screaming Frog’s settings, it can get pretty heavy when added to a website’s normal daily traffic.

Download Punctual Frog for Windows

We built Punctual Frog after fighting a losing battle with the often finicky Windows Task Scheduler. Using Task Scheduler works, of course, and it’s even the way Screaming Frog originally recommended you schedule a crawl. But it’s just so hard to use. You set it up, test it, and then change a setting and it decides to not trigger.

Using the same Screaming Frog feature that lets Task Scheduler work – command line Screaming Frog – Punctual Frog is a quick and reliable downloadable/desktop app that lets you set a website and schedule a crawl. It’s simple, and it just works. And you can download it totally free!

How to make your frogs punctual

Using Punctual Frog is totally easy. No installation is necessary – just unzip the folder, and keep all the files together. To launch the app, just double-click!

Then, set and forget. Enter the website you’d like to crawl in the first box, and then use the “Browse” button to locate your Screaming Frog application. When you’ve got that, enter a date and time and hit “Schedule.”

Once the time hits, Screaming Frog will launch, and start crawling the site you specified! Screaming Frog will use your default settings, so if you’d like to change how the crawl works, you’ll need to change up your defaults.

screaming-frog-default-configuration

Important note: Make sure you set you computer not to sleep. Punctual Frog is punctual, but he’s not going to wake up once he’s sleeping!

Getting some weird Windows warning?

Good. That means your Windows security features are working really well, stopping all software from unknown developers from launching. All you have to do is give the app permission to run – click “More Info”, and then “Run Anyway”.

What do you think? Does this make your frogs more punctual?

Sean Malseed
Sean Malseed
Sean is a developer, technical SEO, and entrepreneur with over 10 years of industry experience. Prior to Greenlane, he founded CircleRank, a successful technical SEO agency in Philadelphia. He has spoken at major digital marketing conferences around the world, including PubCon, SMX, SES/ClickZ, and OMCap. He also co-organizes Philly’s monthly digital marketing event, SEO Grail. …
Read Bio
  • Jennifer

    Hi! This tool looks useful and promising 🙂 There is it also available for MAC OS?

    • Sean Malseed

      Thanks Jennifer! I’m working on the Mac version right now!

  • Ryan McLaughlin

    Hey team, nice work. I’ve used Task Scheduler on a VPS setup in the past which has worked pretty well, but one thing I’ve always wanted was functionality for automated File > Save upon completion of the crawl. Have you looked into whether this might be possible?

    • Sean Malseed

      Yeah I was using task scheduler too. I always forget to close a window or something and then the task doesn’t fire. I hear you about autosaving for sure. Right now it’s not possible, but I’m hoping that the next version of SF will change that.

  • Sebastian

    Can I also schedule the crawl of a urllist.txt?

    • Sean Malseed

      Not yet, but great idea!

  • Drew

    +1 for the Mac version. Looks like a great tool.

  • Daniel

    Dido on the Mac version. Nice work!

    • Sean Malseed

      Thanks Dan!

  • Brian Clark

    So this is AWESOME. Nicely done and thank you!

    Any way to have the tool handle pages/sections of a site that require login?

    Thanks!

  • Kaitlin

    I would like to schedule a crawl in List mode, and then get an email alert with the results, is that possible?

    • Sean Malseed

      Hey Kaitlin! Unfortunately there are no email alerts built in to Screaming Frog. I’d love that too!

  • Canoonic

    Great tool! But… Do you have any plans to develop this tool for OS X?

  • Andy Rich

    Love the tool – thanks so much for developing it.
    Do you have a new download link? The dropbox link is broken currently.
    Thanks
    Andy

© 2017 Greenlane. All rights reserved.

Greenlane's digital marketing headquarters is located just outside of Philadelphia:

2550 Eisenhower Avenue, A203, Eagleville, PA 19403

Privacy Policy    RSS

Subscribe to our Newsletter