No internet connection
  1. Home
  2. Support

Crawl on install goes on for a huge amount of time

By Jeff Reifman @jeffreifman2018-11-26 22:29:37.315Z

Hi, I'm also seeing the post-installation crawl large # and never finishes. I've tried twice.

My site is very simple although the theme has a number of media items, it shouldn't take this long.

Crawl is > 1200 this time and was even higher last time.

Could you please help me debug this? Thanks! ... cool concept

  • 10 replies

There are 10 replies. Estimated reading time: 11 minutes

  1. J
    Jeff Reifman @jeffreifman2018-11-26 22:41:48.821Z

    I tried another plugin just to see if it's faster to install. ... it also says there are 18000+ files.... so ... that might be the problem. Not sure where they are coming from.

    1. J
      In reply tojeffreifman:
      Jeff Reifman @jeffreifman2018-11-27 02:17:39.932Z

      After letting it run to completion with Github deployment, I ran into rate limit errors with the API

      Is there anyway around this?

      2018-11-27 01:58:32 GITHUB: Error creating blob (API limits exceeded?):Github\Exception\RuntimeException: Not Found in /var/www/lookahead/wp-content/plugins/static-html-output-plugin/library/Github/knplabs/github-api/lib/Github/HttpClient/Plugin/GithubExceptionThrower.php:87
      Stack trace:
      #0 /var/www/lookahead/wp-content/plugins/static-html-output-plugin/library/Github/php-http/httplug/src/Promise/HttpFulfilledPromise.php(34): Github\HttpClient\Plugin\GithubExceptionThrower->Github\HttpClient\Plugin{closure}(Object(GuzzleHttp\Psr7\Response))
      #1 /var/www/lookahead/wp-content/plugins/static-html-output-plugin/library/Github/knplabs/github-api/lib/Github/HttpClient/Plugin/GithubExceptionThrower.php(88): Http\Client\Promise\HttpFulfilledPromise->then(Object(Closure))
      #2 /var/www/lookahead/wp-content/plugins/static-html-output-plugin/library/Github/php-http/client-common/src/PluginClient.php(160): Github\HttpClient\Plugin\GithubExceptionThrower->handleRequest(Object(GuzzleHttp\Psr7\Request), Object(Closure), Object(Closure))
      #3 /var/www/lookahead/wp-content/plugins/static-html-output-plugin/library/Github/php-http/client-common/src/PluginClient.php(174): Http\Client\Common\PluginClient->Http\Client\Common{closure}(Object(GuzzleHttp\Psr7\Request))
      #4 /var/www/lookahead/wp-content/plugins/static-html-output-plugin/library/Github/php-http/client-common/src/PluginClient.php(87): Http\Client\Common\PluginClient->Http\Client\Common{closure}(Object(GuzzleHttp\Psr7\Request))
      #5 /var/www/lookahead/wp-content/plugins/static-html-output-plugin/library/Github/php-http/client-common/src/HttpMethodsClient.php(203): Http\Client\Common\PluginClient->sendRequest(Object(GuzzleHttp\Psr7\Request))
      #6 /var/www/lookahead/wp-content/plugins/static-html-output-plugin/library/Github/php-http/client-common/src/HttpMethodsClient.php(192): Http\Client\Common\HttpMethodsClient->sendRequest(Object(GuzzleHttp\Psr7\Request))
      #7 /var/www/lookahead/wp-content/plugins/static-html-output-plugin/library/Github/php-http/client-common/src/HttpMethodsClient.php(107): Http\Client\Common\HttpMethodsClient->send('POST', '/repos/newsclou...', Array, '{"content":"PCF...')
      #8 /var/www/lookahead/wp-content/plugins/static-html-output-plugin/library/Github/knplabs/github-api/lib/Github/Api/AbstractApi.php(166): Http\Client\Common\HttpMethodsClient->post('/repos/newsclou...', Array, '{"content":"PCF...')
      #9 /var/www/lookahead/wp-content/plugins/static-html-output-plugin/library/Github/knplabs/github-api/lib/Github/Api/AcceptHeaderTrait.php(31): Github\Api\AbstractApi->postRaw('/repos/newsclou...', '{"content":"PCF...', Array)
      #10 /var/www/lookahead/wp-content/plugins/static-html-output-plugin/library/Github/knplabs/github-api/lib/Github/Api/AbstractApi.php(148): Github\Api\GitData\Blobs->postRaw('/repos/newsclou...', '{"content":"PCF...', Array)
      #11 /var/www/lookahead/wp-content/plugins/static-html-output-plugin/library/Github/knplabs/github-api/lib/Github/Api/AcceptHeaderTrait.php(26): Github\Api\AbstractApi->post('/repos/newsclou...', Array, Array)
      #12 /var/www/lookahead/wp-content/plugins/static-html-output-plugin/library/Github/knplabs/github-api/lib/Github/Api/GitData/Blobs.php(67): Github\Api\GitData\Blobs->post('/repos/newsclou...', Array)
      #13 /var/www/lookahead/wp-content/plugins/static-html-output-plugin/library/StaticHtmlOutput/GitHub.php(127): Github\Api\GitData\Blobs->create('newscloud', 'lookahead', Array)
      #14 /var/www/lookahead/wp-content/plugins/static-html-output-plugin/library/StaticHtmlOutput.php(440): StaticHtmlOutput_GitHub->upload_blobs(false)
      #15 /var/www/lookahead/wp-content/plugins/static-html-output-plugin/wp-static-html-output.php(147): StaticHtmlOutput_Controller->github_upload_blobs()
      #16 /var/www/lookahead/wp-includes/class-wp-hook.php(286): wp_static_html_output_ajax('')
      #17 /var/www/lookahead/wp-includes/class-wp-hook.php(310): WP_Hook->apply_filters('', Array)
      #18 /var/www/lookahead/wp-includes/plugin.php(453): WP_Hook->do_action(Array)
      #19 /var/www/lookahead/wp-admin/admin-ajax.php(100): do_action('wp_ajax_wp_stat...')

      1. Leon Stafford @leonstafford2018-11-27 04:13:37.779Z

        Hi Jeff,

        Thanks for posting. The latest development build does have rate limit prevention and a bunch of new fixes and features. I’m just resolving one blocking issue with it, then will post it on GitHub Releases page / Slack channel.



      2. In reply tojeffreifman:
        Leon Stafford @leonstafford2018-11-27 12:32:29.907Z

        Hi @jeffreifman,

        Please try this latest development build:

        And let me know if any issues.

        To speed up exports, go to Advanced tab and increase the Crawl increment as high as possible without the server running out of resources.

        Also, ensure that the Discover URLs option is also set under the Crawl tab, it's on by default for new installs, but currently not when upgrading...

        There are speed improvements for the various deploy methods, too.

        Not a thoroughly tested version, though, so please let me know any issues you encounter and I'll fix for the release.



        1. J
          In reply tojeffreifman:
          Jeff Reifman @jeffreifman2018-11-27 20:04:45.665Z

          Thanks for the reply. The link doesn't work. It's not public on github in releases either.

          1. Leon Stafford @leonstafford2018-11-27 21:26:21.640Z

            Sorry Jeff, updated again since. The zip attached to this release is the one:

            1. JJeff Reifman @jeffreifman2018-11-28 19:52:40.767Z

              Posted separately that Github not working with the dev release.

              Also, it's not clear from your new dropdown ... how do I just generate the static site into a different directory on my webserver or a subdirectory? There's no clear option for this. Which option in the drop down does this?

              One thing I've set up is a git repo in a subdirectory on my server ... and I can write a batch script to detect changes and recommit and push. ...

              1. In reply toleonstafford:
                JJeff Reifman @jeffreifman2018-11-29 21:03:24.295Z

                Hi Leon, I sent you an email and posted a few of the issues to Github. Let me know when you get a chance to respond. Thanks!

            2. J
              In reply tojeffreifman:
              Jeff Reifman @jeffreifman2018-11-28 19:44:38.281Z

              With test release:

              FAIL: Unable to complete test upload to GitHub

              I did not add any special access to the personal access key.

              Also, I recommend you switch to deploy keys with write access ... as these can be established for a single repo.

              It's not safe to be installing github personal keys in WP sites.

              1. Leon Stafford @leonstafford2018-11-30 06:28:25.385Z

                Jeff, good point re the deploy keys.

                You can opt not to save your credentials in the plugin at all, just paste the sensitive ones in before a deploy.

                Ideally, people running this plugin can secure their WP via htaccess or run locally, mitigating most security risks.

                With the next release, once I have WP-CLI support built-in, you'll get easier ways to form your own deployment pipelines on the CLI, where you could just do the export/post-processing with the plugin, then do your own GitHub deploy however you like. This will make for much more nicer CI/CD integrations for power users.