bsky-app/bskyweb/embedr-static/robots.txt
bnewbold 58842d03a9
rebased embedr (#3511)
* skeleton of embedr service, based on bskyweb

* embedr container setup

* builds on this branch

* actual routes

* fix embedr go:embed

* tweak embedr dockerfile

* progress on embedr

* fix path params

* tweaks to build process

* try to get embedr dockerfile to install embed deps

* build this branch

* updates to match sam's output HTML

* try to unbreak embedr dockerfile

* small embedr tweak

* docker hack

* get embed.js copied over to embedr

* don't x-frame-options for embed.bsky.app

* bskyembed: remove a console.log

* use html/template for golang snippet generation

* simplify embedr API fetches

* missing file

* Rm console.log fully

---------

Co-authored-by: Dan Abramov <dan.abramov@gmail.com>
2024-04-13 12:20:06 -07:00

9 lines
367 B
Text

# Hello Friends!
# If you are considering bulk or automated crawling, you may want to look in
# to our protocol (API), including a firehose of updates. See: https://atproto.com/
# By default, may crawl anything on this domain. HTTP 429 ("backoff") status
# codes are used for rate-limiting. Up to a handful concurrent requests should
# be ok.
User-Agent: *
Allow: /