heySec Shield · Data sources
Where Shield’s data comes from
What Shield reads
Shield tells people from automated programs. Besides what a visit itself shows, it reads two kinds of published data: lists that describe networks, and what crawler and AI-agent operators publish about their own traffic.
We use a source only where its publisher’s terms allow it. Where a source asks for credit, you find the credit below. The data is refreshed every day.
A network list says where traffic comes from, never who is behind it, and it is one of several inputs to a check. What a check collects from a visit, and how long it is kept, is in our Privacy Policy.
Networks
Shield turns an address into the country and the network it belongs to, and recognises networks that relay or host traffic.
| Source | What it gives | Terms |
|---|---|---|
| IPtoASN | The country and the network name of an address. | Public domain (PDDL v1.0). |
| The Tor Project | The addresses of Tor exit relays, from its Onionoo service. | CC0. |
| Apple | The addresses iCloud Private Relay publishes. | Published by Apple so that others can recognise its traffic. |
| Amazon Web Services, Google Cloud, Microsoft Azure, Oracle Cloud, Linode and Vultr | The address ranges each of them publishes for its cloud. | Published by each provider so that others can recognise its traffic. |
| monosans | The addresses of open proxies. | MIT licence. |
| Spamhaus | Address ranges that Spamhaus advises networks not to accept traffic from, the DROP lists. | Free of charge, with credit. |
The country of an address is the one its network is registered in. That is not always where its users are.
Search engines and other crawlers
A crawler is recognised only by what its operator publishes: a list of its addresses, or a reverse DNS name that leads back to the operator’s own domain. A name in a user agent proves nothing. Each operator below says on its own page why it publishes this.
| Operator | Where it says so |
|---|---|
| Verifying Google requests | |
| Microsoft Bing | How to verify Bingbot |
| Apple | About Applebot |
| OpenAI | Overview of OpenAI crawlers |
| Anthropic | Does Anthropic crawl data from the web |
| Perplexity | Perplexity crawlers |
| Amazon | About Amazonbot |
| DuckDuckGo | About DuckDuckBot |
| Common Crawl | Common Crawl FAQ |
| Mistral | Mistral robots |
| Seznam | About SeznamBot |
| The Pinterest crawler | |
| UptimeRobot | UptimeRobot locations |
AI agents that sign their requests
Some AI agents sign their requests, using HTTP message signatures (RFC 9421) as in the IETF’s Web Bot Auth draft. Shield recognises them by the public keys their operators publish. These are ChatGPT agent from OpenAI, Google-Agent from Google, Browserbase and Anchor Browser.
- ChatGPT agent allowlisting
- Google’s Web Bot Auth documentation
- Browserbase identity
- Anchor Browser and Web Bot Auth
Browserbase and Anchor Browser say that they sign, but not where their keys are published. Shield reads the keys from the operators’ own addresses.
Credits and notices
Where a source asks for credit, it is given here.
Contains data from the Spamhaus DROP lists, © The Spamhaus Project SLU. spamhaus.org/drop
IP to country and network data from IPtoASN, licensed under the Public Domain Dedication and License v1.0.
Tor exit data from the Tor Project’s Onionoo service, available under CC0.
Open proxy list © 2022 monosans, used under the MIT licence.
Licence text
MIT License
Copyright (c) 2022 monosans
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Questions about a source or a credit: [email protected].