mirror of
https://github.com/carlospolop/hacktricks
synced 2024-11-25 06:00:40 +00:00
170 lines
12 KiB
Markdown
170 lines
12 KiB
Markdown
# Uncovering CloudFlare
|
||
|
||
{% hint style="success" %}
|
||
Learn & practice AWS Hacking:<img src="/.gitbook/assets/arte.png" alt="" data-size="line">[**HackTricks Training AWS Red Team Expert (ARTE)**](https://training.hacktricks.xyz/courses/arte)<img src="/.gitbook/assets/arte.png" alt="" data-size="line">\
|
||
Learn & practice GCP Hacking: <img src="/.gitbook/assets/grte.png" alt="" data-size="line">[**HackTricks Training GCP Red Team Expert (GRTE)**<img src="/.gitbook/assets/grte.png" alt="" data-size="line">](https://training.hacktricks.xyz/courses/grte)
|
||
|
||
<details>
|
||
|
||
<summary>Support HackTricks</summary>
|
||
|
||
* Check the [**subscription plans**](https://github.com/sponsors/carlospolop)!
|
||
* **Join the** 💬 [**Discord group**](https://discord.gg/hRep4RUj7f) or the [**telegram group**](https://t.me/peass) or **follow** us on **Twitter** 🐦 [**@hacktricks\_live**](https://twitter.com/hacktricks\_live)**.**
|
||
* **Share hacking tricks by submitting PRs to the** [**HackTricks**](https://github.com/carlospolop/hacktricks) and [**HackTricks Cloud**](https://github.com/carlospolop/hacktricks-cloud) github repos.
|
||
|
||
</details>
|
||
{% endhint %}
|
||
|
||
## Common Techniques to Uncover Cloudflare
|
||
|
||
* You can use some service that gives you the **historical DNS records** of the domain. Maybe the web page is running on an IP address used before.
|
||
* Same could be achieve **checking historical SSL certificates** that could be pointing to the origin IP address.
|
||
* Check also **DNS records of other subdomains pointing directly to IPs**, as it's possible that other subdomains are pointing to the same server (maybe to offer FTP, mail or any other service).
|
||
* If you find a **SSRF inside the web application** you can abuse it to obtain the IP address of the server.
|
||
* Search a unique string of the web page in browsers such as shodan (and maybe google and similar?). Maybe you can find an IP address with that content.
|
||
* In a similar way instead of looking for a uniq string you could search for the favicon icon with the tool: [https://github.com/karma9874/CloudFlare-IP](https://github.com/karma9874/CloudFlare-IP) or with [https://github.com/pielco11/fav-up](https://github.com/pielco11/fav-up)
|
||
* This won't work be very frequently because the server must send the same response when it's accessed by the IP address, but you never know.
|
||
|
||
## Tools to uncover Cloudflare
|
||
|
||
* Search for the domain inside [http://www.crimeflare.org:82/cfs.html](http://www.crimeflare.org:82/cfs.html) or [https://crimeflare.herokuapp.com](https://crimeflare.herokuapp.com). Or use the tool [CloudPeler](https://github.com/zidansec/CloudPeler) (which uses that API)
|
||
* Search for the domain in [https://leaked.site/index.php?resolver/cloudflare.0/](https://leaked.site/index.php?resolver/cloudflare.0/)
|
||
* [**CloudFlair**](https://github.com/christophetd/CloudFlair) is a tool that will search using Censys certificates that contains the domain name, then it will search for IPv4s inside those certificates and finally it will try to access the web page in those IPs.
|
||
* [**CloakQuest3r**](https://github.com/spyboy-productions/CloakQuest3r): CloakQuest3r is a powerful Python tool meticulously crafted to uncover the true IP address of websites safeguarded by Cloudflare and other alternatives, a widely adopted web security and performance enhancement service. Its core mission is to accurately discern the actual IP address of web servers that are concealed behind Cloudflare's protective shield.
|
||
* [Censys](https://search.censys.io/)
|
||
* [Shodan](https://shodan.io/)
|
||
* [Bypass-firewalls-by-DNS-history](https://github.com/vincentcox/bypass-firewalls-by-DNS-history)
|
||
* If you have a set of potential IPs where the web page is located you could use [https://github.com/hakluke/hakoriginfinder](https://github.com/hakluke/hakoriginfinder)
|
||
|
||
```bash
|
||
# You can check if the tool is working with
|
||
prips 1.0.0.0/30 | hakoriginfinder -h one.one.one.one
|
||
|
||
# If you know the company is using AWS you could use the previous tool to search the
|
||
## web page inside the EC2 IPs
|
||
DOMAIN=something.com
|
||
WIDE_REGION=us
|
||
for ir in `curl https://ip-ranges.amazonaws.com/ip-ranges.json | jq -r '.prefixes[] | select(.service=="EC2") | select(.region|test("^us")) | .ip_prefix'`; do
|
||
echo "Checking $ir"
|
||
prips $ir | hakoriginfinder -h "$DOMAIN"
|
||
done
|
||
```
|
||
|
||
## Uncovering Cloudflare from Cloud infrastructure
|
||
|
||
Note that even if this was done for AWS machines, it could be done for any other cloud provider.
|
||
|
||
For a better description of this process check:
|
||
|
||
{% embed url="https://trickest.com/blog/cloudflare-bypass-discover-ip-addresses-aws/?utm_campaign=hacktrics&utm_medium=banner&utm_source=hacktricks" %}
|
||
|
||
```bash
|
||
# Find open ports
|
||
sudo masscan --max-rate 10000 -p80,443 $(curl -s https://ip-ranges.amazonaws.com/ip-ranges.json | jq -r '.prefixes[] | select(.service=="EC2") | .ip_prefix' | tr '\n' ' ') | grep "open" > all_open.txt
|
||
# Format results
|
||
cat all_open.txt | sed 's,.*port \(.*\)/tcp on \(.*\),\2:\1,' | tr -d " " > all_open_formated.txt
|
||
# Search actual web pages
|
||
httpx -silent -threads 200 -l all_open_formated.txt -random-agent -follow-redirects -json -no-color -o webs.json
|
||
# Format web results and remove eternal redirects
|
||
cat webs.json | jq -r "select((.failed==false) and (.chain_status_codes | length) < 9) | .url" | sort -u > aws_webs.json
|
||
|
||
# Search via Host header
|
||
httpx -json -no-color -list aws_webs.json -header Host: cloudflare.malwareworld.com -threads 250 -random-agent -follow-redirects -o web_checks.json
|
||
```
|
||
|
||
## Bypassing Cloudflare through Cloudflare
|
||
|
||
### Authenticated Origin Pulls
|
||
|
||
This mechanism relies on **client** [**SSL certificates**](https://socradar.io/how-to-monitor-your-ssl-certificates-expiration-easily-and-why/) **to authenticate connections** between **Cloudflare’s reverse-proxy** servers and the **origin** server, which is called **mTLS**.
|
||
|
||
Instead of configuring it's own certificate, customers can simple use Cloudflare’s certificate to allow any connection from Cloudflare, **regardless of the tenant**.
|
||
|
||
{% hint style="danger" %}
|
||
Therefore, an attacker could just set a **domain in Cloudflare using Cloudflare's certificate and point** it to the **victim** domain **IP** address. This way, setting his domain completely unprotected, Cloudflare won't protect the requests sent.
|
||
{% endhint %}
|
||
|
||
More info [**here**](https://socradar.io/cloudflare-protection-bypass-vulnerability-on-threat-actors-radar/).
|
||
|
||
### Allowlist Cloudflare IP Addresses
|
||
|
||
This will **reject connections that do not originate from Cloudflare’s** IP address ranges. This is also vulnerable to the previous setup where an attacker just **point his own domain in Cloudflare** to the **victims IP** address and attack it.
|
||
|
||
More info [**here**](https://socradar.io/cloudflare-protection-bypass-vulnerability-on-threat-actors-radar/).
|
||
|
||
## Bypass Cloudflare for scraping
|
||
|
||
### Cache
|
||
|
||
Sometimes you just want to bypass Cloudflare to only scrape the web page. There are some options for this:
|
||
|
||
* Use Google cache: `https://webcache.googleusercontent.com/search?q=cache:https://www.petsathome.com/shop/en/pets/dog`
|
||
* Use other cache services such as [https://archive.org/web/](https://archive.org/web/)
|
||
|
||
### Tools
|
||
|
||
Some tools like the following ones can bypass (or were able to bypass) Cloudflare's protection against scraping:
|
||
|
||
* [https://github.com/sarperavci/CloudflareBypassForScraping](https://github.com/sarperavci/CloudflareBypassForScraping)
|
||
|
||
### Cloudflare Solvers
|
||
|
||
There have been a number of Cloudflare solvers developed:
|
||
|
||
* [FlareSolverr](https://github.com/FlareSolverr/FlareSolverr)
|
||
* [cloudscraper](https://github.com/VeNoMouS/cloudscraper) [Guide here](https://scrapeops.io/python-web-scraping-playbook/python-cloudscraper/)
|
||
* [cloudflare-scrape](https://github.com/Anorov/cloudflare-scrape)
|
||
* [CloudflareSolverRe](https://github.com/RyuzakiH/CloudflareSolverRe)
|
||
* [Cloudflare-IUAM-Solver](https://github.com/ninja-beans/cloudflare-iuam-solver)
|
||
* [cloudflare-bypass](https://github.com/devgianlu/cloudflare-bypass) \[Archived]
|
||
* [CloudflareSolverRe](https://github.com/RyuzakiH/CloudflareSolverRe)
|
||
|
||
### Fortified Headless Browsers <a href="#option-4-scrape-with-fortified-headless-browsers" id="option-4-scrape-with-fortified-headless-browsers"></a>
|
||
|
||
Use a headless browser that isn't deetcted as an automated browser (you might need to customize it for that). Some options are:
|
||
|
||
* **Puppeteer:** The [stealth plugin](https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra-plugin-stealth) for [puppeteer](https://github.com/puppeteer/puppeteer).
|
||
* **Playwright:** The [stealth plugin](https://www.npmjs.com/package/playwright-stealth) is coming to Playwright soon. Follow developments [here](https://github.com/berstend/puppeteer-extra/issues/454) and [here](https://github.com/berstend/puppeteer-extra/tree/master/packages/playwright-extra).
|
||
* **Selenium:** The [undetected-chromedriver](https://github.com/ultrafunkamsterdam/undetected-chromedriver) an optimized Selenium Chromedriver patch.
|
||
|
||
### Smart Proxy With Cloudflare Built-In Bypass <a href="#option-5-smart-proxy-with-cloudflare-built-in-bypass" id="option-5-smart-proxy-with-cloudflare-built-in-bypass"></a>
|
||
|
||
**Smart proxies** proxies are continuously updated by specialized companies, aiming to outmaneuver Cloudflare's security measures (as thats their business).
|
||
|
||
Som of them are:
|
||
|
||
* [ScraperAPI](https://www.scraperapi.com/?fp\_ref=scrapeops)
|
||
* [Scrapingbee](https://www.scrapingbee.com/?fpr=scrapeops)
|
||
* [Oxylabs](https://oxylabs.go2cloud.org/aff\_c?offer\_id=7\&aff\_id=379\&url\_id=32)
|
||
* [Smartproxy](https://prf.hn/click/camref:1100loxdG/\[p\_id:1100l442001]/destination:https%3A%2F%2Fsmartproxy.com%2Fscraping%2Fweb) are noted for their proprietary Cloudflare bypass mechanisms.
|
||
|
||
For those seeking an optimized solution, the [ScrapeOps Proxy Aggregator](https://scrapeops.io/proxy-aggregator/) stands out. This service integrates over 20 proxy providers into a single API, automatically selecting the best and most cost-effective proxy for your target domains, thus offering a superior option for navigating Cloudflare's defenses.
|
||
|
||
### Reverse Engineer Cloudflare Anti-Bot Protection <a href="#option-6-reverse-engineer-cloudflare-anti-bot-protection" id="option-6-reverse-engineer-cloudflare-anti-bot-protection"></a>
|
||
|
||
Reverse engineering Cloudflare's anti-bot measures is a tactic used by smart proxy providers, suitable for extensive web scraping without the high cost of running many headless browsers.
|
||
|
||
**Advantages:** This method allows for the creation of an extremely efficient bypass that specifically targets Cloudflare's checks, ideal for large-scale operations.
|
||
|
||
**Disadvantages:** The downside is the complexity involved in understanding and deceiving Cloudflare's deliberately obscure anti-bot system, requiring ongoing effort to test different strategies and update the bypass as Cloudflare enhances its protections.
|
||
|
||
Find more info about how to do this in the [original article](https://scrapeops.io/web-scraping-playbook/how-to-bypass-cloudflare/).
|
||
|
||
## References
|
||
|
||
* [https://scrapeops.io/web-scraping-playbook/how-to-bypass-cloudflare/](https://scrapeops.io/web-scraping-playbook/how-to-bypass-cloudflare/)
|
||
|
||
{% hint style="success" %}
|
||
Learn & practice AWS Hacking:<img src="/.gitbook/assets/arte.png" alt="" data-size="line">[**HackTricks Training AWS Red Team Expert (ARTE)**](https://training.hacktricks.xyz/courses/arte)<img src="/.gitbook/assets/arte.png" alt="" data-size="line">\
|
||
Learn & practice GCP Hacking: <img src="/.gitbook/assets/grte.png" alt="" data-size="line">[**HackTricks Training GCP Red Team Expert (GRTE)**<img src="/.gitbook/assets/grte.png" alt="" data-size="line">](https://training.hacktricks.xyz/courses/grte)
|
||
|
||
<details>
|
||
|
||
<summary>Support HackTricks</summary>
|
||
|
||
* Check the [**subscription plans**](https://github.com/sponsors/carlospolop)!
|
||
* **Join the** 💬 [**Discord group**](https://discord.gg/hRep4RUj7f) or the [**telegram group**](https://t.me/peass) or **follow** us on **Twitter** 🐦 [**@hacktricks\_live**](https://twitter.com/hacktricks\_live)**.**
|
||
* **Share hacking tricks by submitting PRs to the** [**HackTricks**](https://github.com/carlospolop/hacktricks) and [**HackTricks Cloud**](https://github.com/carlospolop/hacktricks-cloud) github repos.
|
||
|
||
</details>
|
||
{% endhint %}
|