This commit brings the SET Harvester into better conformance with the
HTTP specification. Notably, this makes it possible to use the Harvester
module behind strict layer 7 reverse proxies (such as Cloudflare, Ngrok,
and similar) that require proper HTTP response headers for all replies.
The major change is the addition of a proper HTTP `302 Found` redirect
using the `Location` HTTP header and the addition of an HTML5 document
type declaration (`<!doctype html>`) prepening the body of the reply
sent to a visitor when submitting the SET Harvester's phishing login
form. Smaller changes include correcting misspellings in other HTTP
headers (`Content_type` -> `Content-Type`) in various places.
The original version of pyminifakedns ignored the value of the
Query/Response Flag ("QR flag"), which is the bit immediately prior to
the DNS opcode field. The value of the QR flag should be checked for the
value 0 along with the opcode bits, which should also be zero.
This commit completes porting the internals of the MiniFakeDNS server
class to Python 3. This primarily means converting the Python 2 `str`
types to Python 3's `bytes` objects.
In the process, I've also translated the variable names from their
original Spanish into English, and added explanatory comments for how
the DNS header parsing is accomplished to enhance the educational
potential of SET.
Another small change is the addition of a new core helper function,
`detect_public_ip()`, which makes a couple parts of the codebase a
little more DRY by reducing code duplication across the `set.py` and
`setcore.py` files. This change also makes it possible to parameterize
the IP address that MiniFakeDNS server responds to requests with.
This is a relatively large commit because it refactors SET's interface
to the build-in DNS server it runs. Instead of a block of code inside of
the `setcore.py` file, a new module called `minifakedns` is added, which
houses all of the DNS-related code. Note that this commit *only*
refactors the DNS interface and its internals, it does not actually fix
the exception caused by receiving some DNS query, nor does it complete
the work required to set parameters for the DNS server, such as which IP
address it should respond with. It is just intended to make these
changes easier to introduce moving forward in upcoming chunks of work.
This replaces the `core.start_dns()` function with a new helper, whose
full path is `src.core.minifakedns.start_dns_server()`.
The previous implementation assumed the DNS server would be run from the
main thread, and thus have access to SIGINT, but this was never actually
possible because a `KeyboardInterrupt` was actually intercepted by SET
itself, before the DNS server code ever got to see it. This means that
the DNS server would never shut down cleanly.
This implementation changes that by using a simplistic sentinel value
(called `stop_flag`) that can be set as an instance attribute on the new
`MiniFakeDNS` object. When the sentinel value is `True`, the DNS server
thread will exit its listening loop.
Another change this introduces is the fact that, due to this new design,
the DNS server stops when `core.cleanup_routine()` is run. And, as a
note about that, this was running twice, once upon *startup* rather than
when SET shut down. This looked like a bug to me, and it was causing
problems for this DNS server design, so I removed that extra call and
performed some simplistic tests of various SET functionality to make
sure everything still works. (It seems fine, but might warrant a second
look.)
Finally, note that this commit breaks Python 2 compatibility due to the
use of a context manager handling the UDP socket. Given that the DNS
server was not really functional for some period of time before I
started looking at it more closely, the fact that Python 2 is officially
end-of-life'd (as of January 2020), the relative complexity of writing a
Python 2 and Python 3 implementation, and the fact that many comments
elsewhere in the SET codebase all seem to be nudging towards a Python 3
compatible upgrade, I am choosing to drop support for Python 2 in my own
patches, unless there is some considerable objection.
The DNS server in SET assumes that it's going to be able to bind to port
53 on all interfaces, but on most modern *nix machines, that's no longer
reasonable because `systemd-resolved` runs a stub resolver there. This
means an unhandled exception is raised as soon as `DNS_SERVER=ON` in the
`set.config` file.
While there are other issues with this setting, the first chunk of work
is represented in this patch. It checks to see whether the system is
likely configured in the way systemd recommends, with `/etc/resolv.conf`
set to be a symlink pointing at `/run/systemd/resolve/stub-resolv.conf`.
If so, SET uses its root privileges to automatically reconfigure the
system in such a way as to disable `systemd-resolved`'s stub DNS
resolver, clearing port 53, and then setting the system to use Quad Nine
(`9.9.9.9`) as its DNS resolver. Quad Nine is a free service run by the
security community that provides a fast, anycast public recursive DNS
resolver. This was chosen over Cloudflare (at `1.1.1.1`) or Google
(at `8.8.8.8`) because Quad Nine also provides some malware filtering.
See https://quad9.net for details about this service.
In any event, once the reconfiguration is complete, we register a
cleanup function that will undo our configuration, ceding our own setup
back to the original system configuration just before the DNS server
thread exits, ensuring no permanent changes are made to the system.
This only works on GNU/Linux systems, of course, since Apple macOS do
not run systemd, even though they are POSIX. Of course, Apple also
doesn't run its own local stub resolver, so this is not a problem in
need of a solution of macOS. For this reason, `check_os()` is not used.
The actual `socketserver` variable was sometimes called `server` and
sometimes called `httpd`. This meant that invisible `NameError`
exceptions were being raised in several places when the server's socket
was supposed to be closed (with calls to `socket.close()`.
This commit converges on the variable name `server` for the socket
server, and continues to move code responsible solely for generating
reports into the `generate_reports()` helper function. By doing so, the
functions where the servers are being handled are shorter, making it
easier to spot name errors such as the ones fixed here.
Turns out, the `shutdown_request()` method was not ever catching
`KeyboardInterrupt` exceptions because they were never raised there.
This means we can at least catch the exception in that block, and move
the exception handling closer to where it should be.
That turns out to be in `ssl_server()` itself, which AFAICT was never
going to generate reports the same way that the non-SSL server was,
visible in the `run()` method.
This commit revives the report generation capabilities in `ssl_server()`
by moving the same logic used by `run()` into a new function,
`generate_reports()` and calling it in both code paths.
This is probably a kludge because I am not super familiar with TLS
socket programming or SET generally, but it achieves the result, which
is not to shutdown the (HTTPS) socket server when a TLS client responds
to a TLS Server Hello message with a fatal Alert message.
One example of a client that does this is recent Firefox. What this
means is that if you run SET with `WEBATTACK_SSL=ON` and
`SELF_SIGNED_CERT=ON`, your victim can turn off your HTTPS server simply
by navigating to your attack page.
This is caused by the underlying OpenSSL library raising an error that
`pyopenssl`, in turn, `raise`s through the socket server libraries.
Ultimately, it bubbles up to the `harvester` module through its
`shutdown_request` method, called by the underlying socket server's
`_handle_request_noblock` method. See the backtrace printed in the
comments of Pull Request #696 for a complete example.
The bottom line is that this unhandled exception ultimately causes the
HTTPS server to die before it gets a chance to be useful. Since I assume
that SET doesn't particularly care what certificate validation alerts the
client is sending us, this patch addresses the issue by ignoring every
raised exception from the underlying libraries except for a
`KeyboardInterrupt` so that the SET user can cause a server shutdown
themselves, with the expected `C-c` interrupt signal.
There is probably a more graceful way to handle this, though? Also, note
that this only fixes the HTTPS issues for Python 3. Python 2 exhibits a
different error altogether.
Per Debian policy, Python versions 3 and greater must be called from the
`python3` executable, not the `python` executable. This means
Debian-based systems with Python 3 installed but not Python 2 will fail
to launch SET through `seautomate` because the `python` executable is
hard-coded into the call to `pexpect.spawn()`.
This commit uses the standard library's
`distutils.spawn.find_executable()` method to locate the correct path to
`python3` or, failing that, the correct path to the `python` executable
and uses the result of that call as the first command line word with
which to invoke `setoolkit`.
The `distutils.spawn.find_executable()` method is available at that
exact name in both Python 2 and 3, so this should be portable across all
Python versions.
Note that the shebang line references `python`, and so Python 3-only
systems such as newer Debian-based builds may need to invoke
`seautomate` using an explicit interpreter for it to work:
```sh
sudo python3 ./seautomate /path/to/script/file.txt
```
This resolves the first issue in a line of several preventing the
SSL-capable server from spinning up correctly as described in the
referenced issue number. The cause is simply a missing import when
`setoolkit` is run under Python 3.
See also pymssql/pymssql#668 which describes the discontinuance of the
pymssql project.
This commit addresses the issue by simplistically pinning the dependency
on pymssql to an existing version.
Mass Mailer Attack, SET throws an error:
Something went wrong, printing the error: No module named 'cStringIO'
This fixes it, the change to socket.py fixes the module name,
and the change to smtp_web fixes the exception name.
