Прозрачный HTTP-прокси с Squid 3.5

В течение нескольких дней я пытаюсь понять, как заставить прозрачный HTTP-прокси работать с Squid. Я пытаюсь создать прокси, который принимает интернет-трафик с портов 80 и 443, направляет их через Squid в Privoxy и, наконец, через Tor и возвращает данные. По сути, я хочу "автоматически" возвращать некоторый трафик через Tor без необходимости добавления прокси-сервера к их соединению.

Я знаю, как настроить Privoxy и Tor, но у меня проблемы с таблицами Squid и IP конфигурация.

Вот моя настройка

Загрузить последнюю версию

curl -O http://www.squid-cache.org/Versions/v3/3.5/squid-3.5.22.tar.gz && tar zxvf squid-3.5.22.tar.gz && cd squid-3.5.22

Установить все необходимые пакеты

apt install devscripts build-essential openssl libssl-dev fakeroot libcppunit-dev libsasl2-dev cdbs ccze libfile-readbackwards-perl libcap2 libcap-dev libcap2-dev libnetfilter-conntrack-dev htop ccze sysv-rc-conf -y

Настроить сборку, сделать и установить

./configure \
CHOST="x86_64-pc-linux-gnu" \
CFLAGS="-march=core2 -O2 -pipe" \
CXXFLAGS="${CFLAGS}" \
--build=x86_64-linux-gnu \
--prefix=/usr \
--exec-prefix=/usr \
--bindir=/usr/bin \
--sbindir=/usr/sbin \
--libdir=/usr/lib \
--sharedstatedir=/usr/com \
--includedir=/usr/include \
--localstatedir=/var \
--libexecdir=/usr/lib/squid \
--srcdir=. \
--datadir=/usr/share/squid \
--sysconfdir=/etc/squid \
--infodir=/usr/share/info \
--mandir=/usr/share/man \
--x-includes=/usr/include \
--x-libraries=/usr/lib \
--with-default-user=proxy \
--with-logdir=/var/log/squid \
--with-pidfile=/var/run/squid.pid \
--enable-err-languages=English \
--enable-default-err-language=English \
--enable-storeio=ufs,aufs,diskd \
--enable-linux-netfilter \
--enable-removal-policies=lru,heap \
--enable-gnuregex \
--enable-follow-x-forwarded-for \
--enable-x-accelerator-vary \
--enable-zph-qos \
--enable-delay-pools \
--enable-snmp \
--enable-underscores \
--with-openssl \
--enable-ssl-crtd \
--enable-http-violations \
--enable-async-io=24 \
--enable-storeid-rewrite-helpers \
--with-large-files \
--with-libcap \
--with-netfilter-conntrack \
--with-included-ltdl \
--with-maxfd=65536 \
--with-filedescriptors=65536 \
--with-pthreads \
--without-gnutls \
--without-mit-krb5 \
--without-heimdal-krb5 \
--without-gnugss \
--disable-icap-client \
--disable-wccp \
--disable-wccpv2 \
--disable-dependency-tracking \
--disable-auth --disable-epoll \
--disable-ident-lookups \
--disable-icmp

Разрешить пересылку ip4

echo -e "net.ipv4.ip_forward = 1\nnet.ipv4.conf.default.rp_filter = 0\nnet.ipv4.conf.all.rp_filter = 0\nnet.ipv4.conf.eth0.rp_filter = 0\n" >> /etc/sysctl.conf

Создать сертификаты

mkdir /etc/squid/ssl_certs && cd /etc/squid/ssl_certs
openssl genrsa -out squid.key 2048
openssl req -new -key squid.key -out squid.csr -nodes
openssl x509 -req -days 3652 -in squid.csr -signkey squid.key -out squid.crt
cat squid.crt squid.key > squid.pem

Создать кеш сертификатов

mkdir /var/lib/squid && chown -R proxy:proxy /var/lib/squid/
/usr/lib/squid/ssl_crtd -c -s /var/lib/squid/ssl_db

Измените владельца и права на папки

mkdir -p /var/spool/squid

chown -R proxy:proxy /etc/squid/squid.conf | chown -R proxy:proxy /usr/lib/squid | chown -R proxy:proxy /var/lib/squid/ssl_db/ | chown -R proxy:proxy /var/spool/squid | chown -R proxy:proxy /var/log/squid  | chmod 777 /var/spool/squid | chmod 777 /var/log/squid  | chmod 755 /var/lib/squid/ssl_db/certs | chown proxy:proxy /var/log/squid/

Измените конфигурацию (ниже) и инициализируйте кеш

squid -f /etc/squid/squid.conf -z

Перенаправьте порты 80 и 443

iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-ports 3128
iptables -t nat -A PREROUTING -p tcp --dport 443 -j REDIRECT --to-ports 3129

Моя фактическая конфигурация squid

acl localnet src all

acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl CONNECT method CONNECT

never_direct allow all
always_direct allow all

# Only allow cachemgr access from localhost
http_access allow localhost manager
http_access deny manager

http_access allow localnet
http_access allow localhost

debug_options ALL,2

visible_hostname squid

# stop squid taking forever to restart.
shutdown_lifetime 3
# for clients with a configured proxy.
http_port 3127
# for clients who are sent here via iptables ... REDIRECT.
http_port 3128 tproxy
# for https clients who are sent here via iptables ... REDIRECT
https_port 3129 tproxy ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/etc/squid/ssl_certs/squid.pem

sslcrtd_program /usr/lib/squid/ssl_crtd -s /var/lib/squid/ssl_db -M 4MB sslcrtd_children 8 startup=1 idle=1

# acl step1 at_step SslBump1
# ssl_bump peek step1
# ssl_bump bump all

ssl_bump server-first all
sslproxy_cert_error allow all
sslproxy_flags DONT_VERIFY_PEER

via off
forwarded_for off

request_header_access From deny all
request_header_access Server deny all
request_header_access WWW-Authenticate deny all
request_header_access Link deny all
request_header_access Cache-Control deny all
request_header_access Proxy-Connection deny all
request_header_access X-Cache deny all
request_header_access X-Cache-Lookup deny all
request_header_access Via deny all
request_header_access X-Forwarded-For deny all
request_header_access Pragma deny all
request_header_access Keep-Alive deny all

cache_dir ufs /var/spool/squid 1024 16 256
coredump_dir /var/cache/squid

refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
refresh_pattern .               0       20%     4320

Вы можете заметить, насколько я доброжелателен к настройкам Squid. Это только для тестирования.

Итак, я понял, что ни перехват, ни tproxy не работают. Если я использую ускорение для трафика, отличного от HTTPS, он работает, но ничего больше. Если я использую его как есть, в результате он зависнет на время ожидания клиента, а затем на время ожидания.

Вот пример. Я изменил в / etc / hosts IP для httpbin. org и перенаправил его через ящик squid.

❯ curl -vk https://httpbin.org/ip
*   Trying *******...
* Connected to httpbin.org (*******) port 443 (#0)
* TLS 1.2 connection using TLS_RSA_WITH_AES_256_GCM_SHA384
* Server certificate: ******
* Server certificate: Universe
> GET /ip HTTP/1.1
> Host: httpbin.org
> User-Agent: curl/7.49.1
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< Server: squid/3.5.22
< Mime-Version: 1.0
< Date: Mon, 05 Dec 2016 05:43:50 GMT
< Content-Type: text/html;charset=utf-8
< Content-Length: 3498
< X-Squid-Error: ERR_CONNECT_FAIL 110
< Vary: Accept-Language
< Content-Language: en
< X-Cache: MISS from pipik
< Connection: close

На стороне squid

2016/12/05 05:42:50.362 kid1| 5,2| TcpAcceptor.cc(220) doAccept: New connection on FD 28
2016/12/05 05:42:50.362 kid1| 5,2| TcpAcceptor.cc(295) acceptNext: connection on local=[::]:3129 remote=[::] FD 28 flags=25
2016/12/05 05:42:50.363 kid1| 33,2| client_side.cc(3911) httpsSslBumpAccessCheckDone: sslBump needed for local=*******:3129 remote=############# FD 11 flags=17 method 3
2016/12/05 05:42:50.363 kid1| 11,2| client_side.cc(2347) parseHttpRequest: HTTP Client local=*******:3129 remote=############# FD 11 flags=17
2016/12/05 05:42:50.363 kid1| 11,2| client_side.cc(2348) parseHttpRequest: HTTP Client REQUEST:
---------
CONNECT *******:3129 HTTP/1.1
Host: *******:3129


----------
2016/12/05 05:42:50.363 kid1| 85,2| client_side_request.cc(744) clientAccessCheckDone: The request CONNECT *******:3129 is ALLOWED; last ACL checked: localnet
2016/12/05 05:42:50.363 kid1| 85,2| client_side_request.cc(720) clientAccessCheck2: No adapted_http_access configuration. default: ALLOW
2016/12/05 05:42:50.363 kid1| 85,2| client_side_request.cc(744) clientAccessCheckDone: The request CONNECT *******:3129 is ALLOWED; last ACL checked: localnet
2016/12/05 05:42:50.378 kid1| 83,2| client_side.cc(4284) clientPeekAndSpliceSSL: SSL_accept failed.
2016/12/05 05:42:50.378 kid1| 83,2| client_side.cc(4284) clientPeekAndSpliceSSL: SSL_accept failed.
2016/12/05 05:42:50.378 kid1| 83,2| client_side.cc(4284) clientPeekAndSpliceSSL: SSL_accept failed.
2016/12/05 05:42:50.378 kid1| 83,2| client_side.cc(4284) clientPeekAndSpliceSSL: SSL_accept failed.
2016/12/05 05:42:50.378 kid1| 83,2| client_side.cc(4284) clientPeekAndSpliceSSL: SSL_accept failed.
2016/12/05 05:42:50.378 kid1| 83,2| client_side.cc(4284) clientPeekAndSpliceSSL: SSL_accept failed.
2016/12/05 05:42:50.378 kid1| 83,2| client_side.cc(4284) clientPeekAndSpliceSSL: SSL_accept failed.
2016/12/05 05:42:50.378 kid1| 83,2| client_side.cc(4284) clientPeekAndSpliceSSL: SSL_accept failed.
2016/12/05 05:42:50.378 kid1| 83,2| client_side.cc(4284) clientPeekAndSpliceSSL: SSL_accept failed.
2016/12/05 05:42:50.378 kid1| 83,2| client_side.cc(4284) clientPeekAndSpliceSSL: SSL_accept failed.
2016/12/05 05:42:50.378 kid1| 83,2| client_side.cc(4284) clientPeekAndSpliceSSL: SSL_accept failed.
2016/12/05 05:42:50.378 kid1| 83,2| client_side.cc(4284) clientPeekAndSpliceSSL: SSL_accept failed.
2016/12/05 05:42:50.378 kid1| 83,2| client_side.cc(4284) clientPeekAndSpliceSSL: SSL_accept failed.
2016/12/05 05:42:50.379 kid1| 83,2| client_side.cc(4284) clientPeekAndSpliceSSL: SSL_accept failed.
2016/12/05 05:42:50.379 kid1| 83,2| client_side.cc(4284) clientPeekAndSpliceSSL: SSL_accept failed.
2016/12/05 05:42:50.379 kid1| 83,2| client_side.cc(4284) clientPeekAndSpliceSSL: SSL_accept failed.
2016/12/05 05:42:50.379 kid1| 83,2| client_side.cc(4284) clientPeekAndSpliceSSL: SSL_accept failed.
2016/12/05 05:42:50.379 kid1| 17,2| FwdState.cc(133) FwdState: Forwarding client request local=*******:3129 remote=############# FD 11 flags=17, url=*******:3129
2016/12/05 05:42:50.379 kid1| 44,2| peer_select.cc(280) peerSelectDnsPaths: Found sources for '*******:3129'
2016/12/05 05:42:50.379 kid1| 44,2| peer_select.cc(281) peerSelectDnsPaths:   always_direct = ALLOWED
2016/12/05 05:42:50.379 kid1| 44,2| peer_select.cc(282) peerSelectDnsPaths:    never_direct = DUNNO
2016/12/05 05:42:50.379 kid1| 44,2| peer_select.cc(288) peerSelectDnsPaths:    ORIGINAL_DST = local=############# remote=*******:3129 flags=25
2016/12/05 05:42:50.379 kid1| 44,2| peer_select.cc(295) peerSelectDnsPaths:        timedout = 0
2016/12/05 05:43:50.645 kid1| 4,2| errorpage.cc(1261) BuildContent: No existing error page language negotiated for ERR_CONNECT_FAIL. Using default error file.
2016/12/05 05:43:50.645 kid1| 20,2| store.cc(980) checkCachable: StoreEntry::checkCachable: NO: not cachable
2016/12/05 05:43:50.645 kid1| 20,2| store.cc(980) checkCachable: StoreEntry::checkCachable: NO: not cachable
2016/12/05 05:43:50.845 kid1| 83,2| client_side.cc(3811) clientNegotiateSSL: clientNegotiateSSL: New session 0x29dda60 on FD 11 (#############:59117)
2016/12/05 05:43:50.943 kid1| 11,2| client_side.cc(2347) parseHttpRequest: HTTP Client local=*******:3129 remote=############# FD 11 flags=17
2016/12/05 05:43:50.944 kid1| 11,2| client_side.cc(2348) parseHttpRequest: HTTP Client REQUEST:
---------
GET /ip HTTP/1.1
Host: httpbin.org
User-Agent: curl/7.49.1
Accept: */*


----------
2016/12/05 05:43:50.944 kid1| 33,2| QosConfig.cc(145) doTosLocalMiss: QOS: Preserving TOS on miss, TOS=0
2016/12/05 05:43:50.944 kid1| 33,2| client_side_reply.cc(1534) buildReplyHeader: clientBuildReplyHeader: Connection Keep-Alive not requested by admin or client
2016/12/05 05:43:50.944 kid1| 88,2| client_side_reply.cc(2051) processReplyAccessResult: The reply for GET https://httpbin.org/ip is ALLOWED, because it matched (access_log daemon:/var/log/squid/access.log line)
2016/12/05 05:43:50.944 kid1| 11,2| client_side.cc(1393) sendStartOfMessage: HTTP Client local=*******:3129 remote=############# FD 11 flags=17
2016/12/05 05:43:50.944 kid1| 11,2| client_side.cc(1394) sendStartOfMessage: HTTP Client REPLY:
---------
HTTP/1.1 503 Service Unavailable
Server: squid/3.5.22
Mime-Version: 1.0
Date: Mon, 05 Dec 2016 05:43:50 GMT
Content-Type: text/html;charset=utf-8
Content-Length: 3498
X-Squid-Error: ERR_CONNECT_FAIL 110
Vary: Accept-Language
Content-Language: en
X-Cache: MISS from squid
Connection: close


----------
2016/12/05 05:43:50.944 kid1| 33,2| client_side.cc(817) swanSong: local=*******:3129 remote=############# flags=17
2016/12/05 05:43:50.944 kid1| 20,2| store.cc(980) checkCachable: StoreEntry::checkCachable: NO: not cachable
2016/12/05 05:43:50.944 kid1| 20,2| store.cc(980) checkCachable: StoreEntry::checkCachable: NO: not cachable

я перепробовал так много разных конфигураций, что уже потерялся в том, что работает, а что нет. Я, вероятно, не совсем понимаю связь между iptables и squid, но что бы я ни читал, я всегда оказываюсь здесь.

Я ценю любые предложения.