DoS/DDoS атака, нужна помощь в настройке системы.

24.12.2010 14:53 · отредактировано #110137

Доброго времени суток.

Есть сервер, на котором запущено приложении слушающее 6666 порт. Столкнулись с тем, что через пару часов работы приложения, оно перестаёт пускать к себе юзверей с ошибкой "Too many opened files". Валидных юзверей к программе подключено около 2000 при этом.

Подозреваем что некто нехороший зафлуживает наш сервер, из-за чего выжираются все сокеты, НО в iptables стоит ограничение на кол-во подключений с 1го айпи адреса, поэтому не понимаем, как такое возможно?

Также одна из причин по которой думаем что мы находимся под какой то атакой - расход трафика. В первый день когда столкнулись с "выжранными" сокетами обратили внимание что расход трафика возрос более чем в четыре раза по сравнению с обычным.

При просмотре листинга обратите пожалуйста внимание на кол-во icmp запросов, и на кол-во открытых пассивных tcp соединений(откуда столько? icmp эхо вообще отключено).

Не мог бы кто нибудь помочь идентифицировать вид атаки, подсказать средства защиты, возможно, указать на ошибки сделанные в настройке сервера?
Выкладываю информацию по настройкам iptables+sysctl, +вывод netstat'a.
Правила iptables + sysctl:

text

#!/bin/sh
#######################
##### PREFERENCES #####
#######################
CONN_LIMIT=30
OPEN_MYSQL=false
#external interface ip address
MYSQL_SERVER_IP=""
#ip address of client which will be allowed to connect to this mysql server
MYSQL_CLIENT_IP=""
#######################
###### IpTables #######
#######################
#scan protect
/sbin/iptables -A FORWARD -p tcp --tcp-flags SYN,ACK,FIN,RST RST -m limit --limit 15/minute -d any/0 -j QUEUE
#icmp flood
/sbin/iptables -A FORWARD -p icmp -m limit --limit 10/sec --limit-burst 500 -j DROP
/sbin/iptables -A FORWARD -p icmp -j DROP
#conections per ip
/sbin/iptables -A INPUT -p tcp --dport 22 -m connlimit --connlimit-above $CONN_LIMIT -j DROP
/sbin/iptables -A INPUT -p tcp --dport 80 -m connlimit --connlimit-above $CONN_LIMIT -j DROP
/sbin/iptables -A INPUT -p tcp --dport 6666 -m connlimit --connlimit-above $CONN_LIMIT -j DROP
#syn flood
/sbin/iptables -A INPUT -p tcp --syn --dport 22 -m connlimit --connlimit-above $CONN_LIMIT -j DROP
/sbin/iptables -A INPUT -p tcp --syn --dport 80 -m connlimit --connlimit-above $CONN_LIMIT -j DROP
/sbin/iptables -A INPUT -p tcp --syn --dport 6666 -m connlimit --connlimit-above $CONN_LIMIT -j DROP
#######################
####### Kernel ########
#######################
# syn flood и всё с ним связанное
sysctl -w net.ipv4.tcp_syncookies=1
sysctl -w net.ipv4.tcp_syn_retries=5
sysctl -w net.ipv4.tcp_synack_retries=1
sysctl -w net.ipv4.tcp_max_syn_backlog="4096"
# через 60 секунд с момента наступления тишины в канале сервером выполняется проба
# Если клиент жив, то серверу высылается ответный пакет
sysctl -w net.ipv4.tcp_keepalive_time=60
# если ответа на проверку нет, то с интервалом в 10 секунд повторить
sysctl -w net.ipv4.tcp_keepalive_intvl=10
# сколько раз повторить проверку, после чего закрыть соединение
sysctl -w net.ipv4.tcp_keepalive_probes=3
sysctl -w net.ipv4.netfilter.ip_conntrack_max=65536
sysctl -w net.ipv4.netfilter.ip_conntrack_tcp_timeout_close=10
sysctl -w net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait=30
sysctl -w net.ipv4.netfilter.ip_conntrack_tcp_timeout_last_ack=30
sysctl -w net.ipv4.netfilter.ip_conntrack_tcp_timeout_close_wait=30
sysctl -w net.ipv4.netfilter.ip_conntrack_tcp_timeout_fin_wait=60
sysctl -w net.ipv4.netfilter.ip_conntrack_tcp_timeout_established=30
sysctl -w net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_recv=30
sysctl -w net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_sent=30
#отключаем ответ на icmp запросы для защиты от Ping of Death
sysctl -w net.ipv4.icmp_echo_ignore_all=1
sysctl -w net.ipv4.icmp_echo_ignore_broadcasts=1
#включам проверку подлинности IP адресов для входящих соединений(IP Spoofing Protection)
sysctl -w net.ipv4.conf.all.rp_filter=1
#отключаем IP Source Routing
sysctl -w net.ipv4.conf.all.accept_source_route=0
#считываем новые настройки ядра
sysctl -p
#######################
####### MySQL #########
#######################
if [ "$OPEN_MYSQL" = "true" ]
then
    /sbin/iptables -A INPUT -s $MYSQL_SERVER_IP -i eth0 -p tcp -m tcp --dport 3306 -j ACCEPT
    /sbin/iptables -A INPUT -s $MYSQL_CLIENT_IP -i eth0 -p tcp -m tcp --dport 3306 -j ACCEPT
    /sbin/iptables -A INPUT -i eth0 -p tcp -m tcp --dport 3306 -j DROP
fi
#exit script
exit 0

Вывод netstat -s:

text


Ip:
    141286877 total packets received
    0 forwarded
    0 incoming packets discarded
    141286873 incoming packets delivered
    146374005 requests sent out
Icmp:
  [b]  23534 ICMP messages received[/b]
    7 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 2888
        timeout in transit: 442
        redirects: 243
        echo requests: 19959
  [b]  10336 ICMP messages sent[/b]
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 47
        echo replies: 10289
IcmpMsg:
        InType3: 2888
        InType5: 243
        InType8: 19959
        InType11: 442
        OutType0: 10289
        OutType3: 47
Tcp:
    1084 active connections openings
    [b]38969 passive connection openings[/b]
    482 failed connection attempts
    3427 connection resets received
    949 connections established
    141258893 segments received
    145723683 segments send out
    638667 segments retransmited
    1682 bad segments received.
    3254 resets sent
Udp:
    4399 packets received
    47 packets to unknown port received.
    0 packet receive errors
    4420 packets sent
UdpLite:
TcpExt:
    711 invalid SYN cookies received
    480 resets received for embryonic SYN_RECV sockets
    2 packets pruned from receive queue because of socket buffer overrun
    2 ICMP packets dropped because they were out-of-window
    11054 TCP sockets finished time wait in fast timer
    4 packets rejects in established connections because of timestamp
    8277375 delayed acks sent
    280 delayed acks further delayed because of locked socket
    Quick ack mode was activated 79115 times
    281 times the listen queue of a socket overflowed
    281 SYNs to LISTEN sockets dropped
    7681239 packets directly queued to recvmsg prequeue.
    227616 bytes directly in process context from backlog
    4468882 bytes directly received in process context from prequeue
    19840158 packet headers predicted
    255 packets header predicted and directly queued to user
    93231619 acknowledgments not containing data payload received
    10594564 predicted acknowledgments
    52 times recovered from packet loss due to fast retransmit
    15017 times recovered from packet loss by selective acknowledgements
    4 bad SACK blocks received
    Detected reordering 28 times using FACK
    Detected reordering 45 times using SACK
    Detected reordering 1 times using reno fast retransmit
    Detected reordering 6 times using time stamp
    53 congestion windows fully recovered without slow start
    9 congestion windows partially recovered using Hoe heuristic
    62028 congestion windows recovered without slow start by DSACK
    4636 congestion windows recovered without slow start after partial ack
    10601 TCP data loss events
    TCPLostRetransmit: 384
    165 timeouts after reno fast retransmit
    37785 timeouts after SACK recovery
    1004 timeouts in loss state
    22041 fast retransmits
    1021 forward retransmits
    15539 retransmits in slow start
    515895 other TCP timeouts
    13 classic Reno fast retransmits failed
    1128 SACK retransmits failed
    420 packets collapsed in receive queue due to low socket buffer
    127708 DSACKs sent for old packets
    117 DSACKs sent for out of order packets
    226483 DSACKs received
    78 DSACKs for out of order packets received
    182 connections reset due to unexpected data
    543 connections reset due to early user close
    772 connections aborted due to timeout
    TCPSACKDiscard: 13
    TCPDSACKIgnoredOld: 127066
    TCPDSACKIgnoredNoUndo: 75896
    TCPSpuriousRTOs: 994
    TCPSackShiftFallback: 113041
IpExt:
    InOctets: 1468656105
    OutOctets: 1906573448

netstat -ntu:

text


Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0    235 88.x.x.x:6666      178.206.173.42:3877     ESTABLISHED
tcp        0    465 88.x.x.x:6666      91.212.239.7:17846      ESTABLISHED
tcp        0   1084 88.x.x.x:6666      85.172.124.52:49574     ESTABLISHED
tcp        0     70 88.x.x.x:6666      109.105.166.89:50772    ESTABLISHED
tcp        0     58 88.x.x.x:6666      85.237.37.183:1040      ESTABLISHED
tcp        0    200 88.x.x.x:6666      80.64.175.13:1046       ESTABLISHED
tcp        0    608 88.x.x.x:6666      92.100.126.107:18080    ESTABLISHED
tcp        0      0 88.x.x.x:6666      83.149.3.138:25924      ESTABLISHED
tcp        0      0 88.x.x.x:6666      95.153.182.35:43322     ESTABLISHED
tcp        0    105 88.x.x.x:6666      178.72.99.129:1189      ESTABLISHED
tcp        0    610 88.x.x.x:6666      94.233.233.208:3212     ESTABLISHED
tcp        0    367 88.x.x.x:6666      78.106.13.187:53684     ESTABLISHED
tcp        0    540 88.x.x.x:6666      95.24.77.141:2050       ESTABLISHED
tcp        0      0 88.x.x.x:6666      78.144.250.88:49305     ESTABLISHED
tcp        0    252 88.x.x.x:6666      85.173.84.34:62630      ESTABLISHED
tcp        0    140 88.x.x.x:6666      93.185.192.84:1910      ESTABLISHED
tcp        0    308 88.x.x.x:6666      85.29.185.114:1758      ESTABLISHED
tcp        0   1255 88.x.x.x:6666      188.17.103.121:51193    ESTABLISHED
tcp        0      0 88.x.x.x:6666      178.44.153.227:51272    ESTABLISHED
tcp        0    186 88.x.x.x:6666      188.16.136.31:1586      ESTABLISHED
tcp        0      0 88.x.x.x:6666      92.113.129.202:2864     ESTABLISHED
tcp        0    129 88.x.x.x:6666      109.69.28.14:1803       ESTABLISHED
tcp        0    235 88.x.x.x:6666      95.167.107.2:3621       ESTABLISHED
tcp        0      0 88.x.x.x:6666      188.187.72.232:4838     ESTABLISHED
tcp        0     35 88.x.x.x:6666      95.81.198.141:2556      ESTABLISHED
tcp        0      0 88.x.x.x:6666      95.29.185.43:3015       ESTABLISHED
tcp        0     75 88.x.x.x:6666      95.42.30.210:3703       ESTABLISHED
tcp        0      0 88.x.x.x:6666      89.178.102.51:53296     ESTABLISHED
tcp        0    160 88.x.x.x:6666      109.184.227.184:2365    ESTABLISHED
tcp        0      0 88.x.x.x:6666      178.187.180.27:3358     TIME_WAIT  
tcp        0    273 88.x.x.x:6666      77.35.11.141:50655      ESTABLISHED
tcp        0     36 88.x.x.x:6666      80.64.175.24:59137      ESTABLISHED
tcp        0    154 88.x.x.x:6666      109.171.97.74:50154     ESTABLISHED
tcp        0   1243 88.x.x.x:6666      94.137.196.71:3588      ESTABLISHED
tcp        0      0 88.x.x.x:6666      109.200.108.150:49487   ESTABLISHED
tcp        0    489 88.x.x.x:6666      94.137.7.9:51164        ESTABLISHED
tcp        0    133 88.x.x.x:6666      93.213.28.191:49329     ESTABLISHED
tcp        0      0 88.x.x.x:6666      89.252.17.136:3834      ESTABLISHED
tcp        0    701 88.x.x.x:6666      178.67.67.200:1580      ESTABLISHED
tcp        0      0 88.x.x.x:6666      93.174.224.115:2514     ESTABLISHED
tcp        0    884 88.x.x.x:6666      87.117.185.178:53735    ESTABLISHED
tcp        0    105 88.x.x.x:6666      217.8.238.152:24616     ESTABLISHED
tcp        0    109 88.x.x.x:6666      109.104.175.185:1499    ESTABLISHED
tcp        0      0 88.x.x.x:6666      213.174.2.3:1051        ESTABLISHED
tcp        0    304 88.x.x.x:6666      77.232.142.227:59227    ESTABLISHED
tcp        0    239 88.x.x.x:6666      94.253.77.13:49788      ESTABLISHED
tcp        0    240 88.x.x.x:6666      178.206.107.205:49200   ESTABLISHED
tcp        0    225 88.x.x.x:6666      87.186.52.238:1075      ESTABLISHED
tcp        0      0 88.x.x.x:6666      151.76.170.96:10540     ESTABLISHED
tcp        0      0 88.x.x.x:6666      83.149.3.13:26150       ESTABLISHED
tcp        0   1170 88.x.x.x:6666      92.113.123.175:53649    ESTABLISHED
tcp        0      0 88.x.x.x:6666      77.88.216.196:1903      ESTABLISHED
tcp        0    143 88.x.x.x:6666      188.187.62.136:27628    ESTABLISHED
tcp        0      0 88.x.x.x:6666      93.81.209.152:49435     ESTABLISHED
tcp        0    366 88.x.x.x:6666      178.17.188.69:1340      ESTABLISHED
tcp        0    331 88.x.x.x:6666      91.90.8.84:38368        ESTABLISHED
tcp        0    750 88.x.x.x:6666      178.179.7.223:4384      ESTABLISHED
tcp        0     64 88.x.x.x:6666      78.106.145.240:2198     ESTABLISHED
tcp        0    638 88.x.x.x:6666      79.142.106.54:49389     ESTABLISHED
tcp        0    489 88.x.x.x:6666      95.69.197.31:50018      ESTABLISHED
tcp        0    897 88.x.x.x:6666      109.111.130.141:1065    ESTABLISHED
tcp        0    951 88.x.x.x:6666      46.118.226.31:2688      ESTABLISHED
tcp        0    159 88.x.x.x:6666      95.58.8.64:10197        ESTABLISHED
tcp        0     68 88.x.x.x:6666      213.138.81.103:64313    ESTABLISHED
tcp        0    301 88.x.x.x:6666      93.81.166.209:2447      ESTABLISHED
tcp        0      0 88.x.x.x:6666      213.206.63.13:49205     ESTABLISHED
tcp        0    187 88.x.x.x:6666      77.222.113.22:62841     ESTABLISHED
tcp        0      0 88.x.x.x:6666      92.115.96.40:5641       ESTABLISHED
tcp        0     49 88.x.x.x:6666      178.219.211.129:1159    ESTABLISHED
tcp        0     19 88.x.x.x:6666      91.188.36.91:22936      ESTABLISHED
tcp        0    246 88.x.x.x:6666      95.131.8.202:4648       ESTABLISHED
tcp        0      0 88.x.x.x:6666      95.71.76.124:1983       ESTABLISHED
tcp        0    247 88.x.x.x:6666      109.171.124.79:49309    ESTABLISHED
tcp        0    180 88.x.x.x:6666      95.54.58.228:49206      ESTABLISHED
tcp        0      0 88.x.x.x:6666      109.195.52.70:49526     ESTABLISHED
tcp        0     35 88.x.x.x:6666      109.184.12.21:3993      ESTABLISHED
tcp        0     58 88.x.x.x:6666      80.64.175.107:1089      ESTABLISHED
tcp        0     84 88.x.x.x:6666      79.120.112.27:1150      ESTABLISHED
tcp        0    118 88.x.x.x:6666      188.123.242.15:25353    ESTABLISHED
tcp        0    207 88.x.x.x:6666      85.26.235.183:16078     ESTABLISHED
tcp        0    324 88.x.x.x:6666      83.149.41.252:50798     ESTABLISHED
tcp        0     13 88.x.x.x:6666      85.26.233.245:19829     ESTABLISHED
tcp        0    119 88.x.x.x:6666      91.214.128.21:53416     ESTABLISHED
tcp        0      0 88.x.x.x:6666      46.146.160.126:1150     ESTABLISHED
tcp        0    191 88.x.x.x:6666      80.83.193.41:1135       ESTABLISHED
tcp        0    268 88.x.x.x:6666      80.240.220.65:51993     ESTABLISHED
tcp        0    105 88.x.x.x:6666      95.29.128.198:49514     ESTABLISHED
tcp        0    332 88.x.x.x:6666      178.34.171.183:1818     ESTABLISHED
tcp        0    112 88.x.x.x:6666      213.231.32.94:1841      ESTABLISHED
tcp        0      0 88.x.x.x:6666      213.79.8.40:52163       ESTABLISHED
tcp        0     35 88.x.x.x:6666      93.116.207.93:2212      ESTABLISHED
tcp        0   1185 88.x.x.x:6666      188.16.100.25:1462      ESTABLISHED
tcp        0     23 88.x.x.x:6666      95.70.89.246:2264       ESTABLISHED
tcp        0    417 88.x.x.x:6666      213.87.80.98:62326      ESTABLISHED
tcp        0    434 88.x.x.x:6666      94.180.19.105:3729      ESTABLISHED
tcp        0    194 88.x.x.x:6666      80.64.175.107:3059      ESTABLISHED
tcp        0      0 88.x.x.x:6666      95.78.77.216:52221      ESTABLISHED
tcp        0     35 88.x.x.x:6666      78.106.145.240:53717    ESTABLISHED
tcp        0      0 88.x.x.x:6666      94.50.5.32:63856        ESTABLISHED
tcp        0     83 88.x.x.x:6666      85.140.73.122:1187      ESTABLISHED
tcp        0     35 88.x.x.x:6666      92.248.211.12:2791      ESTABLISHED
tcp        0    878 88.x.x.x:6666      85.175.200.136:1811     ESTABLISHED
tcp        0    122 88.x.x.x:6666      77.122.161.242:4561     ESTABLISHED
tcp        0     38 88.x.x.x:6666      93.72.58.182:49166      ESTABLISHED
tcp        0    630 88.x.x.x:6666      85.26.241.108:61363     ESTABLISHED
tcp        0    151 88.x.x.x:6666      87.228.116.148:57897    ESTABLISHED
tcp        0      0 88.x.x.x:6666      93.120.226.239:50086    ESTABLISHED
tcp        0     35 88.x.x.x:6666      83.174.246.209:1496     ESTABLISHED
tcp        0   1333 88.x.x.x:6666      77.93.41.58:64533       ESTABLISHED
tcp        0      0 88.x.x.x:6666      46.164.193.236:49906    ESTABLISHED
tcp        0      0 88.x.x.x:6666      94.179.83.201:1072      ESTABLISHED
tcp        0    323 88.x.x.x:6666      91.202.109.93:64703     ESTABLISHED
tcp        0     82 88.x.x.x:6666      178.206.61.118:51215    ESTABLISHED
tcp        0      0 88.x.x.x:6666      92.61.64.213:1318       ESTABLISHED
tcp        0     38 88.x.x.x:6666      212.76.188.1:3294       ESTABLISHED
tcp        0     31 88.x.x.x:6666      178.49.15.16:51829      ESTABLISHED
tcp        0      0 88.x.x.x:6666      178.93.157.45:56739     TIME_WAIT  
tcp        0      0 88.x.x.x:6666      109.238.101.126:39464   ESTABLISHED
tcp        0      0 88.x.x.x:6666      46.0.131.67:1058        ESTABLISHED
tcp        0      0 88.x.x.x:6666      77.238.97.242:49234     ESTABLISHED
tcp        0    634 88.x.x.x:6666      95.78.28.8:49294        ESTABLISHED
tcp        0     23 88.x.x.x:6666      195.93.151.4:4715       ESTABLISHED
tcp        0     58 88.x.x.x:6666      78.106.174.38:53058     ESTABLISHED
tcp        0    393 88.x.x.x:6666      94.255.125.7:2085       ESTABLISHED
tcp        0      0 88.x.x.x:6666      178.49.253.14:1351      ESTABLISHED
tcp        0    424 88.x.x.x:6666      46.150.236.39:49318     ESTABLISHED
tcp        0    108 88.x.x.x:6666      93.116.207.93:2307      ESTABLISHED
tcp        0      0 88.x.x.x:6666      213.183.236.98:1346     ESTABLISHED
tcp        0    688 88.x.x.x:6666      188.17.197.68:1040      ESTABLISHED
tcp        0    257 88.x.x.x:6666      80.251.57.9:62680       ESTABLISHED
tcp        0    114 88.x.x.x:6666      94.243.32.152:52870     ESTABLISHED
tcp        0    615 88.x.x.x:6666      88.85.193.219:1227      ESTABLISHED
tcp        0      0 127.0.0.1:56228    127.0.0.1:3306          ESTABLISHED
tcp        0      0 88.x.x.x:6666      94.137.245.109:62588    ESTABLISHED
tcp        0    575 88.x.x.x:6666      94.50.184.111:3510      ESTABLISHED
tcp        0     92 88.x.x.x:6666      188.113.7.139:1043      ESTABLISHED
tcp        0      0 88.x.x.x:6666      188.17.73.12:1075       ESTABLISHED
tcp        0    282 88.x.x.x:6666      46.146.14.164:2231      ESTABLISHED
tcp        0    704 88.x.x.x:6666      93.88.18.224:1084       ESTABLISHED
tcp        0      0 127.0.0.1:3306     127.0.0.1:46466         ESTABLISHED
tcp        0    193 88.x.x.x:6666      95.84.18.113:1132       ESTABLISHED
tcp        0     93 88.x.x.x:6666      95.158.244.195:4543     ESTABLISHED
tcp        0   1577 88.x.x.x:6666      188.18.67.10:4096       ESTABLISHED
tcp        0     70 88.x.x.x:6666      188.17.200.156:51095    ESTABLISHED
tcp        0    671 88.x.x.x:6666      78.106.25.109:1123      ESTABLISHED
tcp        0    209 88.x.x.x:6666      93.124.85.126:10078     ESTABLISHED
tcp        0    187 88.x.x.x:6666      88.147.235.128:41891    ESTABLISHED
tcp        0    305 88.x.x.x:6666      94.180.131.192:50946    ESTABLISHED
tcp        0      0 88.x.x.x:6666      77.73.88.20:52208       ESTABLISHED
tcp        0     32 88.x.x.x:6666      2.60.212.150:1686       ESTABLISHED
tcp        0     81 88.x.x.x:6666      178.34.174.236:1214     ESTABLISHED
tcp        0     70 88.x.x.x:6666      95.190.81.235:1239      ESTABLISHED
tcp        0      0 127.0.0.1:46466         127.0.0.1:3306          ESTABLISHED
tcp        0      0 88.x.x.x:6666      80.78.100.71:50382      ESTABLISHED
tcp        0     61 88.x.x.x:6666      46.42.29.77:1175        ESTABLISHED
tcp        0    186 88.x.x.x:6666      80.64.175.18:55654      ESTABLISHED
tcp        0    208 88.x.x.x:6666      89.189.140.137:50333    ESTABLISHED
tcp        0      0 127.0.0.1:44453         127.0.0.1:3306          ESTABLISHED
tcp        0     70 88.x.x.x:6666      95.68.56.99:4556        ESTABLISHED
tcp        0    191 88.x.x.x:6666      46.138.92.37:1155       ESTABLISHED
tcp        0    258 88.x.x.x:6666      92.113.19.95:1075       ESTABLISHED
tcp        0    234 88.x.x.x:6666      77.41.18.221:3239       ESTABLISHED
tcp        0     86 88.x.x.x:6666      188.18.112.157:49372    ESTABLISHED
tcp        0    910 88.x.x.x:6666      95.105.9.141:3703       ESTABLISHED
tcp        0    578 88.x.x.x:6666      95.71.27.233:1159       ESTABLISHED
tcp        0    455 88.x.x.x:6666      85.140.162.175:18255    ESTABLISHED
tcp        0   1303 88.x.x.x:6666      178.93.41.176:19736     ESTABLISHED
tcp        0  12232 88.x.x.x:6666      91.206.219.83:1485      ESTABLISHED
tcp        0     35 88.x.x.x:6666      77.40.32.49:50439       ESTABLISHED
tcp        0     63 88.x.x.x:6666      195.78.104.5:58701      ESTABLISHED
tcp        0    155 88.x.x.x:6666      62.33.186.242:52404     ESTABLISHED
tcp        0     96 88.x.x.x:6666      195.211.206.228:49391   ESTABLISHED
tcp        0    117 88.x.x.x:6666      95.78.189.210:2696      ESTABLISHED
tcp        0    400 88.x.x.x:6666      62.33.145.164:4815      ESTABLISHED
tcp        0      0 88.x.x.x:6666      178.47.93.124:49562     ESTABLISHED
tcp        0      0 88.x.x.x:6666      92.248.140.241:1467     ESTABLISHED
tcp        0    210 88.x.x.x:6666      95.105.55.219:49486     ESTABLISHED
tcp        0    847 88.x.x.x:6666      94.190.39.33:50088      ESTABLISHED
tcp        0    363 88.x.x.x:6666      188.115.138.226:2055    ESTABLISHED
tcp        0    209 88.x.x.x:6666      81.161.214.148:1053     ESTABLISHED
tcp        0      0 88.x.x.x:6666      85.26.165.89:44287      ESTABLISHED
tcp        0      0 88.x.x.x:6666      91.203.63.25:1129       ESTABLISHED
tcp        0     70 88.x.x.x:6666      87.103.236.121:10716    ESTABLISHED
tcp        0      0 88.x.x.x:6666      89.23.132.4:52251       ESTABLISHED
tcp        0    200 88.x.x.x:6666      178.204.26.59:1299      ESTABLISHED
tcp        0    512 88.x.x.x:6666      93.125.49.66:51026      ESTABLISHED
tcp        0      0 88.x.x.x:6666      92.47.99.72:1743        ESTABLISHED
tcp        0    892 88.x.x.x:6666      95.78.46.59:2357        ESTABLISHED
tcp        0     36 88.x.x.x:6666      78.85.215.3:3266        ESTABLISHED
tcp        0    169 88.x.x.x:6666      194.50.125.38:49445     ESTABLISHED
tcp        0   1303 88.x.x.x:6666      217.175.1.1:51023       ESTABLISHED
tcp        0    332 88.x.x.x:6666      178.94.164.206:2254     ESTABLISHED
tcp        0    509 88.x.x.x:6666      109.198.205.157:49565   ESTABLISHED
tcp        0     14 88.x.x.x:6666      92.61.64.207:49189      ESTABLISHED
tcp        0     70 88.x.x.x:6666      178.123.213.64:14576    ESTABLISHED
tcp        0    209 88.x.x.x:6666      80.237.117.118:63014    ESTABLISHED
tcp        0    158 88.x.x.x:6666      83.239.188.21:1572      ESTABLISHED
tcp        0     23 88.x.x.x:6666      94.77.188.64:49557      ESTABLISHED
tcp        0     58 88.x.x.x:6666      94.251.22.57:1289       ESTABLISHED
tcp        0    124 88.x.x.x:6666      194.246.104.139:38233   ESTABLISHED
tcp        0     50 88.x.x.x:6666      95.72.221.229:49356     ESTABLISHED
tcp        0      0 88.x.x.x:6666      178.47.193.148:55813    ESTABLISHED
tcp        0      0 88.x.x.x:6666      80.64.175.21:1719       ESTABLISHED
tcp        0     35 88.x.x.x:6666      109.126.218.201:51941   ESTABLISHED
tcp        0   1179 88.x.x.x:6666      95.154.119.70:55359     ESTABLISHED
tcp        0    654 88.x.x.x:6666      213.108.172.143:1392    ESTABLISHED
tcp        0    562 88.x.x.x:6666      95.30.46.212:1065       ESTABLISHED
tcp        0    419 88.x.x.x:6666      83.234.74.107:1603      ESTABLISHED
tcp        0    339 88.x.x.x:6666      92.37.171.248:2108      ESTABLISHED
tcp        0    229 88.x.x.x:6666      78.108.79.70:3787       ESTABLISHED
tcp        0    901 88.x.x.x:6666      212.124.27.242:2704     ESTABLISHED
tcp        0      0 88.x.x.x:6666      93.191.14.9:63213       ESTABLISHED
tcp        0      0 88.x.x.x:6666      95.27.22.75:49991       ESTABLISHED
tcp        0     87 88.x.x.x:6666      92.255.66.170:39692     ESTABLISHED
tcp        0    639 88.x.x.x:6666      94.178.70.55:63242      ESTABLISHED
tcp        0     70 88.x.x.x:6666      178.74.196.183:3434     ESTABLISHED
tcp        0    120 88.x.x.x:6666      89.254.232.74:53246     ESTABLISHED
tcp        0    221 88.x.x.x:6666      92.127.147.72:3502      ESTABLISHED
tcp        0   3937 88.x.x.x:6666      77.232.142.241:62193    ESTABLISHED
tcp        0    822 88.x.x.x:6666      82.195.18.50:3249       ESTABLISHED
tcp        0      0 88.x.x.x:6666      93.120.229.244:27797    ESTABLISHED
tcp        0     42 88.x.x.x:6666      78.132.157.221:4374     ESTABLISHED
tcp        0    261 88.x.x.x:6666      178.137.149.134:1372    ESTABLISHED
tcp        0      0 88.x.x.x:6666      62.176.11.190:56191     ESTABLISHED
tcp        0    238 88.x.x.x:6666      188.128.15.234:64206    ESTABLISHED
tcp        0    102 88.x.x.x:6666      77.232.15.242:51113     ESTABLISHED
tcp        0     76 88.x.x.x:6666      92.46.153.107:2049      ESTABLISHED
tcp        0    119 88.x.x.x:6666      46.72.28.181:49308      ESTABLISHED
tcp        0     51 88.x.x.x:6666      87.226.90.9:2677        ESTABLISHED
tcp        0     35 88.x.x.x:6666      89.189.169.49:49675     ESTABLISHED
tcp        0     96 88.x.x.x:6666      109.108.44.228:5045     ESTABLISHED
tcp        0      0 88.x.x.x:6666      88.85.191.198:2018      ESTABLISHED
tcp        0    774 88.x.x.x:6666      46.211.19.43:49655      ESTABLISHED
tcp        0    196 88.x.x.x:6666      178.72.75.207:41775     ESTABLISHED
tcp        0   1259 88.x.x.x:6666      178.72.101.163:57176    ESTABLISHED
tcp        0     70 88.x.x.x:6666      188.16.60.235:4228      ESTABLISHED
tcp        0    360 88.x.x.x:6666      213.138.85.180:1225     ESTABLISHED
tcp        0    133 88.x.x.x:6666      95.24.1.117:49208       ESTABLISHED
tcp        0    176 88.x.x.x:6666      178.187.67.138:50969    ESTABLISHED
tcp        0      0 88.x.x.x:6666      80.92.96.55:49895       ESTABLISHED
tcp        0    490 88.x.x.x:6666      178.49.20.148:52525     ESTABLISHED
tcp        0      0 88.x.x.x:6666      62.84.62.201:62798      ESTABLISHED
tcp        0     35 88.x.x.x:6666      95.42.15.196:49273      ESTABLISHED
tcp        0    640 88.x.x.x:6666      213.138.68.38:3225      ESTABLISHED
tcp        0    221 88.x.x.x:6666      89.178.102.172:64789    ESTABLISHED
tcp        0    829 88.x.x.x:6666      92.124.8.199:1045       ESTABLISHED
tcp        0     71 88.x.x.x:6666      79.133.225.33:51690     ESTABLISHED
tcp        0    243 88.x.x.x:6666      78.84.42.222:49387      ESTABLISHED
tcp        0    221 88.x.x.x:6666      46.146.234.14:49181     ESTABLISHED
tcp        0    526 88.x.x.x:6666      85.26.234.141:60719     ESTABLISHED
tcp        0     23 88.x.x.x:6666      82.200.11.193:10191     ESTABLISHED
tcp        0   1013 88.x.x.x:6666      78.36.220.5:1454        ESTABLISHED
tcp        0      0 88.x.x.x:6666      2.61.79.100:43077       ESTABLISHED
tcp        0      0 88.x.x.x:6666      188.127.240.161:1377    ESTABLISHED
tcp        0    410 88.x.x.x:6666      91.195.230.14:2598      ESTABLISHED
tcp        0      0 88.x.x.x:6666      178.216.73.164:59263    ESTABLISHED
tcp        0     35 88.x.x.x:6666      212.21.8.221:53097      ESTABLISHED
tcp        0     69 88.x.x.x:6666      89.189.191.102:50455    ESTABLISHED
tcp        0     38 88.x.x.x:6666      109.86.61.168:2852      ESTABLISHED
tcp        0    615 88.x.x.x:6666      94.179.27.240:55422     ESTABLISHED
tcp        0    103 88.x.x.x:6666      213.231.61.245:49517    ESTABLISHED
tcp        0    202 88.x.x.x:6666      80.64.175.1:1789        ESTABLISHED
tcp        0     16 88.x.x.x:6666      95.170.181.250:49520    ESTABLISHED
tcp        0    541 88.x.x.x:6666      95.78.44.94:49316       ESTABLISHED
tcp        0    248 88.x.x.x:6666      94.75.150.164:2056      ESTABLISHED
tcp        0   1134 88.x.x.x:6666      95.79.10.152:51538      ESTABLISHED
tcp        0      0 88.x.x.x:6666      85.115.224.161:47116    TIME_WAIT  
tcp        0     38 88.x.x.x:6666      178.64.97.89:1281       ESTABLISHED
tcp        0      0 88.x.x.x:6666      89.178.10.238:3653      ESTABLISHED
tcp        0      0 88.x.x.x:6666      95.73.41.0:49538        ESTABLISHED
tcp        0    588 88.x.x.x:6666      85.159.228.101:49435    ESTABLISHED
tcp        0      0 88.x.x.x:6666      92.112.77.140:49248     ESTABLISHED
tcp        0     70 88.x.x.x:6666      109.86.63.184:49409     ESTABLISHED
tcp        0      0 88.x.x.x:6666      95.134.68.32:3373       ESTABLISHED
tcp        0    740 88.x.x.x:6666      95.154.121.236:62130    ESTABLISHED
tcp        0      0 88.x.x.x:6666      109.72.73.133:61266     ESTABLISHED
tcp        0    284 88.x.x.x:6666      178.49.129.23:45407     ESTABLISHED
tcp        0     93 88.x.x.x:6666      94.103.51.165:50940     ESTABLISHED
tcp        0      0 88.x.x.x:6666      109.61.205.47:3263      ESTABLISHED
tcp        0    437 88.x.x.x:6666      95.179.7.56:1076        ESTABLISHED
tcp        0    780 88.x.x.x:6666      109.196.78.182:3061     ESTABLISHED
tcp        0    370 88.x.x.x:6666      94.248.49.45:1347       ESTABLISHED
tcp        0     13 88.x.x.x:6666      93.116.113.143:1482     ESTABLISHED
tcp        0   1062 88.x.x.x:6666      188.17.103.121:50270    ESTABLISHED
tcp        0      0 88.x.x.x:6666      178.71.39.55:60985      ESTABLISHED
tcp        0      0 88.x.x.x:6666      95.32.251.149:26233     ESTABLISHED
tcp        0     71 88.x.x.x:6666      89.146.81.168:3781      ESTABLISHED
tcp        0      0 88.x.x.x:6666      95.68.112.77:1383       ESTABLISHED
tcp        0    183 88.x.x.x:6666      193.106.202.64:2125     ESTABLISHED
tcp        0    989 88.x.x.x:6666      92.124.226.153:49963    ESTABLISHED
tcp        0    162 88.x.x.x:6666      109.74.166.5:2692       ESTABLISHED
tcp        0    210 88.x.x.x:6666      94.137.199.13:2790      ESTABLISHED
tcp        0     58 88.x.x.x:6666      109.254.67.112:1061     ESTABLISHED
tcp        0     71 88.x.x.x:6666      91.124.31.110:13083     ESTABLISHED
tcp        0     23 88.x.x.x:6666      79.126.69.124:1255      ESTABLISHED
tcp        0      0 88.x.x.x:6666      79.139.211.94:1315      ESTABLISHED
tcp        0    380 88.x.x.x:6666      188.18.174.235:51319    ESTABLISHED
tcp        0     70 88.x.x.x:6666      81.201.254.189:4578     ESTABLISHED
tcp        0    546 88.x.x.x:6666      213.87.88.107:4583      ESTABLISHED
tcp        0     96 88.x.x.x:6666      89.189.169.43:4773      ESTABLISHED
tcp        0     89 88.x.x.x:6666      195.62.92.4:1424        ESTABLISHED
tcp        0      0 88.x.x.x:6666      217.15.200.56:61382     ESTABLISHED
tcp        0      0 127.0.0.1:34634         127.0.0.1:3306          ESTABLISHED
tcp        0    239 88.x.x.x:6666      212.220.2.2:2760        ESTABLISHED
tcp        0    222 88.x.x.x:6666      77.220.60.101:4096      ESTABLISHED
tcp        0      0 88.x.x.x:6666      178.93.140.128:50988    ESTABLISHED
tcp        0     13 88.x.x.x:6666      188.123.231.12:62227    ESTABLISHED
tcp        0    326 88.x.x.x:6666      193.84.90.113:49163     ESTABLISHED
tcp        0     46 88.x.x.x:6666      90.151.43.189:50222     ESTABLISHED
tcp        0    143 88.x.x.x:6666      178.34.30.55:1683       ESTABLISHED
tcp        0   2145 88.x.x.x:6666      94.73.250.140:49782     ESTABLISHED
tcp        0      0 88.x.x.x:6666      178.219.42.12:3936      ESTABLISHED
tcp        0    724 88.x.x.x:6666      85.175.142.56:29220     ESTABLISHED
tcp        0      0 88.x.x.x:6666      62.165.61.67:64600      ESTABLISHED
tcp        0    134 88.x.x.x:6666      77.222.97.214:57498     ESTABLISHED
tcp        0    840 88.x.x.x:6666      2.61.59.144:51325       ESTABLISHED
tcp        0    358 88.x.x.x:6666      77.238.99.66:1057       ESTABLISHED
tcp        0      0 88.x.x.x:6666      217.79.1.253:1237       ESTABLISHED
tcp        0      0 127.0.0.1:3306          127.0.0.1:57727         ESTABLISHED
tcp        0     58 88.x.x.x:6666      178.208.250.46:61957    ESTABLISHED
tcp        0     93 88.x.x.x:6666      79.173.80.119:1050      ESTABLISHED
tcp        0     46 88.x.x.x:6666      85.26.225.108:42076     ESTABLISHED
tcp        0    131 88.x.x.x:6666      217.8.238.251:17266     ESTABLISHED
tcp        0    222 88.x.x.x:6666      109.61.185.42:49368     ESTABLISHED
tcp        0     35 88.x.x.x:6666      95.165.166.135:1035     ESTABLISHED
tcp        0      0 88.x.x.x:6666      217.78.187.94:61080     ESTABLISHED
tcp        0    264 88.x.x.x:6666      81.88.211.109:49966     ESTABLISHED
tcp        0     70 88.x.x.x:6666      2.61.16.21:1890         ESTABLISHED
tcp        0    245 88.x.x.x:6666      77.247.167.116:49345    ESTABLISHED
tcp        0     16 88.x.x.x:6666      109.198.211.249:50296   ESTABLISHED
tcp        0    114 88.x.x.x:6666      89.178.16.175:65224     ESTABLISHED
tcp        0    201 88.x.x.x:6666      78.36.92.185:1384       ESTABLISHED
tcp        0    489 88.x.x.x:6666      178.187.20.85:49181     ESTABLISHED
tcp        0      0 88.x.x.x:6666      188.130.176.4:1269      ESTABLISHED
tcp        0    743 88.x.x.x:6666      95.170.100.6:1777       ESTABLISHED
tcp        0      0 127.0.0.1:57727         127.0.0.1:3306          ESTABLISHED
tcp        0      0 127.0.0.1:53416         127.0.0.1:9014          ESTABLISHED
tcp        0    821 88.x.x.x:6666      77.238.123.208:1194     ESTABLISHED
tcp        0      0 88.x.x.x:6666      46.148.56.105:49475     ESTABLISHED
tcp        0     35 88.x.x.x:6666      109.197.53.62:58365     ESTABLISHED
tcp        0     52 88.x.x.x:6666      77.232.142.219:61253    ESTABLISHED
tcp        0    183 88.x.x.x:6666      178.91.66.125:10555     ESTABLISHED
tcp        0    222 88.x.x.x:6666      195.184.210.38:54688    ESTABLISHED
tcp        0     63 88.x.x.x:6666      92.49.210.216:49178     ESTABLISHED
tcp        0    370 88.x.x.x:6666      77.39.17.74:50203       ESTABLISHED
tcp        0    456 88.x.x.x:6666      109.87.41.180:2569      ESTABLISHED
tcp        0      0 88.x.x.x:6666      95.119.167.248:15446    ESTABLISHED
tcp        0    163 88.x.x.x:6666      93.178.233.135:27473    ESTABLISHED
tcp        0    422 88.x.x.x:6666      178.207.39.19:2477      ESTABLISHED
tcp        0      0 88.x.x.x:6666      90.189.5.63:2933        ESTABLISHED
tcp        0   1878 88.x.x.x:6666      188.17.207.6:1079       ESTABLISHED
tcp        0     23 88.x.x.x:6666      109.169.149.72:49752    ESTABLISHED
tcp        0    211 88.x.x.x:6666      89.237.28.110:58752     ESTABLISHED
tcp        0      0 88.x.x.x:6666      178.95.167.9:1350       ESTABLISHED
tcp        0     54 88.x.x.x:6666      217.118.81.28:23815     ESTABLISHED
tcp        0   1496 88.x.x.x:6666      195.189.105.134:49476   ESTABLISHED
.....

text

 cat /proc/net/ip_conntrack | wc -l
997

24.12.2010 17:51 #110189

Может быть synflood? При такой атаке соединения не устанавливаются, но ресурсы сервера на некоторое время оказываются заняты ожиданием установления соединения.

Что это: http://ru.wikipedia.org/wiki/SYN-%D1%84%D0%BB%D1%83%D0%B4

Попробуйте включить syncookies
[console]
echo "1" > /proc/sys/net/ipv4/tcp_syncookies[/console]

Вот тут еще есть описания борьбы с synflood атаками.
http://tajforum.tj/topic/4692-%D0%BC%D0%B5%D1%82%D0%BE%D0%B4-%D0%B7%D0%B0%D1%89%D0%B8%D1%82%D1%8B-%D0%BE%D1%82-syn-flood-%D0%B0%D1%82%D0%B0%D0%BA%D0%B8/

http://www.opennet.ru/base/sec/linux_dos_guard.txt.html

24.12.2010 18:07 · отредактировано #110199

ESTABLISHED показывает, что соединение установлено, поэтому на synflood не похоже.

По ICMP тоже ничего криминального. Вот стата по нему с обычного десктопа с аптаймом в полтора суток.

text

Icmp:
    24024 ICMP messages received
    3283 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 21060
        timeout in transit: 2421
        source quenches: 308
        redirects: 203
        echo requests: 24
        echo replies: 8
    1642 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 1526
        time exceeded: 6
        echo request: 86
        echo replies: 24

Может, еще надо затюнить TCP idle timeout, но не уверен, что это будет в тему.

24.12.2010 18:16 #110203

elve написал:
Может быть synflood? При такой атаке соединения не устанавливаются, но ресурсы сервера на некоторое время оказываются заняты ожиданием установления соединения.

Что это: http://ru.wikipedia.org/wiki/SYN-%D1%84%D0%BB%D1%83%D0%B4

Попробуйте включить syncookies
[console]
echo "1" > /proc/sys/net/ipv4/tcp_syncookies[/console]

Вот тут еще есть описания борьбы с synflood атаками.
http://tajforum.tj/topic/4692-%D0%BC%D0%B5%D1%82%D0%BE%D0%B4-%D0%B7%D0%B0%D1%89%D0%B8%D1%82%D1%8B-%D0%BE%D1%82-syn-flood-%D0%B0%D1%82%D0%B0%D0%BA%D0%B8/

http://www.opennet.ru/base/sec/linux_dos_guard.txt.html

Синкуки включены, более того стоит ограничение на кол-во син-запросов с одного хоста(в скрипте выше приведены настройки iptables+sysctl, там это отражено), поэтому не знаем что и думать. Насколько мне известно, при атаках типа syn-flood расход трафика не увеличивается так, как это произошло с нами? К примеру, до обнаружения атаки средний расход трафика в сутки был около 2Гб входящего, 4Гб исходящего, сейчас у нас около 6Гб входящего и 25Гб исходящего трафика в сутки.

24.12.2010 20:48 · отредактировано #110240

имя зверушек в студию - сервер:на чём? , приложение:какое ??

возможно само приложение тупит и\или вступает в какое-то гремучее взаимодействие с настройками сервера ...

айпишники в netstat'е чьи ? ваши или "левые" ?

варианты с хитрым руткитом рассматривали ?

проскакивала информация про попытки распространение образов с закладками ботнета ...

а так на лицо попытка построения распределённой сети ...

может вам стоит настроить жёсткую аутентификацию для подключения "своих" юзверей ?.

25.12.2010 01:07 #110308

Неприятно, но не критично, < 1000 хостов. С чего убежденность, что именно синфлуд? tcpdump подтверждает это? Можно попробовать увстановить лимит соединения по времени. Синкуки тоже вещь крайне полезная (при условии именно синфлуда). Можно попробовать так, чтобы сокеты не выжирались быстро:
iptables -I INPUT -p tcp --dport 6666 -i eth0 -m state --state NEW -m recent --set
iptables -I INPUT -p tcp --dport 6666 -i eth0 -m state --state NEW -m recent --update --seconds Х --hitcount Х -j DROP
Где Х кол-во соединений за определенное кол-во времени.