From: Dirk Brenken Date: Sat, 25 Mar 2017 20:49:44 +0000 (+0100) Subject: adblock: update 2.5.0 X-Git-Url: http://git.lede-project.org./?a=commitdiff_plain;h=9e3f88916f21d32fbd234d2248b0ac584ce45121;p=feed%2Fpackages.git adblock: update 2.5.0 * revamp LuCI GUI (see luci repo for details) * integrated runtime statistics in normal adblock ubus service instance (see readme) * simplified boot/init * no longer use raw interface trigger * use only network interface trigger, pre-configured for 'wan/wwan/lan' interfaces (see readme) * fixed a reload issue * removed switch to disable tld compression (seems to be pretty stable) * muted awk parser if source list contains incorrect records * raise dns backend timeout to 20 seconds * documentation update * cosmetics Signed-off-by: Dirk Brenken --- diff --git a/net/adblock/Makefile b/net/adblock/Makefile index a5f4b803fc..8b806acc16 100644 --- a/net/adblock/Makefile +++ b/net/adblock/Makefile @@ -6,7 +6,7 @@ include $(TOPDIR)/rules.mk PKG_NAME:=adblock -PKG_VERSION:=2.4.0 +PKG_VERSION:=2.5.0 PKG_RELEASE:=2 PKG_LICENSE:=GPL-3.0+ PKG_MAINTAINER:=Dirk Brenken diff --git a/net/adblock/files/README.md b/net/adblock/files/README.md index 172551f5cd..2c2ec39821 100644 --- a/net/adblock/files/README.md +++ b/net/adblock/files/README.md @@ -64,8 +64,8 @@ A lot of people already use adblocker plugins within their desktop browsers, but * additional whitelist for manual overrides, located by default in /etc/adblock/adblock.whitelist * quality checks during block list update to ensure a reliable dns backend service * minimal status & error logging to syslog, enable debug logging to receive more output -* procd based init system support (start/stop/restart/reload/suspend/resume) -* procd based hotplug support, the adblock start will be triggered by interface triggers +* procd based init system support (start/stop/restart/reload/suspend/resume/query) +* procd based hotplug support, the adblock start will be solely triggered by network interface triggers * suspend & resume adblock actions temporarily without block list reloading * runtime statistics via ubus service call * query function to quickly identify blocked (sub-)domains, e.g. for whitelisting @@ -73,8 +73,7 @@ A lot of people already use adblocker plugins within their desktop browsers, but * optional: add new adblock sources on your own via uci config ## Prerequisites -* [openwrt](https://openwrt.org), tested with latest stable release (Chaos Calmer) and with current trunk (Designated Driver) -* [LEDE project](https://www.lede-project.org), tested with trunk > r98 +* [LEDE project](https://www.lede-project.org), tested with latest stable release (LEDE 17.01) and with current LEDE snapshot * a usual setup with an enabled dns backend at minimum - dump AP modes without a working dns backend are _not_ supported * a download utility: full versions (with ssl support) of 'wget', 'uclient-fetch', 'aria2c' or 'curl' are supported - the Chaos Calmer built-in busybox wget is not * Chaos Calmer: download & install the external 'wget' package @@ -87,19 +86,11 @@ A lot of people already use adblocker plugins within their desktop browsers, but * enable/disable your favored block list sources in _/etc/config/adblock_ - 'adaway', 'disconnect' and 'yoyo' are enabled by default ## LuCI adblock companion package -* for easy management of the various block list sources and options you can also use a nice & efficient LuCI frontend +* for easy management of the various block list sources and all other adblock options you can also use a nice & efficient LuCI frontend * install 'luci-app-adblock' (_opkg install luci-app-adblock_) * the application is located in LuCI under 'Services' menu * _Thanks to Hannu Nyman for this great adblock LuCI frontend!_ -## Chaos Calmer installation notes -* 'adblock' and 'luci-app-adblock' are _not_ available as .ipk packages in the Chaos Calmer download repository -* download both packages from a development snapshot package directory: - * for 'adblock' look [here](https://downloads.lede-project.org/snapshots/packages/x86_64/packages/) - * for 'luci-app-adblock' look [here](https://downloads.lede-project.org/snapshots/packages/x86_64/luci/) -* manually transfer the packages to your routers temp directory (with tools like _sshfs_ or _winscp_) -* install the packages with _opkg install <...>_ as described above - ## Tweaks * **status/runtime statistics:** the adblock status and runtime statistics are available via ubus service call (see example below) * **debug logging:** for script debugging please set the config option 'adb\_debug' to '1' and check the runtime output with _logread -e "adblock"_ @@ -118,10 +109,9 @@ A lot of people already use adblocker plugins within their desktop browsers, but * usually the pre-configured adblock setup works quite well and no manual config overrides are needed, all listed options apply to the 'global' config section: * adb\_enabled => main switch to enable/disable adblock service (default: '1', enabled) * adb\_debug => enable/disable adblock debug output (default: '0', disabled) - * adb\_iface => restrict the procd interface trigger to a (list of) certain wan interface(s) or disable it at all (default: not set, disabled) + * adb\_iface => set the procd interface trigger to a (list of) lan/wan interface(s) (default: 'wan wwan lan') * adb\_fetch => full path to a different download utility, see example below (default: not set, use wget) * adb\_fetchparm => options for the download utility, see example below (default: not set, use wget options) - * adb\_tldcomp => enable/disable tld compression (default: '1', enabled) ## Examples @@ -155,50 +145,42 @@ curl: **receive adblock statistics via ubus:** 

-ubus call service list '{"name":"adblock_stats"}'
+ubus call service get_data '{"name":"adblock"}
 This will output the active block lists and other runtime information as JSON, e.g.:
 {
-    "adblock_stats": {
-        "instances": {
-            "statistics": {
-                "running": false,
-                "command": [
-                    ""
-                ],
-                "data": {
-                    "active_lists": [
-                        {
-                            "palevo": "14",
-                            "blacklist": "144",
-                            "winspy": "168",
-                            "zeus": "422",
-                            "adaway": "408",
-                            "rolist": "649",
-                            "malwarelist": "1219",
-                            "ransomware": "1495",
-                            "ruadlist": "1791",
-                            "yoyo": "2304",
-                            "openphish": "2139",
-                            "dshield": "154",
-                            "disconnect": "3176",
-                            "spam404": "6251",
-                            "adguard": "11081",
-                            "whocares": "11575",
-                            "winhelp": "10574",
-                            "malware": "13854",
-                            "sysctl": "8539",
-                            "securemecca": "9262",
-                            "shalla": "25358",
-                            "hphosts": "36256"
-                        }
-                    ],
-                    "adblock_version": "2.3.0",
-                    "blocked_domains": "146833",
-                    "dns_backend": "dnsmasq",
-                    "last_rundate": "04.02.2017 21:10:31",
-                    "system": "LEDE Reboot SNAPSHOT r3286-c980147527"
+    "adblock": {
+        "adblock": {
+            "active_lists": [
+                {
+                    "palevo": "14",
+                    "blacklist": "147",
+                    "winspy": "138",
+                    "adaway": "378",
+                    "zeus": "397",
+                    "rolist": "652",
+                    "malwarelist": "1157",
+                    "yoyo": "2272",
+                    "ruadlist": "1793",
+                    "ransomware": "1670",
+                    "dshield": "190",
+                    "openphish": "2672",
+                    "disconnect": "2910",
+                    "spam404": "5715",
+                    "whocares": "8900",
+                    "winhelp": "8528",
+                    "adguard": "10790",
+                    "securemecca": "5080",
+                    "sysctl": "7906",
+                    "malware": "14617",
+                    "hphosts": "12450",
+                    "shalla": "23926"
                 }
-            }
+            ],
+            "adblock_version": "2.5.0",
+            "blocked_domains": "112302",
+            "dns_backend": "dnsmasq",
+            "last_rundate": "25.03.2017 21:09:12",
+            "system": "LEDE Reboot SNAPSHOT r3867-313197d707"
         }
     }
 }
@@ -295,7 +277,7 @@ If your awk one-liner works quite well, add a new source section in adblock conf
## Support -Please join the adblock discussion in this [forum thread](https://forum.openwrt.org/viewtopic.php?id=59803) or contact me by mail +Please join the adblock discussion in this [forum thread](https://forum.lede-project.org/t/adblock-2-x-support-thread/507) or contact me by mail ## Removal * stop all adblock related services with _/etc/init.d/adblock stop_ diff --git a/net/adblock/files/adblock.conf b/net/adblock/files/adblock.conf index 999a2c47a8..0cdb4e1c42 100644 --- a/net/adblock/files/adblock.conf +++ b/net/adblock/files/adblock.conf @@ -4,6 +4,7 @@ config adblock 'global' option adb_enabled '1' option adb_debug '0' + option adb_iface 'wan wwan lan' option adb_whitelist '/etc/adblock/adblock.whitelist' option adb_whitelist_rset '\$1 ~/^([A-Za-z0-9_-]+\.){1,}[A-Za-z]+/{print tolower(\"^\"\$1\"\\\|[.]\"\$1)}' option adb_backup '0' diff --git a/net/adblock/files/adblock.init b/net/adblock/files/adblock.init index cf8a57b151..ee1cd075bf 100755 --- a/net/adblock/files/adblock.init +++ b/net/adblock/files/adblock.init @@ -9,32 +9,24 @@ EXTRA_HELP=" suspend Suspend adblock processing resume Resume adblock processing query Query active blocklists for specific domain" +adb_init="/etc/init.d/adblock" adb_script="/usr/bin/adblock.sh" boot() { - local wanif4 wanif6 cnt=0 max_cnt=30 + local list iface="$(uci -q get adblock.global.adb_iface)" - . "/lib/functions/network.sh" - while [ ${cnt} -le ${max_cnt} ] + for name in ${iface} do - network_find_wan wanif4 - network_find_wan6 wanif6 - if [ -n "${wanif4}" ] || [ -n "${wanif6}" ] || [ ${cnt} -eq ${max_cnt} ] - then - rc_procd start_service - return 0 - else - sleep 1 - cnt=$((cnt+1)) - network_flush_cache - fi + list="${list} network.interface.${name}" done + ubus -t 60 wait_for network.interface ${list} + rc_procd start_service } start_service() { - if [ $(/etc/init.d/adblock enabled; printf ${?}) -eq 0 ] + if [ $("${adb_init}" enabled; printf ${?}) -eq 0 ] then procd_open_instance "adblock" procd_set_param command "${adb_script}" "${@}" @@ -44,11 +36,6 @@ start_service() fi } -reload_service() -{ - rc_procd start_service reload -} - stop_service() { rc_procd "${adb_script}" stop @@ -61,12 +48,12 @@ restart() suspend() { - rc_procd start_service suspend + rc_procd "${adb_script}" suspend } resume() { - rc_procd start_service resume + rc_procd "${adb_script}" resume } query() @@ -78,14 +65,9 @@ service_triggers() { local iface="$(uci -q get adblock.global.adb_iface)" - if [ -z "${iface}" ] - then - procd_add_raw_trigger "interface.*.up" 1000 /etc/init.d/adblock start - else - for name in ${iface} - do - procd_add_interface_trigger "interface.*.up" "${name}" /etc/init.d/adblock start - done - fi - procd_add_config_trigger "config.change" "adblock" /etc/init.d/adblock start + for name in ${iface} + do + procd_add_interface_trigger "interface.*.up" "${name}" "${adb_init}" start + done + procd_add_config_trigger "config.change" "adblock" "${adb_init}" start } diff --git a/net/adblock/files/adblock.sh b/net/adblock/files/adblock.sh index a470d0bf7e..6d8d92255b 100755 --- a/net/adblock/files/adblock.sh +++ b/net/adblock/files/adblock.sh @@ -10,11 +10,11 @@ # LC_ALL=C PATH="/usr/sbin:/usr/bin:/sbin:/bin" -adb_ver="2.4.0-2" +adb_ver="2.5.0" +adb_sysver="$(ubus -S call system board | jsonfilter -e '@.release.description')" adb_enabled=1 adb_debug=0 adb_backup=0 -adb_tldcomp=1 adb_backupdir="/mnt" adb_whitelist="/etc/adblock/adblock.whitelist" adb_whitelist_rset="\$1 ~/^([A-Za-z0-9_-]+\.){1,}[A-Za-z]+/{print tolower(\"^\"\$1\"\\\|[.]\"\$1)}" @@ -40,7 +40,7 @@ f_envload() # set dns backend environment # - while [ ${cnt} -le 10 ] + while [ ${cnt} -le 20 ] do for dns in ${adb_dnslist} do @@ -188,7 +188,6 @@ f_rmdns() rm -f "${adb_backupdir}/${adb_dnsprefix}"*.gz rm -rf "${adb_dnshidedir}" fi - ubus call service delete "{\"name\":\"adblock_stats\",\"instances\":\"statistics\"}" 2>/dev/null } # f_dnsrestart: restart the dns backend @@ -203,7 +202,7 @@ f_dnsrestart() adb_dnsup="$(ubus -S call service list "{\"name\":\"${adb_dns}\"}" | jsonfilter -l1 -e "@.${adb_dns}.instances.*.running")" if [ "${adb_dnsup}" = "true" ] then - return 0 + break fi cnt=$((cnt+1)) sleep 1 @@ -323,7 +322,7 @@ f_log() logger -t "adblock-[${adb_ver}] ${class}" "${log_msg}" if [ "${class}" = "error" ] then - logger -t "adblock-[${adb_ver}] ${class}" "Please check the online documentation 'https://github.com/openwrt/packages/blob/master/net/adblock/files/README.md'" + logger -t "adblock-[${adb_ver}] ${class}" "Please check 'https://github.com/openwrt/packages/blob/master/net/adblock/files/README.md' (${adb_sysver})" f_rmtemp if [ "$(ls -dA "${adb_dnsdir}/${adb_dnsprefix}"* >/dev/null 2>&1)" ] then @@ -340,8 +339,7 @@ f_log() f_main() { local enabled url cnt sum_cnt=0 mem_total=0 - local src_name src_rset shalla_archive list active_lists - local sysver="$(ubus -S call system board | jsonfilter -e '@.release.description')" + local src_name src_rset shalla_archive list active_lists active_triggers mem_total="$(awk '$1 ~ /^MemTotal/ {printf $2}' "/proc/meminfo" 2>/dev/null)" f_log "info " "start adblock processing ..." @@ -398,17 +396,12 @@ f_main() # if [ ${adb_rc} -eq 0 ] && [ -s "${adb_tmpload}" ] then - awk "${src_rset}" "${adb_tmpload}" > "${adb_tmpfile}" + awk "${src_rset}" "${adb_tmpload}" 2>/dev/null > "${adb_tmpfile}" if [ -s "${adb_tmpfile}" ] then - if [ ${adb_tldcomp} -eq 1 ] - then - awk -F "." '{for(f=NF;f > 1;f--) printf "%s.", $f;print $1}' "${adb_tmpfile}" | sort -u > "${adb_tmpload}" - awk '{if(NR==1){tld=$NF};while(getline){if($NF !~ tld"\\."){print tld;tld=$NF}}print tld}' "${adb_tmpload}" > "${adb_tmpfile}" - awk -F "." '{for(f=NF;f > 1;f--) printf "%s.", $f;print $1}' "${adb_tmpfile}" > "${adb_tmpload}" - else - sort -u "${adb_tmpfile}" > "${adb_tmpload}" - fi + awk -F "." '{for(f=NF;f > 1;f--) printf "%s.", $f;print $1}' "${adb_tmpfile}" 2>/dev/null | sort -u > "${adb_tmpload}" + awk '{if(NR==1){tld=$NF};while(getline){if($NF !~ tld"\\."){print tld;tld=$NF}}print tld}' "${adb_tmpload}" 2>/dev/null > "${adb_tmpfile}" + awk -F "." '{for(f=NF;f > 1;f--) printf "%s.", $f;print $1}' "${adb_tmpfile}" 2>/dev/null > "${adb_tmpload}" mv -f "${adb_tmpload}" "${adb_tmpfile}" f_list backup else @@ -424,9 +417,9 @@ f_main() then if [ -s "${adb_tmpdir}/tmp.whitelist" ] then - grep -vf "${adb_tmpdir}/tmp.whitelist" "${adb_tmpfile}" | eval "${adb_dnsformat}" > "${adb_dnsfile}" + grep -vf "${adb_tmpdir}/tmp.whitelist" "${adb_tmpfile}" 2>/dev/null | eval "${adb_dnsformat}" > "${adb_dnsfile}" else - cat "${adb_tmpfile}" | eval "${adb_dnsformat}" > "${adb_dnsfile}" + cat "${adb_tmpfile}" 2>/dev/null | eval "${adb_dnsformat}" > "${adb_dnsfile}" fi adb_rc=${?} if [ ${adb_rc} -ne 0 ] @@ -470,18 +463,24 @@ f_main() f_dnsrestart if [ "${adb_dnsup}" = "true" ] then - f_log "info " "block lists with overall ${sum_cnt} domains loaded successfully (${sysver})" - ubus call service set "{\"name\":\"adblock_stats\", - \"instances\":{\"statistics\":{\"command\":[\"\"], + f_log "info " "block lists with overall ${sum_cnt} domains loaded successfully (${adb_sysver})" + for name in ${adb_iface} + do + active_triggers="${active_triggers}[\"interface.*.up\",[\"if\",[\"eq\",\"interface\",\"${name}\"],[\"run_script\",\"/etc/init.d/adblock\",\"start\"],1000]]," + done + active_triggers="${active_triggers}[\"config.change\",[\"if\",[\"eq\",\"package\",\"adblock\"],[\"run_script\",\"/etc/init.d/adblock\",\"start\"],1000]]" + ubus call service set "{\"name\":\"adblock\", + \"instances\":{\"adblock\":{\"command\":[\"/usr/bin/adblock.sh\"], \"data\":{\"active_lists\":[{${active_lists}}], \"adblock_version\":\"${adb_ver}\", \"blocked_domains\":\"${sum_cnt}\", \"dns_backend\":\"${adb_dns}\", \"last_rundate\":\"$(/bin/date "+%d.%m.%Y %H:%M:%S")\", - \"system\":\"${sysver}\"}}}}" - return 0 + \"system\":\"${adb_sysver}\"}}}, + \"triggers\":[${active_triggers}]}" + else + f_log "error" "dns backend restart with active block lists failed" fi - f_log "error" "dns backend restart with active block lists failed (${sysver})" } # handle different adblock actions