Thibaut VARÈNE [Tue, 6 Sep 2022 15:19:37 +0000 (17:19 +0200)]
Revert "phase1: remove renderable locks: not supported in 0.8.9"
This commit re-enables the network locks that were first introduced in
in
0edb49341 ("phase1: provide upload/download locking"), but which were
disabled in
1e1a32607 as they weren't supported by the buildbot version
used at the time.
The use and benefits of this system is described in
0edb49341.
This reverts commit
1e1a326074018542c4a6f14ffe12984fa9c8a192.
Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
Petr Štetiar [Mon, 15 May 2023 06:08:42 +0000 (08:08 +0200)]
Create README.md
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Sun, 14 May 2023 19:44:53 +0000 (21:44 +0200)]
ci: disable container pushing during pull request builds
Seems like we need to find a different workflow for container
pre-testing.
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Sun, 14 May 2023 18:40:08 +0000 (20:40 +0200)]
ci: fix ghcr.io container pushing
Attempt to fix following:
#11 ERROR: denied: permission_denied: The requested installation does not exist.
------
> pushing ghcr.io/ynezz/buildworker-v3.5.0:master with docker:
------
ERROR: denied: permission_denied: The requested installation does not exist.
Error: buildx failed with: ERROR: denied: permission_denied: The requested installation does not exist.
References: https://github.com/openwrt/buildbot/actions/runs/
4971701948/jobs/
8899889081#step:9:191
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Sun, 14 May 2023 07:12:44 +0000 (09:12 +0200)]
Introduce GitHub actions
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Sat, 26 Mar 2022 09:18:10 +0000 (10:18 +0100)]
phase2: fix just another byte/string inconsistencies
File "/phase2/master.cfg", line 203, in parse_feed_entry
if parts[0].startswith("src-git"):
builtins.TypeError: startswith first arg must be bytes or a tuple of bytes, not str
Fixes: f3fcdbec877e ("phase2: fix obsolete assumption about feed methods")
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Fri, 25 Mar 2022 21:38:54 +0000 (22:38 +0100)]
Bump to latest Buildbot release 3.5.0
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Wed, 23 Feb 2022 17:46:41 +0000 (18:46 +0100)]
phase2: handle missing change sources
In commit
f3fcdbec877e ("phase2: fix obsolete assumption about feed
methods") we've fixed stuck buildmaster for master/phase2 builds, but
the issue wasn't obvious at first sight and some digging was necessary.
So lets make it obvious if/when it happens next time.
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Wed, 23 Feb 2022 17:23:43 +0000 (18:23 +0100)]
phase2: fix obsolete assumption about feed methods
In commit
7fae1e5677e9 ("feeds: use git-src-full to allow Git
versioning") there was `src-git` method of shallow clone replaced with
`src-git-full` and this has caused issue with GitPoller configuration as
it only assumed, that only `src-git` method is going to be used for feed
updates. Fix currently stuck buildbot master by allowing usage of both
Git update methods.
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Wed, 23 Feb 2022 17:22:41 +0000 (18:22 +0100)]
Bump to latest Buildbot release 3.4.1
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Sat, 22 Jan 2022 10:50:11 +0000 (11:50 +0100)]
Bump to latest Buildbot release 3.4.0
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Wed, 3 Nov 2021 13:59:26 +0000 (14:59 +0100)]
ci: docker: deploy automatically only on default branch and tags
In commit
fecd899e8189 ("docker: deploy only manually and on
master/tags") we've introduced manual deployment of the Docker images,
but this is unneeded additional manual step, so rather prefer automatic
deployment. While at it, convert to usage of `rules:` as `only:` is
deprecated and `rules:` are preferred.
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Paul Spooren [Fri, 3 Sep 2021 18:39:47 +0000 (08:39 -1000)]
docker: install `mkisofs` to worker container
x86 targets allow the creation of ISO images, however this is not
possible because the `mkisofs` is missing. Install it.
Signed-off-by: Paul Spooren <mail@aparcar.org>
Paul Spooren [Fri, 3 Sep 2021 18:35:55 +0000 (08:35 -1000)]
docker: sort installed packages alphabetically
Signed-off-by: Paul Spooren <mail@aparcar.org>
Paul Spooren [Sun, 5 Sep 2021 01:51:07 +0000 (15:51 -1000)]
docker: deploy only manually and on master/tags
Unmerged branches shouldn't be deployed as "latest" image, only tags and
`master` branches should.
It is still possible to use containers from tested branches since they
are uploaded to the container registry using their *commit slug*.
Signed-off-by: Paul Spooren <mail@aparcar.org>
Petr Štetiar [Wed, 21 Jul 2021 19:31:28 +0000 (21:31 +0200)]
phase1,phase2: improve round robin builds
There seems to be some issue with database updates, where database
updates are asynchronous and thus the status update of the previous
build (complete_at DB column) might take little bit longer, then
preparation of the next/current build.
This is issue during job prioritization as this might result in the
build of the same builder twice in a row, for example:
2021-06-22 02:42:54+0000 [-] <Build mipsel_24kc number:95 results:success>: build finished
2021-06-22 02:42:55+0000 [-] prioritizeBuilders: mipsel_24kc complete_at: 2021-06-21 20:17:14+00:00
2021-06-22 03:14:30+0000 [-] prioritizeBuilders: mipsel_24kc complete_at: 2021-06-22 02:42:54+00:00
Build finishes at 02:42:54, scheduler then asks for next build at
02:42:55, but the build still has the old complete_at timestamp
2021-06-21 20:17:14 instead of the correct one 2021-06-22 02:42:54, thus
scheduling the build of oldest mipsel_24kc builder one more time.
This is so far very promising workaround attempt which checks latest
builder complete_at in builds table which seems to be updated faster,
thus using greater complete_at value seems to work for now.
References: https://github.com/buildbot/buildbot/issues/4592#issuecomment-
801163587
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Mon, 28 Jun 2021 06:39:14 +0000 (08:39 +0200)]
Bump to latest Buildbot release 3.2.0
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Tony Ambardar [Fri, 26 Mar 2021 09:55:41 +0000 (09:55 +0000)]
docker,worker: install libelf-dev, libdw-dev
The libelf-dev is a previous requirement to support STACK_VALIDATION
during kernel builds (x86/64). Building kernels with BTF debug
information requires both this and libdw-dev be installed.
Signed-off-by: Tony Ambardar <itugrok@yahoo.com>
Petr Štetiar [Thu, 25 Mar 2021 10:21:04 +0000 (11:21 +0100)]
buildworker: provide OpenWrt version in description
So it's clear to anyone which version is actually deployed.
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Thu, 25 Mar 2021 10:03:30 +0000 (11:03 +0100)]
buildmaster: provide OpenWrt version in About section
So it's clear to anyone which version is actually deployed.
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Thu, 25 Mar 2021 08:54:49 +0000 (09:54 +0100)]
phase1,phase2: add file space usage reporting
Add `du` step so it's clear how much space was used by the build.
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Baptiste Jonglez [Wed, 24 Mar 2021 22:28:25 +0000 (23:28 +0100)]
phase1, phase2: exclude temporary source files from sourceupload step
When the dl/ directory is shared with several workers, there can be
temporary "*.hash" and "*.dl" files being created by other workers.
These files should be excluded from the sourceupload step, otherwise it
can cause a failure when rsync tries to read them.
See for example:
https://buildbot.openwrt.org/master/packages/#/builders/14/builds/18/steps/39/logs/stdio
Signed-off-by: Baptiste Jonglez <git@bitsofnetworks.org>
Baptiste Jonglez [Wed, 24 Mar 2021 22:48:13 +0000 (23:48 +0100)]
phase1, phase2: don't mark the build failed if a non-critical step fails
Some steps are informative or optional (haltOnFailure=False), so it makes
no sense to mark the whole build as failed if such a step fails.
Depending on the case, we either mark the build as "warning"
(warnOnFailure=True) or as successful (warnOnFailure=False).
Signed-off-by: Baptiste Jonglez <git@bitsofnetworks.org>
Petr Štetiar [Fri, 19 Mar 2021 20:54:30 +0000 (21:54 +0100)]
phase1,phase2: fix reconfig command by creating twistd.pid
Currently it's not possible to reconfig buildmaster as twistd.pid is
missing so fix it by creating one manually.
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Wed, 17 Mar 2021 09:28:39 +0000 (10:28 +0100)]
buildmaster: add some OpenWrt branding
Welcome to OpenWrt buildbot
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Wed, 17 Mar 2021 08:36:10 +0000 (09:36 +0100)]
Bump to latest Buildbot release 3.0.2
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Wed, 17 Mar 2021 08:35:14 +0000 (09:35 +0100)]
Revert "docker: master/worker: fix issue with latest stable sqlalchemy"
This reverts commit
1d17927e74896df84208da090c80c2d1791d5fe6 as the
issue got fixed in Buildbot version 3.0.2
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Tue, 16 Mar 2021 17:21:15 +0000 (18:21 +0100)]
phase1,phase2: implement round robin builds
Gather newest complete and not skipped build timestamps, reverse sort
them and use that as builder priority, so the newest built targets are
at the bottom.
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Mon, 15 Mar 2021 22:39:55 +0000 (23:39 +0100)]
docker: master/worker: fix issue with latest stable sqlalchemy
Fixes following issue with sqlalchemy version 1.4.0 and too relaxed
dependecies `sqlalchemy>=1.2.0` in buildbot:
Traceback (most recent call last):
File "/usr/local/bin/buildbot", line 8, in <module>
sys.exit(run())
from buildbot.db import enginestrategy
...
File "/usr/local/lib/python3.7/dist-packages/buildbot/db/enginestrategy.py", line 154, in <module>
class BuildbotEngineStrategy(strategies.PlainEngineStrategy):
AttributeError: module 'sqlalchemy.engine.strategies' has no attribute 'PlainEngineStrategy'
References: https://github.com/buildbot/buildbot/issues/5911
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Mon, 15 Mar 2021 21:46:09 +0000 (22:46 +0100)]
phase1,phase2: shuffle order of builds
It seems that once there are new commits, buildbot restarts building all
targets in alphabetical order. The current logic pretty much disables
the building of the alphabetically last targets, so until there is
proper solution just workaround it by randomly shuffling the builders
list.
References: https://github.com/buildbot/buildbot/issues/5193
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Mon, 15 Mar 2021 07:34:58 +0000 (08:34 +0100)]
Bump to latest Buildbot release 3.0.1
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Sat, 13 Mar 2021 23:12:41 +0000 (00:12 +0100)]
ci: buildworker: disabled test temporarily
It seems to be getting stuck for some yet to be investigated reason.
References: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3925#note_528657831
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Sat, 13 Mar 2021 22:33:49 +0000 (23:33 +0100)]
ci: use private runner
Shared runners seems to be unreliable lately and private runner should
be more secure as well.
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Sat, 13 Mar 2021 06:51:03 +0000 (07:51 +0100)]
buildworker: add Python2 package
Which is still needed for 19.07 builds.
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Paul Spooren [Sun, 10 Jan 2021 20:25:38 +0000 (10:25 -1000)]
docker,worker: install gcc-mulitlib
From openwrt/docker[1]:
This is required for luajit, which is in packages feed. While it works
fine for ARMv8 (tested by using Github Actions: aarch64_cortex-a53) and
x84_x64 (tested inside Docker container, it does not work for 32-bits
target and because of that we need to install gcc-multilib.
Luajit fails with the following output: (tested against target:
arm_cortex-a9+vfpv3-d16_musl_eabi inside Docker SDK image)
/usr/include/limits.h:26:10: fatal error: bits/libc-header-start.h: No
such file or directory #include <bits/libc-header-start.h>
^~~~~~~~~~~~~~~~~~~~~~~~~~
[1]: https://github.com/openwrt/docker/pull/74/commits/
f9da188d53fd9809ca4911089611e1f42422d049
Signed-off-by: Paul Spooren <mail@aparcar.org>
Petr Štetiar [Fri, 12 Mar 2021 13:57:44 +0000 (14:57 +0100)]
buildworker: add swig package
The U-Boot build for more and more SoCs is using binman by default to
combine the images (SPL, U-Boot, ...). Binman is build from the U-Boot
project and it needs swig to build. We have multiple patches in OpenWrt
to remove this dependency from U-Boot, but it costs more and more time
to revert back to the old code. We have them in sunxi and rockchip
U-Boot, the Mediatek U-Boot build failed some time ago because of
missing swig in build bots.
In U-Boot 2021.01 the binman usage changed again, so it needs further
adaptations.
Building swig in OpenWrt tools will not be easy. We needs swig with
Python bindings and this version needs the Python development headers to
build.
References: https://lists.infradead.org/pipermail/openwrt-devel/2021-February/033654.html
Suggested-by: Hauke Mehrtens <hauke@hauke-m.de>
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Fri, 12 Mar 2021 13:20:46 +0000 (14:20 +0100)]
treewide: use worker term
[Placeholder for your politically correct commit description]
Suggested-by: Paul Spooren <mail@aparcar.org>
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Tue, 9 Mar 2021 19:32:15 +0000 (20:32 +0100)]
Bump to version 3.0.0
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Sat, 6 Mar 2021 10:35:18 +0000 (11:35 +0100)]
ci: buildslave: fix Docker image tests
Refactoring leftover.
Fixes: 70883902ecf1 ("GitLab CI integration")
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Tue, 2 Mar 2021 12:39:19 +0000 (13:39 +0100)]
GitLab CI integration
For improved automation, further QA etc.
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Sat, 6 Mar 2021 05:45:38 +0000 (06:45 +0100)]
docker: buildslave: make TLS working
It seems like twisted actually needs SSL in the connection string,
otherwise it will get stuck in the following endless loop:
[buildbot_worker.pb.BotFactory#info] Starting factory <buildbot_worker.pb.BotFactory object at 0x7ff40f19e080>
[twisted.application.internet.ClientService#info] Scheduling retry 1 to connect <twisted.internet.endpoints._WrapperEndpoint object at 0x7ff410352978> in 2.
1271975799063436 seconds.
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Fri, 5 Mar 2021 12:06:21 +0000 (13:06 +0100)]
builtmaster,phase1: add missing services dict
Fixes following error:
2021-03-05 11:00:54+0000 [-] error while parsing config file:
...
builtins.KeyError: 'services'
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Fri, 5 Mar 2021 11:42:05 +0000 (12:42 +0100)]
buildmaster: replace horizons with JanitorConfigurator
The old c['logHorizon'] way of configuring is not supported anymore, we
should use JanitorConfigurator. A new __Janitor builder will be created
to help keep an eye on the cleanup activities and for that we need
buildbot-worker package as well.
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Fri, 5 Mar 2021 08:14:55 +0000 (09:14 +0100)]
buildmaster: fix permissions
Otherwise buildbot wouldn't be able to access config.ini or certs and thus fail
to start:
2021-03-05 08:08:18+0000 [-] Loading configuration from '/phase1/master.cfg'
2021-03-05 08:08:18+0000 [-] error while parsing config file:
...
configparser.NoSectionError: No section: 'general'
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Tue, 2 Mar 2021 14:13:22 +0000 (15:13 +0100)]
docker: fix Python cryptography build failure
It seems like recent Python cryptography package versions need Rust
compiler present for OpenSSL wrapper, which leads to following error:
generating cffi module 'build/temp.linux-x86_64-3.7/_openssl.c'
running build_rust
...
error: Can not find Rust compiler
There is following note regarding this issue in the documentation:
If you are on RHEL/CentOS/Fedora/Debian/Ubuntu or another distribution
derived from the preceding list, then you should upgrade pip (in a
virtual environment!) and attempt to install cryptography again before
trying to install the Rust toolchain. These platforms will receive a
binary wheel and require no compiler if you have an updated pip!
So fix this by recommended pip update.
References: https://cryptography.io/en/latest/installation.html#rust
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Thu, 14 Jan 2021 17:32:33 +0000 (18:32 +0100)]
phase1,2: fix missing tmp dir
Silence the missing tmp dir warning in the source.git.
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Thu, 14 Jan 2021 16:27:49 +0000 (17:27 +0100)]
docker: make TLS working
Fixes following errors during setup:
buildslave-phase2_1 | Failed to load application: name 'optionsForClientTLS' is not defined
[-] while starting BuildMaster
...
builtins.ModuleNotFoundError: No module named 'OpenSSL'
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Paul Spooren [Sat, 9 Jan 2021 21:41:53 +0000 (11:41 -1000)]
phase1: make change links clickable
Adding a `revlink` allows to click on changes leading to the
git.openwrt.org commit package. This feature works automatically for
pages like GitHub, however requires this manual step for custom urls.
Signed-off-by: Paul Spooren <mail@aparcar.org>
Paul Spooren [Sun, 10 Jan 2021 20:27:58 +0000 (10:27 -1000)]
docker,worker: install qemu-utils
To convert x86 images to VDI/VMDK the `qemu-img` tools is required,
which is part of Debians `qemu-utils` package. The goal is to remove the
ancient in-tree `qemu-img` version and replace it with a recent host
installed package.
Related: https://github.com/openwrt/openwrt/pull/3739/
Signed-off-by: Paul Spooren <mail@aparcar.org>
[rebased]
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Paul Spooren [Sat, 9 Jan 2021 21:24:15 +0000 (11:24 -1000)]
scripts: remove migrated dumpinfo.pl script
The script was moved over to openwrt.git and is called
`dump-target-info.pl`
Signed-off-by: Paul Spooren <mail@aparcar.org>
[fixed script path in the master build script]
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Thu, 14 Jan 2021 15:22:17 +0000 (16:22 +0100)]
docker: alphabetical sorting of packages
So it's easier to follow.
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Fri, 8 Jan 2021 16:47:12 +0000 (17:47 +0100)]
docker: update to 2.10.1
Removing the mixing with the testing as it seems, that it's not needed
anymore. We can easily just use buildbot from pip.
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Tue, 9 Feb 2021 21:21:15 +0000 (22:21 +0100)]
phase1: fix gitcheckout with corrupted .git
When the container running `git clone` is killed it will leave the .git
directory in unusable state which then leads in the following errors:
if [ -d .git ]; then git checkout -f master; git branch --set-upstream-to origin/master; else exit 0; fi
error: pathspec 'master' did not match any file(s) known to git.
fatal: branch 'master' does not exist
Fix it by removing .git completely if this happens, otherwise it might
simply leave shared build directory in the unusable state.
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Petr Štetiar [Tue, 8 Sep 2020 06:47:34 +0000 (08:47 +0200)]
phase1: dont include stderr output in stageroot variable
Otherwise it can result in strange error messages which are hard to
debug.
Ref: http://lists.openwrt.org/pipermail/openwrt-devel/2020-August/030826.html
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Paul Spooren [Fri, 13 Nov 2020 07:06:03 +0000 (21:06 -1000)]
phase2: use full git history for reproducibility
The SOURCE_DATE_EPOCH variable is used to set reproducible time stamps
for built artifact. As packages get rebuild without changing, they
require individual epochs which are independent of the main tree. To
archive that the git log for each package source path is used.
This mechanism can only work if the full git history is available. Until
now only a shallow copy (--depth 1) would be cloned. With this commit
the `feeds.conf.default` is changed to use the full git repository.
Signed-off-by: Paul Spooren <mail@aparcar.org>
[do not modify feeds.conf.default]
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Fri, 13 Nov 2020 14:03:45 +0000 (15:03 +0100)]
phase1: add separate option for kmod repo embedding
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Sat, 7 Nov 2020 19:31:36 +0000 (20:31 +0100)]
phase1: treat "kmod_archive" as bool option
Fixes: 9ad60fa ("phase1: add config option to enable/disable kmod archive")
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Thibaut VARÈNE [Fri, 28 Feb 2020 23:59:45 +0000 (00:59 +0100)]
rsync.sh: set pipefail
to capture rsync failures when piped through pv
Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
Jo-Philipp Wich [Sat, 7 Nov 2020 19:11:13 +0000 (20:11 +0100)]
phase1: add config option to enable/disable kmod archive
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Paul Spooren [Tue, 30 Jun 2020 23:28:02 +0000 (13:28 -1000)]
phase1: add JSON merge step
The refactored JSON info files require a final merge step which sums up
all created JSON info files of a target into a single `profiles.json`
files.
This patch adds the extra step to run `make json_overview_image_info`
just before calculating the checksums so the `profiles.json` files is
signed as well.
Signed-off-by: Paul Spooren <mail@aparcar.org>
[allow the step to fail to satisfy 18.06 builders]
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Sat, 11 Apr 2020 22:16:26 +0000 (00:16 +0200)]
buildslave: update to Debian 10, introduce TLS support
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Sat, 11 Apr 2020 22:15:43 +0000 (00:15 +0200)]
docker: add example certificates
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Sat, 11 Apr 2020 22:15:26 +0000 (00:15 +0200)]
buildmaster: update to v2.4.1
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Mon, 3 Feb 2020 15:57:12 +0000 (16:57 +0100)]
phase1, phase2: sort and move imports to the top of the file
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Mon, 3 Feb 2020 15:50:47 +0000 (16:50 +0100)]
scripts: cleanup.sh: adapt to buildbot v2 api
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Mon, 3 Feb 2020 15:50:06 +0000 (16:50 +0100)]
phase1, phase2: implement force scheduler logic
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Sun, 2 Feb 2020 18:22:41 +0000 (19:22 +0100)]
phase1, phase2: update master to Debian 10 / Buildbot 2.0.1
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Paul Spooren [Thu, 14 Nov 2019 07:55:30 +0000 (21:55 -1000)]
scripts: signall.sh store usign.key as usign.sec
The signall.sh script signs all files inside a tar via GPG and
signify-openbsd (similar to usign)e and attaches the signatures to the
same archive.
Using more recent versions of signify-openbsd requires a specific naming
schema for keys, private ending with .sec and public with .pub. This was
introduced at
763e1148f68f03cb2fa85d022500acf8c66af222[0].
This patch renames the stored key as usign.sec instead of usign.key. As
of the temporary nature of the key storing, this very unlikely breaks
any existing setups.
[0]: https://github.com/openbsd/src/commit/
763e1148f68f03cb2fa85d022500acf8c66af222
Signed-off-by: Paul Spooren <mail@aparcar.org>
[adjusted commit subject]
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Fri, 18 Oct 2019 14:39:11 +0000 (16:39 +0200)]
phase1: make buildinfo failures "green"
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Fri, 18 Oct 2019 14:16:30 +0000 (16:16 +0200)]
phase1: allow buildinfo target to fail
Not all observed branches provide this target.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Fri, 18 Oct 2019 06:53:41 +0000 (08:53 +0200)]
add helper script to (re)build docker images
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Paul Spooren [Sun, 13 Oct 2019 21:00:39 +0000 (11:00 -1000)]
phase1: use buildinfo instead of prepare
Prepare target is used to compile tools and toolchain, however buildbot
only uses it to create the buildinfo files. This[0] PR splits prepare
and buildinfo allowing the buildbots to run the buildinfo creation only.
This should also fix the current issue with missing json info files:
prepare runs `target/compile` which removes all existing json info files
via `Device/Build`. Only `target/install` creates them via
`Device/Build/image`. However currently the buildbots run
`target/install` before (the second) `target/compile`.
[0]: https://github.com/openwrt/openwrt/pull/2492
Signed-off-by: Paul Spooren <mail@aparcar.org>
Paul Spooren [Tue, 3 Sep 2019 09:59:29 +0000 (23:59 -1000)]
phase1: run prepare instead of diffconfig
This will not only create config.buildinfo as before, but also
version.buildinfo and feeds.buidinfo.
CC: Jo-Philipp Wich <jo@mein.io>
Signed-off-by: Paul Spooren <mail@aparcar.org>
[rename step from "prepare" to "buildinfo"]
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Thu, 18 Jul 2019 13:40:11 +0000 (15:40 +0200)]
phase2: fix copy-paste error in previous commit
Fixes: c17b550 ("phase2: sanitize SDK host command symlinks")
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Thu, 18 Jul 2019 13:30:31 +0000 (15:30 +0200)]
phase2: sanitize SDK host command symlinks
The SDK archive might contain symlinks like
staging_dir/host/bin/gcc -> /builder/arc770_generic/ccache_cc.sh
which may result in host compile failures when a slave has been used
for both phase1 and phase2 builds.
Add a new step which purges such links when running the SDK until
buildroot is eventually fixed to not put suhc links into the archive
in the first place.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Thu, 18 Jul 2019 13:02:11 +0000 (15:02 +0200)]
scripts: cleanup.sh: skip directory symlinks
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Thu, 18 Jul 2019 12:56:47 +0000 (14:56 +0200)]
rsync: remove pid file if it already exists
If the rsync container was forcibly shut down, it may still contain
a stale pid file entry.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Thu, 18 Jul 2019 12:42:14 +0000 (14:42 +0200)]
phase2: implement work dir sharing
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Thu, 18 Jul 2019 12:29:43 +0000 (14:29 +0200)]
treewide: merge phase1 and phase2 cleanup procedures
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Wed, 17 Jul 2019 06:13:24 +0000 (08:13 +0200)]
scripts: signall.sh: use absolute paths when invoking gpg from find
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Tue, 16 Jul 2019 19:28:31 +0000 (21:28 +0200)]
phase2: disable slave side package signing
Since we do not expose secret using keys to buildslaves anymore,
we cannot sign the package indexes on the slave side either.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Tue, 16 Jul 2019 17:18:08 +0000 (19:18 +0200)]
phase1: enable shared work directory by default
Enable work directory sharing for any slave having a concurrent build limit
of one job.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Tue, 16 Jul 2019 17:14:08 +0000 (19:14 +0200)]
phase1: forcibly checkout branch
Pass the -f flag to git checkout in order to ensure that any potentially
existing local changes are overwritten.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Tue, 16 Jul 2019 17:12:29 +0000 (19:12 +0200)]
phase1: automatically setup shared work directory symlink
Until now, the shared work directory symlinks had to be provisioned
manually, this commits automates the setup when work directory sharing
is enabled for the slave.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Mon, 15 Jul 2019 08:36:15 +0000 (10:36 +0200)]
phase1: checkout configured branch
Also fixup the remote since the Git() step does not properly set
it in all cases.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Mon, 15 Jul 2019 06:08:21 +0000 (08:08 +0200)]
docker: buildslave: properly recreate slave if state dir already exists
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Wed, 3 Jul 2019 06:37:49 +0000 (08:37 +0200)]
docker: merge phase1 and phase2 configs into one file
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Wed, 3 Jul 2019 06:36:27 +0000 (08:36 +0200)]
phase2: move phase2 specific options into separate section
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Wed, 3 Jul 2019 06:36:12 +0000 (08:36 +0200)]
phase1: move phase1 specific options into separate section
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Tue, 2 Jul 2019 05:19:05 +0000 (07:19 +0200)]
docker: containerize master and slaves
Introduce docker builds for the buildmaster and slaves, and add
a docker-compose reference setup for quickly spinning up an
example build cluster.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Tue, 2 Jul 2019 16:25:00 +0000 (18:25 +0200)]
scripts: sha2pl: handle usign signatures as well
- Also consider sha256sums.sig and Packages.sig files
- Avoid operation on closed handle when remote list does not exist
- Only add existing additional files to rsync list
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Tue, 2 Jul 2019 10:49:00 +0000 (12:49 +0200)]
phase1: use --contimeout parameter only for rsync daemon connections
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Tue, 2 Jul 2019 09:13:11 +0000 (11:13 +0200)]
treewide: rework signall.sh to read keys from config directly
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Mon, 1 Jul 2019 16:38:18 +0000 (18:38 +0200)]
phase1: prevent ccache/gcc exec recursion when cc_version is unset
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Sun, 30 Jun 2019 19:24:22 +0000 (21:24 +0200)]
phase2: allow overriding config.ini location with env var
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Sun, 30 Jun 2019 19:24:05 +0000 (21:24 +0200)]
phase1: allow overriding config.ini location with env var
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Sun, 30 Jun 2019 18:28:57 +0000 (20:28 +0200)]
phase1: move config.seed into config.ini
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Sun, 30 Jun 2019 18:24:38 +0000 (20:24 +0200)]
phase2: move cleanup.sh to shared script directory
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Sun, 30 Jun 2019 18:24:23 +0000 (20:24 +0200)]
phase1: move cleanup.sh to shared script directory
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Jo-Philipp Wich [Sun, 30 Jun 2019 17:51:57 +0000 (19:51 +0200)]
treewide: rename home_dir to work_dir
Also default to current directory.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>