buildbot.git
5 weeks agobuildmaster: add static apk utility ynezz/add-static-apk
Petr Štetiar [Tue, 24 Sep 2024 20:00:59 +0000 (20:00 +0000)]
buildmaster: add static apk utility

We need a relatively recent `apk` that can sign package indexes. You can
find more details in the referenced issue below. However, it has not
been released yet and is not available as a prebuilt download.
Therefore, until it becomes available, let's build a static version and
use that in the meantime.

Fixes: a94d4e15fdc1 ("add APK signing logic")
References: https://gitlab.alpinelinux.org/alpine/apk-tools/-/issues/11008
Signed-off-by: Petr Štetiar <ynezz@true.cz>
2 months agodocker: bump rsync Debian version v16
Paul Spooren [Wed, 7 Aug 2024 14:49:50 +0000 (16:49 +0200)]
docker: bump rsync Debian version

Signed-off-by: Paul Spooren <mail@aparcar.org>
2 months agoadd APK signing logic
Paul Spooren [Tue, 6 Aug 2024 16:03:21 +0000 (18:03 +0200)]
add APK signing logic

With this commit it's possible to sign APK package indexes
(packages.adb) via the `signall.sh` script, which is run on the
buildmaster. As a consequence `apk` must be available on the
buildmaster. This is the final step to replace OPKG with APK.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2 months agophase1: make IsNoMasterBuild function more flexible
Paul Spooren [Thu, 13 Jun 2024 07:56:38 +0000 (09:56 +0200)]
phase1: make IsNoMasterBuild function more flexible

Work with both master/main

Signed-off-by: Paul Spooren <mail@aparcar.org>
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
2 months agoCI(deps): Bump docker/build-push-action from 5 to 6
dependabot[bot] [Mon, 17 Jun 2024 19:41:44 +0000 (19:41 +0000)]
CI(deps): Bump docker/build-push-action from 5 to 6

Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 5 to 6.
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](https://github.com/docker/build-push-action/compare/v5...v6)

---
updated-dependencies:
- dependency-name: docker/build-push-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2 months agophase1: kmodprepare: add apk compat
Petr Štetiar [Thu, 13 Jun 2024 13:52:17 +0000 (13:52 +0000)]
phase1: kmodprepare: add apk compat

Ansuel reported, that with `USE_APK=y` there are no kmods available in
the build artifacts, so lets fix it by extending the rsync's include
pattern with the `apk` suffix to make it future proof.

Reported-by: Christian Marangi <ansuelsmth@gmail.com>
Signed-off-by: Petr Štetiar <ynezz@true.cz>
2 months agophase1: AnyBranchScheduler: make treeStableTimer configurable
Petr Štetiar [Wed, 12 Jun 2024 18:43:29 +0000 (18:43 +0000)]
phase1: AnyBranchScheduler: make treeStableTimer configurable

The AnyBranchScheduler will currently wait for 15 minutes before
starting a build.  If new changes are made on the same branch during
this interval, the timer will be restarted.

For staging repository we want faster feedback loop, so we're going to
use much lower value, thus lets make this configurable and handled via
Ansible.

Signed-off-by: Petr Štetiar <ynezz@true.cz>
5 months agophase1: make list of builders configurable
Petr Štetiar [Sat, 1 Jun 2024 11:57:11 +0000 (11:57 +0000)]
phase1: make list of builders configurable

Currently we always populate builders (build targets) from the Git tree,
but for staging environment this is usually overkill as we might need
just a bunch of targets, so lets make this configurable.

Signed-off-by: Petr Štetiar <ynezz@true.cz>
5 months agophase1: factor out populateTargetsForBranch
Petr Štetiar [Sat, 1 Jun 2024 11:21:23 +0000 (11:21 +0000)]
phase1: factor out populateTargetsForBranch

Going to make builders (build targets) configurable, so lets factor
current populateTargets into separate function populateTargetsForBranch
which takes a branch as argument. No functional changes.

Signed-off-by: Petr Štetiar <ynezz@true.cz>
5 months agoRevert "Bump buildbot to v3.11.3 release" v15
Petr Štetiar [Sat, 1 Jun 2024 05:54:06 +0000 (05:54 +0000)]
Revert "Bump buildbot to v3.11.3 release"

This reverts commit 44ed9a39fddc4e79ec1276c3e5923766cbe19878 as it
breaks GitPoller.

References: https://github.com/buildbot/buildbot/issues/7663
Signed-off-by: Petr Štetiar <ynezz@true.cz>
5 months agoBump buildbot to v3.11.3 release v14
Petr Štetiar [Wed, 22 May 2024 10:05:50 +0000 (10:05 +0000)]
Bump buildbot to v3.11.3 release

https://github.com/buildbot/buildbot/releases/tag/v3.11.3

Signed-off-by: Petr Štetiar <ynezz@true.cz>
6 months agoBump buildbot to v3.11.1 release v13
Petr Štetiar [Fri, 22 Dec 2023 10:06:54 +0000 (10:06 +0000)]
Bump buildbot to v3.11.1 release

https://github.com/buildbot/buildbot/releases/tag/v3.11.1

Signed-off-by: Petr Štetiar <ynezz@true.cz>
6 months agoRevert "buildmaster: fix Twisted dependency hell by using twisted==22.10.0"
Petr Štetiar [Fri, 22 Dec 2023 11:02:49 +0000 (11:02 +0000)]
Revert "buildmaster: fix Twisted dependency hell by using twisted==22.10.0"

This reverts commit bc17dd9af0d3a56e3e9cd724c41deacc5ae0de14 as it
should be fine in buildbot v3.10.0 release.

Signed-off-by: Petr Štetiar <ynezz@true.cz>
6 months agophase1,phase2: s/master/main for phase{1,2}
Paul Spooren [Tue, 23 May 2023 21:02:48 +0000 (23:02 +0200)]
phase1,phase2: s/master/main for phase{1,2}

More and more projects are switching their repositories to use the
'main' branch instead of the 'master' branch. This also includes many
Linux upstream trees as well. Some trees are even removing their
'master' branches already.

I think this is becoming more and more mainstream and expected of
projects, so we should do the same.

References: https://openwrt.org/voting/2023-02-27
Signed-off-by: Paul Spooren <mail@aparcar.org>
Signed-off-by: Petr Štetiar <ynezz@true.cz> [commit facelift]
6 months agophase2: fix relative paths for scripts and files
Petr Štetiar [Sun, 24 Dec 2023 19:14:14 +0000 (19:14 +0000)]
phase2: fix relative paths for scripts and files

Currently `buildlist` step fails with following:

  ../../../sha2rsync.pl ../../arch-sha256sums bin/packages/aarch64_generic/sha256sums rsynclist
   in dir /builder/aarch64_generic/build/sdk (timeout 1200 secs)
   watching logfiles {}
   argv: [b'../../../sha2rsync.pl', b'../../arch-sha256sums', b'bin/packages/aarch64_generic/sha256sums', b'rsynclist']
   environment:
    ...
    PWD=/builder/aarch64_generic/build/sdk
    ...
  Upon execvpe b'../../../sha2rsync.pl' [b'../../../sha2rsync.pl', b'../../arch-sha256sums', b'bin/packages/aarch64_generic/sha256sums', b'rsynclist'] in environment id 139847367136832
  :Traceback (most recent call last):
  ...
  FileNotFoundError: [Errno 2] No such file or directory: b'../../../sha2rsync.pl'

due to relative paths being off by one:

 worker work dir = /builder/aarch64_cortex-a72/build
 workerdest = "../sha2rsync.pl"
   is absolute path /builder/aarch64_cortex-a72/sha2rsync.pl

thus relative path from:

  FileNotFoundError: [Errno 2] No such file or directory: b'../../../sha2rsync.pl'

in following context:

  PWD=/builder/aarch64_generic/build/sdk
  b'../../../sha2rsync.pl'

is wrong absolute path `/builder/sha2rsync.pl` by one directory level,
thus adjust all those paths accordingly.

Fixes: c3ddb0db167d ("phase2: use sha2rsync.pl for 'targetupload'")
Signed-off-by: Petr Štetiar <ynezz@true.cz>
6 months agorequirements-dev: use black 23.12.1 and ruff 0.1.9
Petr Štetiar [Sat, 23 Dec 2023 06:43:02 +0000 (06:43 +0000)]
requirements-dev: use black 23.12.1 and ruff 0.1.9

Fix ruff's recommendation:

 phase2/master.cfg:328:16: E721 Do not compare types, use `isinstance()`

 "Unlike a direct type comparison, isinstance will also check if an
  object is an instance of a class or a subclass thereof."

Signed-off-by: Petr Štetiar <ynezz@true.cz>
6 months agophase2: fix invalid escape sequence flake8 warning in checksums step
Petr Štetiar [Sat, 23 Dec 2023 07:33:21 +0000 (07:33 +0000)]
phase2: fix invalid escape sequence flake8 warning in checksums step

Resolved a flake8 linting warning related to an invalid escape sequence
in the ShellCommand for calculating checksums:

 phase2/master.cfg:739:28: W605 invalid escape sequence '\('
 phase2/master.cfg:739:32: W605 invalid escape sequence '\)'
 phase2/master.cfg:739:35: W605 invalid escape sequence '\('
 phase2/master.cfg:739:39: W605 invalid escape sequence '\)'

The warning was caused by the use of unescaped parentheses in a regular
expression within a sed command.

Use a raw string (with an 'r' prefix) to treat backslashes as literal
characters, ensuring that the regular expression is correctly
interpreted and flake8 does not raise a warning.

This fix ensures that the code adheres to Python's string handling best
practices and maintains the integrity of the regular expression
functionality.

Fixes: f0faed2970dd ("phase2: compute checksums")
Signed-off-by: Petr Štetiar <ynezz@true.cz>
6 months agophase2: checksums: split long command line
Petr Štetiar [Sun, 24 Dec 2023 09:30:58 +0000 (09:30 +0000)]
phase2: checksums: split long command line

To make it more readable, diffs smaller, easier to review etc.

Signed-off-by: Petr Štetiar <ynezz@true.cz>
6 months agoAdd linting with flake8 tool
Petr Štetiar [Sat, 23 Dec 2023 06:13:02 +0000 (06:13 +0000)]
Add linting with flake8 tool

flake8 is a Python tool that glues together pycodestyle, pyflakes,
mccabe, and third-party plugins to check the style and quality of some
Python code.

Currently we've issue in phase2 in checksum step:

 /bin/sh: 2: Syntax error: ")" unexpected

And it seems, that flake8 is able to spot places which might lead to
such issues during runtime:

 phase2/master.cfg:733:151: W605 invalid escape sequence '\('
 phase2/master.cfg:733:155: W605 invalid escape sequence '\)'
 phase2/master.cfg:733:158: W605 invalid escape sequence '\('
 phase2/master.cfg:733:162: W605 invalid escape sequence '\)'

So lets enable flake8 checking on the CI so we can spot similar places
in the future and address them before deployment.

We dont want to make current ongoing work on phase2 code harder, thus we
don't touch that part yet, so we whitelist most of the checks for now.

References: f0faed2970dd ("phase2: compute checksums")
Signed-off-by: Petr Štetiar <ynezz@true.cz>
6 months agophase2: fix stray closing parenthesis
Petr Štetiar [Sat, 23 Dec 2023 07:14:20 +0000 (07:14 +0000)]
phase2: fix stray closing parenthesis

Due to the stray closing parenthesis introduced in commit f0faed2970dd
("phase2: compute checksums") we have currently we've failing checksum
step in phase2:

 argv: b'cd bin/packages/mipsel_24kc_24kf; find . -type f -not -name \'sha256sums\' -printf "%P\n" | sort | xargs -r ../../../staging_dir/host/bin/mkhash -n sha256 | sed -ne \'s!^\\(.*\\) \\(.*\\)$!\x01 *\x02!p\' > sha256sums)'
 environment:
  ...snip...
  using PTY: False
 /bin/sh: 2: Syntax error: ")" unexpected
 program finished with exit code 2

Fixes: f0faed2970dd ("phase2: compute checksums")
Reported-by: Hannu Nyman <hannu.nyman@iki.fi>
Signed-off-by: Petr Štetiar <ynezz@true.cz>
6 months agophase2: dlsha2rsyncpl: fix sha2rsync.pl script location
Petr Štetiar [Sun, 24 Dec 2023 08:27:03 +0000 (08:27 +0000)]
phase2: dlsha2rsyncpl: fix sha2pl script location

Fixes following error:

 File 'sha2rsync.pl' not available at master

Fixes: c3ddb0db167d ("phase2: use sha2rsync.pl for 'targetupload'")
Signed-off-by: Petr Štetiar <ynezz@true.cz>
6 months agophase2: update default sdk pattern to a more relaxed pattern
Christian Marangi [Fri, 12 Apr 2024 13:09:45 +0000 (15:09 +0200)]
phase2: update default sdk pattern to a more relaxed pattern

Update default sdk pattern to a more relaxed pattern to make it future
proof if in the future the compression extension will change again.

Needed currently with the migration from .xz to .zst.

Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
6 months agoCI(deps): Bump docker/login-action from 2 to 3
dependabot[bot] [Fri, 12 Apr 2024 13:18:15 +0000 (13:18 +0000)]
CI(deps): Bump docker/login-action from 2 to 3

Bumps [docker/login-action](https://github.com/docker/login-action) from 2 to 3.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/v2...v3)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
6 months agoCI(deps): Bump docker/build-push-action from 4 to 5
dependabot[bot] [Fri, 12 Apr 2024 13:18:13 +0000 (13:18 +0000)]
CI(deps): Bump docker/build-push-action from 4 to 5

Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 4 to 5.
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](https://github.com/docker/build-push-action/compare/v4...v5)

---
updated-dependencies:
- dependency-name: docker/build-push-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
6 months agoCI(deps): Bump docker/metadata-action from 4 to 5
dependabot[bot] [Fri, 12 Apr 2024 13:18:12 +0000 (13:18 +0000)]
CI(deps): Bump docker/metadata-action from 4 to 5

Bumps [docker/metadata-action](https://github.com/docker/metadata-action) from 4 to 5.
- [Release notes](https://github.com/docker/metadata-action/releases)
- [Upgrade guide](https://github.com/docker/metadata-action/blob/master/UPGRADE.md)
- [Commits](https://github.com/docker/metadata-action/compare/v4...v5)

---
updated-dependencies:
- dependency-name: docker/metadata-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
6 months agoCI(deps): Bump actions/setup-python from 4 to 5
dependabot[bot] [Fri, 12 Apr 2024 13:18:09 +0000 (13:18 +0000)]
CI(deps): Bump actions/setup-python from 4 to 5

Bumps [actions/setup-python](https://github.com/actions/setup-python) from 4 to 5.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](https://github.com/actions/setup-python/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
6 months agoCI(deps): Bump actions/checkout from 3 to 4
dependabot[bot] [Fri, 12 Apr 2024 13:18:06 +0000 (13:18 +0000)]
CI(deps): Bump actions/checkout from 3 to 4

Bumps [actions/checkout](https://github.com/actions/checkout) from 3 to 4.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
6 months agoCI: add dependabot scan
Christian Marangi [Fri, 12 Apr 2024 13:16:53 +0000 (15:16 +0200)]
CI: add dependabot scan

Add dependabot scan to warn and propose updates to our github actions.

Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
10 months agophase2: max_builds is always set to 1 v12
Thibaut VARÈNE [Fri, 22 Dec 2023 13:32:02 +0000 (14:32 +0100)]
phase2: max_builds is always set to 1

Align with phase1 (ac0d815)

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
10 months agobuildmaster: fix Twisted dependency hell by using twisted==22.10.0 v11
Petr Štetiar [Fri, 22 Dec 2023 10:35:23 +0000 (10:35 +0000)]
buildmaster: fix Twisted dependency hell by using twisted==22.10.0

Force twisted==22.10.0 which fixes following buildbot master startup
issue:

Unhandled error in Deferred:

Traceback (most recent call last):
  File "/.../site-packages/buildbot/scripts/create_master.py", line 84, in createDB
    master = BuildMaster(config['basedir'])
  File "/.../site-packages/buildbot/master.py", line 102, in __init__
    self._services_d = self.create_child_services()
  File "/.../site-packages/twisted/internet/defer.py", line 2245, in unwindGenerator
    return _cancellableInlineCallbacks(gen)
  File "/.../site-packages/twisted/internet/defer.py", line 2157, in _cancellableInlineCallbacks
    _inlineCallbacks(None, gen, status, _copy_context())
--- <exception caught here> ---
  File "/.../site-packages/twisted/internet/defer.py", line 1997, in _inlineCallbacks
    result = context.run(gen.send, result)
  File "/.../site-packages/buildbot/master.py", line 188, in create_child_services
    self.www = wwwservice.WWWService()
  File "/.../site-packages/buildbot/www/service.py", line 196, in __init__
    self.apps = get_plugins('www', None, load_now=True)
  File "/.../site-packages/buildbot/plugins/db.py", line 356, in get_plugins
    return _DB.add_namespace(namespace, interface, check_extras, load_now)
  File "/.../site-packages/buildbot/plugins/db.py", line 306, in add_namespace
    tempo.load()
  File "/.../site-packages/buildbot/plugins/db.py", line 242, in load
    self._tree.load()
  File "/.../site-packages/buildbot/plugins/db.py", line 112, in load
    child.load()
  File "/.../site-packages/buildbot/plugins/db.py", line 45, in load
    self._value = self._loader(self._entry)
  File "/.../site-packages/buildbot/plugins/db.py", line 214, in _load_entry
    raise PluginDBError('Requirements are not satisfied '
buildbot.errors.PluginDBError: Requirements are not satisfied for buildbot.www:base:
The 'zope-interface>=5' distribution was not found and is required by Twisted

This commit should be reverted once we bump to buildbot >= 3.10 which
has this workaround integrated.

References: https://github.com/buildbot/buildbot/commit/94e2d59c23472f3fe640437630309bea518c5b9e
Signed-off-by: Petr Štetiar <ynezz@true.cz>
10 months agoci: improve QA with cram based tests
Petr Štetiar [Fri, 22 Dec 2023 09:46:01 +0000 (09:46 +0000)]
ci: improve QA with cram based tests

Currently we've broken master container but we're not aware about it as
current tests are very lame, so lets improve it a bit with some more
reliable, extensible solution.

Signed-off-by: Petr Štetiar <ynezz@true.cz>
10 months agobuildmaster: entry.sh: fix errors by checking for dir existence
Petr Štetiar [Fri, 22 Dec 2023 09:28:40 +0000 (09:28 +0000)]
buildmaster: entry.sh: fix errors by checking for dir existence

Get rid of following annyoing errors during testing:

  chown: cannot access '/config': No such file or directory
  chown: cannot access '/certs': No such file or directory
  chmod: cannot access '/config': No such file or directory
  chmod: cannot access '/certs': No such file or directory

As those dirs doesn't exist in the container, they're being provided as
volumes during deployment.

Signed-off-by: Petr Štetiar <ynezz@true.cz>
11 months agophase2: remove unused git_ssh plumbing v10
Thibaut VARÈNE [Wed, 15 Nov 2023 14:12:17 +0000 (15:12 +0100)]
phase2: remove unused git_ssh plumbing

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
11 months agoscripts: remove unused expire.sh
Thibaut VARÈNE [Wed, 15 Nov 2023 12:45:18 +0000 (13:45 +0100)]
scripts: remove unused expire.sh

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
11 months agophase2: remove unused tree_expire option and steps
Thibaut VARÈNE [Wed, 15 Nov 2023 12:44:42 +0000 (13:44 +0100)]
phase2: remove unused tree_expire option and steps

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
11 months agophase2: rsync: use --size-only instead of --checksum for sourceupload
Thibaut VARÈNE [Wed, 15 Nov 2023 11:21:41 +0000 (12:21 +0100)]
phase2: rsync: use --size-only instead of --checksum for sourceupload

Align with phase1 (62a01a1)

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
11 months agophase2: use sha2rsync.pl for 'targetupload'
Thibaut VARÈNE [Wed, 15 Nov 2023 10:50:01 +0000 (11:50 +0100)]
phase2: use sha2rsync.pl for 'targetupload'

Align with phase1 (3246628)

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
11 months agophase2: compute checksums
Thibaut VARÈNE [Tue, 14 Nov 2023 17:58:43 +0000 (18:58 +0100)]
phase2: compute checksums

This will be necessary to get rid of 'rsync --checksum' and use
sha2rsync.pl instead, as on phase1, thereby easing the load on the rsync
server.

This uses the same construct as "make checksum" in the main repo.

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
11 months agophase2: reduce verbosity of rsync and use rsync.sh helper
Thibaut VARÈNE [Wed, 15 Nov 2023 10:50:01 +0000 (11:50 +0100)]
phase2: reduce verbosity of rsync and use sh helper

Align with phase1 (e55b76f)

This patch:
- removes '--progress' rsync parameter
- uses the wrapper script 'rsync.sh'

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
11 months agophase2: don't enable rsync compression where unnecessary
Thibaut VARÈNE [Wed, 15 Nov 2023 09:35:44 +0000 (10:35 +0100)]
phase2: don't enable rsync compression where unnecessary

Align with phase1 (cf7b9ba)

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
11 months agophase2: regroup common rsync options and add timeout
Thibaut VARÈNE [Tue, 14 Nov 2023 11:23:11 +0000 (12:23 +0100)]
phase2: regroup common rsync options and add timeout

Align with phase1 (3deb6c0)

"-4 -v --timeout=120"

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
11 months agophase2: report ccache stats at end of build
Thibaut VARÈNE [Tue, 14 Nov 2023 10:20:34 +0000 (11:20 +0100)]
phase2: report ccache stats at end of build

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
11 months agoRevert "phase2: use full git history for reproducibility"
Thibaut VARÈNE [Mon, 13 Nov 2023 16:32:34 +0000 (17:32 +0100)]
Revert "phase2: use full git history for reproducibility"

Following openwrt/11bb5337b8d8b5018e48f0df415efb99e2f49d0d we no longer
need the full git history.

This reverts commit 5b96616d056e26adbd50cc73a5e51b8449110b7d.

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
11 months agophase2: use Interpolate instead of WithProperties
Thibaut VARÈNE [Sat, 11 Nov 2023 16:26:41 +0000 (17:26 +0100)]
phase2: use Interpolate instead of WithProperties

Align with phase1 (d1a92ba)

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
11 months agophase2: s/SetProperty/SetPropertyFromCommand/
Thibaut VARÈNE [Sat, 11 Nov 2023 15:28:07 +0000 (16:28 +0100)]
phase2: s/SetProperty/SetPropertyFromCommand/

Align with phase1 (3141c97)

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
11 months agophase2: do not exceed nproc build concurrency
Thibaut VARÈNE [Sat, 11 Nov 2023 15:12:37 +0000 (16:12 +0100)]
phase2: do not exceed nproc build concurrency

Align with phase1 (2ad0478)

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
11 months agophase2: remove unused 'other_builds' property
Thibaut VARÈNE [Sat, 11 Nov 2023 15:11:25 +0000 (16:11 +0100)]
phase2: remove unused 'other_builds' property

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
11 months agophase1: treat all branches equally for building
Thibaut VARÈNE [Tue, 14 Nov 2023 13:23:50 +0000 (14:23 +0100)]
phase1: treat all branches equally for building

Following discussion here:
https://lists.openwrt.org/pipermail/openwrt-devel/2023-November/041769.html

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
11 months agophase1: raise priority of tag builds
Thibaut VARÈNE [Mon, 13 Nov 2023 14:23:49 +0000 (15:23 +0100)]
phase1: raise priority of tag builds

Currently the buildmaster would only order tag builds within their own
branch, meaning that if e.g. a higher priority branch has normal
buildrequests (i.e. buildrequests comming from the AnyBranchScheduler),
and a lower priority branch has "tag" buildrequests (i.e. from the
Triggerable scheduler), the former builderequests would be served first,
deferring the build of e.g. release tags.

We want forced builds (release tag) to have maximum priority, regardless
of branch priority. This commit attempts to solve this problem by
leveraging the newly (as of buildbot 3.9.0) introduced "priority"
scheduler parameter, by raising the Triggerable scheduler buildrequests
priority, and then considering this higher priority in Builders' order.

The net result is that Builders are now prioritized if they have
pending higher priority buildrequest, still preserving the branch order.
In other words, tag requests are front run while preserving branch order,
meaning that if two branches have tag buildrequests, the higher priority
branch is still served first.

This requires buildbot 3.9.0

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
11 months agoci: split container push steps to separate job and add deploy tag
Christian Marangi [Tue, 14 Nov 2023 13:53:12 +0000 (14:53 +0100)]
ci: split container push steps to separate job and add deploy tag

Split container push related steps to separate jobs and add deploy tag.

This is to better organize the workflow and drop additional checks for
single steps moving them to the single job.
Also we use a feature of github to better track changes deployed to our
buildbot.

Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
11 months agoci: generalize container test step
Christian Marangi [Tue, 14 Nov 2023 13:49:37 +0000 (14:49 +0100)]
ci: generalize container test step

Generalize container test step by using include feature of matrix
strategy and defining additional values for container command test and
config verification.

Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
11 months agoci: move git short sha length to ENV
Christian Marangi [Tue, 14 Nov 2023 13:47:26 +0000 (14:47 +0100)]
ci: move git short sha length to ENV

Move git short sha length to ENV to make it easier to configure in the
future if needed.

Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
11 months agodocker,worker: install zstd
Paul Spooren [Tue, 14 Nov 2023 11:27:08 +0000 (12:27 +0100)]
docker,worker: install zstd

Faster compression with partly better rates. Should be used eventually
for ImageBuilders etc.

Signed-off-by: Paul Spooren <mail@aparcar.org>
15 months agodocker,worker: install pyelftools v9
Ben Whitten [Thu, 27 Jul 2023 20:25:56 +0000 (21:25 +0100)]
docker,worker: install pyelftools

UBoot version of at least 2023.07.02 that use binman also have a dependency of pyelftools on the build host.

Signed-off-by: Ben Whitten <BWhitten@users.noreply.github.com>
Signed-off-by: Petr Štetiar <ynezz@true.cz> [whitespace fix]
15 months agophase1: workaround dlprune recursive directory removal v8
Petr Štetiar [Sat, 22 Jul 2023 08:53:45 +0000 (10:53 +0200)]
phase1: workaround dlprune recursive directory removal

Workarounds following issue:

 find: cannot delete ‘dl/ath10k-ct-firmware-2020-11-08’: Directory not empty

References: #13
Signed-off-by: Petr Štetiar <ynezz@true.cz>
15 months agophase1: prepareFactory: use variables instead of list index
Petr Štetiar [Sun, 18 Jun 2023 05:32:22 +0000 (07:32 +0200)]
phase1: prepareFactory: use variables instead of list index

Thus make the code more readable.

Signed-off-by: Petr Štetiar <ynezz@true.cz>
15 months agophase1: populateTargets: log branch name as well
Petr Štetiar [Sun, 18 Jun 2023 05:28:01 +0000 (07:28 +0200)]
phase1: populateTargets: log branch name as well

So the progress is more verbose.

Signed-off-by: Petr Štetiar <ynezz@true.cz>
15 months agophase1: do not leak targets between branches
Petr Štetiar [Sun, 18 Jun 2023 05:26:17 +0000 (07:26 +0200)]
phase1: do not leak targets between branches

Robert noticed, that after rename of `ipq807x` target in main branch to
`qualcommax/ipq807x` subtarget, that buildbot is trying to build this
new `qualcommax/ipq807x` subtarget on `openwrt-23.05` branch as well.

Thibaut later explained, that this is by design, his initial idea was to
find exhaustive list of all targets and let the `checkarch` step do the
final triaging.

I find this approach confusing, because if the subtarget is not present
in that branch, we shouldn't have builder for it configured as well.
Furthermore wasting roughly 5 minutes of precious buildworker time to
checkout all feeds and then just find out, that we're not going to use
those seems suboptimal as well.

So lets fix it by using builders for targets as present in respective
branches.

Fixes: #14
Reported-by: Robert Marko <robimarko@gmail.com>
Signed-off-by: Petr Štetiar <ynezz@true.cz>
16 months agophase1: reformat with black
Petr Štetiar [Sun, 18 Jun 2023 05:19:32 +0000 (07:19 +0200)]
phase1: reformat with black

Making everything consistent, more readable, CI guarded.

Signed-off-by: Petr Štetiar <ynezz@true.cz>
17 months agophase1: dlprune: fix cannot delete ‘dl/’: Not a directory v7
Petr Štetiar [Thu, 1 Jun 2023 05:56:49 +0000 (07:56 +0200)]
phase1: dlprune: fix cannot delete ‘dl/’: Not a directory

dlprune currently fails with following error:

 find: cannot delete ‘dl/’: Not a directory

Initial idea was to delete only stuff under download directory and not
the download directory itself, so lets adjust the find command
accordingly.

Fixes: 68b20ed67b5e ("phase1: prune unused files from dl/")
Signed-off-by: Petr Štetiar <ynezz@true.cz>
17 months agophase1: satisfy getver.sh requirements v6
Thibaut VARÈNE [Tue, 23 May 2023 18:12:09 +0000 (20:12 +0200)]
phase1: satisfy getver.sh requirements

scripts/getver.sh requires upstream branch tracking to correctly compute
revision number. Buildbot Git() step does not set branch tracking as it
checks out individual commits and then reset the target branch to it, so
in order for getver.sh to work, this workaround forcefully sets the
upstream branch.

This fixes malformed version code in filenames as observed on
openwrt-23.05:

r23001+3-38c150612c
instead of:
r23004-7f0db09513

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
17 months agophase1: cleanup redundant steps v5
Thibaut VARÈNE [Mon, 22 May 2023 18:20:12 +0000 (20:20 +0200)]
phase1: cleanup redundant steps

These 3 rm steps are unnecessary since these folders are already cleaned
by the Git() step. Thus reduce clutter in already pretty busy Factory.

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
17 months agophase1: workaround gitverify 1st build failures on fresh workers v4
Petr Štetiar [Tue, 16 May 2023 18:41:05 +0000 (20:41 +0200)]
phase1: workaround gitverify 1st build failures on fresh workers

Currently every 1st build on ephemeral build worker results in the
failure of the `gitverify` build step as the stock buildbot Git() build
step for some reasons just clones the Git repository into detached HEAD
state.

On the 2nd build using the same build worker Git() checkouts the branch
and thus makes `gitverify` step happy and the builds then continues
normally.

This needs to be fixed properly, either by adjusting the `gitverify`
check or adding suitable `mode` into Git() build step, but this needs
more time and testing.

So for now, lets simply workaround that issue by running Git() step two
times.

References: #5
Signed-off-by: Petr Štetiar <ynezz@true.cz>
17 months agodocker,worker: install g++-multilib
Alois Klink [Wed, 23 Nov 2022 17:23:48 +0000 (17:23 +0000)]
docker,worker: install g++-multilib

From [openwrt/docker@5484951][1] (GitHub PR [#89][2]):

Node fails to cross-compile from a 64-bit build machine to 32-bit host
with the following error:

```
  In file included from /usr/include/c++/8/memory:62,
  from ../deps/v8/src/libplatform/default-foreground-task-runner.h:8,
  from ../deps/v8/src/libplatform/default-foreground-task-runner.cc:5:
  /usr/include/c++/8/bits/stl_algobase.h:59:10: fatal error: bits/c++config.h:
    No such file or directory
  #include <bits/c++config.h>
          ^~~~~~~~~~~~~~~~~~
  compilation terminated.
```

On Debian, `g++-multilib` can be installed to fix this.

[1]: https://gitlab.com/openwrt/docker/-/commit/54849510d7802028b94757051cca6d004a9ca1d1
[2]: https://github.com/openwrt/docker/pull/89

Fixes: https://github.com/openwrt/packages/issues/18476
Fixes: https://forum.openwrt.org/t/why-arent-the-node-and-node-npm-packages-available-on-arm-cortex-a9-vfpv3-d16-in-22-03-2/142722
Signed-off-by: Alois Klink <alois@aloisklink.com>
17 months agoci: add basic config checking with ruff v3
Petr Štetiar [Tue, 16 May 2023 10:57:56 +0000 (12:57 +0200)]
ci: add basic config checking with ruff

Signed-off-by: Petr Štetiar <ynezz@true.cz>
17 months agophase1: actually make rsync -4 configurable
Thibaut VARÈNE [Mon, 15 May 2023 20:08:50 +0000 (22:08 +0200)]
phase1: actually make rsync -4 configurable

54e80d5ce introduced an interpolate bug that would evaluate an empty
string as being part of the arguments passed to rsync, resulting in a
failure as rsync interprets that empty string as a source.

This commit fixes this by instead appending '4' to regular shortopts
when necessary.

Fixes: 54e80d5ce
Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
Signed-off-by: Petr Štetiar <ynezz@true.cz>
17 months agophase1: perform kmodupload only when needed
Thibaut VARÈNE [Tue, 16 May 2023 08:13:11 +0000 (10:13 +0200)]
phase1: perform kmodupload only when needed

Fixes: 909e899
Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
Signed-off-by: Petr Štetiar <ynezz@true.cz>
17 months agophase1: fix crash on exception during Git update v2
Petr Štetiar [Mon, 15 May 2023 14:18:16 +0000 (16:18 +0200)]
phase1: fix crash on exception during Git update

Fixes following exception:

 2023-05-15 14:08:56+0000 [-] <Build master_apm821xx/nand number:1 results:success> build got exception when running step Git(name='git', repourl='https://git.openwrt.org/openwrt/openwrt.git', mode='full', method='fresh', locks=renderer(<function NetLockDl at 0x7f5d33351ee0>), haltOnFailure=True)
 2023-05-15 14:08:56+0000 [-] Unhandled Error
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/twisted/internet/defer.py", line 1792, in gotResult
    _inlineCallbacks(r, gen, status, context)
  File "/usr/local/lib/python3.9/dist-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
    result = context.run(
  File "/usr/local/lib/python3.9/dist-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.9/dist-packages/buildbot/process/build.py", line 576, in _start_next_step_impl
    log.err(e)
--- <exception caught here> ---
  File "/usr/local/lib/python3.9/dist-packages/buildbot/process/build.py", line 562, in _start_next_step_impl
    results = yield step.startStep(self.conn)
builtins.KeyError: None

as None is valid property value, but cant be used as dict index.

Signed-off-by: Petr Štetiar <ynezz@true.cz>
17 months agoci: allow pushing of containers during pull requests
Petr Štetiar [Mon, 15 May 2023 07:25:07 +0000 (09:25 +0200)]
ci: allow pushing of containers during pull requests

Make it possible to push containers in pull requests in private forks,
so the container can be actually runtime tested before merging the
changes.

Signed-off-by: Petr Štetiar <ynezz@true.cz>
17 months agobuildworker,buildmaster: bump Debian to version 11
Petr Štetiar [Mon, 15 May 2023 07:14:48 +0000 (09:14 +0200)]
buildworker,buildmaster: bump Debian to version 11

Debian 10 LTS support ends on 6/2024, so it makes no sense to use it as
a base for 23.05 release, so lets switch to Debian 11 which should've
LTS support till 6/2026.

Signed-off-by: Petr Štetiar <ynezz@true.cz>
17 months agoBump buildbot version to latest stable release 3.8.0
Petr Štetiar [Mon, 15 May 2023 07:12:16 +0000 (09:12 +0200)]
Bump buildbot version to latest stable release 3.8.0

Signed-off-by: Petr Štetiar <ynezz@true.cz>
17 months agobuildmaster: fix unpinned buildbot package version
Petr Štetiar [Mon, 15 May 2023 06:53:58 +0000 (08:53 +0200)]
buildmaster: fix unpinned buildbot package version

It seems, that current pip3 install pinning doesnt work properly as it
can result in the following package versions installation:

 buildbot-3.6.1 buildbot-console-view-3.5.0 buildbot-grid-view-3.5.0
 buildbot-waterfall-view-3.5.0 buildbot-worker-3.5.0 buildbot-www-3.5.0

So lets pin the buildbot package itself to BUILDBOT_VERSION as well.

References: https://gitlab.com/openwrt/buildbot/-/jobs/3324049571#L1158
Signed-off-by: Petr Štetiar <ynezz@true.cz>
17 months agophase1: make 'rsync -4' worker-configurable
Thibaut VARÈNE [Tue, 15 Nov 2022 09:44:56 +0000 (10:44 +0100)]
phase1: make 'rsync -4' worker-configurable

We set '-4' in rsync_defopts, asking rsync to "prefer" ipv4 whenever
possible on _all_ workers. This was historically done because some
workers had flaky ipv6 connectivity, however in the future the reverse
may be true, with worker having worse networking over ipv4 than ipv6.

This change introduces an 'rsync_ipv4' worker configuration option:
when set to a true value, the old rsync behavior (adding '-4') is used,
when unset or set to false, the extra rsync argument is not used.

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
17 months agophase1: call host ccache in stats
Thibaut VARÈNE [Fri, 28 Oct 2022 09:05:21 +0000 (11:05 +0200)]
phase1: call host ccache in stats

We do not use the staging_dir ccache on buildbots (config seed contains
CONFIG_CCACHE=n), only the host tool if present.
Also only execute this step if 'ccache_command' is set.

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
17 months agophase1: tag builders with their intended branch name
Thibaut VARÈNE [Thu, 27 Oct 2022 20:35:09 +0000 (22:35 +0200)]
phase1: tag builders with their intended branch name

This helps sorting them in waterfall and builders views.

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
17 months agophase1: documentation update
Thibaut VARÈNE [Thu, 27 Oct 2022 18:31:19 +0000 (20:31 +0200)]
phase1: documentation update

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
17 months agophase1: allow restricting builders to tag builds only
Thibaut VARÈNE [Thu, 27 Oct 2022 12:56:21 +0000 (14:56 +0200)]
phase1: allow restricting builders to tag builds only

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
17 months agophase1: refactor worker parsing
Thibaut VARÈNE [Thu, 27 Oct 2022 12:32:30 +0000 (14:32 +0200)]
phase1: refactor worker parsing

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
17 months agophase1: disable debug messages
Thibaut VARÈNE [Thu, 27 Oct 2022 12:11:13 +0000 (14:11 +0200)]
phase1: disable debug messages

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
17 months agophase1: prioritize builders by branch order
Thibaut VARÈNE [Thu, 27 Oct 2022 11:27:56 +0000 (13:27 +0200)]
phase1: prioritize builders by branch order

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
17 months agophase1: switch to separate builders per branch per target
Thibaut VARÈNE [Thu, 27 Oct 2022 11:09:17 +0000 (13:09 +0200)]
phase1: switch to separate builders per branch per target

This will simplify reporting and ensure consistent build statistics.
It will also simplify prioritizing builders by branch.

This also introduces a single "trigger" scheduler that will handle all
trigger requests for each branch/target combination, speeding up the
forcing of builds.

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
17 months agophase1: fix ForceBuild validation logic
Thibaut VARÈNE [Thu, 27 Oct 2022 08:20:39 +0000 (10:20 +0200)]
phase1: fix ForceBuild validation logic

We shouldn't use the local GitPoller repo to find the commit hash of a
tag, since this repo may not yet have been updated at the time the new
tag value is fetched from the remote repo.

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
17 months agophase1: run ForceBuilder on local worker
Thibaut VARÈNE [Wed, 26 Oct 2022 20:53:12 +0000 (22:53 +0200)]
phase1: run ForceBuilder on local worker

There is no reason to send this over the network to some remote worker,
incurring latency for a purely housekeeping task that should be done on
the buildmaster host.

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
17 months agophase1: prune unused files from dl/
Thibaut VARÈNE [Wed, 26 Oct 2022 19:35:18 +0000 (21:35 +0200)]
phase1: prune unused files from dl/

Remove files that haven't been accessed in >15 days from dl/

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
17 months agophase1: workers are running a single build
Thibaut VARÈNE [Wed, 26 Oct 2022 19:34:19 +0000 (21:34 +0200)]
phase1: workers are running a single build

Worker lock on dl/ is now unnecessary

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
17 months agophase1: Fix the force scheduler process
Thibaut VARÈNE [Wed, 26 Oct 2022 18:51:20 +0000 (20:51 +0200)]
phase1: Fix the force scheduler process

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
17 months agophase1: always front-run the 00_force_build builder
Thibaut VARÈNE [Wed, 26 Oct 2022 17:30:25 +0000 (19:30 +0200)]
phase1: always front-run the 00_force_build builder

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
17 months agophase1: fix ForceScheduler
Thibaut VARÈNE [Wed, 26 Oct 2022 12:43:15 +0000 (14:43 +0200)]
phase1: fix ForceScheduler

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
17 months agophase1: don't log env on statistics steps
Thibaut VARÈNE [Wed, 26 Oct 2022 07:46:22 +0000 (09:46 +0200)]
phase1: don't log env on statistics steps

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
17 months agophase1: consistent step styling
Thibaut VARÈNE [Tue, 25 Oct 2022 20:09:55 +0000 (22:09 +0200)]
phase1: consistent step styling

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
17 months agophase1: gitcheckout workaround is no longer needed
Thibaut VARÈNE [Tue, 25 Oct 2022 19:58:53 +0000 (21:58 +0200)]
phase1: gitcheckout workaround is no longer needed

And it has adverse effects (slows/breaks Git step).

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
17 months agophase1: move shared dl to same folder as shared-workdir
Thibaut VARÈNE [Tue, 25 Oct 2022 18:41:21 +0000 (20:41 +0200)]
phase1: move shared dl to same folder as shared-workdir

The previous command moved it to $HOME/dl, which could pollute the user's
home folder during test runs. This brings the "shared dl" folder in line
with the shared workdir folder, all within buildbot folder hierarchy.

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
17 months agophase1: perform rsync steps only if configured
Thibaut VARÈNE [Tue, 25 Oct 2022 17:48:46 +0000 (19:48 +0200)]
phase1: perform rsync steps only if configured

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
17 months agophase1: adjust steps descriptions
Thibaut VARÈNE [Tue, 25 Oct 2022 17:25:39 +0000 (19:25 +0200)]
phase1: adjust steps descriptions

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
17 months agophase1: hide/skip ccache steps if not available
Thibaut VARÈNE [Tue, 25 Oct 2022 13:41:06 +0000 (15:41 +0200)]
phase1: hide/skip ccache steps if not available

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
17 months agophase1: FlattenList doesn't work on renderables
Thibaut VARÈNE [Tue, 25 Oct 2022 11:08:50 +0000 (13:08 +0200)]
phase1: FlattenList doesn't work on renderables

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
17 months agophase1: update config.ini.example
Thibaut VARÈNE [Mon, 24 Oct 2022 16:17:46 +0000 (18:17 +0200)]
phase1: update config.ini.example

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
17 months agophase1: populateConfig(): quiet git clone
Thibaut VARÈNE [Mon, 24 Oct 2022 16:03:26 +0000 (18:03 +0200)]
phase1: populateConfig(): quiet git clone

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
17 months agophase1: make scheduler and changesource branch-aware
Thibaut VARÈNE [Mon, 24 Oct 2022 15:09:43 +0000 (17:09 +0200)]
phase1: make scheduler and changesource branch-aware

The ForcedScheduler for tags still needs work

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>
17 months agophase1: branch-aware signall.sh
Thibaut VARÈNE [Mon, 24 Oct 2022 14:56:45 +0000 (16:56 +0200)]
phase1: branch-aware signall.sh

Signed-off-by: Thibaut VARÈNE <hacks@slashdirt.org>