1129594 - Add more info to allthethings.json

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Armen [:armenzg]]

We need to modify allthethings.json for bug 1112777 to determine how to connect all builders.

This came from the analysis in this comment:
https://bugzilla.mozilla.org/show_bug.cgi?id=1112777#c9

I think the steps to run this script is:
hg clone https://hg.mozilla.org/build/braindump
cd community
./setup_buildbot_environment.sh
cd ~/.mozilla/releng/repos/buildbot-configs
source ../../venv/bin/activate
../braindump/buildbot-related/dump_allthethings.sh

allthethings.json will be generate under buildbot-configs.

What the script does is to setup a bunch of buildbot masters and then call this (with a list of all masters):
python $(dirname $0)/dump_master_json.py -o $OUTPUT ${MASTER_DIRS[*]} [1]

dump_master_json.py simply loads a python dictionaries for each master called 'c' (for example [3]).
That script then combines the dumps of all the masters [3].

Perhaps we could try to get a dump of c for each master and see where we can find the info we need.

[1] https://hg.mozilla.org/build/braindump/file/961db9340928/buildbot-related/dump_master_json.py
[2] http://hg.mozilla.org/build/buildbot-configs/file/default/mozilla-tests/tests_master.cfg#l25
http://mxr.mozilla.org/build/search?string=c%2B%3D%2BBuildmasterConfig
[3] https://hg.mozilla.org/build/braindump/file/961db9340928/buildbot-related/dump_master_json.py#l79

Comment 1

[Armen [:armenzg]]

To generate all of the data so we can inspect it we have to do the following:
* checkout braindump
* cd braindump/community
* ./setup_buildbot_environment.sh
* source ~/.mozilla/releng/venv/bin/activate

So far the same as the initial comment. Once we activate the virtual environment:
* cd ~/.mozilla/releng/repos/buildbot-configs
* /home/armenzg/.mozilla/releng/repos/braindump/buildbot-related/dump_masters.sh > dump_masters.txt

NOTE: It needs to be an absolute path.
NOTE: This script can take several minutes even on a powerful computer.

I will attach the file so we can discuss it.

Comment 2

[Armen [:armenzg]]

Attachment: dump_masters.txt.tar.gz

Comment 3

[Armen [:armenzg]]

It seems that we have two schedulers [1][2] listening to "mozilla-central-win32-opt-unittest".
I can't spot anything to make the relations we need.

[1]
<buildbot.schedulers.basic.Scheduler> {'builderNames': ['Windows 7 32-bit mozilla-central opt test mochitest-1',
                  'Windows 7 32-bit mozilla-central opt test mochitest-2',
                  'Windows 7 32-bit mozilla-central opt test mochitest-3',
                  'Windows 7 32-bit mozilla-central opt test mochitest-4',
                  'Windows 7 32-bit mozilla-central opt test mochitest-5',
                  'Windows 7 32-bit mozilla-central opt test mochitest-browser-chrome-1',
                  'Windows 7 32-bit mozilla-central opt test mochitest-browser-chrome-2',
                  'Windows 7 32-bit mozilla-central opt test mochitest-browser-chrome-3',
                  'Windows 7 32-bit mozilla-central opt test mochitest-other',
                  'Windows 7 32-bit mozilla-central opt test reftest',
                  'Windows 7 32-bit mozilla-central opt test jsreftest',
                  'Windows 7 32-bit mozilla-central opt test crashtest',
                  'Windows 7 32-bit mozilla-central opt test xpcshell',
                  'Windows 7 32-bit mozilla-central opt test cppunit',
                  'Windows 7 32-bit mozilla-central opt test mochitest-devtools-chrome',
                  'Windows 7 32-bit mozilla-central opt test reftest-no-accel',
                  'Windows 7 32-bit mozilla-central opt test jetpack',
                  'Windows 7 32-bit mozilla-central opt test marionette',
                  'Windows 7 32-bit mozilla-central opt test web-platform-tests-1',
                  'Windows 7 32-bit mozilla-central opt test web-platform-tests-2',
                  'Windows 7 32-bit mozilla-central opt test web-platform-tests-3',
                  'Windows 7 32-bit mozilla-central opt test web-platform-tests-4',
                  'Windows 7 32-bit mozilla-central opt test web-platform-tests-reftests',
                  'Windows 7 32-bit mozilla-central opt test jittest',
                  'Windows 7 32-bit mozilla-central opt test mochitest-gl',
                  'Windows 7 32-bit mozilla-central opt test mochitest-e10s-browser-chrome-1',
                  'Windows 7 32-bit mozilla-central opt test mochitest-e10s-browser-chrome-2',
                  'Windows 7 32-bit mozilla-central opt test mochitest-e10s-browser-chrome-3'],
 'change_filter': <ChangeFilter on branch == mozilla-central-win32-opt-unittest>,
 'fileIsImportant': None,
[2]
<buildbot.schedulers.basic.Scheduler> {'builderNames': ['Windows XP 32-bit mozilla-central opt test mochitest-1',
                  'Windows XP 32-bit mozilla-central opt test mochitest-2',
                  'Windows XP 32-bit mozilla-central opt test mochitest-3',
                  'Windows XP 32-bit mozilla-central opt test mochitest-4',
                  'Windows XP 32-bit mozilla-central opt test mochitest-5',
                  'Windows XP 32-bit mozilla-central opt test mochitest-browser-chrome-1',
                  'Windows XP 32-bit mozilla-central opt test mochitest-browser-chrome-2',
                  'Windows XP 32-bit mozilla-central opt test mochitest-browser-chrome-3',
                  'Windows XP 32-bit mozilla-central opt test mochitest-other',
                  'Windows XP 32-bit mozilla-central opt test reftest',
                  'Windows XP 32-bit mozilla-central opt test jsreftest',
                  'Windows XP 32-bit mozilla-central opt test crashtest',
                  'Windows XP 32-bit mozilla-central opt test xpcshell',
                  'Windows XP 32-bit mozilla-central opt test cppunit',
                  'Windows XP 32-bit mozilla-central opt test mochitest-devtools-chrome',
                  'Windows XP 32-bit mozilla-central opt test jetpack',
                  'Windows XP 32-bit mozilla-central opt test marionette',
                  'Windows XP 32-bit mozilla-central opt test web-platform-tests-1',
                  'Windows XP 32-bit mozilla-central opt test web-platform-tests-2',
                  'Windows XP 32-bit mozilla-central opt test web-platform-tests-3',
                  'Windows XP 32-bit mozilla-central opt test web-platform-tests-4',
                  'Windows XP 32-bit mozilla-central opt test web-platform-tests-reftests',
                  'Windows XP 32-bit mozilla-central opt test jittest',
                  'Windows XP 32-bit mozilla-central opt test mochitest-gl'],
 'change_filter': <ChangeFilter on branch == mozilla-central-win32-opt-unittest>,
 'fileIsImportant': None,
 'name': 'tests-mozilla-central-xp-ix-opt-unittest',
 'properties': {'scheduler': 'tests-mozilla-central-xp-ix-opt-unittest'},
 'treeStableTimer': None}
 'name': 'tests-mozilla-central-win7-ix-opt-unittest',
 'properties': {'scheduler': 'tests-mozilla-central-win7-ix-opt-unittest'},
 'treeStableTimer': None}

Comment 4

[Alice Scarpa [:adusca]]

I found that [0] has the build names for a given platform, e.g. 'win32-debug': { ... 'base_name': 'WINNT 5.2 %(branch)s leak test' ...}, and [1] has the tests names for a given platform, e.g. PLATFORMS['win32']['win7-ix'] = {'name': "Windows 7 32-bit"}. This is potentially useful, maybe the mapping between the test names and platform codes can easily be added to the json file by modifying master.cfg.
[0] - https://hg.mozilla.org/build/buildbot-configs/file/62928117cb72/mozilla/config.py
[1] - https://hg.mozilla.org/build/buildbot-configs/file/62928117cb72/mozilla-tests/config.py

Comment 5

[Armen [:armenzg]]

(In reply to Alice Scarpa[:adusca] from comment #4)
> I found that [0] has the build names for a given platform, e.g.
> 'win32-debug': { ... 'base_name': 'WINNT 5.2 %(branch)s leak test' ...}, and
> [1] has the tests names for a given platform, e.g.
> PLATFORMS['win32']['win7-ix'] = {'name': "Windows 7 32-bit"}. This is
> potentially useful, maybe the mapping between the test names and platform
> codes can easily be added to the json file by modifying master.cfg.
> [0] -
> https://hg.mozilla.org/build/buildbot-configs/file/62928117cb72/mozilla/
> config.py
> [1] -
> https://hg.mozilla.org/build/buildbot-configs/file/62928117cb72/mozilla-
> tests/config.py

Yes, Alice. You're on the right track.
All of these tools generate the data based on the buildbot-configs code.
I am hoping to aim directly at the source.
I briefly mentioned some of this in here:
https://bugzilla.mozilla.org/show_bug.cgi?id=1112777#c2

I hope to have something good for you next week!

Comment 6

[Alice Scarpa [:adusca]]

I did some investigation and, in your log (dump_masters.txt), every build job has SendChangeSteps which send notifications on given branches (e.g. [1]). We could add the SendChangeStep information to the builders section of allthethings.json and thus collect those branches.

Also, as noticed in comment 5, Schedulers have ChangeFilters listening on a given branch as well. We could add that information to the schedulers section of allthethings.json.

It would then be possible to match a test job to a build job: We find the scheduler for the test job, find the branch it listens on, and find what build job publishes sendchanges to that branch.

[1] SendChangeStep {'branch': 'mozilla-beta-android-talos' ... } {}
    SendChangeStep {'branch': 'mozilla-beta-android-opt-unittest' ... } {}

Comment 7

[Alice Scarpa [:adusca]]

Attachment: Added triggers field

I implemented the changes suggested in my previous comment. Now we have a 'triggers' field for builders [1] and 'filter_changes' for schedulers [2].
     
Unfortunately, there is a problem with that. My method cannot find triggers for mozharness jobs, because they don't have a 'SendChangeStep' step. Instead, mozharness jobs just call the mozharness script via a MockCommand step [3], and the script itself [4] sends the relevant messages.
     
We could deal with that by duplicating the mozharness logic (very fragile). Another way would be to include into allthethings.json the relevant parameters used to call mozharness. We could then extract the logic mozharness uses to compute the sendchange branch into a separate method, import mozharness to use that method and use it to deduce the triggers from the script arguments. But then the information would not be directly available from allthethings.json. Which way do you think is better? Any other ideas?
     
[1]
     "Android 2.3 Debug mozilla-beta build": {
          "properties": {
            "branch": "mozilla-beta",
            "platform": "android-debug",
            "product": "mobile",
            "slavebuilddir": "m-beta-and-d-00000000000000000",
            "stage_platform": "android-debug"
          },
          "shortname": "mozilla-beta-android-debug",
          "slavebuilddir": "m-beta-and-d-00000000000000000",
          "slavepool": "2e490369b232e800931337b9b1186f5027eac1e6",
          "triggers": [
            "mozilla-beta-android-debug-unittest"
          ]
        },
     
 [2]
    "tests-mozilla-beta-panda_android-debug-unittest": {
          "downstream": [
            "Android 4.0 Panda mozilla-beta debug test crashtest",
            "Android 4.0 Panda mozilla-beta debug test jsreftest-1",
            "Android 4.0 Panda mozilla-beta debug test jsreftest-2",
            "Android 4.0 Panda mozilla-beta debug test jsreftest-3",
            "Android 4.0 Panda mozilla-beta debug test mochitest-1",
            "Android 4.0 Panda mozilla-beta debug test mochitest-2",
            "Android 4.0 Panda mozilla-beta debug test mochitest-3",
            "Android 4.0 Panda mozilla-beta debug test mochitest-4",
            "Android 4.0 Panda mozilla-beta debug test mochitest-5",
            "Android 4.0 Panda mozilla-beta debug test mochitest-6",
            "Android 4.0 Panda mozilla-beta debug test mochitest-7",
            "Android 4.0 Panda mozilla-beta debug test mochitest-8",
            "Android 4.0 Panda mozilla-beta debug test plain-reftest-1",
            "Android 4.0 Panda mozilla-beta debug test plain-reftest-2",
            "Android 4.0 Panda mozilla-beta debug test plain-reftest-3",
            "Android 4.0 Panda mozilla-beta debug test plain-reftest-4",
            "Android 4.0 Panda mozilla-beta debug test plain-reftest-5",
            "Android 4.0 Panda mozilla-beta debug test plain-reftest-6",
            "Android 4.0 Panda mozilla-beta debug test plain-reftest-7",
            "Android 4.0 Panda mozilla-beta debug test plain-reftest-8"
          ],
          "filter_branches": [
            [
              "mozilla-beta-android-debug-unittest"
            ]
          ]
        },
     
[3]
    MockCommand {'command': ['/tools/buildbot/bin/python',  <WithProperties "%(script_repo_checkout)s/scripts/fx_desktop_build.py">,  '--config',  'builds/releng_base_linux_32_builds.py',  '--branch',  'ash',  '--build-pool',  'production'], 'description': None, 'descriptionDone': None, 'env': {'CCACHE_COMPRESS': '1',  'CCACHE_DIR': '/builds/ccache',  'CCACHE_UMASK': '002',  'DISPLAY': ':2',  'HG_SHARE_BASE_DIR': '/builds/hg-shared',  'LC_ALL': 'C',  'LD_LIBRARY_PATH': '/tools/gcc-4.3.3/installed/lib',  'MOZ_AUTOMATION': '1',  'MOZ_CRASHREPORTER_NO_REPORT': '1',  'MOZ_OBJDIR': 'obj-firefox',  'MOZ_SIGNING_SERVERS': 'gpg:osslsigncode:signcode:mar:jar:b2gmar:signing.build.mozilla.org:8080,gpg:mar:dmg:mac-signing.build.mozilla.org:8080,gpg:mar:dmgv2:mac-v2-signing.build.mozilla.org:8080',  'MOZ_SIGN_CMD': <WithProperties "python %(toolsdir)s/release/signing/signtool.py --cachedir %(basedir)s/signing_cache -t %(basedir)s/token -n %(basedir)s/nonce -c %(toolsdir)s/release/signing/host.cert -H gpg:osslsigncode:signcode:mar:jar:b2gmar:signing.build.mozilla.org:8080 -H gpg:mar:dmg:mac-signing.build.mozilla.org:8080 -H gpg:mar:dmgv2:mac-v2-signing.build.mozilla.org:8080">,  'MOZ_SYMBOLS_EXTRA_BUILDID': 'ash',  'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin',  'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py',  'PROPERTIES_FILE': <WithProperties "%(basedir)s/buildprops.json">,  'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org',  'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/',  'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_rsa',  'SYMBOL_SERVER_USER': 'ffxbld',  'TINDERBOX_OUTPUT': '1',  'TOOLTOOL_CACHE': '/builds/tooltool_cache',  'TOOLTOOL_HOME': '/builds'}, 'haltOnFailure': True, 'log_eval_func': <function eval_func>, 'logfiles': {}, 'maxTime': 19800, 'mock': False, 'mock_args': ['--unpriv'], 'mock_login': 'mock_mozilla', 'mock_workdir_mutator': <function <lambda>>, 'mock_workdir_prefix': '%(basedir)s/', 'name': 'run_script', 'target': None, 'timeout': 10800, 'usePTY': 'slave-config', 'warnOnWarnings': True, 'workdir': '.'} {}
     
[4] https://github.com/mozilla/build-mozharness/blob/master/mozharness/mozilla/building/buildbase.py#L1521

Comment 8

[Armen [:armenzg]]

I will be looking at this today. Thanks Alice!

Comment 9

[Armen [:armenzg]]

I will test your code locally and reply back with my results.

> Unfortunately, there is a problem with that. My method cannot find triggers
> for mozharness jobs, because they don't have a 'SendChangeStep' step.
> Instead, mozharness jobs just call the mozharness script via a MockCommand
> step [3], and the script itself [4] sends the relevant messages.
>      
> We could deal with that by duplicating the mozharness logic (very fragile).
> Another way would be to include into allthethings.json the relevant
> parameters used to call mozharness. We could then extract the logic
> mozharness uses to compute the sendchange branch into a separate method,
> import mozharness to use that method and use it to deduce the triggers from
> the script arguments. But then the information would not be directly
> available from allthethings.json. Which way do you think is better? Any
> other ideas?

I will think about it and talk with catlee/jlund about this.

For now, we can create a module in braindump that we'll copy the branch generation code.
Eventually that code could be put into mozharness or be a python importable package (who knows).
From looking at the mozharness code, I'm not happy on how it has been broken down (the branch should be calculated inside of buildbot.py rather than outside).

The actual sendchanges format does not really change much over time.
So we could build them ourselves as long as we know the following:
* repo_name
* stage_platform
* talos or unittest
* if unittests then 'opt' or 'debug' or 'pgo'

Various examples:
talos_branch = "%s-%s-%s%s" % (self.branch, self.stage_platform,
build_type, 'talos')
unittest_branch = "%s-%s-%s" % (self.branch, platform_and_build_type,
'unittest')
branch = '%s-%s-debug-unittest' %
(self.buildbot_config["properties"]["branch"], platform)
branch = '%s-%s-opt-unittest' %
(self.buildbot_config["properties"]["branch"],
self.buildbot_config["properties"]["platform"])


https://github.com/mozilla/build-mozharness/blob/master/mozharness/mozilla/building/buildbase.py#L1556
https://github.com/mozilla/build-mozharness/blob/master/mozharness/mozilla/building/buildbase.py#L1579
https://github.com/mozilla/build-mozharness/blob/master/mozharness/mozilla/buildbot.py#L151
https://github.com/mozilla/build-mozharness/blob/master/mozharness/mozilla/buildbot.py#L153

Comment 10

[Armen [:armenzg]]

Comment on attachment 8560933 [details] [diff] [review]
Added triggers field

Review of attachment 8560933 [details] [diff] [review]:
-----------------------------------------------------------------

It is looking good.
I will attach the differences.

Comment 11

[Armen [:armenzg]]

Attachment: differences.txt

We almost got this.
If you notice in the output, we got filter_branches to be null (e.g. null) or simply have the repo_path (e.g. "projects/ux").
I don't if those values would be an issue or not.

I wonder if we could use a more meaningful key than "filter_changes".

In order to generate the differences of using the patch attached (after running setup-buildbot.sh first):
cd ~/.mozilla/releng/repos/buildbot-configs
source ~/.mozilla/releng/venv/bin/activate
~/.mozilla/releng/repos/braindump/buildbot-related/dump_allthethings.sh
mv allthethings.json allthething.old.json
cd ../braindump
wget https://bug1129594.bugzilla.mozilla.org/attachment.cgi?id=8560933
patch -p1 < attachment.cgi?id=8560933
cd -
~/.mozilla/releng/repos/braindump/buildbot-related/dump_allthethings.sh
diff -pU8 allthethings.old.json allthethings.json > ~/moz/patches/differences.txt

Comment 12

[Armen [:armenzg]]

Comment on attachment 8561523 [details] [diff] [review]
differences.txt

catlee: adusca's patch (attachment 8560933 [details] [diff] [review]) generates this newer version of allthethings.json.

Would this work for you?
It adds the sendchange branches as a "triggers" entry for the build jobs [1].
It also adds "filter_changes" to the set of builders that listen to that branch.[2]

Should we add "filter_changes" indiscriminately to all schedulers? or only those starting with "tests-"?

[1]
-      "slavepool": "2e490369b232e800931337b9b1186f5027eac1e6"
+      "slavepool": "2e490369b232e800931337b9b1186f5027eac1e6", 
+      "triggers": [
+        "mozilla-beta-android-talos", 
+        "mozilla-beta-android-opt-unittest"
+      ]
     }, 

[2]
     "tests-mozilla-beta-android-talos": {
       "downstream": [
         "Android 4.0 Panda mozilla-beta talos remote-tp4m_nochrome", 
         "Android 4.0 Panda mozilla-beta talos remote-trobocheck2", 
         "Android 4.0 Panda mozilla-beta talos remote-trobopan", 
         "Android 4.0 Panda mozilla-beta talos remote-troboprovider", 
         "Android 4.0 Panda mozilla-beta talos remote-tspaint", 
         "Android 4.0 Panda mozilla-beta talos remote-tsvgx"
+      ], 
+      "filter_branches": [
+        [
+          "mozilla-beta-android-talos"
+        ]
       ]
     },

Comment 13

[Alice Scarpa [:adusca]]

Attachment: Added mozharness logic

I copied the logic from [1] and [2] into dump_master_json.py. It's very hardcoded, but it works. How can I improve that?

[1] https://github.com/mozilla/build-mozharness/blob/0a4f14aa76bb3419b9fe5ad28105dd9796c4d4c5/mozharness/mozilla/building/buildbase.py#L1521

[2] https://github.com/mozilla/build-mozharness/blob/master/mozharness/mozilla/buildbot.py#L142

Comment 14

[Alice Scarpa [:adusca]]

Attachment: allthethings.diff

The diff between allthethings.json generated by my lasted patch and the original one.

Comment 15

[Armen [:armenzg]]

Comment on attachment 8562084 [details] [diff] [review]
allthethings.diff

Marking as patch to make bugzilla gives more visualization options.

Comment 16

[Armen [:armenzg]]

As much as I'm sad to say this, we should stop pursuing this bug as-is - there's too much bending we need to do to make it work. Thanks to adusca we have been able to see how deep the rabbit hole is.
However, sometimes in software development we have to step back and decide which battles to pick.
I would take the first iteration you did if catlee is OK with it.
The reason I say all this, it's because the number of build jobs that are going to run mozharness is going to increase in the next few months, hence, we would fall out of date; it's very unfortunate the way that mozharness does the sendchanges.

I filed bug 1131610 in order to make things better for the future.

adusca says on IRC, if I understood correctly, that she would take new allthethings.json (as a one-off) to generate the data she needs for bug 1130103.

Comment 17

[Alice Scarpa [:adusca]]

Attachment: Added just triggered_by (f.k.a. filter_branches)

Here's an alternate version of the previous patch with just what I need for bug 1130103 and no messy mozharness copy-pasting.

Comment 18

[Armen [:armenzg]]

Comment on attachment 8561523 [details] [diff] [review]
differences.txt

We will land. Let us know later on if this was not correct.

Comment 19

[Armen [:armenzg]]

Comment on attachment 8565192 [details] [diff] [review]
Added just triggered_by (f.k.a. filter_branches)

r=me

Comment 20

[Armen [:armenzg]]

Landed: https://hg.mozilla.org/build/braindump/rev/263752819282

I assume allthethings.json will take some time to show these changes.

Comment 21

[Armen [:armenzg]]

We did the changes we needed in here.
Thanks to Alice!