Closed Bug 1129594 Opened 9 years ago Closed 9 years ago

Add more info to allthethings.json

Categories

(Release Engineering :: General, defect)

x86_64
Linux
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: armenzg, Assigned: adusca)

References

Details

Attachments

(4 files, 2 obsolete files)

We need to modify allthethings.json for bug 1112777 to determine how to connect all builders.

This came from the analysis in this comment:
https://bugzilla.mozilla.org/show_bug.cgi?id=1112777#c9

I think the steps to run this script is:
hg clone https://hg.mozilla.org/build/braindump
cd community
./setup_buildbot_environment.sh
cd ~/.mozilla/releng/repos/buildbot-configs
source ../../venv/bin/activate
../braindump/buildbot-related/dump_allthethings.sh

allthethings.json will be generate under buildbot-configs.

What the script does is to setup a bunch of buildbot masters and then call this (with a list of all masters):
python $(dirname $0)/dump_master_json.py -o $OUTPUT ${MASTER_DIRS[*]} [1]

dump_master_json.py simply loads a python dictionaries for each master called 'c' (for example [3]).
That script then combines the dumps of all the masters [3].

Perhaps we could try to get a dump of c for each master and see where we can find the info we need.

[1] https://hg.mozilla.org/build/braindump/file/961db9340928/buildbot-related/dump_master_json.py
[2] http://hg.mozilla.org/build/buildbot-configs/file/default/mozilla-tests/tests_master.cfg#l25
http://mxr.mozilla.org/build/search?string=c%2B%3D%2BBuildmasterConfig
[3] https://hg.mozilla.org/build/braindump/file/961db9340928/buildbot-related/dump_master_json.py#l79
To generate all of the data so we can inspect it we have to do the following:
* checkout braindump
* cd braindump/community
* ./setup_buildbot_environment.sh
* source ~/.mozilla/releng/venv/bin/activate

So far the same as the initial comment. Once we activate the virtual environment:
* cd ~/.mozilla/releng/repos/buildbot-configs
* /home/armenzg/.mozilla/releng/repos/braindump/buildbot-related/dump_masters.sh > dump_masters.txt

NOTE: It needs to be an absolute path.
NOTE: This script can take several minutes even on a powerful computer.

I will attach the file so we can discuss it.
It seems that we have two schedulers [1][2] listening to "mozilla-central-win32-opt-unittest".
I can't spot anything to make the relations we need.

[1]
<buildbot.schedulers.basic.Scheduler> {'builderNames': ['Windows 7 32-bit mozilla-central opt test mochitest-1',
                  'Windows 7 32-bit mozilla-central opt test mochitest-2',
                  'Windows 7 32-bit mozilla-central opt test mochitest-3',
                  'Windows 7 32-bit mozilla-central opt test mochitest-4',
                  'Windows 7 32-bit mozilla-central opt test mochitest-5',
                  'Windows 7 32-bit mozilla-central opt test mochitest-browser-chrome-1',
                  'Windows 7 32-bit mozilla-central opt test mochitest-browser-chrome-2',
                  'Windows 7 32-bit mozilla-central opt test mochitest-browser-chrome-3',
                  'Windows 7 32-bit mozilla-central opt test mochitest-other',
                  'Windows 7 32-bit mozilla-central opt test reftest',
                  'Windows 7 32-bit mozilla-central opt test jsreftest',
                  'Windows 7 32-bit mozilla-central opt test crashtest',
                  'Windows 7 32-bit mozilla-central opt test xpcshell',
                  'Windows 7 32-bit mozilla-central opt test cppunit',
                  'Windows 7 32-bit mozilla-central opt test mochitest-devtools-chrome',
                  'Windows 7 32-bit mozilla-central opt test reftest-no-accel',
                  'Windows 7 32-bit mozilla-central opt test jetpack',
                  'Windows 7 32-bit mozilla-central opt test marionette',
                  'Windows 7 32-bit mozilla-central opt test web-platform-tests-1',
                  'Windows 7 32-bit mozilla-central opt test web-platform-tests-2',
                  'Windows 7 32-bit mozilla-central opt test web-platform-tests-3',
                  'Windows 7 32-bit mozilla-central opt test web-platform-tests-4',
                  'Windows 7 32-bit mozilla-central opt test web-platform-tests-reftests',
                  'Windows 7 32-bit mozilla-central opt test jittest',
                  'Windows 7 32-bit mozilla-central opt test mochitest-gl',
                  'Windows 7 32-bit mozilla-central opt test mochitest-e10s-browser-chrome-1',
                  'Windows 7 32-bit mozilla-central opt test mochitest-e10s-browser-chrome-2',
                  'Windows 7 32-bit mozilla-central opt test mochitest-e10s-browser-chrome-3'],
 'change_filter': <ChangeFilter on branch == mozilla-central-win32-opt-unittest>,
 'fileIsImportant': None,
[2]
<buildbot.schedulers.basic.Scheduler> {'builderNames': ['Windows XP 32-bit mozilla-central opt test mochitest-1',
                  'Windows XP 32-bit mozilla-central opt test mochitest-2',
                  'Windows XP 32-bit mozilla-central opt test mochitest-3',
                  'Windows XP 32-bit mozilla-central opt test mochitest-4',
                  'Windows XP 32-bit mozilla-central opt test mochitest-5',
                  'Windows XP 32-bit mozilla-central opt test mochitest-browser-chrome-1',
                  'Windows XP 32-bit mozilla-central opt test mochitest-browser-chrome-2',
                  'Windows XP 32-bit mozilla-central opt test mochitest-browser-chrome-3',
                  'Windows XP 32-bit mozilla-central opt test mochitest-other',
                  'Windows XP 32-bit mozilla-central opt test reftest',
                  'Windows XP 32-bit mozilla-central opt test jsreftest',
                  'Windows XP 32-bit mozilla-central opt test crashtest',
                  'Windows XP 32-bit mozilla-central opt test xpcshell',
                  'Windows XP 32-bit mozilla-central opt test cppunit',
                  'Windows XP 32-bit mozilla-central opt test mochitest-devtools-chrome',
                  'Windows XP 32-bit mozilla-central opt test jetpack',
                  'Windows XP 32-bit mozilla-central opt test marionette',
                  'Windows XP 32-bit mozilla-central opt test web-platform-tests-1',
                  'Windows XP 32-bit mozilla-central opt test web-platform-tests-2',
                  'Windows XP 32-bit mozilla-central opt test web-platform-tests-3',
                  'Windows XP 32-bit mozilla-central opt test web-platform-tests-4',
                  'Windows XP 32-bit mozilla-central opt test web-platform-tests-reftests',
                  'Windows XP 32-bit mozilla-central opt test jittest',
                  'Windows XP 32-bit mozilla-central opt test mochitest-gl'],
 'change_filter': <ChangeFilter on branch == mozilla-central-win32-opt-unittest>,
 'fileIsImportant': None,
 'name': 'tests-mozilla-central-xp-ix-opt-unittest',
 'properties': {'scheduler': 'tests-mozilla-central-xp-ix-opt-unittest'},
 'treeStableTimer': None}
 'name': 'tests-mozilla-central-win7-ix-opt-unittest',
 'properties': {'scheduler': 'tests-mozilla-central-win7-ix-opt-unittest'},
 'treeStableTimer': None}
I found that [0] has the build names for a given platform, e.g. 'win32-debug': { ... 'base_name': 'WINNT 5.2 %(branch)s leak test' ...}, and [1] has the tests names for a given platform, e.g. PLATFORMS['win32']['win7-ix'] = {'name': "Windows 7 32-bit"}. This is potentially useful, maybe the mapping between the test names and platform codes can easily be added to the json file by modifying master.cfg.
[0] - https://hg.mozilla.org/build/buildbot-configs/file/62928117cb72/mozilla/config.py
[1] - https://hg.mozilla.org/build/buildbot-configs/file/62928117cb72/mozilla-tests/config.py
(In reply to Alice Scarpa[:adusca] from comment #4)
> I found that [0] has the build names for a given platform, e.g.
> 'win32-debug': { ... 'base_name': 'WINNT 5.2 %(branch)s leak test' ...}, and
> [1] has the tests names for a given platform, e.g.
> PLATFORMS['win32']['win7-ix'] = {'name': "Windows 7 32-bit"}. This is
> potentially useful, maybe the mapping between the test names and platform
> codes can easily be added to the json file by modifying master.cfg.
> [0] -
> https://hg.mozilla.org/build/buildbot-configs/file/62928117cb72/mozilla/
> config.py
> [1] -
> https://hg.mozilla.org/build/buildbot-configs/file/62928117cb72/mozilla-
> tests/config.py

Yes, Alice. You're on the right track.
All of these tools generate the data based on the buildbot-configs code.
I am hoping to aim directly at the source.
I briefly mentioned some of this in here:
https://bugzilla.mozilla.org/show_bug.cgi?id=1112777#c2

I hope to have something good for you next week!
I did some investigation and, in your log (dump_masters.txt), every build job has SendChangeSteps which send notifications on given branches (e.g. [1]). We could add the SendChangeStep information to the builders section of allthethings.json and thus collect those branches.

Also, as noticed in comment 5, Schedulers have ChangeFilters listening on a given branch as well. We could add that information to the schedulers section of allthethings.json.

It would then be possible to match a test job to a build job: We find the scheduler for the test job, find the branch it listens on, and find what build job publishes sendchanges to that branch.

[1] SendChangeStep {'branch': 'mozilla-beta-android-talos' ... } {}
    SendChangeStep {'branch': 'mozilla-beta-android-opt-unittest' ... } {}
Attached patch Added triggers field (obsolete) — Splinter Review
I implemented the changes suggested in my previous comment. Now we have a 'triggers' field for builders [1] and 'filter_changes' for schedulers [2].
     
Unfortunately, there is a problem with that. My method cannot find triggers for mozharness jobs, because they don't have a 'SendChangeStep' step. Instead, mozharness jobs just call the mozharness script via a MockCommand step [3], and the script itself [4] sends the relevant messages.
     
We could deal with that by duplicating the mozharness logic (very fragile). Another way would be to include into allthethings.json the relevant parameters used to call mozharness. We could then extract the logic mozharness uses to compute the sendchange branch into a separate method, import mozharness to use that method and use it to deduce the triggers from the script arguments. But then the information would not be directly available from allthethings.json. Which way do you think is better? Any other ideas?
     
[1]
     "Android 2.3 Debug mozilla-beta build": {
          "properties": {
            "branch": "mozilla-beta",
            "platform": "android-debug",
            "product": "mobile",
            "slavebuilddir": "m-beta-and-d-00000000000000000",
            "stage_platform": "android-debug"
          },
          "shortname": "mozilla-beta-android-debug",
          "slavebuilddir": "m-beta-and-d-00000000000000000",
          "slavepool": "2e490369b232e800931337b9b1186f5027eac1e6",
          "triggers": [
            "mozilla-beta-android-debug-unittest"
          ]
        },
     
 [2]
    "tests-mozilla-beta-panda_android-debug-unittest": {
          "downstream": [
            "Android 4.0 Panda mozilla-beta debug test crashtest",
            "Android 4.0 Panda mozilla-beta debug test jsreftest-1",
            "Android 4.0 Panda mozilla-beta debug test jsreftest-2",
            "Android 4.0 Panda mozilla-beta debug test jsreftest-3",
            "Android 4.0 Panda mozilla-beta debug test mochitest-1",
            "Android 4.0 Panda mozilla-beta debug test mochitest-2",
            "Android 4.0 Panda mozilla-beta debug test mochitest-3",
            "Android 4.0 Panda mozilla-beta debug test mochitest-4",
            "Android 4.0 Panda mozilla-beta debug test mochitest-5",
            "Android 4.0 Panda mozilla-beta debug test mochitest-6",
            "Android 4.0 Panda mozilla-beta debug test mochitest-7",
            "Android 4.0 Panda mozilla-beta debug test mochitest-8",
            "Android 4.0 Panda mozilla-beta debug test plain-reftest-1",
            "Android 4.0 Panda mozilla-beta debug test plain-reftest-2",
            "Android 4.0 Panda mozilla-beta debug test plain-reftest-3",
            "Android 4.0 Panda mozilla-beta debug test plain-reftest-4",
            "Android 4.0 Panda mozilla-beta debug test plain-reftest-5",
            "Android 4.0 Panda mozilla-beta debug test plain-reftest-6",
            "Android 4.0 Panda mozilla-beta debug test plain-reftest-7",
            "Android 4.0 Panda mozilla-beta debug test plain-reftest-8"
          ],
          "filter_branches": [
            [
              "mozilla-beta-android-debug-unittest"
            ]
          ]
        },
     
[3]
    MockCommand {'command': ['/tools/buildbot/bin/python',  <WithProperties "%(script_repo_checkout)s/scripts/fx_desktop_build.py">,  '--config',  'builds/releng_base_linux_32_builds.py',  '--branch',  'ash',  '--build-pool',  'production'], 'description': None, 'descriptionDone': None, 'env': {'CCACHE_COMPRESS': '1',  'CCACHE_DIR': '/builds/ccache',  'CCACHE_UMASK': '002',  'DISPLAY': ':2',  'HG_SHARE_BASE_DIR': '/builds/hg-shared',  'LC_ALL': 'C',  'LD_LIBRARY_PATH': '/tools/gcc-4.3.3/installed/lib',  'MOZ_AUTOMATION': '1',  'MOZ_CRASHREPORTER_NO_REPORT': '1',  'MOZ_OBJDIR': 'obj-firefox',  'MOZ_SIGNING_SERVERS': 'gpg:osslsigncode:signcode:mar:jar:b2gmar:signing.build.mozilla.org:8080,gpg:mar:dmg:mac-signing.build.mozilla.org:8080,gpg:mar:dmgv2:mac-v2-signing.build.mozilla.org:8080',  'MOZ_SIGN_CMD': <WithProperties "python %(toolsdir)s/release/signing/signtool.py --cachedir %(basedir)s/signing_cache -t %(basedir)s/token -n %(basedir)s/nonce -c %(toolsdir)s/release/signing/host.cert -H gpg:osslsigncode:signcode:mar:jar:b2gmar:signing.build.mozilla.org:8080 -H gpg:mar:dmg:mac-signing.build.mozilla.org:8080 -H gpg:mar:dmgv2:mac-v2-signing.build.mozilla.org:8080">,  'MOZ_SYMBOLS_EXTRA_BUILDID': 'ash',  'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin',  'POST_SYMBOL_UPLOAD_CMD': '/usr/local/bin/post-symbol-upload.py',  'PROPERTIES_FILE': <WithProperties "%(basedir)s/buildprops.json">,  'SYMBOL_SERVER_HOST': 'symbolpush.mozilla.org',  'SYMBOL_SERVER_PATH': '/mnt/netapp/breakpad/symbols_ffx/',  'SYMBOL_SERVER_SSH_KEY': '/home/mock_mozilla/.ssh/ffxbld_rsa',  'SYMBOL_SERVER_USER': 'ffxbld',  'TINDERBOX_OUTPUT': '1',  'TOOLTOOL_CACHE': '/builds/tooltool_cache',  'TOOLTOOL_HOME': '/builds'}, 'haltOnFailure': True, 'log_eval_func': <function eval_func>, 'logfiles': {}, 'maxTime': 19800, 'mock': False, 'mock_args': ['--unpriv'], 'mock_login': 'mock_mozilla', 'mock_workdir_mutator': <function <lambda>>, 'mock_workdir_prefix': '%(basedir)s/', 'name': 'run_script', 'target': None, 'timeout': 10800, 'usePTY': 'slave-config', 'warnOnWarnings': True, 'workdir': '.'} {}
     
[4] https://github.com/mozilla/build-mozharness/blob/master/mozharness/mozilla/building/buildbase.py#L1521
Attachment #8560933 - Flags: feedback?(armenzg)
I will be looking at this today. Thanks Alice!
I will test your code locally and reply back with my results.

> Unfortunately, there is a problem with that. My method cannot find triggers
> for mozharness jobs, because they don't have a 'SendChangeStep' step.
> Instead, mozharness jobs just call the mozharness script via a MockCommand
> step [3], and the script itself [4] sends the relevant messages.
>      
> We could deal with that by duplicating the mozharness logic (very fragile).
> Another way would be to include into allthethings.json the relevant
> parameters used to call mozharness. We could then extract the logic
> mozharness uses to compute the sendchange branch into a separate method,
> import mozharness to use that method and use it to deduce the triggers from
> the script arguments. But then the information would not be directly
> available from allthethings.json. Which way do you think is better? Any
> other ideas?

I will think about it and talk with catlee/jlund about this.

For now, we can create a module in braindump that we'll copy the branch generation code.
Eventually that code could be put into mozharness or be a python importable package (who knows).
From looking at the mozharness code, I'm not happy on how it has been broken down (the branch should be calculated inside of buildbot.py rather than outside).

The actual sendchanges format does not really change much over time.
So we could build them ourselves as long as we know the following:
* repo_name
* stage_platform
* talos or unittest
* if unittests then 'opt' or 'debug' or 'pgo'

Various examples:
talos_branch = "%s-%s-%s%s" % (self.branch, self.stage_platform,
build_type, 'talos')
unittest_branch = "%s-%s-%s" % (self.branch, platform_and_build_type,
'unittest')
branch = '%s-%s-debug-unittest' %
(self.buildbot_config["properties"]["branch"], platform)
branch = '%s-%s-opt-unittest' %
(self.buildbot_config["properties"]["branch"],
self.buildbot_config["properties"]["platform"])


https://github.com/mozilla/build-mozharness/blob/master/mozharness/mozilla/building/buildbase.py#L1556
https://github.com/mozilla/build-mozharness/blob/master/mozharness/mozilla/building/buildbase.py#L1579
https://github.com/mozilla/build-mozharness/blob/master/mozharness/mozilla/buildbot.py#L151
https://github.com/mozilla/build-mozharness/blob/master/mozharness/mozilla/buildbot.py#L153
Comment on attachment 8560933 [details] [diff] [review]
Added triggers field

Review of attachment 8560933 [details] [diff] [review]:
-----------------------------------------------------------------

It is looking good.
I will attach the differences.
Attachment #8560933 - Flags: feedback?(armenzg) → feedback+
Attached patch differences.txtSplinter Review
We almost got this.
If you notice in the output, we got filter_branches to be null (e.g. null) or simply have the repo_path (e.g. "projects/ux").
I don't if those values would be an issue or not.

I wonder if we could use a more meaningful key than "filter_changes".

In order to generate the differences of using the patch attached (after running setup-buildbot.sh first):
cd ~/.mozilla/releng/repos/buildbot-configs
source ~/.mozilla/releng/venv/bin/activate
~/.mozilla/releng/repos/braindump/buildbot-related/dump_allthethings.sh
mv allthethings.json allthething.old.json
cd ../braindump
wget https://bug1129594.bugzilla.mozilla.org/attachment.cgi?id=8560933
patch -p1 < attachment.cgi?id=8560933
cd -
~/.mozilla/releng/repos/braindump/buildbot-related/dump_allthethings.sh
diff -pU8 allthethings.old.json allthethings.json > ~/moz/patches/differences.txt
Attachment #8561523 - Flags: feedback?
Comment on attachment 8561523 [details] [diff] [review]
differences.txt

catlee: adusca's patch (attachment 8560933 [details] [diff] [review]) generates this newer version of allthethings.json.

Would this work for you?
It adds the sendchange branches as a "triggers" entry for the build jobs [1].
It also adds "filter_changes" to the set of builders that listen to that branch.[2]

Should we add "filter_changes" indiscriminately to all schedulers? or only those starting with "tests-"?

[1]
-      "slavepool": "2e490369b232e800931337b9b1186f5027eac1e6"
+      "slavepool": "2e490369b232e800931337b9b1186f5027eac1e6", 
+      "triggers": [
+        "mozilla-beta-android-talos", 
+        "mozilla-beta-android-opt-unittest"
+      ]
     }, 

[2]
     "tests-mozilla-beta-android-talos": {
       "downstream": [
         "Android 4.0 Panda mozilla-beta talos remote-tp4m_nochrome", 
         "Android 4.0 Panda mozilla-beta talos remote-trobocheck2", 
         "Android 4.0 Panda mozilla-beta talos remote-trobopan", 
         "Android 4.0 Panda mozilla-beta talos remote-troboprovider", 
         "Android 4.0 Panda mozilla-beta talos remote-tspaint", 
         "Android 4.0 Panda mozilla-beta talos remote-tsvgx"
+      ], 
+      "filter_branches": [
+        [
+          "mozilla-beta-android-talos"
+        ]
       ]
     },
Attachment #8561523 - Flags: feedback? → feedback?(catlee)
Attached patch Added mozharness logic (obsolete) — Splinter Review
I copied the logic from [1] and [2] into dump_master_json.py. It's very hardcoded, but it works. How can I improve that?

[1] https://github.com/mozilla/build-mozharness/blob/0a4f14aa76bb3419b9fe5ad28105dd9796c4d4c5/mozharness/mozilla/building/buildbase.py#L1521

[2] https://github.com/mozilla/build-mozharness/blob/master/mozharness/mozilla/buildbot.py#L142
Attachment #8560933 - Attachment is obsolete: true
Attachment #8562079 - Flags: feedback?(armenzg)
The diff between allthethings.json generated by my lasted patch and the original one.
Comment on attachment 8562084 [details] [diff] [review]
allthethings.diff

Marking as patch to make bugzilla gives more visualization options.
Attachment #8562084 - Attachment is patch: true
Attachment #8561523 - Attachment is patch: true
Depends on: 1131610
As much as I'm sad to say this, we should stop pursuing this bug as-is - there's too much bending we need to do to make it work. Thanks to adusca we have been able to see how deep the rabbit hole is.
However, sometimes in software development we have to step back and decide which battles to pick.
I would take the first iteration you did if catlee is OK with it.
The reason I say all this, it's because the number of build jobs that are going to run mozharness is going to increase in the next few months, hence, we would fall out of date; it's very unfortunate the way that mozharness does the sendchanges.

I filed bug 1131610 in order to make things better for the future.

adusca says on IRC, if I understood correctly, that she would take new allthethings.json (as a one-off) to generate the data she needs for bug 1130103.
Attachment #8562079 - Flags: feedback?(armenzg)
Here's an alternate version of the previous patch with just what I need for bug 1130103 and no messy mozharness copy-pasting.
Attachment #8562079 - Attachment is obsolete: true
Comment on attachment 8561523 [details] [diff] [review]
differences.txt

We will land. Let us know later on if this was not correct.
Attachment #8561523 - Flags: feedback?(catlee)
Comment on attachment 8565192 [details] [diff] [review]
Added just triggered_by (f.k.a. filter_branches)

r=me
Attachment #8565192 - Flags: review+
Assignee: nobody → alicescarpa
Landed: https://hg.mozilla.org/build/braindump/rev/263752819282

I assume allthethings.json will take some time to show these changes.
We did the changes we needed in here.
Thanks to Alice!
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Component: Tools → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: