Back to build
Raw log | Switch to full mode | Login

vpython -u /b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/tools/device_recovery.py --blacklist-file /b/s/w/ir/cache/builder/src/out/bad_devices.json --known-devices-file /home/chrome-bot/.android/known_devices.json --adb-path /b/s/w/ir/cache/builder/src/third_party/android_tools/sdk/platform-tools/adb -v in dir /b/s/w/ir/cache/builder: allow_subannotations: False cmd: ['vpython', '-u', '/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/tools/device_recovery.py', '--blacklist-file', '/b/s/w/ir/cache/builder/src/out/bad_devices.json', '--known-devices-file', '/home/chrome-bot/.android/known_devices.json', '--adb-path', '/b/s/w/ir/cache/builder/src/third_party/android_tools/sdk/platform-tools/adb', '-v'] cwd: '/b/s/w/ir/cache/builder' env: {'CHROME_HEADLESS': '1', 'PATH': '/b/s/w/ir/cache/builder/src/third_party/android_tools/sdk/platform-tools:/b/s/w/ir/cache/builder/src/build/android:%(PATH)s'} env_prefixes: {} env_suffixes: {} infra_step: True name_tokens: ('device_recovery',) ok_ret: frozenset([0]) trigger_specs: () full environment: ADDRFAM: inet BOTO_CONFIG: /b/s/w/ir/tmp/gsutil_task/.boto BUILDBUCKET_EXPERIMENTAL: FALSE CHROME_HEADLESS: 1 CIPD_CACHE_DIR: /b/s/cipd_cache/cache CIPD_PROTOCOL: v2 DEVSHELL_CLIENT_PORT: 55814 DOCKER_CONFIG: /b/s/w/ir/tmp/docker_cfg_task DOCKER_TMPDIR: /b/s/w/ir/tmp/docker_tmp_task GIT_CONFIG_NOSYSTEM: 1 GIT_TERMINAL_PROMPT: 0 HOME: /home/chrome-bot IFACE: em1 INFRA_GIT_WRAPPER_HOME: /b/s/w/ir/tmp/git_home_task LANG: en_US.UTF-8 LANGUAGE: en_US: LOGDOG_COORDINATOR_HOST: logs.chromium.org LOGDOG_STREAM_PREFIX: buildbucket/cr-buildbucket.appspot.com/8918537712723199760 LOGDOG_STREAM_PROJECT: chromium LOGDOG_STREAM_SERVER_PATH: unix:/b/s/w/ir/tmp/ld.sock LOGICAL: em1 LOGNAME: chrome-bot LUCI_CONTEXT: /b/s/w/ittHt3yW/luci_context.940774895 MAC_CHROMIUM_TMPDIR: /b/s/w/ir/tmp/t MAIL: /var/mail/chrome-bot METHOD: dhcp NO_GCE_CHECK: False PATH: /b/s/w/ir/cache/builder/src/third_party/android_tools/sdk/platform-tools:/b/s/w/ir/cache/builder/src/build/android:/b/s/w/ir/cipd_bin_packages:/b/s/w/ir/cipd_bin_packages/bin:/b/s/cipd_cache/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games PWD: /b/s/w/ir/k PYTHONIOENCODING: UTF-8 PYTHONUNBUFFERED: 1 SHELL: /bin/bash SHLVL: 2 SWARMING_BOT_ID: build4-b9 SWARMING_HEADLESS: 1 SWARMING_SERVER: https://chromium-swarm.appspot.com SWARMING_TASK_ID: 43afd33f8245bb11 TEMP: /b/s/w/ir/tmp/t TEMPDIR: /b/s/w/ir/tmp/t TERM: linux TMP: /b/s/w/ir/tmp/t TMPDIR: /b/s/w/ir/tmp/t UPSTART_EVENTS: local-filesystems net-device-up UPSTART_INSTANCE: UPSTART_JOB: swarming USER: chrome-bot VPYTHON_VIRTUALENV_ROOT: /b/s/w/ir/cache/vpython XDG_RUNTIME_DIR: /run/user/1000 XDG_SESSION_ID: c1 _: /b/s/w/ir/cipd_bin_packages/vpython I 0.018s Main Expected devices: I 0.018s Main 0acc25be25995b8a I 0.018s Main 0d88ca9443e4bb01 I 0.019s Main 06c14b05003b815c I 0.019s Main 06ad9a16003bb372 I 0.019s Main 0aa167a825995d01 I 0.019s Main 06a9b452003beda2 I 0.019s Main 0acca87043e4ba6d I 0.019s Main 06b32a05003b9923 I 1.741s TimeoutThread-1-for-MainThread condition '<lambda>' met (0.0s) I 4.030s blacklisting_device_status(0acc25be25995b8a) Adding 0acc25be25995b8a to blacklist /b/s/w/ir/cache/builder/src/out/bad_devices.json for reason: offline I 4.032s blacklisting_device_status(0acca87043e4ba6d) Adding 0acca87043e4ba6d to blacklist /b/s/w/ir/cache/builder/src/out/bad_devices.json for reason: offline I 4.465s Main Resetting blacklist /b/s/w/ir/cache/builder/src/out/bad_devices.json I 4.475s Main kill 15 14569 (adb [adb -L tcp:5037 fork-server server --reply-fd 8]) I 4.475s Main kill 15 14590 (adb [/b/s/w/ir/cache/builder/src/third_party/android_tools/sdk/platform-tools/adb -s 0d88ca9443e4bb01 logcat -v threadtime]) I 4.475s Main kill 15 14591 (adb [/b/s/w/ir/cache/builder/src/third_party/android_tools/sdk/platform-tools/adb -s 06ad9a16003bb372 logcat -v threadtime]) I 4.475s Main kill 15 14619 (adb [/b/s/w/ir/cache/builder/src/third_party/android_tools/sdk/platform-tools/adb -s 0aa167a825995d01 logcat -v threadtime]) I 4.475s Main kill 15 14748 (adb []) I 4.475s Main kill 15 14749 (adb []) I 4.486s Main kill 3 14590 (adb []) I 4.486s Main kill 3 14591 (adb []) I 4.486s Main kill 3 14619 (adb []) I 4.486s Main kill 3 14748 (adb []) I 4.486s Main kill 3 14749 (adb []) I 4.496s Main kill 9 14590 (adb []) I 4.497s Main kill 9 14591 (adb []) I 4.497s Main kill 9 14619 (adb []) I 4.497s Main kill 9 14748 (adb []) I 4.497s Main kill 9 14749 (adb []) E 4.507s Main Unable to kill 14590 (adb []) E 4.507s Main Unable to kill 14591 (adb []) E 4.508s Main Unable to kill 14619 (adb []) E 4.508s Main Unable to kill 14748 (adb []) E 4.508s Main Unable to kill 14749 (adb []) W 7.574s Main USB reset disabled for 0acc25be25995b8a (crbug.com/642914) W 7.574s Main USB reset disabled for 06c14b05003b815c (crbug.com/642914) W 7.574s Main USB reset disabled for 0aa167a825995d01 (crbug.com/642914) W 7.574s Main USB reset disabled for 06a9b452003beda2 (crbug.com/642914) W 7.574s Main USB reset disabled for 0acca87043e4ba6d (crbug.com/642914) W 7.574s Main USB reset disabled for 06b32a05003b9923 (crbug.com/642914) I 7.841s TimeoutThread-1-for-RecoverDevice(0acc25be25995b8a) condition 'sd_card_ready' met (0.3s) I 7.850s TimeoutThread-1-for-RecoverDevice(0acca87043e4ba6d) condition 'sd_card_ready' met (0.3s) I 7.894s TimeoutThread-1-for-RecoverDevice(06ad9a16003bb372) condition 'sd_card_ready' met (0.3s) I 8.470s TimeoutThread-1-for-RecoverDevice(0acc25be25995b8a) condition 'pm_ready' met (0.9s) I 8.505s TimeoutThread-1-for-RecoverDevice(0acca87043e4ba6d) condition 'pm_ready' met (0.9s) I 8.512s TimeoutThread-1-for-RecoverDevice(06ad9a16003bb372) condition 'pm_ready' met (0.9s) I 8.534s TimeoutThread-1-for-RecoverDevice(0acc25be25995b8a) condition 'boot_completed' met (1.0s) I 8.546s TimeoutThread-1-for-RecoverDevice(0acca87043e4ba6d) condition 'boot_completed' met (1.0s) I 8.557s TimeoutThread-1-for-RecoverDevice(06ad9a16003bb372) condition 'boot_completed' met (1.0s) I 8.998s TimeoutThread-1-for-RecoverDevice(0acc25be25995b8a) condition 'device_offline' met (0.5s) I 9.039s TimeoutThread-1-for-RecoverDevice(0acca87043e4ba6d) condition 'device_offline' met (0.5s) I 9.074s TimeoutThread-1-for-RecoverDevice(06ad9a16003bb372) condition 'device_offline' met (0.5s) I 12.893s TimeoutThread-1-for-RecoverDevice(0aa167a825995d01) condition 'sd_card_ready' met (5.3s) W 13.508s TimeoutThread-1-for-RecoverDevice(0aa167a825995d01) exit status of shell command 'pm path android' missing. E 13.508s TimeoutThread-1-for-RecoverDevice(0aa167a825995d01) Partial output found: W 13.508s TimeoutThread-1-for-RecoverDevice(0aa167a825995d01) Attempting to run in large_output mode. W 13.508s TimeoutThread-1-for-RecoverDevice(0aa167a825995d01) Use RunShellCommand(..., large_output=True) for shell commands that expect a lot of output. E 13.547s RecoverDevice(0aa167a825995d01) Failure while waiting for 0aa167a825995d01. Attempting to recover. Traceback (most recent call last): File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/tools/device_recovery.py", line 75, in RecoverDevice device.WaitUntilFullyBooted(retries=0) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper retry_if_func=retry_if_func) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 158, in Run error_log_func=error_log_func) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 198, in JoinAll self._JoinAll(watcher, timeout) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 170, in _JoinAll thread.ReraiseIfException() File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 93, in run self._ret = self._func(*self._args, **self._kwargs) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 151, in <lambda> child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs), File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl return f(*args, **kwargs) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/device_utils.py", line 833, in WaitUntilFullyBooted timeout_retry.WaitFor(pm_ready) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 98, in WaitFor result = condition() File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/device_utils.py", line 817, in pm_ready return self._GetApplicationPathsInternal('android', skip_cache=True) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/device_utils.py", line 664, in _GetApplicationPathsInternal ['pm', 'path', package], check_return=should_check_return) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper return impl() File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl return f(*args, **kwargs) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/device_utils.py", line 1192, in RunShellCommand output = handle_large_output(cmd, large_output) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/device_utils.py", line 1168, in handle_large_output return handle_large_output(cmd, True) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/device_utils.py", line 1157, in handle_large_output handle_large_command(cmd) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/device_utils.py", line 1143, in handle_large_command return handle_check_return(cmd) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/device_utils.py", line 1134, in handle_check_return return run(cmd) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/device_utils.py", line 1130, in run return self.adb.Shell(cmd) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 533, in Shell output = self._RunDeviceAdbCmd(args, timeout, retries, check_error=False) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 316, in _RunDeviceAdbCmd check_error=check_error) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper return impl() File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl return f(*args, **kwargs) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 293, in _RunAdbCmd raise device_errors.DeviceUnreachableError(device_serial) DeviceUnreachableError: 0aa167a825995d01 C 13.606s delete_temporary_file(0aa167a825995d01) (TimeoutThread-1-for-delete_temporary_file(0aa167a825995d01)) Exception on _RunAdbCmd(<class 'devil.android.sdk.adb_wrapper.AdbWrapper'>, ['shell', 'rm -f /data/local/tmp/temp_file-a26dc0f75ed88'], check_error=False, retries=2, device_serial=0aa167a825995d01, timeout=30), attempt 1 of 3: DeviceUnreachableError('0aa167a825995d01',) E 13.611s RecoverDevice(0aa167a825995d01) Failed to reboot 0aa167a825995d01. Traceback (most recent call last): File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/tools/device_recovery.py", line 83, in RecoverDevice device.Reboot(block=False, timeout=5, retries=0) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper retry_if_func=retry_if_func) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 158, in Run error_log_func=error_log_func) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 198, in JoinAll self._JoinAll(watcher, timeout) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 170, in _JoinAll thread.ReraiseIfException() File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 93, in run self._ret = self._func(*self._args, **self._kwargs) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 151, in <lambda> child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs), File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl return f(*args, **kwargs) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/device_utils.py", line 859, in Reboot self.adb.Reboot() File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 913, in Reboot self._RunDeviceAdbCmd(cmd, timeout, retries) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 316, in _RunDeviceAdbCmd check_error=check_error) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper return impl() File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl return f(*args, **kwargs) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 293, in _RunAdbCmd raise device_errors.DeviceUnreachableError(device_serial) DeviceUnreachableError: 0aa167a825995d01 I 13.611s RecoverDevice(0aa167a825995d01) Adding 0aa167a825995d01 to blacklist /b/s/w/ir/cache/builder/src/out/bad_devices.json for reason: reboot_failure C 13.670s delete_temporary_file(0aa167a825995d01) (TimeoutThread-2-for-delete_temporary_file(0aa167a825995d01)) Exception on _RunAdbCmd(<class 'devil.android.sdk.adb_wrapper.AdbWrapper'>, ['shell', 'rm -f /data/local/tmp/temp_file-a26dc0f75ed88'], check_error=False, retries=2, device_serial=0aa167a825995d01, timeout=30), attempt 2 of 3: DeviceUnreachableError('0aa167a825995d01',) W 13.734s delete_temporary_file(0aa167a825995d01) Failed to delete temporary file /data/local/tmp/temp_file-a26dc0f75ed88: 0aa167a825995d01 I 14.686s TimeoutThread-1-for-RecoverDevice(0aa167a825995d01) condition 'sd_card_ready' met (1.1s) I 15.493s TimeoutThread-1-for-RecoverDevice(0aa167a825995d01) condition 'pm_ready' not met (1.9s) I 19.228s TimeoutThread-1-for-RecoverDevice(0acc25be25995b8a) condition 'sd_card_ready' met (10.2s) I 19.419s TimeoutThread-1-for-RecoverDevice(0acca87043e4ba6d) condition 'sd_card_ready' met (10.4s) I 19.855s TimeoutThread-1-for-RecoverDevice(0acc25be25995b8a) condition 'pm_ready' not met (10.9s) I 20.178s TimeoutThread-1-for-RecoverDevice(0acca87043e4ba6d) condition 'pm_ready' not met (11.1s) I 21.109s TimeoutThread-1-for-RecoverDevice(0aa167a825995d01) condition 'pm_ready' met (7.5s) I 21.151s TimeoutThread-1-for-RecoverDevice(0aa167a825995d01) condition 'boot_completed' not met (7.5s) I 21.291s TimeoutThread-1-for-RecoverDevice(06ad9a16003bb372) condition 'sd_card_ready' met (12.2s) I 22.283s TimeoutThread-1-for-RecoverDevice(06ad9a16003bb372) condition 'pm_ready' not met (13.2s) I 25.454s TimeoutThread-1-for-RecoverDevice(0acc25be25995b8a) condition 'pm_ready' not met (16.5s) I 25.807s TimeoutThread-1-for-RecoverDevice(0acca87043e4ba6d) condition 'pm_ready' not met (16.8s) I 26.199s TimeoutThread-1-for-RecoverDevice(0aa167a825995d01) condition 'boot_completed' met (12.6s) I 27.894s TimeoutThread-1-for-RecoverDevice(06ad9a16003bb372) condition 'pm_ready' not met (18.8s) I 31.051s TimeoutThread-1-for-RecoverDevice(0acc25be25995b8a) condition 'pm_ready' met (22.0s) I 31.094s TimeoutThread-1-for-RecoverDevice(0acc25be25995b8a) condition 'boot_completed' met (22.1s) I 31.423s TimeoutThread-1-for-RecoverDevice(0acca87043e4ba6d) condition 'pm_ready' met (22.4s) I 31.464s TimeoutThread-1-for-RecoverDevice(0acca87043e4ba6d) condition 'boot_completed' met (22.4s) I 33.505s TimeoutThread-1-for-RecoverDevice(06ad9a16003bb372) condition 'pm_ready' met (24.4s) I 33.545s TimeoutThread-1-for-RecoverDevice(06ad9a16003bb372) condition 'boot_completed' met (24.5s) E 36.659s RecoverDevice(06c14b05003b815c) Failure while waiting for 06c14b05003b815c. Attempting to recover. Traceback (most recent call last): File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/tools/device_recovery.py", line 75, in RecoverDevice device.WaitUntilFullyBooted(retries=0) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper retry_if_func=retry_if_func) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 158, in Run error_log_func=error_log_func) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 198, in JoinAll self._JoinAll(watcher, timeout) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 170, in _JoinAll thread.ReraiseIfException() File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 93, in run self._ret = self._func(*self._args, **self._kwargs) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 151, in <lambda> child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs), File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl return f(*args, **kwargs) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/device_utils.py", line 831, in WaitUntilFullyBooted self.adb.WaitForDevice() File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 862, in WaitForDevice self._RunDeviceAdbCmd(['wait-for-device'], timeout, retries) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 316, in _RunDeviceAdbCmd check_error=check_error) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper return impl() File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl return f(*args, **kwargs) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 277, in _RunAdbCmd timeout, env=cls._ADB_ENV) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/cmd_helper.py", line 425, in GetCmdStatusAndOutputWithTimeout raise TimeoutError(output.getvalue()) CommandTimeoutError: Timeout E 36.663s RecoverDevice(06a9b452003beda2) Failure while waiting for 06a9b452003beda2. Attempting to recover. Traceback (most recent call last): File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/tools/device_recovery.py", line 75, in RecoverDevice device.WaitUntilFullyBooted(retries=0) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper retry_if_func=retry_if_func) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 158, in Run error_log_func=error_log_func) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 198, in JoinAll self._JoinAll(watcher, timeout) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 170, in _JoinAll thread.ReraiseIfException() File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 93, in run self._ret = self._func(*self._args, **self._kwargs) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 151, in <lambda> child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs), File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl return f(*args, **kwargs) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/device_utils.py", line 831, in WaitUntilFullyBooted self.adb.WaitForDevice() File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 862, in WaitForDevice self._RunDeviceAdbCmd(['wait-for-device'], timeout, retries) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 316, in _RunDeviceAdbCmd check_error=check_error) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper return impl() File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl return f(*args, **kwargs) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 277, in _RunAdbCmd timeout, env=cls._ADB_ENV) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/cmd_helper.py", line 425, in GetCmdStatusAndOutputWithTimeout raise TimeoutError(output.getvalue()) CommandTimeoutError: Timeout E 36.664s RecoverDevice(06b32a05003b9923) Failure while waiting for 06b32a05003b9923. Attempting to recover. Traceback (most recent call last): File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/tools/device_recovery.py", line 75, in RecoverDevice device.WaitUntilFullyBooted(retries=0) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper retry_if_func=retry_if_func) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 158, in Run error_log_func=error_log_func) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 198, in JoinAll self._JoinAll(watcher, timeout) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 170, in _JoinAll thread.ReraiseIfException() File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 93, in run self._ret = self._func(*self._args, **self._kwargs) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 151, in <lambda> child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs), File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl return f(*args, **kwargs) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/device_utils.py", line 831, in WaitUntilFullyBooted self.adb.WaitForDevice() File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 862, in WaitForDevice self._RunDeviceAdbCmd(['wait-for-device'], timeout, retries) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 316, in _RunDeviceAdbCmd check_error=check_error) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper return impl() File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl return f(*args, **kwargs) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 277, in _RunAdbCmd timeout, env=cls._ADB_ENV) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/cmd_helper.py", line 425, in GetCmdStatusAndOutputWithTimeout raise TimeoutError(output.getvalue()) CommandTimeoutError: Timeout E 36.723s RecoverDevice(06c14b05003b815c) Failed to reboot 06c14b05003b815c. Traceback (most recent call last): File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/tools/device_recovery.py", line 83, in RecoverDevice device.Reboot(block=False, timeout=5, retries=0) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper retry_if_func=retry_if_func) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 158, in Run error_log_func=error_log_func) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 198, in JoinAll self._JoinAll(watcher, timeout) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 170, in _JoinAll thread.ReraiseIfException() File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 93, in run self._ret = self._func(*self._args, **self._kwargs) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 151, in <lambda> child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs), File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl return f(*args, **kwargs) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/device_utils.py", line 859, in Reboot self.adb.Reboot() File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 913, in Reboot self._RunDeviceAdbCmd(cmd, timeout, retries) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 316, in _RunDeviceAdbCmd check_error=check_error) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper return impl() File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl return f(*args, **kwargs) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 293, in _RunAdbCmd raise device_errors.DeviceUnreachableError(device_serial) DeviceUnreachableError: 06c14b05003b815c I 36.723s RecoverDevice(06c14b05003b815c) Adding 06c14b05003b815c to blacklist /b/s/w/ir/cache/builder/src/out/bad_devices.json for reason: reboot_failure E 36.727s RecoverDevice(06a9b452003beda2) Failed to reboot 06a9b452003beda2. Traceback (most recent call last): File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/tools/device_recovery.py", line 83, in RecoverDevice device.Reboot(block=False, timeout=5, retries=0) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper retry_if_func=retry_if_func) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 158, in Run error_log_func=error_log_func) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 198, in JoinAll self._JoinAll(watcher, timeout) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 170, in _JoinAll thread.ReraiseIfException() File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 93, in run self._ret = self._func(*self._args, **self._kwargs) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 151, in <lambda> child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs), File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl return f(*args, **kwargs) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/device_utils.py", line 859, in Reboot self.adb.Reboot() File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 913, in Reboot self._RunDeviceAdbCmd(cmd, timeout, retries) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 316, in _RunDeviceAdbCmd check_error=check_error) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper return impl() File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl return f(*args, **kwargs) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 293, in _RunAdbCmd raise device_errors.DeviceUnreachableError(device_serial) DeviceUnreachableError: 06a9b452003beda2 I 36.727s RecoverDevice(06a9b452003beda2) Adding 06a9b452003beda2 to blacklist /b/s/w/ir/cache/builder/src/out/bad_devices.json for reason: reboot_failure E 36.728s RecoverDevice(06b32a05003b9923) Failed to reboot 06b32a05003b9923. Traceback (most recent call last): File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/tools/device_recovery.py", line 83, in RecoverDevice device.Reboot(block=False, timeout=5, retries=0) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper retry_if_func=retry_if_func) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 158, in Run error_log_func=error_log_func) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 198, in JoinAll self._JoinAll(watcher, timeout) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 170, in _JoinAll thread.ReraiseIfException() File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 93, in run self._ret = self._func(*self._args, **self._kwargs) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 151, in <lambda> child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs), File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl return f(*args, **kwargs) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/device_utils.py", line 859, in Reboot self.adb.Reboot() File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 913, in Reboot self._RunDeviceAdbCmd(cmd, timeout, retries) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 316, in _RunDeviceAdbCmd check_error=check_error) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper return impl() File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl return f(*args, **kwargs) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 293, in _RunAdbCmd raise device_errors.DeviceUnreachableError(device_serial) DeviceUnreachableError: 06b32a05003b9923 I 36.728s RecoverDevice(06b32a05003b9923) Adding 06b32a05003b9923 to blacklist /b/s/w/ir/cache/builder/src/out/bad_devices.json for reason: reboot_failure I 96.761s RecoverDevice(06c14b05003b815c) Still working on WaitUntilFullyBooted(06c14b05003b815c, retries=0, timeout=300) I 96.773s RecoverDevice(06a9b452003beda2) Still working on WaitUntilFullyBooted(06a9b452003beda2, retries=0, timeout=300) I 96.790s RecoverDevice(06b32a05003b9923) Still working on WaitUntilFullyBooted(06b32a05003b9923, retries=0, timeout=300) I 156.798s RecoverDevice(06c14b05003b815c) Still working on WaitUntilFullyBooted(06c14b05003b815c, retries=0, timeout=300) I 156.810s RecoverDevice(06a9b452003beda2) Still working on WaitUntilFullyBooted(06a9b452003beda2, retries=0, timeout=300) I 156.827s RecoverDevice(06b32a05003b9923) Still working on WaitUntilFullyBooted(06b32a05003b9923, retries=0, timeout=300) I 216.834s RecoverDevice(06c14b05003b815c) Still working on WaitUntilFullyBooted(06c14b05003b815c, retries=0, timeout=300) I 216.847s RecoverDevice(06a9b452003beda2) Still working on WaitUntilFullyBooted(06a9b452003beda2, retries=0, timeout=300) I 216.864s RecoverDevice(06b32a05003b9923) Still working on WaitUntilFullyBooted(06b32a05003b9923, retries=0, timeout=300) I 276.872s RecoverDevice(06c14b05003b815c) Still working on WaitUntilFullyBooted(06c14b05003b815c, retries=0, timeout=300) I 276.883s RecoverDevice(06a9b452003beda2) Still working on WaitUntilFullyBooted(06a9b452003beda2, retries=0, timeout=300) I 276.901s RecoverDevice(06b32a05003b9923) Still working on WaitUntilFullyBooted(06b32a05003b9923, retries=0, timeout=300) E 322.063s RecoverDevice(06c14b05003b815c) Timed out while waiting for 06c14b05003b815c. Traceback (most recent call last): File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/tools/device_recovery.py", line 111, in RecoverDevice retries=0, timeout=device.REBOOT_DEFAULT_TIMEOUT) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper retry_if_func=retry_if_func) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 158, in Run error_log_func=error_log_func) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 198, in JoinAll self._JoinAll(watcher, timeout) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 170, in _JoinAll thread.ReraiseIfException() File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 93, in run self._ret = self._func(*self._args, **self._kwargs) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 151, in <lambda> child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs), File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl return f(*args, **kwargs) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/device_utils.py", line 831, in WaitUntilFullyBooted self.adb.WaitForDevice() File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 862, in WaitForDevice self._RunDeviceAdbCmd(['wait-for-device'], timeout, retries) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 316, in _RunDeviceAdbCmd check_error=check_error) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper return impl() File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl return f(*args, **kwargs) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 277, in _RunAdbCmd timeout, env=cls._ADB_ENV) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/cmd_helper.py", line 425, in GetCmdStatusAndOutputWithTimeout raise TimeoutError(output.getvalue()) CommandTimeoutError: Timeout I 322.063s RecoverDevice(06c14b05003b815c) Adding 06c14b05003b815c to blacklist /b/s/w/ir/cache/builder/src/out/bad_devices.json for reason: reboot_timeout E 322.074s RecoverDevice(06a9b452003beda2) Timed out while waiting for 06a9b452003beda2. Traceback (most recent call last): File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/tools/device_recovery.py", line 111, in RecoverDevice retries=0, timeout=device.REBOOT_DEFAULT_TIMEOUT) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper retry_if_func=retry_if_func) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 158, in Run error_log_func=error_log_func) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 198, in JoinAll self._JoinAll(watcher, timeout) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 170, in _JoinAll thread.ReraiseIfException() File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 93, in run self._ret = self._func(*self._args, **self._kwargs) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 151, in <lambda> child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs), File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl return f(*args, **kwargs) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/device_utils.py", line 831, in WaitUntilFullyBooted self.adb.WaitForDevice() File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 862, in WaitForDevice self._RunDeviceAdbCmd(['wait-for-device'], timeout, retries) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 316, in _RunDeviceAdbCmd check_error=check_error) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper return impl() File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl return f(*args, **kwargs) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 277, in _RunAdbCmd timeout, env=cls._ADB_ENV) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/cmd_helper.py", line 425, in GetCmdStatusAndOutputWithTimeout raise TimeoutError(output.getvalue()) CommandTimeoutError: Timeout I 322.074s RecoverDevice(06a9b452003beda2) Adding 06a9b452003beda2 to blacklist /b/s/w/ir/cache/builder/src/out/bad_devices.json for reason: reboot_timeout E 322.092s RecoverDevice(06b32a05003b9923) Timed out while waiting for 06b32a05003b9923. Traceback (most recent call last): File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/tools/device_recovery.py", line 111, in RecoverDevice retries=0, timeout=device.REBOOT_DEFAULT_TIMEOUT) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper retry_if_func=retry_if_func) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 158, in Run error_log_func=error_log_func) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 198, in JoinAll self._JoinAll(watcher, timeout) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 170, in _JoinAll thread.ReraiseIfException() File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 93, in run self._ret = self._func(*self._args, **self._kwargs) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 151, in <lambda> child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs), File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl return f(*args, **kwargs) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/device_utils.py", line 831, in WaitUntilFullyBooted self.adb.WaitForDevice() File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 862, in WaitForDevice self._RunDeviceAdbCmd(['wait-for-device'], timeout, retries) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 316, in _RunDeviceAdbCmd check_error=check_error) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper return impl() File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl return f(*args, **kwargs) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 277, in _RunAdbCmd timeout, env=cls._ADB_ENV) File "/b/s/w/ir/cache/builder/src/third_party/catapult/devil/devil/utils/cmd_helper.py", line 425, in GetCmdStatusAndOutputWithTimeout raise TimeoutError(output.getvalue()) CommandTimeoutError: Timeout I 322.092s RecoverDevice(06b32a05003b9923) Adding 06b32a05003b9923 to blacklist /b/s/w/ir/cache/builder/src/out/bad_devices.json for reason: reboot_timeout