Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........................XX........X.X.X..................X] duration: 30.130919505s FAIL (2025-09-22 03:47:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:47:19.113260644 (565h39m39.605693089s) [upload=497648]: Starting upload Sep 22 03:47:19.113327358 (565h39m39.605759806s) [upload=497648]: Done starting upload Sep 22 03:47:19.113519545 (565h39m39.605951979s) [upload=497648]: Uploading segments... Sep 22 03:47:19.113758321 (565h39m39.606190758s) [upload=497648]: Got next segment. Inline: false Sep 22 03:47:19.113807108 (565h39m39.606239542s) [upload=497648]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:47:19.113812587 (565h39m39.606245029s) [upload=497648]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:47:19.291699841 (565h39m39.784132286s) [seg_pos={0 0}, upload=497648]: Begin upload segment... Sep 22 03:47:19.291780171 (565h39m39.784212613s) [seg_pos={0 0}, upload=497648]: Done begin upload segment. Sep 22 03:47:19.291799704 (565h39m39.784232151s) [noise=true , address=45.91.116.10:4082 , piece_id=3WKYBWDBVQZW6YRS73OVBS4PFNBZUHLJDBQ4AWR3S5JQ3KF26SKQ, seg_pos={0 0}, upload=497648]: Uploading piece... Sep 22 03:47:19.291810667 (565h39m39.784243114s) [noise=true , address=45.91.116.13:4153 , piece_id=ZNANBZKTJYBN5SZ7TCXAYTFYUIE2B3A4WXIVNL6FQRR2QBAKUIVQ, seg_pos={0 0}, upload=497648]: Uploading piece... Sep 22 03:47:19.292481936 (565h39m39.784914388s) [noise=true , address=45.91.116.13:4155 , piece_id=H4OJRSOA7NOM2HNNKQ4XB4O4VFR5GXL6CNF6TNWXEBMU7CEA5CQQ, seg_pos={0 0}, upload=497648]: Uploading piece... Sep 22 03:47:19.292766290 (565h39m39.785198737s) [noise=true , address=45.91.116.9:4262 , piece_id=ZV3WB7ZIZN45AIWIG6RWL5UX2TXVGNMBDI3Q2Z77X452FK6R7INA, seg_pos={0 0}, upload=497648]: Uploading piece... Sep 22 03:47:19.293122951 (565h39m39.785555394s) [noise=true , address=45.91.116.9:4261 , piece_id=OPC57SHPLSYVQAUI2NTQR3TW3VE7YSUYR75LTITVICFB5AG2Q2XQ, seg_pos={0 0}, upload=497648]: Uploading piece... Sep 22 03:47:19.293192250 (565h39m39.785624695s) [noise=true , address=77.90.31.16:4733 , piece_id=BRZKGDY7NC6XCDGBMZUCFQ2THTNSTXFP244RDYWM56N6NTJMXBNA, seg_pos={0 0}, upload=497648]: Uploading piece... Sep 22 03:47:19.293665833 (565h39m39.786098276s) [noise=true , address=145.79.227.15:4874 , piece_id=MD22P4HUKA2FNC3CBOHWLIRA5FTYBAMXO6CJ6CMZLRNDV2CT3P5Q, seg_pos={0 0}, upload=497648]: Uploading piece... Sep 22 03:47:19.293718852 (565h39m39.786151303s) [noise=true , address=145.79.227.6:4258 , piece_id=2PJFMC66DGNHOX462YPRLWFIXMDAHM5OCLFTQBOMMVUDZO4ICHCQ, seg_pos={0 0}, upload=497648]: Uploading piece... Sep 22 03:47:19.294198102 (565h39m39.786630545s) [noise=true , address=45.91.116.5:4298 , piece_id=7SCDSYIC5SFOGISRFJRVIHLC7IOXUW3G2LUCHN43GLRJ4CE6WGWA, seg_pos={0 0}, upload=497648]: Uploading piece... Sep 22 03:47:19.294390119 (565h39m39.786822561s) [noise=true , address=45.91.116.6:4975 , piece_id=HJLWUPSDJUTYJJVPST33RFRDNUQEURLDLXYYXRJNB4D7OIEU3C3Q, seg_pos={0 0}, upload=497648]: Uploading piece... Sep 22 03:47:19.294560635 (565h39m39.786993078s) [noise=true , address=145.79.227.7:4295 , piece_id=GMB6XEKOB5BYJLSZN7F3PMUQWTXZVFAPC35X4EYHKMOQK6D6X2JA, seg_pos={0 0}, upload=497648]: Uploading piece... Sep 22 03:47:19.294969814 (565h39m39.787402256s) [noise=true , address=145.79.227.14:4840 , piece_id=W754XUR4WOJTLXEAX4OAJYXDWNJTCXH6G6RD7OCLVORJXILRIFAA, seg_pos={0 0}, upload=497648]: Uploading piece... Sep 22 03:47:19.295343832 (565h39m39.787776289s) [noise=true , address=77.90.31.17:4771 , piece_id=LNYVTFUYRRH3FLS5TY5W3SCX4JZHXCOG2HEUWLCBATEDSVIHCZ3Q, seg_pos={0 0}, upload=497648]: Uploading piece... Sep 22 03:47:19.295711740 (565h39m39.788144185s) [noise=true , address=45.91.116.11:4119 , piece_id=XLXQUF363RSR5DQYCRRO2RBFMK6NFJI2E4V3GBFIDONI4TOEJNGA, seg_pos={0 0}, upload=497648]: Uploading piece... Sep 22 03:47:19.296047129 (565h39m39.788479572s) [noise=true , address=45.91.116.5:4297 , piece_id=JKQ33Q2IAQVK7XDCVM2DR7XKG7OCHNBXRO7SIQR7BBKYCZFNZLPQ, seg_pos={0 0}, upload=497648]: Uploading piece... Sep 22 03:47:19.296305630 (565h39m39.788738064s) [noise=true , address=77.90.31.10:4517 , piece_id=IC6CZORXKKLJJA4L2PUQ5L6CC4P34SB4TBBSWZUHXOAKR5DWSRZA, seg_pos={0 0}, upload=497648]: Uploading piece... Sep 22 03:47:19.901856308 (565h39m40.394288754s) [upload=497648]: Next returned nil segment Sep 22 03:47:19.901877862 (565h39m40.394310308s) [upload=497648]: Waiting for error group managing segments... Sep 22 03:47:20.037832946 (565h39m40.530265383s) [noise=true , address=77.90.31.16:4733 , piece_id=BRZKGDY7NC6XCDGBMZUCFQ2THTNSTXFP244RDYWM56N6NTJMXBNA, seg_pos={0 0}, upload=497648]: Done uploading piece. err: <nil> Sep 22 03:47:20.038248106 (565h39m40.530680551s) [noise=true , address=45.91.116.9:4262 , piece_id=ZV3WB7ZIZN45AIWIG6RWL5UX2TXVGNMBDI3Q2Z77X452FK6R7INA, seg_pos={0 0}, upload=497648]: Done uploading piece. err: <nil> Sep 22 03:47:20.039735833 (565h39m40.532168276s) [noise=true , address=45.91.116.9:4261 , piece_id=OPC57SHPLSYVQAUI2NTQR3TW3VE7YSUYR75LTITVICFB5AG2Q2XQ, seg_pos={0 0}, upload=497648]: Done uploading piece. err: <nil> Sep 22 03:47:20.040427428 (565h39m40.532859879s) [noise=true , address=45.91.116.5:4297 , piece_id=JKQ33Q2IAQVK7XDCVM2DR7XKG7OCHNBXRO7SIQR7BBKYCZFNZLPQ, seg_pos={0 0}, upload=497648]: Done uploading piece. err: <nil> Sep 22 03:47:20.041032921 (565h39m40.533465365s) [noise=true , address=77.90.31.10:4517 , piece_id=IC6CZORXKKLJJA4L2PUQ5L6CC4P34SB4TBBSWZUHXOAKR5DWSRZA, seg_pos={0 0}, upload=497648]: Done uploading piece. err: <nil> Sep 22 03:47:20.041562800 (565h39m40.533995233s) [noise=true , address=45.91.116.10:4082 , piece_id=3WKYBWDBVQZW6YRS73OVBS4PFNBZUHLJDBQ4AWR3S5JQ3KF26SKQ, seg_pos={0 0}, upload=497648]: Done uploading piece. err: <nil> Sep 22 03:47:20.042262927 (565h39m40.534695362s) [noise=true , address=45.91.116.6:4975 , piece_id=HJLWUPSDJUTYJJVPST33RFRDNUQEURLDLXYYXRJNB4D7OIEU3C3Q, seg_pos={0 0}, upload=497648]: Done uploading piece. err: <nil> Sep 22 03:47:20.042448400 (565h39m40.534880835s) [noise=true , address=45.91.116.13:4153 , piece_id=ZNANBZKTJYBN5SZ7TCXAYTFYUIE2B3A4WXIVNL6FQRR2QBAKUIVQ, seg_pos={0 0}, upload=497648]: Done uploading piece. err: <nil> Sep 22 03:47:20.043333560 (565h39m40.535766s) [noise=true , address=45.91.116.11:4119 , piece_id=XLXQUF363RSR5DQYCRRO2RBFMK6NFJI2E4V3GBFIDONI4TOEJNGA, seg_pos={0 0}, upload=497648]: Done uploading piece. err: <nil> Sep 22 03:47:20.050574755 (565h39m40.543007186s) [noise=true , address=45.91.116.13:4155 , piece_id=H4OJRSOA7NOM2HNNKQ4XB4O4VFR5GXL6CNF6TNWXEBMU7CEA5CQQ, seg_pos={0 0}, upload=497648]: Done uploading piece. err: <nil> Sep 22 03:47:20.054036721 (565h39m40.546469167s) [noise=true , address=45.91.116.5:4298 , piece_id=7SCDSYIC5SFOGISRFJRVIHLC7IOXUW3G2LUCHN43GLRJ4CE6WGWA, seg_pos={0 0}, upload=497648]: Done uploading piece. err: <nil> Sep 22 03:47:20.057760534 (565h39m40.550192984s) [noise=true , address=77.90.31.17:4771 , piece_id=LNYVTFUYRRH3FLS5TY5W3SCX4JZHXCOG2HEUWLCBATEDSVIHCZ3Q, seg_pos={0 0}, upload=497648]: Done uploading piece. err: <nil> Sep 22 03:47:48.801625731 (565h40m9.294058165s) [noise=true , address=145.79.227.15:4874 , piece_id=MD22P4HUKA2FNC3CBOHWLIRA5FTYBAMXO6CJ6CMZLRNDV2CT3P5Q, seg_pos={0 0}, upload=497648]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:47:48.801653583 (565h40m9.294086018s) [noise=true , address=145.79.227.6:4258 , piece_id=2PJFMC66DGNHOX462YPRLWFIXMDAHM5OCLFTQBOMMVUDZO4ICHCQ, seg_pos={0 0}, upload=497648]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: rpc: dial tcp 145.79.227.6:4258: i/o timeout; conn closed: op Sep 22 03:47:48.801767497 (565h40m9.294199939s) [noise=true , address=145.79.227.14:4840 , piece_id=W754XUR4WOJTLXEAX4OAJYXDWNJTCXH6G6RD7OCLVORJXILRIFAA, seg_pos={0 0}, upload=497648]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; context deadline exceeded Sep 22 03:47:48.801813396 (565h40m9.294245838s) [noise=true , address=145.79.227.7:4295 , piece_id=GMB6XEKOB5BYJLSZN7F3PMUQWTXZVFAPC35X4EYHKMOQK6D6X2JA, seg_pos={0 0}, upload=497648]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:47:48.801869087 (565h40m9.29430154s) [seg_pos={0 0}, upload=497648]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:47:48.930686637 (565h40m9.423119077s) [upload=497648]: Done uploading segments... Sep 22 03:47:48.930725302 (565h40m9.423157751s) [upload=497648]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.130956252s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..X.X.......XX....X..X...X.......X...XX..........X.......X..] duration: 2.66741156s FAIL (2025-09-22 03:47:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.667501874s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........................XX........X.X.X..................] duration: 2.409965723s FAIL (2025-09-22 03:46:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:33:59.485342710 (565h26m19.977775146s) [upload=497444]: Starting upload Sep 22 03:33:59.485420892 (565h26m19.977853329s) [upload=497444]: Done starting upload Sep 22 03:33:59.485539660 (565h26m19.977972092s) [upload=497444]: Uploading segments... Sep 22 03:33:59.485714076 (565h26m19.978146509s) [upload=497444]: Got next segment. Inline: false Sep 22 03:33:59.485813900 (565h26m19.978246333s) [upload=497444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:33:59.485821999 (565h26m19.978254434s) [upload=497444]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:33:59.595141740 (565h26m20.087574174s) [seg_pos={0 0}, upload=497444]: Begin upload segment... Sep 22 03:33:59.595226098 (565h26m20.087658531s) [seg_pos={0 0}, upload=497444]: Done begin upload segment. Sep 22 03:33:59.595253117 (565h26m20.087685562s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.595807440 (565h26m20.088239889s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596269486 (565h26m20.088701937s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596610761 (565h26m20.089043209s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596697470 (565h26m20.089129917s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.597355109 (565h26m20.089787556s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598119615 (565h26m20.090552066s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598953736 (565h26m20.091386188s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599655367 (565h26m20.092087809s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599993860 (565h26m20.092426306s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.600640516 (565h26m20.093072958s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601107222 (565h26m20.093539671s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601842149 (565h26m20.0942746s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.602793768 (565h26m20.095226222s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603229497 (565h26m20.095661946s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603695255 (565h26m20.096127703s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:34:00.229252838 (565h26m20.721685298s) [upload=497444]: Next returned nil segment Sep 22 03:34:00.229281454 (565h26m20.72171391s) [upload=497444]: Waiting for error group managing segments... Sep 22 03:34:00.367601956 (565h26m20.860034403s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.368894773 (565h26m20.861327222s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.391796558 (565h26m20.884228999s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393337713 (565h26m20.885770148s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393367500 (565h26m20.885799936s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394210027 (565h26m20.886642463s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394934320 (565h26m20.887366757s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396404984 (565h26m20.888837431s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396633830 (565h26m20.889066265s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.397583959 (565h26m20.8900164s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420259441 (565h26m20.9126919s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420371917 (565h26m20.912804356s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:29.213490723 (565h26m49.705923175s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213577765 (565h26m49.706010194s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213714723 (565h26m49.706147169s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213752753 (565h26m49.7061852s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213811756 (565h26m49.706244208s) [seg_pos={0 0}, upload=497444]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:34:29.316998815 (565h26m49.809431249s) [upload=497444]: Done uploading segments... Sep 22 03:34:29.317035357 (565h26m49.809467809s) [upload=497444]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.409996602s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...X.X.......XX....X..X...X.......X...XX..........X.......X.] duration: 2.801045031s FAIL (2025-09-22 03:46:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.801108245s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............................XX........X.X.X.................] duration: 2.384054166s FAIL (2025-09-22 03:46:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:33:59.485342710 (565h26m19.977775146s) [upload=497444]: Starting upload Sep 22 03:33:59.485420892 (565h26m19.977853329s) [upload=497444]: Done starting upload Sep 22 03:33:59.485539660 (565h26m19.977972092s) [upload=497444]: Uploading segments... Sep 22 03:33:59.485714076 (565h26m19.978146509s) [upload=497444]: Got next segment. Inline: false Sep 22 03:33:59.485813900 (565h26m19.978246333s) [upload=497444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:33:59.485821999 (565h26m19.978254434s) [upload=497444]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:33:59.595141740 (565h26m20.087574174s) [seg_pos={0 0}, upload=497444]: Begin upload segment... Sep 22 03:33:59.595226098 (565h26m20.087658531s) [seg_pos={0 0}, upload=497444]: Done begin upload segment. Sep 22 03:33:59.595253117 (565h26m20.087685562s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.595807440 (565h26m20.088239889s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596269486 (565h26m20.088701937s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596610761 (565h26m20.089043209s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596697470 (565h26m20.089129917s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.597355109 (565h26m20.089787556s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598119615 (565h26m20.090552066s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598953736 (565h26m20.091386188s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599655367 (565h26m20.092087809s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599993860 (565h26m20.092426306s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.600640516 (565h26m20.093072958s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601107222 (565h26m20.093539671s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601842149 (565h26m20.0942746s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.602793768 (565h26m20.095226222s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603229497 (565h26m20.095661946s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603695255 (565h26m20.096127703s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:34:00.229252838 (565h26m20.721685298s) [upload=497444]: Next returned nil segment Sep 22 03:34:00.229281454 (565h26m20.72171391s) [upload=497444]: Waiting for error group managing segments... Sep 22 03:34:00.367601956 (565h26m20.860034403s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.368894773 (565h26m20.861327222s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.391796558 (565h26m20.884228999s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393337713 (565h26m20.885770148s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393367500 (565h26m20.885799936s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394210027 (565h26m20.886642463s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394934320 (565h26m20.887366757s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396404984 (565h26m20.888837431s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396633830 (565h26m20.889066265s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.397583959 (565h26m20.8900164s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420259441 (565h26m20.9126919s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420371917 (565h26m20.912804356s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:29.213490723 (565h26m49.705923175s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213577765 (565h26m49.706010194s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213714723 (565h26m49.706147169s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213752753 (565h26m49.7061852s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213811756 (565h26m49.706244208s) [seg_pos={0 0}, upload=497444]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:34:29.316998815 (565h26m49.809431249s) [upload=497444]: Done uploading segments... Sep 22 03:34:29.317035357 (565h26m49.809467809s) [upload=497444]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.384113627s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....X.X.......XX....X..X...X.......X...XX..........X.......X] duration: 30.001651135s FAIL (2025-09-22 03:45:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002375308s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............................XX........X.X.X................] duration: 2.540522339s FAIL (2025-09-22 03:45:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:33:59.485342710 (565h26m19.977775146s) [upload=497444]: Starting upload Sep 22 03:33:59.485420892 (565h26m19.977853329s) [upload=497444]: Done starting upload Sep 22 03:33:59.485539660 (565h26m19.977972092s) [upload=497444]: Uploading segments... Sep 22 03:33:59.485714076 (565h26m19.978146509s) [upload=497444]: Got next segment. Inline: false Sep 22 03:33:59.485813900 (565h26m19.978246333s) [upload=497444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:33:59.485821999 (565h26m19.978254434s) [upload=497444]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:33:59.595141740 (565h26m20.087574174s) [seg_pos={0 0}, upload=497444]: Begin upload segment... Sep 22 03:33:59.595226098 (565h26m20.087658531s) [seg_pos={0 0}, upload=497444]: Done begin upload segment. Sep 22 03:33:59.595253117 (565h26m20.087685562s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.595807440 (565h26m20.088239889s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596269486 (565h26m20.088701937s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596610761 (565h26m20.089043209s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596697470 (565h26m20.089129917s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.597355109 (565h26m20.089787556s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598119615 (565h26m20.090552066s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598953736 (565h26m20.091386188s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599655367 (565h26m20.092087809s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599993860 (565h26m20.092426306s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.600640516 (565h26m20.093072958s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601107222 (565h26m20.093539671s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601842149 (565h26m20.0942746s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.602793768 (565h26m20.095226222s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603229497 (565h26m20.095661946s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603695255 (565h26m20.096127703s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:34:00.229252838 (565h26m20.721685298s) [upload=497444]: Next returned nil segment Sep 22 03:34:00.229281454 (565h26m20.72171391s) [upload=497444]: Waiting for error group managing segments... Sep 22 03:34:00.367601956 (565h26m20.860034403s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.368894773 (565h26m20.861327222s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.391796558 (565h26m20.884228999s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393337713 (565h26m20.885770148s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393367500 (565h26m20.885799936s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394210027 (565h26m20.886642463s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394934320 (565h26m20.887366757s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396404984 (565h26m20.888837431s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396633830 (565h26m20.889066265s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.397583959 (565h26m20.8900164s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420259441 (565h26m20.9126919s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420371917 (565h26m20.912804356s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:29.213490723 (565h26m49.705923175s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213577765 (565h26m49.706010194s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213714723 (565h26m49.706147169s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213752753 (565h26m49.7061852s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213811756 (565h26m49.706244208s) [seg_pos={0 0}, upload=497444]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:34:29.316998815 (565h26m49.809431249s) [upload=497444]: Done uploading segments... Sep 22 03:34:29.317035357 (565h26m49.809467809s) [upload=497444]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.540574361s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X....X.X.......XX....X..X...X.......X...XX..........X.......] duration: 2.899749484s FAIL (2025-09-22 03:44:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.899783221s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............................XX........X.X.X...............] duration: 2.369704413s FAIL (2025-09-22 03:44:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:33:59.485342710 (565h26m19.977775146s) [upload=497444]: Starting upload Sep 22 03:33:59.485420892 (565h26m19.977853329s) [upload=497444]: Done starting upload Sep 22 03:33:59.485539660 (565h26m19.977972092s) [upload=497444]: Uploading segments... Sep 22 03:33:59.485714076 (565h26m19.978146509s) [upload=497444]: Got next segment. Inline: false Sep 22 03:33:59.485813900 (565h26m19.978246333s) [upload=497444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:33:59.485821999 (565h26m19.978254434s) [upload=497444]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:33:59.595141740 (565h26m20.087574174s) [seg_pos={0 0}, upload=497444]: Begin upload segment... Sep 22 03:33:59.595226098 (565h26m20.087658531s) [seg_pos={0 0}, upload=497444]: Done begin upload segment. Sep 22 03:33:59.595253117 (565h26m20.087685562s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.595807440 (565h26m20.088239889s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596269486 (565h26m20.088701937s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596610761 (565h26m20.089043209s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596697470 (565h26m20.089129917s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.597355109 (565h26m20.089787556s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598119615 (565h26m20.090552066s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598953736 (565h26m20.091386188s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599655367 (565h26m20.092087809s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599993860 (565h26m20.092426306s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.600640516 (565h26m20.093072958s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601107222 (565h26m20.093539671s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601842149 (565h26m20.0942746s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.602793768 (565h26m20.095226222s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603229497 (565h26m20.095661946s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603695255 (565h26m20.096127703s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:34:00.229252838 (565h26m20.721685298s) [upload=497444]: Next returned nil segment Sep 22 03:34:00.229281454 (565h26m20.72171391s) [upload=497444]: Waiting for error group managing segments... Sep 22 03:34:00.367601956 (565h26m20.860034403s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.368894773 (565h26m20.861327222s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.391796558 (565h26m20.884228999s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393337713 (565h26m20.885770148s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393367500 (565h26m20.885799936s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394210027 (565h26m20.886642463s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394934320 (565h26m20.887366757s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396404984 (565h26m20.888837431s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396633830 (565h26m20.889066265s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.397583959 (565h26m20.8900164s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420259441 (565h26m20.9126919s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420371917 (565h26m20.912804356s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:29.213490723 (565h26m49.705923175s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213577765 (565h26m49.706010194s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213714723 (565h26m49.706147169s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213752753 (565h26m49.7061852s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213811756 (565h26m49.706244208s) [seg_pos={0 0}, upload=497444]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:34:29.316998815 (565h26m49.809431249s) [upload=497444]: Done uploading segments... Sep 22 03:34:29.317035357 (565h26m49.809467809s) [upload=497444]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.369794614s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............................XX........X.X.X..............] duration: 2.641155982s FAIL (2025-09-22 03:44:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:33:59.485342710 (565h26m19.977775146s) [upload=497444]: Starting upload Sep 22 03:33:59.485420892 (565h26m19.977853329s) [upload=497444]: Done starting upload Sep 22 03:33:59.485539660 (565h26m19.977972092s) [upload=497444]: Uploading segments... Sep 22 03:33:59.485714076 (565h26m19.978146509s) [upload=497444]: Got next segment. Inline: false Sep 22 03:33:59.485813900 (565h26m19.978246333s) [upload=497444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:33:59.485821999 (565h26m19.978254434s) [upload=497444]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:33:59.595141740 (565h26m20.087574174s) [seg_pos={0 0}, upload=497444]: Begin upload segment... Sep 22 03:33:59.595226098 (565h26m20.087658531s) [seg_pos={0 0}, upload=497444]: Done begin upload segment. Sep 22 03:33:59.595253117 (565h26m20.087685562s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.595807440 (565h26m20.088239889s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596269486 (565h26m20.088701937s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596610761 (565h26m20.089043209s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596697470 (565h26m20.089129917s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.597355109 (565h26m20.089787556s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598119615 (565h26m20.090552066s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598953736 (565h26m20.091386188s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599655367 (565h26m20.092087809s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599993860 (565h26m20.092426306s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.600640516 (565h26m20.093072958s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601107222 (565h26m20.093539671s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601842149 (565h26m20.0942746s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.602793768 (565h26m20.095226222s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603229497 (565h26m20.095661946s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603695255 (565h26m20.096127703s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:34:00.229252838 (565h26m20.721685298s) [upload=497444]: Next returned nil segment Sep 22 03:34:00.229281454 (565h26m20.72171391s) [upload=497444]: Waiting for error group managing segments... Sep 22 03:34:00.367601956 (565h26m20.860034403s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.368894773 (565h26m20.861327222s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.391796558 (565h26m20.884228999s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393337713 (565h26m20.885770148s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393367500 (565h26m20.885799936s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394210027 (565h26m20.886642463s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394934320 (565h26m20.887366757s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396404984 (565h26m20.888837431s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396633830 (565h26m20.889066265s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.397583959 (565h26m20.8900164s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420259441 (565h26m20.9126919s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420371917 (565h26m20.912804356s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:29.213490723 (565h26m49.705923175s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213577765 (565h26m49.706010194s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213714723 (565h26m49.706147169s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213752753 (565h26m49.7061852s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213811756 (565h26m49.706244208s) [seg_pos={0 0}, upload=497444]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:34:29.316998815 (565h26m49.809431249s) [upload=497444]: Done uploading segments... Sep 22 03:34:29.317035357 (565h26m49.809467809s) [upload=497444]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.641192895s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X....X.X.......XX....X..X...X.......X...XX..........X......] duration: 3.099630274s FAIL (2025-09-22 03:43:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.099658232s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..X....X.X.......XX....X..X...X.......X...XX..........X.....] duration: 2.664363935s FAIL (2025-09-22 03:43:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.664611915s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [................................XX........X.X.X.............] duration: 2.493317064s FAIL (2025-09-22 03:43:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:33:59.485342710 (565h26m19.977775146s) [upload=497444]: Starting upload Sep 22 03:33:59.485420892 (565h26m19.977853329s) [upload=497444]: Done starting upload Sep 22 03:33:59.485539660 (565h26m19.977972092s) [upload=497444]: Uploading segments... Sep 22 03:33:59.485714076 (565h26m19.978146509s) [upload=497444]: Got next segment. Inline: false Sep 22 03:33:59.485813900 (565h26m19.978246333s) [upload=497444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:33:59.485821999 (565h26m19.978254434s) [upload=497444]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:33:59.595141740 (565h26m20.087574174s) [seg_pos={0 0}, upload=497444]: Begin upload segment... Sep 22 03:33:59.595226098 (565h26m20.087658531s) [seg_pos={0 0}, upload=497444]: Done begin upload segment. Sep 22 03:33:59.595253117 (565h26m20.087685562s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.595807440 (565h26m20.088239889s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596269486 (565h26m20.088701937s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596610761 (565h26m20.089043209s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596697470 (565h26m20.089129917s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.597355109 (565h26m20.089787556s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598119615 (565h26m20.090552066s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598953736 (565h26m20.091386188s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599655367 (565h26m20.092087809s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599993860 (565h26m20.092426306s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.600640516 (565h26m20.093072958s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601107222 (565h26m20.093539671s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601842149 (565h26m20.0942746s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.602793768 (565h26m20.095226222s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603229497 (565h26m20.095661946s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603695255 (565h26m20.096127703s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:34:00.229252838 (565h26m20.721685298s) [upload=497444]: Next returned nil segment Sep 22 03:34:00.229281454 (565h26m20.72171391s) [upload=497444]: Waiting for error group managing segments... Sep 22 03:34:00.367601956 (565h26m20.860034403s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.368894773 (565h26m20.861327222s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.391796558 (565h26m20.884228999s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393337713 (565h26m20.885770148s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393367500 (565h26m20.885799936s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394210027 (565h26m20.886642463s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394934320 (565h26m20.887366757s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396404984 (565h26m20.888837431s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396633830 (565h26m20.889066265s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.397583959 (565h26m20.8900164s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420259441 (565h26m20.9126919s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420371917 (565h26m20.912804356s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:29.213490723 (565h26m49.705923175s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213577765 (565h26m49.706010194s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213714723 (565h26m49.706147169s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213752753 (565h26m49.7061852s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213811756 (565h26m49.706244208s) [seg_pos={0 0}, upload=497444]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:34:29.316998815 (565h26m49.809431249s) [upload=497444]: Done uploading segments... Sep 22 03:34:29.317035357 (565h26m49.809467809s) [upload=497444]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.493346211s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...X....X.X.......XX....X..X...X.......X...XX..........X....] duration: 3.091904742s FAIL (2025-09-22 03:42:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.091923321s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................XX........X.X.X............] duration: 2.69495796s FAIL (2025-09-22 03:42:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:33:59.485342710 (565h26m19.977775146s) [upload=497444]: Starting upload Sep 22 03:33:59.485420892 (565h26m19.977853329s) [upload=497444]: Done starting upload Sep 22 03:33:59.485539660 (565h26m19.977972092s) [upload=497444]: Uploading segments... Sep 22 03:33:59.485714076 (565h26m19.978146509s) [upload=497444]: Got next segment. Inline: false Sep 22 03:33:59.485813900 (565h26m19.978246333s) [upload=497444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:33:59.485821999 (565h26m19.978254434s) [upload=497444]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:33:59.595141740 (565h26m20.087574174s) [seg_pos={0 0}, upload=497444]: Begin upload segment... Sep 22 03:33:59.595226098 (565h26m20.087658531s) [seg_pos={0 0}, upload=497444]: Done begin upload segment. Sep 22 03:33:59.595253117 (565h26m20.087685562s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.595807440 (565h26m20.088239889s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596269486 (565h26m20.088701937s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596610761 (565h26m20.089043209s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596697470 (565h26m20.089129917s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.597355109 (565h26m20.089787556s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598119615 (565h26m20.090552066s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598953736 (565h26m20.091386188s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599655367 (565h26m20.092087809s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599993860 (565h26m20.092426306s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.600640516 (565h26m20.093072958s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601107222 (565h26m20.093539671s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601842149 (565h26m20.0942746s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.602793768 (565h26m20.095226222s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603229497 (565h26m20.095661946s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603695255 (565h26m20.096127703s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:34:00.229252838 (565h26m20.721685298s) [upload=497444]: Next returned nil segment Sep 22 03:34:00.229281454 (565h26m20.72171391s) [upload=497444]: Waiting for error group managing segments... Sep 22 03:34:00.367601956 (565h26m20.860034403s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.368894773 (565h26m20.861327222s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.391796558 (565h26m20.884228999s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393337713 (565h26m20.885770148s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393367500 (565h26m20.885799936s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394210027 (565h26m20.886642463s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394934320 (565h26m20.887366757s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396404984 (565h26m20.888837431s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396633830 (565h26m20.889066265s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.397583959 (565h26m20.8900164s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420259441 (565h26m20.9126919s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420371917 (565h26m20.912804356s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:29.213490723 (565h26m49.705923175s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213577765 (565h26m49.706010194s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213714723 (565h26m49.706147169s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213752753 (565h26m49.7061852s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213811756 (565h26m49.706244208s) [seg_pos={0 0}, upload=497444]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:34:29.316998815 (565h26m49.809431249s) [upload=497444]: Done uploading segments... Sep 22 03:34:29.317035357 (565h26m49.809467809s) [upload=497444]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.694995161s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....X....X.X.......XX....X..X...X.......X...XX..........X...] duration: 2.782739683s FAIL (2025-09-22 03:41:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.782835383s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................XX........X.X.X...........] duration: 2.496086209s FAIL (2025-09-22 03:41:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:33:59.485342710 (565h26m19.977775146s) [upload=497444]: Starting upload Sep 22 03:33:59.485420892 (565h26m19.977853329s) [upload=497444]: Done starting upload Sep 22 03:33:59.485539660 (565h26m19.977972092s) [upload=497444]: Uploading segments... Sep 22 03:33:59.485714076 (565h26m19.978146509s) [upload=497444]: Got next segment. Inline: false Sep 22 03:33:59.485813900 (565h26m19.978246333s) [upload=497444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:33:59.485821999 (565h26m19.978254434s) [upload=497444]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:33:59.595141740 (565h26m20.087574174s) [seg_pos={0 0}, upload=497444]: Begin upload segment... Sep 22 03:33:59.595226098 (565h26m20.087658531s) [seg_pos={0 0}, upload=497444]: Done begin upload segment. Sep 22 03:33:59.595253117 (565h26m20.087685562s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.595807440 (565h26m20.088239889s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596269486 (565h26m20.088701937s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596610761 (565h26m20.089043209s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596697470 (565h26m20.089129917s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.597355109 (565h26m20.089787556s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598119615 (565h26m20.090552066s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598953736 (565h26m20.091386188s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599655367 (565h26m20.092087809s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599993860 (565h26m20.092426306s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.600640516 (565h26m20.093072958s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601107222 (565h26m20.093539671s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601842149 (565h26m20.0942746s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.602793768 (565h26m20.095226222s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603229497 (565h26m20.095661946s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603695255 (565h26m20.096127703s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:34:00.229252838 (565h26m20.721685298s) [upload=497444]: Next returned nil segment Sep 22 03:34:00.229281454 (565h26m20.72171391s) [upload=497444]: Waiting for error group managing segments... Sep 22 03:34:00.367601956 (565h26m20.860034403s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.368894773 (565h26m20.861327222s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.391796558 (565h26m20.884228999s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393337713 (565h26m20.885770148s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393367500 (565h26m20.885799936s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394210027 (565h26m20.886642463s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394934320 (565h26m20.887366757s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396404984 (565h26m20.888837431s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396633830 (565h26m20.889066265s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.397583959 (565h26m20.8900164s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420259441 (565h26m20.9126919s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420371917 (565h26m20.912804356s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:29.213490723 (565h26m49.705923175s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213577765 (565h26m49.706010194s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213714723 (565h26m49.706147169s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213752753 (565h26m49.7061852s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213811756 (565h26m49.706244208s) [seg_pos={0 0}, upload=497444]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:34:29.316998815 (565h26m49.809431249s) [upload=497444]: Done uploading segments... Sep 22 03:34:29.317035357 (565h26m49.809467809s) [upload=497444]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.496114608s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X..................................XX........X.X.X..........] duration: 2.422737916s FAIL (2025-09-22 03:41:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:33:59.485342710 (565h26m19.977775146s) [upload=497444]: Starting upload Sep 22 03:33:59.485420892 (565h26m19.977853329s) [upload=497444]: Done starting upload Sep 22 03:33:59.485539660 (565h26m19.977972092s) [upload=497444]: Uploading segments... Sep 22 03:33:59.485714076 (565h26m19.978146509s) [upload=497444]: Got next segment. Inline: false Sep 22 03:33:59.485813900 (565h26m19.978246333s) [upload=497444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:33:59.485821999 (565h26m19.978254434s) [upload=497444]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:33:59.595141740 (565h26m20.087574174s) [seg_pos={0 0}, upload=497444]: Begin upload segment... Sep 22 03:33:59.595226098 (565h26m20.087658531s) [seg_pos={0 0}, upload=497444]: Done begin upload segment. Sep 22 03:33:59.595253117 (565h26m20.087685562s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.595807440 (565h26m20.088239889s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596269486 (565h26m20.088701937s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596610761 (565h26m20.089043209s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596697470 (565h26m20.089129917s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.597355109 (565h26m20.089787556s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598119615 (565h26m20.090552066s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598953736 (565h26m20.091386188s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599655367 (565h26m20.092087809s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599993860 (565h26m20.092426306s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.600640516 (565h26m20.093072958s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601107222 (565h26m20.093539671s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601842149 (565h26m20.0942746s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.602793768 (565h26m20.095226222s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603229497 (565h26m20.095661946s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603695255 (565h26m20.096127703s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:34:00.229252838 (565h26m20.721685298s) [upload=497444]: Next returned nil segment Sep 22 03:34:00.229281454 (565h26m20.72171391s) [upload=497444]: Waiting for error group managing segments... Sep 22 03:34:00.367601956 (565h26m20.860034403s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.368894773 (565h26m20.861327222s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.391796558 (565h26m20.884228999s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393337713 (565h26m20.885770148s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393367500 (565h26m20.885799936s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394210027 (565h26m20.886642463s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394934320 (565h26m20.887366757s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396404984 (565h26m20.888837431s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396633830 (565h26m20.889066265s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.397583959 (565h26m20.8900164s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420259441 (565h26m20.9126919s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420371917 (565h26m20.912804356s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:29.213490723 (565h26m49.705923175s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213577765 (565h26m49.706010194s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213714723 (565h26m49.706147169s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213752753 (565h26m49.7061852s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213811756 (565h26m49.706244208s) [seg_pos={0 0}, upload=497444]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:34:29.316998815 (565h26m49.809431249s) [upload=497444]: Done uploading segments... Sep 22 03:34:29.317035357 (565h26m49.809467809s) [upload=497444]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.422782365s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....X....X.X.......XX....X..X...X.......X...XX..........X..] duration: 2.750702011s FAIL (2025-09-22 03:40:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.750807685s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X..................................XX........X.X.X.........] duration: 2.459807846s FAIL (2025-09-22 03:40:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:33:59.485342710 (565h26m19.977775146s) [upload=497444]: Starting upload Sep 22 03:33:59.485420892 (565h26m19.977853329s) [upload=497444]: Done starting upload Sep 22 03:33:59.485539660 (565h26m19.977972092s) [upload=497444]: Uploading segments... Sep 22 03:33:59.485714076 (565h26m19.978146509s) [upload=497444]: Got next segment. Inline: false Sep 22 03:33:59.485813900 (565h26m19.978246333s) [upload=497444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:33:59.485821999 (565h26m19.978254434s) [upload=497444]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:33:59.595141740 (565h26m20.087574174s) [seg_pos={0 0}, upload=497444]: Begin upload segment... Sep 22 03:33:59.595226098 (565h26m20.087658531s) [seg_pos={0 0}, upload=497444]: Done begin upload segment. Sep 22 03:33:59.595253117 (565h26m20.087685562s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.595807440 (565h26m20.088239889s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596269486 (565h26m20.088701937s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596610761 (565h26m20.089043209s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596697470 (565h26m20.089129917s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.597355109 (565h26m20.089787556s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598119615 (565h26m20.090552066s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598953736 (565h26m20.091386188s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599655367 (565h26m20.092087809s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599993860 (565h26m20.092426306s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.600640516 (565h26m20.093072958s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601107222 (565h26m20.093539671s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601842149 (565h26m20.0942746s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.602793768 (565h26m20.095226222s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603229497 (565h26m20.095661946s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603695255 (565h26m20.096127703s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:34:00.229252838 (565h26m20.721685298s) [upload=497444]: Next returned nil segment Sep 22 03:34:00.229281454 (565h26m20.72171391s) [upload=497444]: Waiting for error group managing segments... Sep 22 03:34:00.367601956 (565h26m20.860034403s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.368894773 (565h26m20.861327222s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.391796558 (565h26m20.884228999s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393337713 (565h26m20.885770148s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393367500 (565h26m20.885799936s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394210027 (565h26m20.886642463s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394934320 (565h26m20.887366757s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396404984 (565h26m20.888837431s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396633830 (565h26m20.889066265s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.397583959 (565h26m20.8900164s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420259441 (565h26m20.9126919s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420371917 (565h26m20.912804356s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:29.213490723 (565h26m49.705923175s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213577765 (565h26m49.706010194s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213714723 (565h26m49.706147169s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213752753 (565h26m49.7061852s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213811756 (565h26m49.706244208s) [seg_pos={0 0}, upload=497444]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:34:29.316998815 (565h26m49.809431249s) [upload=497444]: Done uploading segments... Sep 22 03:34:29.317035357 (565h26m49.809467809s) [upload=497444]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.459846212s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X..................................XX........X.X.X........] duration: 2.373869096s FAIL (2025-09-22 03:39:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:33:59.485342710 (565h26m19.977775146s) [upload=497444]: Starting upload Sep 22 03:33:59.485420892 (565h26m19.977853329s) [upload=497444]: Done starting upload Sep 22 03:33:59.485539660 (565h26m19.977972092s) [upload=497444]: Uploading segments... Sep 22 03:33:59.485714076 (565h26m19.978146509s) [upload=497444]: Got next segment. Inline: false Sep 22 03:33:59.485813900 (565h26m19.978246333s) [upload=497444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:33:59.485821999 (565h26m19.978254434s) [upload=497444]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:33:59.595141740 (565h26m20.087574174s) [seg_pos={0 0}, upload=497444]: Begin upload segment... Sep 22 03:33:59.595226098 (565h26m20.087658531s) [seg_pos={0 0}, upload=497444]: Done begin upload segment. Sep 22 03:33:59.595253117 (565h26m20.087685562s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.595807440 (565h26m20.088239889s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596269486 (565h26m20.088701937s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596610761 (565h26m20.089043209s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596697470 (565h26m20.089129917s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.597355109 (565h26m20.089787556s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598119615 (565h26m20.090552066s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598953736 (565h26m20.091386188s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599655367 (565h26m20.092087809s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599993860 (565h26m20.092426306s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.600640516 (565h26m20.093072958s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601107222 (565h26m20.093539671s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601842149 (565h26m20.0942746s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.602793768 (565h26m20.095226222s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603229497 (565h26m20.095661946s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603695255 (565h26m20.096127703s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:34:00.229252838 (565h26m20.721685298s) [upload=497444]: Next returned nil segment Sep 22 03:34:00.229281454 (565h26m20.72171391s) [upload=497444]: Waiting for error group managing segments... Sep 22 03:34:00.367601956 (565h26m20.860034403s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.368894773 (565h26m20.861327222s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.391796558 (565h26m20.884228999s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393337713 (565h26m20.885770148s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393367500 (565h26m20.885799936s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394210027 (565h26m20.886642463s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394934320 (565h26m20.887366757s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396404984 (565h26m20.888837431s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396633830 (565h26m20.889066265s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.397583959 (565h26m20.8900164s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420259441 (565h26m20.9126919s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420371917 (565h26m20.912804356s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:29.213490723 (565h26m49.705923175s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213577765 (565h26m49.706010194s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213714723 (565h26m49.706147169s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213752753 (565h26m49.7061852s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213811756 (565h26m49.706244208s) [seg_pos={0 0}, upload=497444]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:34:29.316998815 (565h26m49.809431249s) [upload=497444]: Done uploading segments... Sep 22 03:34:29.317035357 (565h26m49.809467809s) [upload=497444]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.373900947s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [......X....X.X.......XX....X..X...X.......X...XX..........X.] duration: 2.859139682s FAIL (2025-09-22 03:39:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.859167297s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.......X....X.X.......XX....X..X...X.......X...XX..........X] duration: 30.000933539s FAIL (2025-09-22 03:39:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000958918s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X..................................XX........X.X.X.......] duration: 2.415679045s FAIL (2025-09-22 03:39:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:33:59.485342710 (565h26m19.977775146s) [upload=497444]: Starting upload Sep 22 03:33:59.485420892 (565h26m19.977853329s) [upload=497444]: Done starting upload Sep 22 03:33:59.485539660 (565h26m19.977972092s) [upload=497444]: Uploading segments... Sep 22 03:33:59.485714076 (565h26m19.978146509s) [upload=497444]: Got next segment. Inline: false Sep 22 03:33:59.485813900 (565h26m19.978246333s) [upload=497444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:33:59.485821999 (565h26m19.978254434s) [upload=497444]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:33:59.595141740 (565h26m20.087574174s) [seg_pos={0 0}, upload=497444]: Begin upload segment... Sep 22 03:33:59.595226098 (565h26m20.087658531s) [seg_pos={0 0}, upload=497444]: Done begin upload segment. Sep 22 03:33:59.595253117 (565h26m20.087685562s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.595807440 (565h26m20.088239889s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596269486 (565h26m20.088701937s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596610761 (565h26m20.089043209s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596697470 (565h26m20.089129917s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.597355109 (565h26m20.089787556s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598119615 (565h26m20.090552066s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598953736 (565h26m20.091386188s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599655367 (565h26m20.092087809s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599993860 (565h26m20.092426306s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.600640516 (565h26m20.093072958s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601107222 (565h26m20.093539671s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601842149 (565h26m20.0942746s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.602793768 (565h26m20.095226222s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603229497 (565h26m20.095661946s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603695255 (565h26m20.096127703s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:34:00.229252838 (565h26m20.721685298s) [upload=497444]: Next returned nil segment Sep 22 03:34:00.229281454 (565h26m20.72171391s) [upload=497444]: Waiting for error group managing segments... Sep 22 03:34:00.367601956 (565h26m20.860034403s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.368894773 (565h26m20.861327222s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.391796558 (565h26m20.884228999s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393337713 (565h26m20.885770148s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393367500 (565h26m20.885799936s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394210027 (565h26m20.886642463s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394934320 (565h26m20.887366757s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396404984 (565h26m20.888837431s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396633830 (565h26m20.889066265s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.397583959 (565h26m20.8900164s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420259441 (565h26m20.9126919s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420371917 (565h26m20.912804356s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:29.213490723 (565h26m49.705923175s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213577765 (565h26m49.706010194s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213714723 (565h26m49.706147169s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213752753 (565h26m49.7061852s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213811756 (565h26m49.706244208s) [seg_pos={0 0}, upload=497444]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:34:29.316998815 (565h26m49.809431249s) [upload=497444]: Done uploading segments... Sep 22 03:34:29.317035357 (565h26m49.809467809s) [upload=497444]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.415711033s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X..................................XX........X.X.X......] duration: 2.392916162s FAIL (2025-09-22 03:38:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:33:59.485342710 (565h26m19.977775146s) [upload=497444]: Starting upload Sep 22 03:33:59.485420892 (565h26m19.977853329s) [upload=497444]: Done starting upload Sep 22 03:33:59.485539660 (565h26m19.977972092s) [upload=497444]: Uploading segments... Sep 22 03:33:59.485714076 (565h26m19.978146509s) [upload=497444]: Got next segment. Inline: false Sep 22 03:33:59.485813900 (565h26m19.978246333s) [upload=497444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:33:59.485821999 (565h26m19.978254434s) [upload=497444]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:33:59.595141740 (565h26m20.087574174s) [seg_pos={0 0}, upload=497444]: Begin upload segment... Sep 22 03:33:59.595226098 (565h26m20.087658531s) [seg_pos={0 0}, upload=497444]: Done begin upload segment. Sep 22 03:33:59.595253117 (565h26m20.087685562s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.595807440 (565h26m20.088239889s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596269486 (565h26m20.088701937s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596610761 (565h26m20.089043209s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596697470 (565h26m20.089129917s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.597355109 (565h26m20.089787556s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598119615 (565h26m20.090552066s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598953736 (565h26m20.091386188s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599655367 (565h26m20.092087809s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599993860 (565h26m20.092426306s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.600640516 (565h26m20.093072958s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601107222 (565h26m20.093539671s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601842149 (565h26m20.0942746s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.602793768 (565h26m20.095226222s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603229497 (565h26m20.095661946s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603695255 (565h26m20.096127703s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:34:00.229252838 (565h26m20.721685298s) [upload=497444]: Next returned nil segment Sep 22 03:34:00.229281454 (565h26m20.72171391s) [upload=497444]: Waiting for error group managing segments... Sep 22 03:34:00.367601956 (565h26m20.860034403s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.368894773 (565h26m20.861327222s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.391796558 (565h26m20.884228999s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393337713 (565h26m20.885770148s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393367500 (565h26m20.885799936s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394210027 (565h26m20.886642463s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394934320 (565h26m20.887366757s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396404984 (565h26m20.888837431s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396633830 (565h26m20.889066265s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.397583959 (565h26m20.8900164s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420259441 (565h26m20.9126919s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420371917 (565h26m20.912804356s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:29.213490723 (565h26m49.705923175s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213577765 (565h26m49.706010194s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213714723 (565h26m49.706147169s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213752753 (565h26m49.7061852s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213811756 (565h26m49.706244208s) [seg_pos={0 0}, upload=497444]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:34:29.316998815 (565h26m49.809431249s) [upload=497444]: Done uploading segments... Sep 22 03:34:29.317035357 (565h26m49.809467809s) [upload=497444]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.392965025s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X.......X....X.X.......XX....X..X...X.......X...XX..........] duration: 2.617423132s FAIL (2025-09-22 03:38:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.617449103s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X.......X....X.X.......XX....X..X...X.......X...XX.........] duration: 2.753322532s FAIL (2025-09-22 03:38:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.753348222s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X..................................XX........X.X.X.....] duration: 2.489667292s FAIL (2025-09-22 03:37:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:33:59.485342710 (565h26m19.977775146s) [upload=497444]: Starting upload Sep 22 03:33:59.485420892 (565h26m19.977853329s) [upload=497444]: Done starting upload Sep 22 03:33:59.485539660 (565h26m19.977972092s) [upload=497444]: Uploading segments... Sep 22 03:33:59.485714076 (565h26m19.978146509s) [upload=497444]: Got next segment. Inline: false Sep 22 03:33:59.485813900 (565h26m19.978246333s) [upload=497444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:33:59.485821999 (565h26m19.978254434s) [upload=497444]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:33:59.595141740 (565h26m20.087574174s) [seg_pos={0 0}, upload=497444]: Begin upload segment... Sep 22 03:33:59.595226098 (565h26m20.087658531s) [seg_pos={0 0}, upload=497444]: Done begin upload segment. Sep 22 03:33:59.595253117 (565h26m20.087685562s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.595807440 (565h26m20.088239889s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596269486 (565h26m20.088701937s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596610761 (565h26m20.089043209s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596697470 (565h26m20.089129917s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.597355109 (565h26m20.089787556s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598119615 (565h26m20.090552066s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598953736 (565h26m20.091386188s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599655367 (565h26m20.092087809s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599993860 (565h26m20.092426306s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.600640516 (565h26m20.093072958s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601107222 (565h26m20.093539671s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601842149 (565h26m20.0942746s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.602793768 (565h26m20.095226222s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603229497 (565h26m20.095661946s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603695255 (565h26m20.096127703s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:34:00.229252838 (565h26m20.721685298s) [upload=497444]: Next returned nil segment Sep 22 03:34:00.229281454 (565h26m20.72171391s) [upload=497444]: Waiting for error group managing segments... Sep 22 03:34:00.367601956 (565h26m20.860034403s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.368894773 (565h26m20.861327222s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.391796558 (565h26m20.884228999s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393337713 (565h26m20.885770148s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393367500 (565h26m20.885799936s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394210027 (565h26m20.886642463s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394934320 (565h26m20.887366757s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396404984 (565h26m20.888837431s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396633830 (565h26m20.889066265s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.397583959 (565h26m20.8900164s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420259441 (565h26m20.9126919s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420371917 (565h26m20.912804356s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:29.213490723 (565h26m49.705923175s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213577765 (565h26m49.706010194s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213714723 (565h26m49.706147169s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213752753 (565h26m49.7061852s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213811756 (565h26m49.706244208s) [seg_pos={0 0}, upload=497444]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:34:29.316998815 (565h26m49.809431249s) [upload=497444]: Done uploading segments... Sep 22 03:34:29.317035357 (565h26m49.809467809s) [upload=497444]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.489699344s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......X..................................XX........X.X.X....] duration: 2.475532447s FAIL (2025-09-22 03:37:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:33:59.485342710 (565h26m19.977775146s) [upload=497444]: Starting upload Sep 22 03:33:59.485420892 (565h26m19.977853329s) [upload=497444]: Done starting upload Sep 22 03:33:59.485539660 (565h26m19.977972092s) [upload=497444]: Uploading segments... Sep 22 03:33:59.485714076 (565h26m19.978146509s) [upload=497444]: Got next segment. Inline: false Sep 22 03:33:59.485813900 (565h26m19.978246333s) [upload=497444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:33:59.485821999 (565h26m19.978254434s) [upload=497444]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:33:59.595141740 (565h26m20.087574174s) [seg_pos={0 0}, upload=497444]: Begin upload segment... Sep 22 03:33:59.595226098 (565h26m20.087658531s) [seg_pos={0 0}, upload=497444]: Done begin upload segment. Sep 22 03:33:59.595253117 (565h26m20.087685562s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.595807440 (565h26m20.088239889s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596269486 (565h26m20.088701937s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596610761 (565h26m20.089043209s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596697470 (565h26m20.089129917s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.597355109 (565h26m20.089787556s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598119615 (565h26m20.090552066s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598953736 (565h26m20.091386188s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599655367 (565h26m20.092087809s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599993860 (565h26m20.092426306s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.600640516 (565h26m20.093072958s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601107222 (565h26m20.093539671s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601842149 (565h26m20.0942746s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.602793768 (565h26m20.095226222s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603229497 (565h26m20.095661946s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603695255 (565h26m20.096127703s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:34:00.229252838 (565h26m20.721685298s) [upload=497444]: Next returned nil segment Sep 22 03:34:00.229281454 (565h26m20.72171391s) [upload=497444]: Waiting for error group managing segments... Sep 22 03:34:00.367601956 (565h26m20.860034403s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.368894773 (565h26m20.861327222s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.391796558 (565h26m20.884228999s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393337713 (565h26m20.885770148s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393367500 (565h26m20.885799936s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394210027 (565h26m20.886642463s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394934320 (565h26m20.887366757s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396404984 (565h26m20.888837431s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396633830 (565h26m20.889066265s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.397583959 (565h26m20.8900164s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420259441 (565h26m20.9126919s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420371917 (565h26m20.912804356s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:29.213490723 (565h26m49.705923175s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213577765 (565h26m49.706010194s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213714723 (565h26m49.706147169s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213752753 (565h26m49.7061852s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213811756 (565h26m49.706244208s) [seg_pos={0 0}, upload=497444]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:34:29.316998815 (565h26m49.809431249s) [upload=497444]: Done uploading segments... Sep 22 03:34:29.317035357 (565h26m49.809467809s) [upload=497444]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.475569826s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..X.......X....X.X.......XX....X..X...X.......X...XX........] duration: 2.773731886s FAIL (2025-09-22 03:37:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.773761644s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X......X..................................XX........X.X.X...] duration: 2.452346925s FAIL (2025-09-22 03:36:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:33:59.485342710 (565h26m19.977775146s) [upload=497444]: Starting upload Sep 22 03:33:59.485420892 (565h26m19.977853329s) [upload=497444]: Done starting upload Sep 22 03:33:59.485539660 (565h26m19.977972092s) [upload=497444]: Uploading segments... Sep 22 03:33:59.485714076 (565h26m19.978146509s) [upload=497444]: Got next segment. Inline: false Sep 22 03:33:59.485813900 (565h26m19.978246333s) [upload=497444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:33:59.485821999 (565h26m19.978254434s) [upload=497444]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:33:59.595141740 (565h26m20.087574174s) [seg_pos={0 0}, upload=497444]: Begin upload segment... Sep 22 03:33:59.595226098 (565h26m20.087658531s) [seg_pos={0 0}, upload=497444]: Done begin upload segment. Sep 22 03:33:59.595253117 (565h26m20.087685562s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.595807440 (565h26m20.088239889s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596269486 (565h26m20.088701937s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596610761 (565h26m20.089043209s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596697470 (565h26m20.089129917s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.597355109 (565h26m20.089787556s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598119615 (565h26m20.090552066s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598953736 (565h26m20.091386188s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599655367 (565h26m20.092087809s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599993860 (565h26m20.092426306s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.600640516 (565h26m20.093072958s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601107222 (565h26m20.093539671s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601842149 (565h26m20.0942746s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.602793768 (565h26m20.095226222s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603229497 (565h26m20.095661946s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603695255 (565h26m20.096127703s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:34:00.229252838 (565h26m20.721685298s) [upload=497444]: Next returned nil segment Sep 22 03:34:00.229281454 (565h26m20.72171391s) [upload=497444]: Waiting for error group managing segments... Sep 22 03:34:00.367601956 (565h26m20.860034403s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.368894773 (565h26m20.861327222s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.391796558 (565h26m20.884228999s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393337713 (565h26m20.885770148s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393367500 (565h26m20.885799936s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394210027 (565h26m20.886642463s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394934320 (565h26m20.887366757s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396404984 (565h26m20.888837431s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396633830 (565h26m20.889066265s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.397583959 (565h26m20.8900164s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420259441 (565h26m20.9126919s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420371917 (565h26m20.912804356s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:29.213490723 (565h26m49.705923175s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213577765 (565h26m49.706010194s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213714723 (565h26m49.706147169s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213752753 (565h26m49.7061852s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213811756 (565h26m49.706244208s) [seg_pos={0 0}, upload=497444]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:34:29.316998815 (565h26m49.809431249s) [upload=497444]: Done uploading segments... Sep 22 03:34:29.317035357 (565h26m49.809467809s) [upload=497444]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.45238129s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...X.......X....X.X.......XX....X..X...X.......X...XX.......] duration: 2.856007856s FAIL (2025-09-22 03:36:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.856027661s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [XX......X..................................XX........X.X.X..] duration: 2.648876992s FAIL (2025-09-22 03:35:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:33:59.485342710 (565h26m19.977775146s) [upload=497444]: Starting upload Sep 22 03:33:59.485420892 (565h26m19.977853329s) [upload=497444]: Done starting upload Sep 22 03:33:59.485539660 (565h26m19.977972092s) [upload=497444]: Uploading segments... Sep 22 03:33:59.485714076 (565h26m19.978146509s) [upload=497444]: Got next segment. Inline: false Sep 22 03:33:59.485813900 (565h26m19.978246333s) [upload=497444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:33:59.485821999 (565h26m19.978254434s) [upload=497444]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:33:59.595141740 (565h26m20.087574174s) [seg_pos={0 0}, upload=497444]: Begin upload segment... Sep 22 03:33:59.595226098 (565h26m20.087658531s) [seg_pos={0 0}, upload=497444]: Done begin upload segment. Sep 22 03:33:59.595253117 (565h26m20.087685562s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.595807440 (565h26m20.088239889s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596269486 (565h26m20.088701937s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596610761 (565h26m20.089043209s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596697470 (565h26m20.089129917s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.597355109 (565h26m20.089787556s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598119615 (565h26m20.090552066s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598953736 (565h26m20.091386188s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599655367 (565h26m20.092087809s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599993860 (565h26m20.092426306s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.600640516 (565h26m20.093072958s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601107222 (565h26m20.093539671s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601842149 (565h26m20.0942746s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.602793768 (565h26m20.095226222s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603229497 (565h26m20.095661946s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603695255 (565h26m20.096127703s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:34:00.229252838 (565h26m20.721685298s) [upload=497444]: Next returned nil segment Sep 22 03:34:00.229281454 (565h26m20.72171391s) [upload=497444]: Waiting for error group managing segments... Sep 22 03:34:00.367601956 (565h26m20.860034403s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.368894773 (565h26m20.861327222s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.391796558 (565h26m20.884228999s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393337713 (565h26m20.885770148s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393367500 (565h26m20.885799936s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394210027 (565h26m20.886642463s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394934320 (565h26m20.887366757s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396404984 (565h26m20.888837431s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396633830 (565h26m20.889066265s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.397583959 (565h26m20.8900164s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420259441 (565h26m20.9126919s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420371917 (565h26m20.912804356s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:29.213490723 (565h26m49.705923175s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213577765 (565h26m49.706010194s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213714723 (565h26m49.706147169s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213752753 (565h26m49.7061852s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213811756 (565h26m49.706244208s) [seg_pos={0 0}, upload=497444]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:34:29.316998815 (565h26m49.809431249s) [upload=497444]: Done uploading segments... Sep 22 03:34:29.317035357 (565h26m49.809467809s) [upload=497444]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.648906486s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.XX......X..................................XX........X.X.X.] duration: 2.433234444s FAIL (2025-09-22 03:35:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:33:59.485342710 (565h26m19.977775146s) [upload=497444]: Starting upload Sep 22 03:33:59.485420892 (565h26m19.977853329s) [upload=497444]: Done starting upload Sep 22 03:33:59.485539660 (565h26m19.977972092s) [upload=497444]: Uploading segments... Sep 22 03:33:59.485714076 (565h26m19.978146509s) [upload=497444]: Got next segment. Inline: false Sep 22 03:33:59.485813900 (565h26m19.978246333s) [upload=497444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:33:59.485821999 (565h26m19.978254434s) [upload=497444]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:33:59.595141740 (565h26m20.087574174s) [seg_pos={0 0}, upload=497444]: Begin upload segment... Sep 22 03:33:59.595226098 (565h26m20.087658531s) [seg_pos={0 0}, upload=497444]: Done begin upload segment. Sep 22 03:33:59.595253117 (565h26m20.087685562s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.595807440 (565h26m20.088239889s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596269486 (565h26m20.088701937s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596610761 (565h26m20.089043209s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596697470 (565h26m20.089129917s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.597355109 (565h26m20.089787556s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598119615 (565h26m20.090552066s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598953736 (565h26m20.091386188s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599655367 (565h26m20.092087809s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599993860 (565h26m20.092426306s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.600640516 (565h26m20.093072958s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601107222 (565h26m20.093539671s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601842149 (565h26m20.0942746s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.602793768 (565h26m20.095226222s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603229497 (565h26m20.095661946s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603695255 (565h26m20.096127703s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:34:00.229252838 (565h26m20.721685298s) [upload=497444]: Next returned nil segment Sep 22 03:34:00.229281454 (565h26m20.72171391s) [upload=497444]: Waiting for error group managing segments... Sep 22 03:34:00.367601956 (565h26m20.860034403s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.368894773 (565h26m20.861327222s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.391796558 (565h26m20.884228999s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393337713 (565h26m20.885770148s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393367500 (565h26m20.885799936s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394210027 (565h26m20.886642463s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394934320 (565h26m20.887366757s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396404984 (565h26m20.888837431s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396633830 (565h26m20.889066265s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.397583959 (565h26m20.8900164s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420259441 (565h26m20.9126919s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420371917 (565h26m20.912804356s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:29.213490723 (565h26m49.705923175s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213577765 (565h26m49.706010194s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213714723 (565h26m49.706147169s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213752753 (565h26m49.7061852s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213811756 (565h26m49.706244208s) [seg_pos={0 0}, upload=497444]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:34:29.316998815 (565h26m49.809431249s) [upload=497444]: Done uploading segments... Sep 22 03:34:29.317035357 (565h26m49.809467809s) [upload=497444]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.433333564s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....X.......X....X.X.......XX....X..X...X.......X...XX......] duration: 3.067703375s FAIL (2025-09-22 03:35:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.067742569s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....X.......X....X.X.......XX....X..X...X.......X...XX.....] duration: 2.768814582s FAIL (2025-09-22 03:34:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.768844331s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..XX......X..................................XX........X.X.X] duration: 30.106387176s FAIL (2025-09-22 03:34:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:33:59.485342710 (565h26m19.977775146s) [upload=497444]: Starting upload Sep 22 03:33:59.485420892 (565h26m19.977853329s) [upload=497444]: Done starting upload Sep 22 03:33:59.485539660 (565h26m19.977972092s) [upload=497444]: Uploading segments... Sep 22 03:33:59.485714076 (565h26m19.978146509s) [upload=497444]: Got next segment. Inline: false Sep 22 03:33:59.485813900 (565h26m19.978246333s) [upload=497444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:33:59.485821999 (565h26m19.978254434s) [upload=497444]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:33:59.595141740 (565h26m20.087574174s) [seg_pos={0 0}, upload=497444]: Begin upload segment... Sep 22 03:33:59.595226098 (565h26m20.087658531s) [seg_pos={0 0}, upload=497444]: Done begin upload segment. Sep 22 03:33:59.595253117 (565h26m20.087685562s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.595807440 (565h26m20.088239889s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596269486 (565h26m20.088701937s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596610761 (565h26m20.089043209s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.596697470 (565h26m20.089129917s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.597355109 (565h26m20.089787556s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598119615 (565h26m20.090552066s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.598953736 (565h26m20.091386188s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599655367 (565h26m20.092087809s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.599993860 (565h26m20.092426306s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.600640516 (565h26m20.093072958s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601107222 (565h26m20.093539671s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.601842149 (565h26m20.0942746s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.602793768 (565h26m20.095226222s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603229497 (565h26m20.095661946s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:33:59.603695255 (565h26m20.096127703s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Uploading piece... Sep 22 03:34:00.229252838 (565h26m20.721685298s) [upload=497444]: Next returned nil segment Sep 22 03:34:00.229281454 (565h26m20.72171391s) [upload=497444]: Waiting for error group managing segments... Sep 22 03:34:00.367601956 (565h26m20.860034403s) [noise=true , address=77.90.31.16:4734 , piece_id=W3FIPHDVLBQQF2XZOTHRWWXMRDCWFFFBWG3PC2E2INYVNK6D2FCQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.368894773 (565h26m20.861327222s) [noise=true , address=45.91.116.5:4297 , piece_id=VJKQZYI5ZT3P7WYWBYQ4ZFTVKGKN5TRZXMA5O32SH5UTD3RTQIGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.391796558 (565h26m20.884228999s) [noise=true , address=45.91.116.9:4262 , piece_id=RWQHQRPRZA72O3KIFVO6JCTCKK7HYUYIUY6DQBRGMJLIOAWERSVQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393337713 (565h26m20.885770148s) [noise=true , address=77.90.31.13:4627 , piece_id=UHVEG6XZGSJ6YXQGWN7YCGFSFBJ4NME6AS74XYGUMXFHJWXWNU2A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.393367500 (565h26m20.885799936s) [noise=true , address=45.91.116.14:4189 , piece_id=X7AXSCEVR6J7AM3AU5GKFEUOYBOKEVAR6GF7RXHFL4KEFP55YT6A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394210027 (565h26m20.886642463s) [noise=true , address=77.90.31.13:4626 , piece_id=NYOS3NZRU2TPKXHFE2CLAFGTI3QVAPCFKXZD66JIJ2ZVEURIFMBQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.394934320 (565h26m20.887366757s) [noise=true , address=45.91.116.13:4155 , piece_id=ISR3JF7DIRU4DNHTIOLR32L73OBCFIVSQP7KZZ777QGPDFG5GUGA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396404984 (565h26m20.888837431s) [noise=true , address=77.90.31.11:4554 , piece_id=XNT3ZFW3MX3UV7ZKC3NYJHZ7N7BHM2G7H3CNGEXLKGTYB5CCQ4JQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.396633830 (565h26m20.889066265s) [noise=true , address=45.91.116.8:4045 , piece_id=36EKJCJOKLOYXLU5UWWDJ66PROEANIEL6XSZY5L6GECMHUM4G73A, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.397583959 (565h26m20.8900164s) [noise=true , address=45.91.116.5:4298 , piece_id=FBQQBAVQBMT5RRIDOKIHPP73WR7BORWXIFKA2JORBZWTO7QI2YUA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420259441 (565h26m20.9126919s) [noise=true , address=45.91.116.4:4937 , piece_id=PNWH4Z46G4IUSSIBNYDEQURNPQ2ZJZMNU7MF446PJ2XX6Q23ZMYQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:00.420371917 (565h26m20.912804356s) [noise=true , address=77.90.31.18:4805 , piece_id=SINR7FP3YL76SHBL6JADAHSISA73UJEPNCATTJEQFZGFVE6RPSJQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: <nil> Sep 22 03:34:29.213490723 (565h26m49.705923175s) [noise=true , address=145.79.227.11:4731 , piece_id=RV5Q2AH6DQH4CLUXHFD2R2KZCO2LGGHMLCFOJ6PROG4HJJML2LWA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213577765 (565h26m49.706010194s) [noise=true , address=145.79.227.17:4970 , piece_id=ZAZ5Q6SQ2L4KD5SQXMVJZZTRKARTBT4TMQTGKKFF35OJFHSTOCGQ, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213714723 (565h26m49.706147169s) [noise=true , address=145.79.227.7:4293 , piece_id=NV7H4IEMFALEQRNQG7OCD2N25XXLBNZFXSMHZ6KZWM6F3WODM6RA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213752753 (565h26m49.7061852s) [noise=true , address=145.79.227.16:4912 , piece_id=RZBE3IHHV3ZZGIBIXZQWT3AAMYHAJKCIDAGYRZDFVUW4XKH64EQA, seg_pos={0 0}, upload=497444]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:34:29.213811756 (565h26m49.706244208s) [seg_pos={0 0}, upload=497444]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:34:29.316998815 (565h26m49.809431249s) [upload=497444]: Done uploading segments... Sep 22 03:34:29.317035357 (565h26m49.809467809s) [upload=497444]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.106422035s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [......X.......X....X.X.......XX....X..X...X.......X...XX....] duration: 2.786073151s FAIL (2025-09-22 03:33:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.786179702s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.......X.......X....X.X.......XX....X..X...X.......X...XX...] duration: 2.842972148s FAIL (2025-09-22 03:33:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.842998509s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...XX......X..................................XX........X.X.] duration: 2.395717179s FAIL (2025-09-22 03:32:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:32:19.924750849 (565h24m40.4171833s) [upload=497418]: Starting upload Sep 22 03:32:19.924867354 (565h24m40.417299807s) [upload=497418]: Done starting upload Sep 22 03:32:19.925203449 (565h24m40.4176359s) [upload=497418]: Uploading segments... Sep 22 03:32:19.925287243 (565h24m40.417719695s) [upload=497418]: Got next segment. Inline: false Sep 22 03:32:19.925336709 (565h24m40.417769166s) [upload=497418]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:32:19.925342383 (565h24m40.417774837s) [upload=497418]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:32:20.049021010 (565h24m40.541453468s) [seg_pos={0 0}, upload=497418]: Begin upload segment... Sep 22 03:32:20.049226405 (565h24m40.541658861s) [seg_pos={0 0}, upload=497418]: Done begin upload segment. Sep 22 03:32:20.049319505 (565h24m40.541752013s) [noise=true , address=45.91.116.12:4225 , piece_id=UBFB7FK2O52JEGKVOLPYUC6VZXO2Y3QGS5ETQDGAIKNGVYK2LAMQ, seg_pos={0 0}, upload=497418]: Uploading piece... Sep 22 03:32:20.049590243 (565h24m40.5420227s) [noise=true , address=45.91.116.4:4937 , piece_id=PJ2O4P73DTE7KEZTOYCKWCGVDPGNVHXM7SNJ5YYNHWU7NXPWGDDA, seg_pos={0 0}, upload=497418]: Uploading piece... Sep 22 03:32:20.050346652 (565h24m40.542779107s) [noise=true , address=145.79.227.4:4185 , piece_id=CXG5NFMZW6I5JNOVYKI3H4APLE5Z3SJ67QE7YWJ7PKEMEWGPNXQQ, seg_pos={0 0}, upload=497418]: Uploading piece... Sep 22 03:32:20.050474413 (565h24m40.542906868s) [noise=true , address=45.91.116.12:4227 , piece_id=ASB3EY2OHTPM3JAULT5FJWINKTETEZQS6OJWQ6VBVJ4KFDUFZJBQ, seg_pos={0 0}, upload=497418]: Uploading piece... Sep 22 03:32:20.051033932 (565h24m40.543466386s) [noise=true , address=77.90.31.11:4553 , piece_id=JQGXEEJZKLA2BFRUMXTSSDFAY3DBX2QCGGI6NSP46QQ5E2LT543Q, seg_pos={0 0}, upload=497418]: Uploading piece... Sep 22 03:32:20.051206357 (565h24m40.543638812s) [noise=true , address=45.91.116.11:4117 , piece_id=CCWYPE3HIR5MFVNXMCCKLDG3RQ3GIK2PJ5CPTHDH4GP3Y2X7D2JA, seg_pos={0 0}, upload=497418]: Uploading piece... Sep 22 03:32:20.051571288 (565h24m40.544003745s) [noise=true , address=145.79.227.7:4295 , piece_id=DDBQZCH3K3IW5PAORJTB5VLHLGRT5YTJCZPAU2B5QQJ2ZTNS5UBQ, seg_pos={0 0}, upload=497418]: Uploading piece... Sep 22 03:32:20.052104521 (565h24m40.544536976s) [noise=true , address=77.90.31.21:4879 , piece_id=VQ32RSF4Y7QYDUE7ZS4IHXUPCQTHICDMY6RKNGQB4ZNBUY477B3A, seg_pos={0 0}, upload=497418]: Uploading piece... Sep 22 03:32:20.052442884 (565h24m40.544875341s) [noise=true , address=45.91.116.10:4081 , piece_id=WWG4TZHOCUIGMCPG3KVGVVBTWHKAIOJ7UMSNJZZLII3PF2PN3WRA, seg_pos={0 0}, upload=497418]: Uploading piece... Sep 22 03:32:20.053564401 (565h24m40.545996858s) [noise=true , address=145.79.227.12:4767 , piece_id=V47AW7EETZPI7EKPR3MUT353J5JQ5NMB4FCKQWMK7FO2IXEXB45Q, seg_pos={0 0}, upload=497418]: Uploading piece... Sep 22 03:32:20.054548877 (565h24m40.546981339s) [noise=true , address=77.90.31.11:4554 , piece_id=SRZQJELC3PIZJXOOLGPDEWASTULELLE2MOLUXUWZRXXFQSHS3MEQ, seg_pos={0 0}, upload=497418]: Uploading piece... Sep 22 03:32:20.054674502 (565h24m40.547106963s) [noise=true , address=145.79.227.18:4005 , piece_id=ZUI3BZMHCTXQBWFNRHTHJUQ6YYKT7VU7ZBCHDIGVP2P7EV7LJYDQ, seg_pos={0 0}, upload=497418]: Uploading piece... Sep 22 03:32:20.055674190 (565h24m40.548106646s) [noise=true , address=77.90.31.13:4628 , piece_id=3LYIT6QKJLUKCCXFADPGO55ZMZGGMCDCUOT4AY7VWCDAAUWZ255A, seg_pos={0 0}, upload=497418]: Uploading piece... Sep 22 03:32:20.056474029 (565h24m40.548906486s) [noise=true , address=45.91.116.9:4263 , piece_id=MZML33QFHP5AMNRNZAPBYNYZ7XRJ6LP27BM7M4Q5TXHLQPQZQI7A, seg_pos={0 0}, upload=497418]: Uploading piece... Sep 22 03:32:20.057242805 (565h24m40.549675263s) [noise=true , address=77.90.31.19:4843 , piece_id=4JIYKOTCK3VWJT45XLMPROAA5B5JGAR5X3X3VHA5V3XHCB6QONXA, seg_pos={0 0}, upload=497418]: Uploading piece... Sep 22 03:32:20.058081509 (565h24m40.550513961s) [noise=true , address=45.91.116.14:4189 , piece_id=ATKGMNAWM7T6AF2A5DCLD5XFNEZISNJONPJRBBFSOGIOAGLAOO7A, seg_pos={0 0}, upload=497418]: Uploading piece... Sep 22 03:32:20.681856516 (565h24m41.17428895s) [upload=497418]: Next returned nil segment Sep 22 03:32:20.681875499 (565h24m41.174307942s) [upload=497418]: Waiting for error group managing segments... Sep 22 03:32:20.815278977 (565h24m41.307711413s) [noise=true , address=45.91.116.11:4117 , piece_id=CCWYPE3HIR5MFVNXMCCKLDG3RQ3GIK2PJ5CPTHDH4GP3Y2X7D2JA, seg_pos={0 0}, upload=497418]: Done uploading piece. err: <nil> Sep 22 03:32:20.818426564 (565h24m41.310858998s) [noise=true , address=45.91.116.12:4227 , piece_id=ASB3EY2OHTPM3JAULT5FJWINKTETEZQS6OJWQ6VBVJ4KFDUFZJBQ, seg_pos={0 0}, upload=497418]: Done uploading piece. err: <nil> Sep 22 03:32:20.818740943 (565h24m41.311173379s) [noise=true , address=45.91.116.4:4937 , piece_id=PJ2O4P73DTE7KEZTOYCKWCGVDPGNVHXM7SNJ5YYNHWU7NXPWGDDA, seg_pos={0 0}, upload=497418]: Done uploading piece. err: <nil> Sep 22 03:32:20.819331645 (565h24m41.311764077s) [noise=true , address=45.91.116.10:4081 , piece_id=WWG4TZHOCUIGMCPG3KVGVVBTWHKAIOJ7UMSNJZZLII3PF2PN3WRA, seg_pos={0 0}, upload=497418]: Done uploading piece. err: <nil> Sep 22 03:32:20.820018812 (565h24m41.312451245s) [noise=true , address=77.90.31.11:4554 , piece_id=SRZQJELC3PIZJXOOLGPDEWASTULELLE2MOLUXUWZRXXFQSHS3MEQ, seg_pos={0 0}, upload=497418]: Done uploading piece. err: <nil> Sep 22 03:32:20.822997673 (565h24m41.315430136s) [noise=true , address=77.90.31.21:4879 , piece_id=VQ32RSF4Y7QYDUE7ZS4IHXUPCQTHICDMY6RKNGQB4ZNBUY477B3A, seg_pos={0 0}, upload=497418]: Done uploading piece. err: <nil> Sep 22 03:32:20.824203131 (565h24m41.316635559s) [noise=true , address=45.91.116.12:4225 , piece_id=UBFB7FK2O52JEGKVOLPYUC6VZXO2Y3QGS5ETQDGAIKNGVYK2LAMQ, seg_pos={0 0}, upload=497418]: Done uploading piece. err: <nil> Sep 22 03:32:20.824546249 (565h24m41.316978688s) [noise=true , address=45.91.116.9:4263 , piece_id=MZML33QFHP5AMNRNZAPBYNYZ7XRJ6LP27BM7M4Q5TXHLQPQZQI7A, seg_pos={0 0}, upload=497418]: Done uploading piece. err: <nil> Sep 22 03:32:20.830417562 (565h24m41.322850007s) [noise=true , address=45.91.116.14:4189 , piece_id=ATKGMNAWM7T6AF2A5DCLD5XFNEZISNJONPJRBBFSOGIOAGLAOO7A, seg_pos={0 0}, upload=497418]: Done uploading piece. err: <nil> Sep 22 03:32:20.838100731 (565h24m41.330533178s) [noise=true , address=77.90.31.11:4553 , piece_id=JQGXEEJZKLA2BFRUMXTSSDFAY3DBX2QCGGI6NSP46QQ5E2LT543Q, seg_pos={0 0}, upload=497418]: Done uploading piece. err: <nil> Sep 22 03:32:20.856331262 (565h24m41.348763696s) [noise=true , address=77.90.31.19:4843 , piece_id=4JIYKOTCK3VWJT45XLMPROAA5B5JGAR5X3X3VHA5V3XHCB6QONXA, seg_pos={0 0}, upload=497418]: Done uploading piece. err: <nil> Sep 22 03:32:20.870354991 (565h24m41.362787553s) [noise=true , address=77.90.31.13:4628 , piece_id=3LYIT6QKJLUKCCXFADPGO55ZMZGGMCDCUOT4AY7VWCDAAUWZ255A, seg_pos={0 0}, upload=497418]: Done uploading piece. err: <nil> Sep 22 03:32:49.610745767 (565h25m10.103178197s) [noise=true , address=145.79.227.4:4185 , piece_id=CXG5NFMZW6I5JNOVYKI3H4APLE5Z3SJ67QE7YWJ7PKEMEWGPNXQQ, seg_pos={0 0}, upload=497418]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: rpc: dial tcp 145.79.227.4:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:32:49.610832472 (565h25m10.103264917s) [noise=true , address=145.79.227.18:4005 , piece_id=ZUI3BZMHCTXQBWFNRHTHJUQ6YYKT7VU7ZBCHDIGVP2P7EV7LJYDQ, seg_pos={0 0}, upload=497418]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; context deadline exceeded Sep 22 03:32:49.610955875 (565h25m10.103388319s) [noise=true , address=145.79.227.12:4767 , piece_id=V47AW7EETZPI7EKPR3MUT353J5JQ5NMB4FCKQWMK7FO2IXEXB45Q, seg_pos={0 0}, upload=497418]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:32:49.610974978 (565h25m10.10340742s) [noise=true , address=145.79.227.7:4295 , piece_id=DDBQZCH3K3IW5PAORJTB5VLHLGRT5YTJCZPAU2B5QQJ2ZTNS5UBQ, seg_pos={0 0}, upload=497418]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: rpc: dial tcp 145.79.227.7:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:32:49.611018571 (565h25m10.103451015s) [seg_pos={0 0}, upload=497418]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:32:49.724143868 (565h25m10.216576304s) [upload=497418]: Done uploading segments... Sep 22 03:32:49.724173258 (565h25m10.216605693s) [upload=497418]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.395748411s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....XX......X..................................XX........X.X] duration: 30.115303013s FAIL (2025-09-22 03:32:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:32:19.924750849 (565h24m40.4171833s) [upload=497418]: Starting upload Sep 22 03:32:19.924867354 (565h24m40.417299807s) [upload=497418]: Done starting upload Sep 22 03:32:19.925203449 (565h24m40.4176359s) [upload=497418]: Uploading segments... Sep 22 03:32:19.925287243 (565h24m40.417719695s) [upload=497418]: Got next segment. Inline: false Sep 22 03:32:19.925336709 (565h24m40.417769166s) [upload=497418]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:32:19.925342383 (565h24m40.417774837s) [upload=497418]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:32:20.049021010 (565h24m40.541453468s) [seg_pos={0 0}, upload=497418]: Begin upload segment... Sep 22 03:32:20.049226405 (565h24m40.541658861s) [seg_pos={0 0}, upload=497418]: Done begin upload segment. Sep 22 03:32:20.049319505 (565h24m40.541752013s) [noise=true , address=45.91.116.12:4225 , piece_id=UBFB7FK2O52JEGKVOLPYUC6VZXO2Y3QGS5ETQDGAIKNGVYK2LAMQ, seg_pos={0 0}, upload=497418]: Uploading piece... Sep 22 03:32:20.049590243 (565h24m40.5420227s) [noise=true , address=45.91.116.4:4937 , piece_id=PJ2O4P73DTE7KEZTOYCKWCGVDPGNVHXM7SNJ5YYNHWU7NXPWGDDA, seg_pos={0 0}, upload=497418]: Uploading piece... Sep 22 03:32:20.050346652 (565h24m40.542779107s) [noise=true , address=145.79.227.4:4185 , piece_id=CXG5NFMZW6I5JNOVYKI3H4APLE5Z3SJ67QE7YWJ7PKEMEWGPNXQQ, seg_pos={0 0}, upload=497418]: Uploading piece... Sep 22 03:32:20.050474413 (565h24m40.542906868s) [noise=true , address=45.91.116.12:4227 , piece_id=ASB3EY2OHTPM3JAULT5FJWINKTETEZQS6OJWQ6VBVJ4KFDUFZJBQ, seg_pos={0 0}, upload=497418]: Uploading piece... Sep 22 03:32:20.051033932 (565h24m40.543466386s) [noise=true , address=77.90.31.11:4553 , piece_id=JQGXEEJZKLA2BFRUMXTSSDFAY3DBX2QCGGI6NSP46QQ5E2LT543Q, seg_pos={0 0}, upload=497418]: Uploading piece... Sep 22 03:32:20.051206357 (565h24m40.543638812s) [noise=true , address=45.91.116.11:4117 , piece_id=CCWYPE3HIR5MFVNXMCCKLDG3RQ3GIK2PJ5CPTHDH4GP3Y2X7D2JA, seg_pos={0 0}, upload=497418]: Uploading piece... Sep 22 03:32:20.051571288 (565h24m40.544003745s) [noise=true , address=145.79.227.7:4295 , piece_id=DDBQZCH3K3IW5PAORJTB5VLHLGRT5YTJCZPAU2B5QQJ2ZTNS5UBQ, seg_pos={0 0}, upload=497418]: Uploading piece... Sep 22 03:32:20.052104521 (565h24m40.544536976s) [noise=true , address=77.90.31.21:4879 , piece_id=VQ32RSF4Y7QYDUE7ZS4IHXUPCQTHICDMY6RKNGQB4ZNBUY477B3A, seg_pos={0 0}, upload=497418]: Uploading piece... Sep 22 03:32:20.052442884 (565h24m40.544875341s) [noise=true , address=45.91.116.10:4081 , piece_id=WWG4TZHOCUIGMCPG3KVGVVBTWHKAIOJ7UMSNJZZLII3PF2PN3WRA, seg_pos={0 0}, upload=497418]: Uploading piece... Sep 22 03:32:20.053564401 (565h24m40.545996858s) [noise=true , address=145.79.227.12:4767 , piece_id=V47AW7EETZPI7EKPR3MUT353J5JQ5NMB4FCKQWMK7FO2IXEXB45Q, seg_pos={0 0}, upload=497418]: Uploading piece... Sep 22 03:32:20.054548877 (565h24m40.546981339s) [noise=true , address=77.90.31.11:4554 , piece_id=SRZQJELC3PIZJXOOLGPDEWASTULELLE2MOLUXUWZRXXFQSHS3MEQ, seg_pos={0 0}, upload=497418]: Uploading piece... Sep 22 03:32:20.054674502 (565h24m40.547106963s) [noise=true , address=145.79.227.18:4005 , piece_id=ZUI3BZMHCTXQBWFNRHTHJUQ6YYKT7VU7ZBCHDIGVP2P7EV7LJYDQ, seg_pos={0 0}, upload=497418]: Uploading piece... Sep 22 03:32:20.055674190 (565h24m40.548106646s) [noise=true , address=77.90.31.13:4628 , piece_id=3LYIT6QKJLUKCCXFADPGO55ZMZGGMCDCUOT4AY7VWCDAAUWZ255A, seg_pos={0 0}, upload=497418]: Uploading piece... Sep 22 03:32:20.056474029 (565h24m40.548906486s) [noise=true , address=45.91.116.9:4263 , piece_id=MZML33QFHP5AMNRNZAPBYNYZ7XRJ6LP27BM7M4Q5TXHLQPQZQI7A, seg_pos={0 0}, upload=497418]: Uploading piece... Sep 22 03:32:20.057242805 (565h24m40.549675263s) [noise=true , address=77.90.31.19:4843 , piece_id=4JIYKOTCK3VWJT45XLMPROAA5B5JGAR5X3X3VHA5V3XHCB6QONXA, seg_pos={0 0}, upload=497418]: Uploading piece... Sep 22 03:32:20.058081509 (565h24m40.550513961s) [noise=true , address=45.91.116.14:4189 , piece_id=ATKGMNAWM7T6AF2A5DCLD5XFNEZISNJONPJRBBFSOGIOAGLAOO7A, seg_pos={0 0}, upload=497418]: Uploading piece... Sep 22 03:32:20.681856516 (565h24m41.17428895s) [upload=497418]: Next returned nil segment Sep 22 03:32:20.681875499 (565h24m41.174307942s) [upload=497418]: Waiting for error group managing segments... Sep 22 03:32:20.815278977 (565h24m41.307711413s) [noise=true , address=45.91.116.11:4117 , piece_id=CCWYPE3HIR5MFVNXMCCKLDG3RQ3GIK2PJ5CPTHDH4GP3Y2X7D2JA, seg_pos={0 0}, upload=497418]: Done uploading piece. err: <nil> Sep 22 03:32:20.818426564 (565h24m41.310858998s) [noise=true , address=45.91.116.12:4227 , piece_id=ASB3EY2OHTPM3JAULT5FJWINKTETEZQS6OJWQ6VBVJ4KFDUFZJBQ, seg_pos={0 0}, upload=497418]: Done uploading piece. err: <nil> Sep 22 03:32:20.818740943 (565h24m41.311173379s) [noise=true , address=45.91.116.4:4937 , piece_id=PJ2O4P73DTE7KEZTOYCKWCGVDPGNVHXM7SNJ5YYNHWU7NXPWGDDA, seg_pos={0 0}, upload=497418]: Done uploading piece. err: <nil> Sep 22 03:32:20.819331645 (565h24m41.311764077s) [noise=true , address=45.91.116.10:4081 , piece_id=WWG4TZHOCUIGMCPG3KVGVVBTWHKAIOJ7UMSNJZZLII3PF2PN3WRA, seg_pos={0 0}, upload=497418]: Done uploading piece. err: <nil> Sep 22 03:32:20.820018812 (565h24m41.312451245s) [noise=true , address=77.90.31.11:4554 , piece_id=SRZQJELC3PIZJXOOLGPDEWASTULELLE2MOLUXUWZRXXFQSHS3MEQ, seg_pos={0 0}, upload=497418]: Done uploading piece. err: <nil> Sep 22 03:32:20.822997673 (565h24m41.315430136s) [noise=true , address=77.90.31.21:4879 , piece_id=VQ32RSF4Y7QYDUE7ZS4IHXUPCQTHICDMY6RKNGQB4ZNBUY477B3A, seg_pos={0 0}, upload=497418]: Done uploading piece. err: <nil> Sep 22 03:32:20.824203131 (565h24m41.316635559s) [noise=true , address=45.91.116.12:4225 , piece_id=UBFB7FK2O52JEGKVOLPYUC6VZXO2Y3QGS5ETQDGAIKNGVYK2LAMQ, seg_pos={0 0}, upload=497418]: Done uploading piece. err: <nil> Sep 22 03:32:20.824546249 (565h24m41.316978688s) [noise=true , address=45.91.116.9:4263 , piece_id=MZML33QFHP5AMNRNZAPBYNYZ7XRJ6LP27BM7M4Q5TXHLQPQZQI7A, seg_pos={0 0}, upload=497418]: Done uploading piece. err: <nil> Sep 22 03:32:20.830417562 (565h24m41.322850007s) [noise=true , address=45.91.116.14:4189 , piece_id=ATKGMNAWM7T6AF2A5DCLD5XFNEZISNJONPJRBBFSOGIOAGLAOO7A, seg_pos={0 0}, upload=497418]: Done uploading piece. err: <nil> Sep 22 03:32:20.838100731 (565h24m41.330533178s) [noise=true , address=77.90.31.11:4553 , piece_id=JQGXEEJZKLA2BFRUMXTSSDFAY3DBX2QCGGI6NSP46QQ5E2LT543Q, seg_pos={0 0}, upload=497418]: Done uploading piece. err: <nil> Sep 22 03:32:20.856331262 (565h24m41.348763696s) [noise=true , address=77.90.31.19:4843 , piece_id=4JIYKOTCK3VWJT45XLMPROAA5B5JGAR5X3X3VHA5V3XHCB6QONXA, seg_pos={0 0}, upload=497418]: Done uploading piece. err: <nil> Sep 22 03:32:20.870354991 (565h24m41.362787553s) [noise=true , address=77.90.31.13:4628 , piece_id=3LYIT6QKJLUKCCXFADPGO55ZMZGGMCDCUOT4AY7VWCDAAUWZ255A, seg_pos={0 0}, upload=497418]: Done uploading piece. err: <nil> Sep 22 03:32:49.610745767 (565h25m10.103178197s) [noise=true , address=145.79.227.4:4185 , piece_id=CXG5NFMZW6I5JNOVYKI3H4APLE5Z3SJ67QE7YWJ7PKEMEWGPNXQQ, seg_pos={0 0}, upload=497418]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: rpc: dial tcp 145.79.227.4:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:32:49.610832472 (565h25m10.103264917s) [noise=true , address=145.79.227.18:4005 , piece_id=ZUI3BZMHCTXQBWFNRHTHJUQ6YYKT7VU7ZBCHDIGVP2P7EV7LJYDQ, seg_pos={0 0}, upload=497418]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; context deadline exceeded Sep 22 03:32:49.610955875 (565h25m10.103388319s) [noise=true , address=145.79.227.12:4767 , piece_id=V47AW7EETZPI7EKPR3MUT353J5JQ5NMB4FCKQWMK7FO2IXEXB45Q, seg_pos={0 0}, upload=497418]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:32:49.610974978 (565h25m10.10340742s) [noise=true , address=145.79.227.7:4295 , piece_id=DDBQZCH3K3IW5PAORJTB5VLHLGRT5YTJCZPAU2B5QQJ2ZTNS5UBQ, seg_pos={0 0}, upload=497418]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: rpc: dial tcp 145.79.227.7:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:32:49.611018571 (565h25m10.103451015s) [seg_pos={0 0}, upload=497418]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:32:49.724143868 (565h25m10.216576304s) [upload=497418]: Done uploading segments... Sep 22 03:32:49.724173258 (565h25m10.216605693s) [upload=497418]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.115338652s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [........X.......X....X.X.......XX....X..X...X.......X...XX..] duration: 2.709772219s FAIL (2025-09-22 03:32:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.709903285s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.........X.......X....X.X.......XX....X..X...X.......X...XX.] duration: 2.880585545s FAIL (2025-09-22 03:31:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.880611289s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....XX......X..................................XX........X.] duration: 2.392177904s FAIL (2025-09-22 03:31:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:30:38.253684130 (565h22m58.746116562s) [upload=497393]: Starting upload Sep 22 03:30:38.253763306 (565h22m58.746195751s) [upload=497393]: Done starting upload Sep 22 03:30:38.254015830 (565h22m58.746448274s) [upload=497393]: Uploading segments... Sep 22 03:30:38.254316362 (565h22m58.74674886s) [upload=497393]: Got next segment. Inline: false Sep 22 03:30:38.254403775 (565h22m58.746836217s) [upload=497393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:30:38.254410064 (565h22m58.746842507s) [upload=497393]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:30:38.382784826 (565h22m58.875217258s) [seg_pos={0 0}, upload=497393]: Begin upload segment... Sep 22 03:30:38.382835375 (565h22m58.875267821s) [seg_pos={0 0}, upload=497393]: Done begin upload segment. Sep 22 03:30:38.382858677 (565h22m58.875291119s) [noise=true , address=145.79.227.15:4874 , piece_id=P5KWLBGV62V6RL6NGAC52Q3T3BBUR4MPBC6WMJ7XSO7HUM5TGLWQ, seg_pos={0 0}, upload=497393]: Uploading piece... Sep 22 03:30:38.382878583 (565h22m58.875311041s) [noise=true , address=77.90.31.12:4592 , piece_id=BLAEIKVMY5NS4MHXPKKMSR7PILM22P26B5NQ7LIOMUZDCBWPOTIA, seg_pos={0 0}, upload=497393]: Uploading piece... Sep 22 03:30:38.383230282 (565h22m58.875662728s) [noise=true , address=77.90.31.13:4626 , piece_id=42D4DBRMZXRP3FWOM2DLJCC2CKR2D2BNG2FDHRYMW4333BRWAK5A, seg_pos={0 0}, upload=497393]: Uploading piece... Sep 22 03:30:38.383519716 (565h22m58.875952159s) [noise=true , address=145.79.227.3:4149 , piece_id=VH74CVVUPCKJ454QLNGTL7ODEDKGIA4UEZ475QEVHBBZFYZ3JQZQ, seg_pos={0 0}, upload=497393]: Uploading piece... Sep 22 03:30:38.383529441 (565h22m58.875961888s) [noise=true , address=45.91.116.13:4155 , piece_id=CE5HB226XOLG2YDRB5NWD3ZLAKRMNRR6KVBGD537TPWWKO2DMSOQ, seg_pos={0 0}, upload=497393]: Uploading piece... Sep 22 03:30:38.383872047 (565h22m58.876304481s) [noise=true , address=145.79.227.14:4837 , piece_id=6OFMPXAFB7EIPJ554U62AJFNLQK3F4Z4QRRN2AJG7R3V246XVPMA, seg_pos={0 0}, upload=497393]: Uploading piece... Sep 22 03:30:38.383961181 (565h22m58.876393625s) [noise=true , address=45.91.116.4:4940 , piece_id=XH26GGIKZIWUYSO733TUF5OPZHATRFEJVD6QNSB5XLTVLOCDVHRQ, seg_pos={0 0}, upload=497393]: Uploading piece... Sep 22 03:30:38.384177835 (565h22m58.876610276s) [noise=true , address=45.91.116.8:4045 , piece_id=AZD4EI7SQEF2C7JYQPIRNT6DUZ5X64AOEIBRQSL3272TIQXLYRUQ, seg_pos={0 0}, upload=497393]: Uploading piece... Sep 22 03:30:38.384502484 (565h22m58.876934926s) [noise=true , address=45.91.116.7:4009 , piece_id=S2HT5AFAYOWFUUPI4T5EZGKKBRHVALLNILRFXEQDNX4KKAALLEMQ, seg_pos={0 0}, upload=497393]: Uploading piece... Sep 22 03:30:38.384871801 (565h22m58.877304231s) [noise=true , address=45.91.116.11:4118 , piece_id=SVAVGCPHYRAR7HLY4HNVA3PLTNWEJMGWGZZUXGQTV44HH7OTNLMQ, seg_pos={0 0}, upload=497393]: Uploading piece... Sep 22 03:30:38.385586582 (565h22m58.878019028s) [noise=true , address=45.91.116.10:4082 , piece_id=MFWLJU44PIW2RP25XOEUZM2OKOHTEVHIIUXNYYOKBOZ5DGRKAQYA, seg_pos={0 0}, upload=497393]: Uploading piece... Sep 22 03:30:38.385952994 (565h22m58.878385437s) [noise=true , address=77.90.31.21:4879 , piece_id=JR7NPRPP3F6XJJV3ELJFVPMZDKT5YTUTTGAKKZOGO2PMYVK46S4Q, seg_pos={0 0}, upload=497393]: Uploading piece... Sep 22 03:30:38.390221329 (565h22m58.882653761s) [noise=true , address=45.91.116.11:4117 , piece_id=5WVIYBNXTWR35AN7EV2QCW64PESYEVEOKSCNELUJL3NFGYV7BTFA, seg_pos={0 0}, upload=497393]: Uploading piece... Sep 22 03:30:38.395801346 (565h22m58.888233781s) [noise=true , address=45.91.116.12:4227 , piece_id=KIGSN5EFLY5KJY5SCJRTBLWCHYJ74VVAU2GL5STPSG2X6YBNMTUA, seg_pos={0 0}, upload=497393]: Uploading piece... Sep 22 03:30:38.398069619 (565h22m58.890502052s) [noise=true , address=45.91.116.14:4189 , piece_id=DVPTWCQ7ZHEYJBIQNMVKZ5MMILCCILNMP6SIOFKKT5P5ZZFSGMWA, seg_pos={0 0}, upload=497393]: Uploading piece... Sep 22 03:30:38.400199342 (565h22m58.892631777s) [noise=true , address=145.79.227.18:4006 , piece_id=4K3KVJ6PLUCLZXTEKUTMRZTKLGL5KI45DFDB2ZUJNYXQOUTWHQWA, seg_pos={0 0}, upload=497393]: Uploading piece... Sep 22 03:30:38.989325371 (565h22m59.481757806s) [upload=497393]: Next returned nil segment Sep 22 03:30:38.989343716 (565h22m59.481776164s) [upload=497393]: Waiting for error group managing segments... Sep 22 03:30:39.130528584 (565h22m59.622961012s) [noise=true , address=45.91.116.4:4940 , piece_id=XH26GGIKZIWUYSO733TUF5OPZHATRFEJVD6QNSB5XLTVLOCDVHRQ, seg_pos={0 0}, upload=497393]: Done uploading piece. err: <nil> Sep 22 03:30:39.130567403 (565h22m59.622999839s) [noise=true , address=45.91.116.8:4045 , piece_id=AZD4EI7SQEF2C7JYQPIRNT6DUZ5X64AOEIBRQSL3272TIQXLYRUQ, seg_pos={0 0}, upload=497393]: Done uploading piece. err: <nil> Sep 22 03:30:39.131070313 (565h22m59.623502748s) [noise=true , address=45.91.116.10:4082 , piece_id=MFWLJU44PIW2RP25XOEUZM2OKOHTEVHIIUXNYYOKBOZ5DGRKAQYA, seg_pos={0 0}, upload=497393]: Done uploading piece. err: <nil> Sep 22 03:30:39.135930198 (565h22m59.628362634s) [noise=true , address=45.91.116.11:4118 , piece_id=SVAVGCPHYRAR7HLY4HNVA3PLTNWEJMGWGZZUXGQTV44HH7OTNLMQ, seg_pos={0 0}, upload=497393]: Done uploading piece. err: <nil> Sep 22 03:30:39.137666810 (565h22m59.630099245s) [noise=true , address=45.91.116.11:4117 , piece_id=5WVIYBNXTWR35AN7EV2QCW64PESYEVEOKSCNELUJL3NFGYV7BTFA, seg_pos={0 0}, upload=497393]: Done uploading piece. err: <nil> Sep 22 03:30:39.139344426 (565h22m59.631776861s) [noise=true , address=45.91.116.7:4009 , piece_id=S2HT5AFAYOWFUUPI4T5EZGKKBRHVALLNILRFXEQDNX4KKAALLEMQ, seg_pos={0 0}, upload=497393]: Done uploading piece. err: <nil> Sep 22 03:30:39.141607776 (565h22m59.634040212s) [noise=true , address=77.90.31.21:4879 , piece_id=JR7NPRPP3F6XJJV3ELJFVPMZDKT5YTUTTGAKKZOGO2PMYVK46S4Q, seg_pos={0 0}, upload=497393]: Done uploading piece. err: <nil> Sep 22 03:30:39.143160656 (565h22m59.635593091s) [noise=true , address=45.91.116.14:4189 , piece_id=DVPTWCQ7ZHEYJBIQNMVKZ5MMILCCILNMP6SIOFKKT5P5ZZFSGMWA, seg_pos={0 0}, upload=497393]: Done uploading piece. err: <nil> Sep 22 03:30:39.143965172 (565h22m59.636397622s) [noise=true , address=45.91.116.12:4227 , piece_id=KIGSN5EFLY5KJY5SCJRTBLWCHYJ74VVAU2GL5STPSG2X6YBNMTUA, seg_pos={0 0}, upload=497393]: Done uploading piece. err: <nil> Sep 22 03:30:39.146760718 (565h22m59.639193154s) [noise=true , address=77.90.31.13:4626 , piece_id=42D4DBRMZXRP3FWOM2DLJCC2CKR2D2BNG2FDHRYMW4333BRWAK5A, seg_pos={0 0}, upload=497393]: Done uploading piece. err: <nil> Sep 22 03:30:39.149262646 (565h22m59.641695089s) [noise=true , address=77.90.31.12:4592 , piece_id=BLAEIKVMY5NS4MHXPKKMSR7PILM22P26B5NQ7LIOMUZDCBWPOTIA, seg_pos={0 0}, upload=497393]: Done uploading piece. err: <nil> Sep 22 03:30:39.178886890 (565h22m59.671319322s) [noise=true , address=45.91.116.13:4155 , piece_id=CE5HB226XOLG2YDRB5NWD3ZLAKRMNRR6KVBGD537TPWWKO2DMSOQ, seg_pos={0 0}, upload=497393]: Done uploading piece. err: <nil> Sep 22 03:31:07.974118417 (565h23m28.466550859s) [noise=true , address=145.79.227.15:4874 , piece_id=P5KWLBGV62V6RL6NGAC52Q3T3BBUR4MPBC6WMJ7XSO7HUM5TGLWQ, seg_pos={0 0}, upload=497393]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:31:07.974180669 (565h23m28.466613097s) [noise=true , address=145.79.227.3:4149 , piece_id=VH74CVVUPCKJ454QLNGTL7ODEDKGIA4UEZ475QEVHBBZFYZ3JQZQ, seg_pos={0 0}, upload=497393]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:31:07.974209924 (565h23m28.466642357s) [noise=true , address=145.79.227.18:4006 , piece_id=4K3KVJ6PLUCLZXTEKUTMRZTKLGL5KI45DFDB2ZUJNYXQOUTWHQWA, seg_pos={0 0}, upload=497393]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: rpc: dial tcp 145.79.227.18:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:31:07.974255115 (565h23m28.466687544s) [noise=true , address=145.79.227.14:4837 , piece_id=6OFMPXAFB7EIPJ554U62AJFNLQK3F4Z4QRRN2AJG7R3V246XVPMA, seg_pos={0 0}, upload=497393]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:31:07.974284622 (565h23m28.466717067s) [seg_pos={0 0}, upload=497393]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:31:08.076146881 (565h23m28.56857932s) [upload=497393]: Done uploading segments... Sep 22 03:31:08.076179027 (565h23m28.56861147s) [upload=497393]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.392207587s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..........X.......X....X.X.......XX....X..X...X.......X...XX] duration: 30.001194445s FAIL (2025-09-22 03:31:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001221968s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X.....XX......X..................................XX........X] duration: 30.104372599s FAIL (2025-09-22 03:31:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:30:38.253684130 (565h22m58.746116562s) [upload=497393]: Starting upload Sep 22 03:30:38.253763306 (565h22m58.746195751s) [upload=497393]: Done starting upload Sep 22 03:30:38.254015830 (565h22m58.746448274s) [upload=497393]: Uploading segments... Sep 22 03:30:38.254316362 (565h22m58.74674886s) [upload=497393]: Got next segment. Inline: false Sep 22 03:30:38.254403775 (565h22m58.746836217s) [upload=497393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:30:38.254410064 (565h22m58.746842507s) [upload=497393]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:30:38.382784826 (565h22m58.875217258s) [seg_pos={0 0}, upload=497393]: Begin upload segment... Sep 22 03:30:38.382835375 (565h22m58.875267821s) [seg_pos={0 0}, upload=497393]: Done begin upload segment. Sep 22 03:30:38.382858677 (565h22m58.875291119s) [noise=true , address=145.79.227.15:4874 , piece_id=P5KWLBGV62V6RL6NGAC52Q3T3BBUR4MPBC6WMJ7XSO7HUM5TGLWQ, seg_pos={0 0}, upload=497393]: Uploading piece... Sep 22 03:30:38.382878583 (565h22m58.875311041s) [noise=true , address=77.90.31.12:4592 , piece_id=BLAEIKVMY5NS4MHXPKKMSR7PILM22P26B5NQ7LIOMUZDCBWPOTIA, seg_pos={0 0}, upload=497393]: Uploading piece... Sep 22 03:30:38.383230282 (565h22m58.875662728s) [noise=true , address=77.90.31.13:4626 , piece_id=42D4DBRMZXRP3FWOM2DLJCC2CKR2D2BNG2FDHRYMW4333BRWAK5A, seg_pos={0 0}, upload=497393]: Uploading piece... Sep 22 03:30:38.383519716 (565h22m58.875952159s) [noise=true , address=145.79.227.3:4149 , piece_id=VH74CVVUPCKJ454QLNGTL7ODEDKGIA4UEZ475QEVHBBZFYZ3JQZQ, seg_pos={0 0}, upload=497393]: Uploading piece... Sep 22 03:30:38.383529441 (565h22m58.875961888s) [noise=true , address=45.91.116.13:4155 , piece_id=CE5HB226XOLG2YDRB5NWD3ZLAKRMNRR6KVBGD537TPWWKO2DMSOQ, seg_pos={0 0}, upload=497393]: Uploading piece... Sep 22 03:30:38.383872047 (565h22m58.876304481s) [noise=true , address=145.79.227.14:4837 , piece_id=6OFMPXAFB7EIPJ554U62AJFNLQK3F4Z4QRRN2AJG7R3V246XVPMA, seg_pos={0 0}, upload=497393]: Uploading piece... Sep 22 03:30:38.383961181 (565h22m58.876393625s) [noise=true , address=45.91.116.4:4940 , piece_id=XH26GGIKZIWUYSO733TUF5OPZHATRFEJVD6QNSB5XLTVLOCDVHRQ, seg_pos={0 0}, upload=497393]: Uploading piece... Sep 22 03:30:38.384177835 (565h22m58.876610276s) [noise=true , address=45.91.116.8:4045 , piece_id=AZD4EI7SQEF2C7JYQPIRNT6DUZ5X64AOEIBRQSL3272TIQXLYRUQ, seg_pos={0 0}, upload=497393]: Uploading piece... Sep 22 03:30:38.384502484 (565h22m58.876934926s) [noise=true , address=45.91.116.7:4009 , piece_id=S2HT5AFAYOWFUUPI4T5EZGKKBRHVALLNILRFXEQDNX4KKAALLEMQ, seg_pos={0 0}, upload=497393]: Uploading piece... Sep 22 03:30:38.384871801 (565h22m58.877304231s) [noise=true , address=45.91.116.11:4118 , piece_id=SVAVGCPHYRAR7HLY4HNVA3PLTNWEJMGWGZZUXGQTV44HH7OTNLMQ, seg_pos={0 0}, upload=497393]: Uploading piece... Sep 22 03:30:38.385586582 (565h22m58.878019028s) [noise=true , address=45.91.116.10:4082 , piece_id=MFWLJU44PIW2RP25XOEUZM2OKOHTEVHIIUXNYYOKBOZ5DGRKAQYA, seg_pos={0 0}, upload=497393]: Uploading piece... Sep 22 03:30:38.385952994 (565h22m58.878385437s) [noise=true , address=77.90.31.21:4879 , piece_id=JR7NPRPP3F6XJJV3ELJFVPMZDKT5YTUTTGAKKZOGO2PMYVK46S4Q, seg_pos={0 0}, upload=497393]: Uploading piece... Sep 22 03:30:38.390221329 (565h22m58.882653761s) [noise=true , address=45.91.116.11:4117 , piece_id=5WVIYBNXTWR35AN7EV2QCW64PESYEVEOKSCNELUJL3NFGYV7BTFA, seg_pos={0 0}, upload=497393]: Uploading piece... Sep 22 03:30:38.395801346 (565h22m58.888233781s) [noise=true , address=45.91.116.12:4227 , piece_id=KIGSN5EFLY5KJY5SCJRTBLWCHYJ74VVAU2GL5STPSG2X6YBNMTUA, seg_pos={0 0}, upload=497393]: Uploading piece... Sep 22 03:30:38.398069619 (565h22m58.890502052s) [noise=true , address=45.91.116.14:4189 , piece_id=DVPTWCQ7ZHEYJBIQNMVKZ5MMILCCILNMP6SIOFKKT5P5ZZFSGMWA, seg_pos={0 0}, upload=497393]: Uploading piece... Sep 22 03:30:38.400199342 (565h22m58.892631777s) [noise=true , address=145.79.227.18:4006 , piece_id=4K3KVJ6PLUCLZXTEKUTMRZTKLGL5KI45DFDB2ZUJNYXQOUTWHQWA, seg_pos={0 0}, upload=497393]: Uploading piece... Sep 22 03:30:38.989325371 (565h22m59.481757806s) [upload=497393]: Next returned nil segment Sep 22 03:30:38.989343716 (565h22m59.481776164s) [upload=497393]: Waiting for error group managing segments... Sep 22 03:30:39.130528584 (565h22m59.622961012s) [noise=true , address=45.91.116.4:4940 , piece_id=XH26GGIKZIWUYSO733TUF5OPZHATRFEJVD6QNSB5XLTVLOCDVHRQ, seg_pos={0 0}, upload=497393]: Done uploading piece. err: <nil> Sep 22 03:30:39.130567403 (565h22m59.622999839s) [noise=true , address=45.91.116.8:4045 , piece_id=AZD4EI7SQEF2C7JYQPIRNT6DUZ5X64AOEIBRQSL3272TIQXLYRUQ, seg_pos={0 0}, upload=497393]: Done uploading piece. err: <nil> Sep 22 03:30:39.131070313 (565h22m59.623502748s) [noise=true , address=45.91.116.10:4082 , piece_id=MFWLJU44PIW2RP25XOEUZM2OKOHTEVHIIUXNYYOKBOZ5DGRKAQYA, seg_pos={0 0}, upload=497393]: Done uploading piece. err: <nil> Sep 22 03:30:39.135930198 (565h22m59.628362634s) [noise=true , address=45.91.116.11:4118 , piece_id=SVAVGCPHYRAR7HLY4HNVA3PLTNWEJMGWGZZUXGQTV44HH7OTNLMQ, seg_pos={0 0}, upload=497393]: Done uploading piece. err: <nil> Sep 22 03:30:39.137666810 (565h22m59.630099245s) [noise=true , address=45.91.116.11:4117 , piece_id=5WVIYBNXTWR35AN7EV2QCW64PESYEVEOKSCNELUJL3NFGYV7BTFA, seg_pos={0 0}, upload=497393]: Done uploading piece. err: <nil> Sep 22 03:30:39.139344426 (565h22m59.631776861s) [noise=true , address=45.91.116.7:4009 , piece_id=S2HT5AFAYOWFUUPI4T5EZGKKBRHVALLNILRFXEQDNX4KKAALLEMQ, seg_pos={0 0}, upload=497393]: Done uploading piece. err: <nil> Sep 22 03:30:39.141607776 (565h22m59.634040212s) [noise=true , address=77.90.31.21:4879 , piece_id=JR7NPRPP3F6XJJV3ELJFVPMZDKT5YTUTTGAKKZOGO2PMYVK46S4Q, seg_pos={0 0}, upload=497393]: Done uploading piece. err: <nil> Sep 22 03:30:39.143160656 (565h22m59.635593091s) [noise=true , address=45.91.116.14:4189 , piece_id=DVPTWCQ7ZHEYJBIQNMVKZ5MMILCCILNMP6SIOFKKT5P5ZZFSGMWA, seg_pos={0 0}, upload=497393]: Done uploading piece. err: <nil> Sep 22 03:30:39.143965172 (565h22m59.636397622s) [noise=true , address=45.91.116.12:4227 , piece_id=KIGSN5EFLY5KJY5SCJRTBLWCHYJ74VVAU2GL5STPSG2X6YBNMTUA, seg_pos={0 0}, upload=497393]: Done uploading piece. err: <nil> Sep 22 03:30:39.146760718 (565h22m59.639193154s) [noise=true , address=77.90.31.13:4626 , piece_id=42D4DBRMZXRP3FWOM2DLJCC2CKR2D2BNG2FDHRYMW4333BRWAK5A, seg_pos={0 0}, upload=497393]: Done uploading piece. err: <nil> Sep 22 03:30:39.149262646 (565h22m59.641695089s) [noise=true , address=77.90.31.12:4592 , piece_id=BLAEIKVMY5NS4MHXPKKMSR7PILM22P26B5NQ7LIOMUZDCBWPOTIA, seg_pos={0 0}, upload=497393]: Done uploading piece. err: <nil> Sep 22 03:30:39.178886890 (565h22m59.671319322s) [noise=true , address=45.91.116.13:4155 , piece_id=CE5HB226XOLG2YDRB5NWD3ZLAKRMNRR6KVBGD537TPWWKO2DMSOQ, seg_pos={0 0}, upload=497393]: Done uploading piece. err: <nil> Sep 22 03:31:07.974118417 (565h23m28.466550859s) [noise=true , address=145.79.227.15:4874 , piece_id=P5KWLBGV62V6RL6NGAC52Q3T3BBUR4MPBC6WMJ7XSO7HUM5TGLWQ, seg_pos={0 0}, upload=497393]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:31:07.974180669 (565h23m28.466613097s) [noise=true , address=145.79.227.3:4149 , piece_id=VH74CVVUPCKJ454QLNGTL7ODEDKGIA4UEZ475QEVHBBZFYZ3JQZQ, seg_pos={0 0}, upload=497393]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:31:07.974209924 (565h23m28.466642357s) [noise=true , address=145.79.227.18:4006 , piece_id=4K3KVJ6PLUCLZXTEKUTMRZTKLGL5KI45DFDB2ZUJNYXQOUTWHQWA, seg_pos={0 0}, upload=497393]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: rpc: dial tcp 145.79.227.18:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:31:07.974255115 (565h23m28.466687544s) [noise=true , address=145.79.227.14:4837 , piece_id=6OFMPXAFB7EIPJ554U62AJFNLQK3F4Z4QRRN2AJG7R3V246XVPMA, seg_pos={0 0}, upload=497393]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:31:07.974284622 (565h23m28.466717067s) [seg_pos={0 0}, upload=497393]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:31:08.076146881 (565h23m28.56857932s) [upload=497393]: Done uploading segments... Sep 22 03:31:08.076179027 (565h23m28.56861147s) [upload=497393]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.104486238s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X..........X.......X....X.X.......XX....X..X...X.......X...X] duration: 30.001719781s FAIL (2025-09-22 03:30:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001749444s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X.....XX......X..................................XX........] duration: 2.755523242s FAIL (2025-09-22 03:30:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:24:49.723373695 (565h17m10.215806124s) [upload=497309]: Starting upload Sep 22 03:24:49.723442877 (565h17m10.215875318s) [upload=497309]: Done starting upload Sep 22 03:24:49.723559207 (565h17m10.215991652s) [upload=497309]: Uploading segments... Sep 22 03:24:49.723695446 (565h17m10.216127887s) [upload=497309]: Got next segment. Inline: false Sep 22 03:24:49.723736149 (565h17m10.216168577s) [upload=497309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:24:49.723740861 (565h17m10.216173289s) [upload=497309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:24:49.871368134 (565h17m10.363800581s) [seg_pos={0 0}, upload=497309]: Begin upload segment... Sep 22 03:24:49.871435764 (565h17m10.363868213s) [seg_pos={0 0}, upload=497309]: Done begin upload segment. Sep 22 03:24:49.871487100 (565h17m10.363919548s) [noise=true , address=45.91.116.12:4225 , piece_id=NDHKLBODOXW7I6BVSUYSI4HD7JIA7RIENQTHGACCHZVWJV7OUZAQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.871946535 (565h17m10.364378979s) [noise=true , address=45.91.116.4:4940 , piece_id=4OETXZLBJNC4LOCW4JZSEETA6TCOKE6VBTKSQNZXFBGF5XCBMMXA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.872276404 (565h17m10.364708849s) [noise=true , address=77.90.31.19:4843 , piece_id=WABF6QYOWYRBKAETKWVTFENKFXFHBZGDHO45NM4C4HAQNZTSNL3A, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.872412852 (565h17m10.364845295s) [noise=true , address=77.90.31.10:4518 , piece_id=Q3ADWC5FIKIPHW62GZYUJSA2A5KEIBSTSA44QYGHIG7LKKU7WV3Q, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.872744232 (565h17m10.365176677s) [noise=true , address=45.91.116.11:4117 , piece_id=7QQGDENCX3VD6FA27UGWP5G2OY7TUZDPLWGB3KIIE522ROQPSGLA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.873123905 (565h17m10.365556356s) [noise=true , address=45.91.116.4:4939 , piece_id=N5H73PWM3J4TJ7IAF6Y2D5YBOE4N6TEDTOKDXNKISAHE5QU7UKUA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.873493977 (565h17m10.36592642s) [noise=true , address=145.79.227.16:4910 , piece_id=5OWHUIEMBGTSEELOPIVPIFIRC3NOT6TH2PMBXFHI5UR24VCGUQHA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.873817544 (565h17m10.366249987s) [noise=true , address=45.91.116.9:4263 , piece_id=557UZ2CO2FRCEQ5VMKRN7A2MZ7MVWDC3T6U7FDTFESTLTVEIKJLA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.874416869 (565h17m10.366849332s) [noise=true , address=45.91.116.13:4155 , piece_id=LDJUFT6P32JN6JG7X37OLMQIDB3YOGBVHINLI4PLMNZYF2U7OWVA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.874450690 (565h17m10.366883132s) [noise=true , address=145.79.227.16:4909 , piece_id=OJXZAJ2KMOWV55PUE5YIL36HUY4N5ZO267PUW2JBZKWWBZ5RBDJA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.874846463 (565h17m10.367278909s) [noise=true , address=45.91.116.7:4011 , piece_id=72BQFEAUQAK3N7B5NIRM4FUSGLSNO6KJW7GDBYT32U3FAI3OLZEQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.875190183 (565h17m10.367622627s) [noise=true , address=77.90.31.19:4841 , piece_id=XKJT2XATWPNLIWJYBZAOJAXAK2LQLTYBFF6QY3T67OYNVRDWPBMA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.875511139 (565h17m10.367943588s) [noise=true , address=145.79.227.5:4221 , piece_id=H5VJX26IHETQULTOMWQ5MH2RKRHDIPG75KVIGUNS7PRITJH6AB6Q, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.875869578 (565h17m10.368302022s) [noise=true , address=145.79.227.12:4767 , piece_id=W6KQ4VTOAWVWBBEWSLO7SLGZT7C5YHIPMSQIAFTWCRVK7E6XKHGA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.876186798 (565h17m10.368619242s) [noise=true , address=77.90.31.15:4698 , piece_id=QCEUAGDNIOYBBXHD3GNPCWVG6MG5VFHKCEUHLXJJ3SDBWXRAGDJQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.885649743 (565h17m10.378082172s) [noise=true , address=45.91.116.8:4045 , piece_id=FMGLHYRQQAE3BFWFOL7H6Z52SHWC4VAFH6IHO7IFJBHRGR7KRBRQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:50.485219822 (565h17m10.977652262s) [upload=497309]: Next returned nil segment Sep 22 03:24:50.485258559 (565h17m10.977691003s) [upload=497309]: Waiting for error group managing segments... Sep 22 03:24:50.626215248 (565h17m11.118647677s) [noise=true , address=45.91.116.9:4263 , piece_id=557UZ2CO2FRCEQ5VMKRN7A2MZ7MVWDC3T6U7FDTFESTLTVEIKJLA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.626240268 (565h17m11.118672703s) [noise=true , address=45.91.116.11:4117 , piece_id=7QQGDENCX3VD6FA27UGWP5G2OY7TUZDPLWGB3KIIE522ROQPSGLA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.627351929 (565h17m11.119784361s) [noise=true , address=45.91.116.4:4939 , piece_id=N5H73PWM3J4TJ7IAF6Y2D5YBOE4N6TEDTOKDXNKISAHE5QU7UKUA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.630822831 (565h17m11.123255268s) [noise=true , address=45.91.116.12:4225 , piece_id=NDHKLBODOXW7I6BVSUYSI4HD7JIA7RIENQTHGACCHZVWJV7OUZAQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.634925323 (565h17m11.127357763s) [noise=true , address=45.91.116.8:4045 , piece_id=FMGLHYRQQAE3BFWFOL7H6Z52SHWC4VAFH6IHO7IFJBHRGR7KRBRQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.635470447 (565h17m11.12790288s) [noise=true , address=77.90.31.19:4841 , piece_id=XKJT2XATWPNLIWJYBZAOJAXAK2LQLTYBFF6QY3T67OYNVRDWPBMA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.637898075 (565h17m11.130330508s) [noise=true , address=45.91.116.13:4155 , piece_id=LDJUFT6P32JN6JG7X37OLMQIDB3YOGBVHINLI4PLMNZYF2U7OWVA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.640511390 (565h17m11.132943826s) [noise=true , address=77.90.31.10:4518 , piece_id=Q3ADWC5FIKIPHW62GZYUJSA2A5KEIBSTSA44QYGHIG7LKKU7WV3Q, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.642303091 (565h17m11.134735523s) [noise=true , address=45.91.116.7:4011 , piece_id=72BQFEAUQAK3N7B5NIRM4FUSGLSNO6KJW7GDBYT32U3FAI3OLZEQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.652686616 (565h17m11.145119058s) [noise=true , address=77.90.31.19:4843 , piece_id=WABF6QYOWYRBKAETKWVTFENKFXFHBZGDHO45NM4C4HAQNZTSNL3A, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.654812594 (565h17m11.14724503s) [noise=true , address=45.91.116.4:4940 , piece_id=4OETXZLBJNC4LOCW4JZSEETA6TCOKE6VBTKSQNZXFBGF5XCBMMXA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.668718247 (565h17m11.161150738s) [noise=true , address=77.90.31.15:4698 , piece_id=QCEUAGDNIOYBBXHD3GNPCWVG6MG5VFHKCEUHLXJJ3SDBWXRAGDJQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:52.419474418 (565h17m12.911906861s) [noise=true , address=145.79.227.16:4910 , piece_id=5OWHUIEMBGTSEELOPIVPIFIRC3NOT6TH2PMBXFHI5UR24VCGUQHA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: rpc: dial tcp 145.79.227.16:4910: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:24:52.419494168 (565h17m12.911926604s) [noise=true , address=145.79.227.16:4909 , piece_id=OJXZAJ2KMOWV55PUE5YIL36HUY4N5ZO267PUW2JBZKWWBZ5RBDJA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:25:19.448227337 (565h17m39.940659785s) [noise=true , address=145.79.227.5:4221 , piece_id=H5VJX26IHETQULTOMWQ5MH2RKRHDIPG75KVIGUNS7PRITJH6AB6Q, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 22 03:25:19.448346534 (565h17m39.940778963s) [noise=true , address=145.79.227.12:4767 , piece_id=W6KQ4VTOAWVWBBEWSLO7SLGZT7C5YHIPMSQIAFTWCRVK7E6XKHGA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:25:19.448373202 (565h17m39.940805632s) [seg_pos={0 0}, upload=497309]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:25:19.549520173 (565h17m40.041952616s) [upload=497309]: Done uploading segments... Sep 22 03:25:19.549557698 (565h17m40.041990148s) [upload=497309]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.75555837s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X..........X.......X....X.X.......XX....X..X...X.......X...] duration: 2.915562905s FAIL (2025-09-22 03:29:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.915590196s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X.....XX......X..................................XX.......] duration: 2.453578314s FAIL (2025-09-22 03:29:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:24:49.723373695 (565h17m10.215806124s) [upload=497309]: Starting upload Sep 22 03:24:49.723442877 (565h17m10.215875318s) [upload=497309]: Done starting upload Sep 22 03:24:49.723559207 (565h17m10.215991652s) [upload=497309]: Uploading segments... Sep 22 03:24:49.723695446 (565h17m10.216127887s) [upload=497309]: Got next segment. Inline: false Sep 22 03:24:49.723736149 (565h17m10.216168577s) [upload=497309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:24:49.723740861 (565h17m10.216173289s) [upload=497309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:24:49.871368134 (565h17m10.363800581s) [seg_pos={0 0}, upload=497309]: Begin upload segment... Sep 22 03:24:49.871435764 (565h17m10.363868213s) [seg_pos={0 0}, upload=497309]: Done begin upload segment. Sep 22 03:24:49.871487100 (565h17m10.363919548s) [noise=true , address=45.91.116.12:4225 , piece_id=NDHKLBODOXW7I6BVSUYSI4HD7JIA7RIENQTHGACCHZVWJV7OUZAQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.871946535 (565h17m10.364378979s) [noise=true , address=45.91.116.4:4940 , piece_id=4OETXZLBJNC4LOCW4JZSEETA6TCOKE6VBTKSQNZXFBGF5XCBMMXA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.872276404 (565h17m10.364708849s) [noise=true , address=77.90.31.19:4843 , piece_id=WABF6QYOWYRBKAETKWVTFENKFXFHBZGDHO45NM4C4HAQNZTSNL3A, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.872412852 (565h17m10.364845295s) [noise=true , address=77.90.31.10:4518 , piece_id=Q3ADWC5FIKIPHW62GZYUJSA2A5KEIBSTSA44QYGHIG7LKKU7WV3Q, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.872744232 (565h17m10.365176677s) [noise=true , address=45.91.116.11:4117 , piece_id=7QQGDENCX3VD6FA27UGWP5G2OY7TUZDPLWGB3KIIE522ROQPSGLA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.873123905 (565h17m10.365556356s) [noise=true , address=45.91.116.4:4939 , piece_id=N5H73PWM3J4TJ7IAF6Y2D5YBOE4N6TEDTOKDXNKISAHE5QU7UKUA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.873493977 (565h17m10.36592642s) [noise=true , address=145.79.227.16:4910 , piece_id=5OWHUIEMBGTSEELOPIVPIFIRC3NOT6TH2PMBXFHI5UR24VCGUQHA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.873817544 (565h17m10.366249987s) [noise=true , address=45.91.116.9:4263 , piece_id=557UZ2CO2FRCEQ5VMKRN7A2MZ7MVWDC3T6U7FDTFESTLTVEIKJLA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.874416869 (565h17m10.366849332s) [noise=true , address=45.91.116.13:4155 , piece_id=LDJUFT6P32JN6JG7X37OLMQIDB3YOGBVHINLI4PLMNZYF2U7OWVA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.874450690 (565h17m10.366883132s) [noise=true , address=145.79.227.16:4909 , piece_id=OJXZAJ2KMOWV55PUE5YIL36HUY4N5ZO267PUW2JBZKWWBZ5RBDJA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.874846463 (565h17m10.367278909s) [noise=true , address=45.91.116.7:4011 , piece_id=72BQFEAUQAK3N7B5NIRM4FUSGLSNO6KJW7GDBYT32U3FAI3OLZEQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.875190183 (565h17m10.367622627s) [noise=true , address=77.90.31.19:4841 , piece_id=XKJT2XATWPNLIWJYBZAOJAXAK2LQLTYBFF6QY3T67OYNVRDWPBMA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.875511139 (565h17m10.367943588s) [noise=true , address=145.79.227.5:4221 , piece_id=H5VJX26IHETQULTOMWQ5MH2RKRHDIPG75KVIGUNS7PRITJH6AB6Q, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.875869578 (565h17m10.368302022s) [noise=true , address=145.79.227.12:4767 , piece_id=W6KQ4VTOAWVWBBEWSLO7SLGZT7C5YHIPMSQIAFTWCRVK7E6XKHGA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.876186798 (565h17m10.368619242s) [noise=true , address=77.90.31.15:4698 , piece_id=QCEUAGDNIOYBBXHD3GNPCWVG6MG5VFHKCEUHLXJJ3SDBWXRAGDJQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.885649743 (565h17m10.378082172s) [noise=true , address=45.91.116.8:4045 , piece_id=FMGLHYRQQAE3BFWFOL7H6Z52SHWC4VAFH6IHO7IFJBHRGR7KRBRQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:50.485219822 (565h17m10.977652262s) [upload=497309]: Next returned nil segment Sep 22 03:24:50.485258559 (565h17m10.977691003s) [upload=497309]: Waiting for error group managing segments... Sep 22 03:24:50.626215248 (565h17m11.118647677s) [noise=true , address=45.91.116.9:4263 , piece_id=557UZ2CO2FRCEQ5VMKRN7A2MZ7MVWDC3T6U7FDTFESTLTVEIKJLA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.626240268 (565h17m11.118672703s) [noise=true , address=45.91.116.11:4117 , piece_id=7QQGDENCX3VD6FA27UGWP5G2OY7TUZDPLWGB3KIIE522ROQPSGLA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.627351929 (565h17m11.119784361s) [noise=true , address=45.91.116.4:4939 , piece_id=N5H73PWM3J4TJ7IAF6Y2D5YBOE4N6TEDTOKDXNKISAHE5QU7UKUA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.630822831 (565h17m11.123255268s) [noise=true , address=45.91.116.12:4225 , piece_id=NDHKLBODOXW7I6BVSUYSI4HD7JIA7RIENQTHGACCHZVWJV7OUZAQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.634925323 (565h17m11.127357763s) [noise=true , address=45.91.116.8:4045 , piece_id=FMGLHYRQQAE3BFWFOL7H6Z52SHWC4VAFH6IHO7IFJBHRGR7KRBRQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.635470447 (565h17m11.12790288s) [noise=true , address=77.90.31.19:4841 , piece_id=XKJT2XATWPNLIWJYBZAOJAXAK2LQLTYBFF6QY3T67OYNVRDWPBMA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.637898075 (565h17m11.130330508s) [noise=true , address=45.91.116.13:4155 , piece_id=LDJUFT6P32JN6JG7X37OLMQIDB3YOGBVHINLI4PLMNZYF2U7OWVA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.640511390 (565h17m11.132943826s) [noise=true , address=77.90.31.10:4518 , piece_id=Q3ADWC5FIKIPHW62GZYUJSA2A5KEIBSTSA44QYGHIG7LKKU7WV3Q, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.642303091 (565h17m11.134735523s) [noise=true , address=45.91.116.7:4011 , piece_id=72BQFEAUQAK3N7B5NIRM4FUSGLSNO6KJW7GDBYT32U3FAI3OLZEQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.652686616 (565h17m11.145119058s) [noise=true , address=77.90.31.19:4843 , piece_id=WABF6QYOWYRBKAETKWVTFENKFXFHBZGDHO45NM4C4HAQNZTSNL3A, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.654812594 (565h17m11.14724503s) [noise=true , address=45.91.116.4:4940 , piece_id=4OETXZLBJNC4LOCW4JZSEETA6TCOKE6VBTKSQNZXFBGF5XCBMMXA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.668718247 (565h17m11.161150738s) [noise=true , address=77.90.31.15:4698 , piece_id=QCEUAGDNIOYBBXHD3GNPCWVG6MG5VFHKCEUHLXJJ3SDBWXRAGDJQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:52.419474418 (565h17m12.911906861s) [noise=true , address=145.79.227.16:4910 , piece_id=5OWHUIEMBGTSEELOPIVPIFIRC3NOT6TH2PMBXFHI5UR24VCGUQHA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: rpc: dial tcp 145.79.227.16:4910: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:24:52.419494168 (565h17m12.911926604s) [noise=true , address=145.79.227.16:4909 , piece_id=OJXZAJ2KMOWV55PUE5YIL36HUY4N5ZO267PUW2JBZKWWBZ5RBDJA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:25:19.448227337 (565h17m39.940659785s) [noise=true , address=145.79.227.5:4221 , piece_id=H5VJX26IHETQULTOMWQ5MH2RKRHDIPG75KVIGUNS7PRITJH6AB6Q, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 22 03:25:19.448346534 (565h17m39.940778963s) [noise=true , address=145.79.227.12:4767 , piece_id=W6KQ4VTOAWVWBBEWSLO7SLGZT7C5YHIPMSQIAFTWCRVK7E6XKHGA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:25:19.448373202 (565h17m39.940805632s) [seg_pos={0 0}, upload=497309]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:25:19.549520173 (565h17m40.041952616s) [upload=497309]: Done uploading segments... Sep 22 03:25:19.549557698 (565h17m40.041990148s) [upload=497309]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.453613058s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)