Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.344426838s FAIL (2025-07-09 01:37:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== cessful: 13 optimal: 13 errs: <nil> Jul 9 01:12:51.706672230 (376h1m51.590085861s) [upload=300705]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:12:51.706677984 (376h1m51.590091617s) [upload=300705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:12:51.883113268 (376h1m51.76652692s) [upload=300705]: Done uploading segments... Jul 9 01:12:51.883135264 (376h1m51.766548909s) [upload=300705]: Upload finished. err: <nil> Jul 9 01:13:20.653341358 (376h2m20.536754991s) [upload=300711]: Starting upload Jul 9 01:13:20.653504915 (376h2m20.536918556s) [upload=300711]: Done starting upload Jul 9 01:13:20.653690252 (376h2m20.537103868s) [upload=300711]: Uploading segments... Jul 9 01:13:20.653990712 (376h2m20.537404332s) [upload=300711]: Got next segment. Inline: false Jul 9 01:13:20.654059122 (376h2m20.537472753s) [upload=300711]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:13:20.654065941 (376h2m20.537479574s) [upload=300711]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:13:20.769797764 (376h2m20.653211413s) [seg_pos={0 0}, upload=300711]: Begin upload segment... Jul 9 01:13:20.769876747 (376h2m20.653290386s) [seg_pos={0 0}, upload=300711]: Done begin upload segment. Jul 9 01:13:20.769954018 (376h2m20.653367648s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770052187 (376h2m20.653465935s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770659155 (376h2m20.6540728s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770677751 (376h2m20.654091394s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771126425 (376h2m20.654540056s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771277890 (376h2m20.654691526s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771476972 (376h2m20.654890603s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771804253 (376h2m20.655217892s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771829179 (376h2m20.655242816s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771909583 (376h2m20.655323221s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.772476215 (376h2m20.655889855s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.773678060 (376h2m20.657091699s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.774188246 (376h2m20.657601884s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.774573193 (376h2m20.657986824s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.775340796 (376h2m20.658754434s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.775786189 (376h2m20.659199809s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:21.378914824 (376h2m21.262328447s) [upload=300711]: Next returned nil segment Jul 9 01:13:21.378938224 (376h2m21.262351854s) [upload=300711]: Waiting for error group managing segments... Jul 9 01:13:21.517933925 (376h2m21.401347538s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.518663136 (376h2m21.402076749s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.522226581 (376h2m21.40564021s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.535639134 (376h2m21.419052751s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.539204781 (376h2m21.422618407s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.545311795 (376h2m21.428725407s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.548650688 (376h2m21.432064308s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.548766114 (376h2m21.432179732s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.552618914 (376h2m21.436032531s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.556234201 (376h2m21.439647819s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.556714631 (376h2m21.440128256s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.584297058 (376h2m21.467710677s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:50.371498503 (376h2m50.254912119s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:13:50.371544262 (376h2m50.254957891s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection Jul 9 01:13:50.371667019 (376h2m50.255080637s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection Jul 9 01:13:50.371684908 (376h2m50.255098535s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection Jul 9 01:13:50.371707098 (376h2m50.255120766s) [seg_pos={0 0}, upload=300711]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:13:50.484313152 (376h2m50.367726771s) [upload=300711]: Done uploading segments... Jul 9 01:13:50.484347418 (376h2m50.367761038s) [upload=300711]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.344457656s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 25 out of 60 is failed (threshold=3) [X...........XXXXX...X.........X..X...XXX..XXXX.XXXXX..X.XXX.] duration: 46.054409217s FAIL (2025-07-09 01:37:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 46.054436027s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X.X......X.X.XXXXX.......XX....................X..X..X.....] duration: 4.240055387s FAIL (2025-07-09 01:37:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.240129721s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X.X.X......X.X.XXXXX.......XX....................X..X..X....] duration: 2.618912724s FAIL (2025-07-09 01:36:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.618938184s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 15.231001042s FAIL (2025-07-09 01:36:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== cessful: 13 optimal: 13 errs: <nil> Jul 9 01:12:51.706672230 (376h1m51.590085861s) [upload=300705]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:12:51.706677984 (376h1m51.590091617s) [upload=300705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:12:51.883113268 (376h1m51.76652692s) [upload=300705]: Done uploading segments... Jul 9 01:12:51.883135264 (376h1m51.766548909s) [upload=300705]: Upload finished. err: <nil> Jul 9 01:13:20.653341358 (376h2m20.536754991s) [upload=300711]: Starting upload Jul 9 01:13:20.653504915 (376h2m20.536918556s) [upload=300711]: Done starting upload Jul 9 01:13:20.653690252 (376h2m20.537103868s) [upload=300711]: Uploading segments... Jul 9 01:13:20.653990712 (376h2m20.537404332s) [upload=300711]: Got next segment. Inline: false Jul 9 01:13:20.654059122 (376h2m20.537472753s) [upload=300711]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:13:20.654065941 (376h2m20.537479574s) [upload=300711]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:13:20.769797764 (376h2m20.653211413s) [seg_pos={0 0}, upload=300711]: Begin upload segment... Jul 9 01:13:20.769876747 (376h2m20.653290386s) [seg_pos={0 0}, upload=300711]: Done begin upload segment. Jul 9 01:13:20.769954018 (376h2m20.653367648s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770052187 (376h2m20.653465935s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770659155 (376h2m20.6540728s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770677751 (376h2m20.654091394s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771126425 (376h2m20.654540056s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771277890 (376h2m20.654691526s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771476972 (376h2m20.654890603s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771804253 (376h2m20.655217892s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771829179 (376h2m20.655242816s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771909583 (376h2m20.655323221s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.772476215 (376h2m20.655889855s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.773678060 (376h2m20.657091699s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.774188246 (376h2m20.657601884s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.774573193 (376h2m20.657986824s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.775340796 (376h2m20.658754434s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.775786189 (376h2m20.659199809s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:21.378914824 (376h2m21.262328447s) [upload=300711]: Next returned nil segment Jul 9 01:13:21.378938224 (376h2m21.262351854s) [upload=300711]: Waiting for error group managing segments... Jul 9 01:13:21.517933925 (376h2m21.401347538s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.518663136 (376h2m21.402076749s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.522226581 (376h2m21.40564021s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.535639134 (376h2m21.419052751s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.539204781 (376h2m21.422618407s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.545311795 (376h2m21.428725407s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.548650688 (376h2m21.432064308s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.548766114 (376h2m21.432179732s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.552618914 (376h2m21.436032531s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.556234201 (376h2m21.439647819s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.556714631 (376h2m21.440128256s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.584297058 (376h2m21.467710677s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:50.371498503 (376h2m50.254912119s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:13:50.371544262 (376h2m50.254957891s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection Jul 9 01:13:50.371667019 (376h2m50.255080637s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection Jul 9 01:13:50.371684908 (376h2m50.255098535s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection Jul 9 01:13:50.371707098 (376h2m50.255120766s) [seg_pos={0 0}, upload=300711]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:13:50.484313152 (376h2m50.367726771s) [upload=300711]: Done uploading segments... Jul 9 01:13:50.484347418 (376h2m50.367761038s) [upload=300711]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 15.231031296s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 25 out of 60 is failed (threshold=3) [.X...........XXXXX...X.........X..X...XXX..XXXX.XXXXX..X.XXX] duration: 1m0.001894752s FAIL (2025-07-09 01:36:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00194385s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XX.X.X......X.X.XXXXX.......XX....................X..X..X...] duration: 20.743853877s FAIL (2025-07-09 01:36:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.74388414s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.483893597s FAIL (2025-07-09 01:35:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== cessful: 13 optimal: 13 errs: <nil> Jul 9 01:12:51.706672230 (376h1m51.590085861s) [upload=300705]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:12:51.706677984 (376h1m51.590091617s) [upload=300705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:12:51.883113268 (376h1m51.76652692s) [upload=300705]: Done uploading segments... Jul 9 01:12:51.883135264 (376h1m51.766548909s) [upload=300705]: Upload finished. err: <nil> Jul 9 01:13:20.653341358 (376h2m20.536754991s) [upload=300711]: Starting upload Jul 9 01:13:20.653504915 (376h2m20.536918556s) [upload=300711]: Done starting upload Jul 9 01:13:20.653690252 (376h2m20.537103868s) [upload=300711]: Uploading segments... Jul 9 01:13:20.653990712 (376h2m20.537404332s) [upload=300711]: Got next segment. Inline: false Jul 9 01:13:20.654059122 (376h2m20.537472753s) [upload=300711]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:13:20.654065941 (376h2m20.537479574s) [upload=300711]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:13:20.769797764 (376h2m20.653211413s) [seg_pos={0 0}, upload=300711]: Begin upload segment... Jul 9 01:13:20.769876747 (376h2m20.653290386s) [seg_pos={0 0}, upload=300711]: Done begin upload segment. Jul 9 01:13:20.769954018 (376h2m20.653367648s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770052187 (376h2m20.653465935s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770659155 (376h2m20.6540728s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770677751 (376h2m20.654091394s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771126425 (376h2m20.654540056s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771277890 (376h2m20.654691526s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771476972 (376h2m20.654890603s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771804253 (376h2m20.655217892s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771829179 (376h2m20.655242816s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771909583 (376h2m20.655323221s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.772476215 (376h2m20.655889855s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.773678060 (376h2m20.657091699s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.774188246 (376h2m20.657601884s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.774573193 (376h2m20.657986824s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.775340796 (376h2m20.658754434s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.775786189 (376h2m20.659199809s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:21.378914824 (376h2m21.262328447s) [upload=300711]: Next returned nil segment Jul 9 01:13:21.378938224 (376h2m21.262351854s) [upload=300711]: Waiting for error group managing segments... Jul 9 01:13:21.517933925 (376h2m21.401347538s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.518663136 (376h2m21.402076749s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.522226581 (376h2m21.40564021s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.535639134 (376h2m21.419052751s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.539204781 (376h2m21.422618407s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.545311795 (376h2m21.428725407s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.548650688 (376h2m21.432064308s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.548766114 (376h2m21.432179732s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.552618914 (376h2m21.436032531s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.556234201 (376h2m21.439647819s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.556714631 (376h2m21.440128256s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.584297058 (376h2m21.467710677s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:50.371498503 (376h2m50.254912119s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:13:50.371544262 (376h2m50.254957891s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection Jul 9 01:13:50.371667019 (376h2m50.255080637s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection Jul 9 01:13:50.371684908 (376h2m50.255098535s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection Jul 9 01:13:50.371707098 (376h2m50.255120766s) [seg_pos={0 0}, upload=300711]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:13:50.484313152 (376h2m50.367726771s) [upload=300711]: Done uploading segments... Jul 9 01:13:50.484347418 (376h2m50.367761038s) [upload=300711]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.483921984s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 25 out of 60 is failed (threshold=3) [X.X...........XXXXX...X.........X..X...XXX..XXXX.XXXXX..X.XX] duration: 1m0.001867054s FAIL (2025-07-09 01:35:21 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001886982s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.XX.X.X......X.X.XXXXX.......XX....................X..X..X..] duration: 20.90780575s FAIL (2025-07-09 01:35:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.907828121s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.294937895s FAIL (2025-07-09 01:34:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== cessful: 13 optimal: 13 errs: <nil> Jul 9 01:12:51.706672230 (376h1m51.590085861s) [upload=300705]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:12:51.706677984 (376h1m51.590091617s) [upload=300705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:12:51.883113268 (376h1m51.76652692s) [upload=300705]: Done uploading segments... Jul 9 01:12:51.883135264 (376h1m51.766548909s) [upload=300705]: Upload finished. err: <nil> Jul 9 01:13:20.653341358 (376h2m20.536754991s) [upload=300711]: Starting upload Jul 9 01:13:20.653504915 (376h2m20.536918556s) [upload=300711]: Done starting upload Jul 9 01:13:20.653690252 (376h2m20.537103868s) [upload=300711]: Uploading segments... Jul 9 01:13:20.653990712 (376h2m20.537404332s) [upload=300711]: Got next segment. Inline: false Jul 9 01:13:20.654059122 (376h2m20.537472753s) [upload=300711]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:13:20.654065941 (376h2m20.537479574s) [upload=300711]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:13:20.769797764 (376h2m20.653211413s) [seg_pos={0 0}, upload=300711]: Begin upload segment... Jul 9 01:13:20.769876747 (376h2m20.653290386s) [seg_pos={0 0}, upload=300711]: Done begin upload segment. Jul 9 01:13:20.769954018 (376h2m20.653367648s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770052187 (376h2m20.653465935s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770659155 (376h2m20.6540728s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770677751 (376h2m20.654091394s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771126425 (376h2m20.654540056s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771277890 (376h2m20.654691526s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771476972 (376h2m20.654890603s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771804253 (376h2m20.655217892s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771829179 (376h2m20.655242816s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771909583 (376h2m20.655323221s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.772476215 (376h2m20.655889855s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.773678060 (376h2m20.657091699s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.774188246 (376h2m20.657601884s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.774573193 (376h2m20.657986824s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.775340796 (376h2m20.658754434s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.775786189 (376h2m20.659199809s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:21.378914824 (376h2m21.262328447s) [upload=300711]: Next returned nil segment Jul 9 01:13:21.378938224 (376h2m21.262351854s) [upload=300711]: Waiting for error group managing segments... Jul 9 01:13:21.517933925 (376h2m21.401347538s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.518663136 (376h2m21.402076749s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.522226581 (376h2m21.40564021s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.535639134 (376h2m21.419052751s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.539204781 (376h2m21.422618407s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.545311795 (376h2m21.428725407s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.548650688 (376h2m21.432064308s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.548766114 (376h2m21.432179732s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.552618914 (376h2m21.436032531s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.556234201 (376h2m21.439647819s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.556714631 (376h2m21.440128256s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.584297058 (376h2m21.467710677s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:50.371498503 (376h2m50.254912119s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:13:50.371544262 (376h2m50.254957891s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection Jul 9 01:13:50.371667019 (376h2m50.255080637s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection Jul 9 01:13:50.371684908 (376h2m50.255098535s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection Jul 9 01:13:50.371707098 (376h2m50.255120766s) [seg_pos={0 0}, upload=300711]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:13:50.484313152 (376h2m50.367726771s) [upload=300711]: Done uploading segments... Jul 9 01:13:50.484347418 (376h2m50.367761038s) [upload=300711]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.294976959s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.36815053s FAIL (2025-07-09 01:34:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== cessful: 13 optimal: 13 errs: <nil> Jul 9 01:12:51.706672230 (376h1m51.590085861s) [upload=300705]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:12:51.706677984 (376h1m51.590091617s) [upload=300705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:12:51.883113268 (376h1m51.76652692s) [upload=300705]: Done uploading segments... Jul 9 01:12:51.883135264 (376h1m51.766548909s) [upload=300705]: Upload finished. err: <nil> Jul 9 01:13:20.653341358 (376h2m20.536754991s) [upload=300711]: Starting upload Jul 9 01:13:20.653504915 (376h2m20.536918556s) [upload=300711]: Done starting upload Jul 9 01:13:20.653690252 (376h2m20.537103868s) [upload=300711]: Uploading segments... Jul 9 01:13:20.653990712 (376h2m20.537404332s) [upload=300711]: Got next segment. Inline: false Jul 9 01:13:20.654059122 (376h2m20.537472753s) [upload=300711]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:13:20.654065941 (376h2m20.537479574s) [upload=300711]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:13:20.769797764 (376h2m20.653211413s) [seg_pos={0 0}, upload=300711]: Begin upload segment... Jul 9 01:13:20.769876747 (376h2m20.653290386s) [seg_pos={0 0}, upload=300711]: Done begin upload segment. Jul 9 01:13:20.769954018 (376h2m20.653367648s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770052187 (376h2m20.653465935s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770659155 (376h2m20.6540728s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770677751 (376h2m20.654091394s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771126425 (376h2m20.654540056s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771277890 (376h2m20.654691526s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771476972 (376h2m20.654890603s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771804253 (376h2m20.655217892s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771829179 (376h2m20.655242816s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771909583 (376h2m20.655323221s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.772476215 (376h2m20.655889855s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.773678060 (376h2m20.657091699s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.774188246 (376h2m20.657601884s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.774573193 (376h2m20.657986824s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.775340796 (376h2m20.658754434s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.775786189 (376h2m20.659199809s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:21.378914824 (376h2m21.262328447s) [upload=300711]: Next returned nil segment Jul 9 01:13:21.378938224 (376h2m21.262351854s) [upload=300711]: Waiting for error group managing segments... Jul 9 01:13:21.517933925 (376h2m21.401347538s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.518663136 (376h2m21.402076749s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.522226581 (376h2m21.40564021s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.535639134 (376h2m21.419052751s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.539204781 (376h2m21.422618407s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.545311795 (376h2m21.428725407s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.548650688 (376h2m21.432064308s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.548766114 (376h2m21.432179732s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.552618914 (376h2m21.436032531s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.556234201 (376h2m21.439647819s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.556714631 (376h2m21.440128256s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.584297058 (376h2m21.467710677s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:50.371498503 (376h2m50.254912119s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:13:50.371544262 (376h2m50.254957891s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection Jul 9 01:13:50.371667019 (376h2m50.255080637s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection Jul 9 01:13:50.371684908 (376h2m50.255098535s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection Jul 9 01:13:50.371707098 (376h2m50.255120766s) [seg_pos={0 0}, upload=300711]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:13:50.484313152 (376h2m50.367726771s) [upload=300711]: Done uploading segments... Jul 9 01:13:50.484347418 (376h2m50.367761038s) [upload=300711]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.368181156s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..XX.X.X......X.X.XXXXX.......XX....................X..X..X.] duration: 2.733723196s FAIL (2025-07-09 01:34:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.733759707s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 25 out of 60 is failed (threshold=3) [XX.X...........XXXXX...X.........X..X...XXX..XXXX.XXXXX..X.X] duration: 1m0.001603577s FAIL (2025-07-09 01:34:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001632503s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 11.430970913s FAIL (2025-07-09 01:34:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== cessful: 13 optimal: 13 errs: <nil> Jul 9 01:12:51.706672230 (376h1m51.590085861s) [upload=300705]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:12:51.706677984 (376h1m51.590091617s) [upload=300705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:12:51.883113268 (376h1m51.76652692s) [upload=300705]: Done uploading segments... Jul 9 01:12:51.883135264 (376h1m51.766548909s) [upload=300705]: Upload finished. err: <nil> Jul 9 01:13:20.653341358 (376h2m20.536754991s) [upload=300711]: Starting upload Jul 9 01:13:20.653504915 (376h2m20.536918556s) [upload=300711]: Done starting upload Jul 9 01:13:20.653690252 (376h2m20.537103868s) [upload=300711]: Uploading segments... Jul 9 01:13:20.653990712 (376h2m20.537404332s) [upload=300711]: Got next segment. Inline: false Jul 9 01:13:20.654059122 (376h2m20.537472753s) [upload=300711]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:13:20.654065941 (376h2m20.537479574s) [upload=300711]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:13:20.769797764 (376h2m20.653211413s) [seg_pos={0 0}, upload=300711]: Begin upload segment... Jul 9 01:13:20.769876747 (376h2m20.653290386s) [seg_pos={0 0}, upload=300711]: Done begin upload segment. Jul 9 01:13:20.769954018 (376h2m20.653367648s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770052187 (376h2m20.653465935s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770659155 (376h2m20.6540728s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770677751 (376h2m20.654091394s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771126425 (376h2m20.654540056s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771277890 (376h2m20.654691526s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771476972 (376h2m20.654890603s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771804253 (376h2m20.655217892s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771829179 (376h2m20.655242816s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771909583 (376h2m20.655323221s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.772476215 (376h2m20.655889855s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.773678060 (376h2m20.657091699s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.774188246 (376h2m20.657601884s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.774573193 (376h2m20.657986824s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.775340796 (376h2m20.658754434s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.775786189 (376h2m20.659199809s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:21.378914824 (376h2m21.262328447s) [upload=300711]: Next returned nil segment Jul 9 01:13:21.378938224 (376h2m21.262351854s) [upload=300711]: Waiting for error group managing segments... Jul 9 01:13:21.517933925 (376h2m21.401347538s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.518663136 (376h2m21.402076749s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.522226581 (376h2m21.40564021s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.535639134 (376h2m21.419052751s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.539204781 (376h2m21.422618407s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.545311795 (376h2m21.428725407s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.548650688 (376h2m21.432064308s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.548766114 (376h2m21.432179732s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.552618914 (376h2m21.436032531s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.556234201 (376h2m21.439647819s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.556714631 (376h2m21.440128256s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.584297058 (376h2m21.467710677s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:50.371498503 (376h2m50.254912119s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:13:50.371544262 (376h2m50.254957891s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection Jul 9 01:13:50.371667019 (376h2m50.255080637s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection Jul 9 01:13:50.371684908 (376h2m50.255098535s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection Jul 9 01:13:50.371707098 (376h2m50.255120766s) [seg_pos={0 0}, upload=300711]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:13:50.484313152 (376h2m50.367726771s) [upload=300711]: Done uploading segments... Jul 9 01:13:50.484347418 (376h2m50.367761038s) [upload=300711]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 11.431005242s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...XX.X.X......X.X.XXXXX.......XX....................X..X..X] duration: 30.000507582s FAIL (2025-07-09 01:34:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000533703s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [....XX.X.X......X.X.XXXXX.......XX....................X..X..] duration: 7.714821883s FAIL (2025-07-09 01:33:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.714839439s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.29162088s FAIL (2025-07-09 01:32:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== cessful: 13 optimal: 13 errs: <nil> Jul 9 01:12:51.706672230 (376h1m51.590085861s) [upload=300705]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:12:51.706677984 (376h1m51.590091617s) [upload=300705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:12:51.883113268 (376h1m51.76652692s) [upload=300705]: Done uploading segments... Jul 9 01:12:51.883135264 (376h1m51.766548909s) [upload=300705]: Upload finished. err: <nil> Jul 9 01:13:20.653341358 (376h2m20.536754991s) [upload=300711]: Starting upload Jul 9 01:13:20.653504915 (376h2m20.536918556s) [upload=300711]: Done starting upload Jul 9 01:13:20.653690252 (376h2m20.537103868s) [upload=300711]: Uploading segments... Jul 9 01:13:20.653990712 (376h2m20.537404332s) [upload=300711]: Got next segment. Inline: false Jul 9 01:13:20.654059122 (376h2m20.537472753s) [upload=300711]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:13:20.654065941 (376h2m20.537479574s) [upload=300711]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:13:20.769797764 (376h2m20.653211413s) [seg_pos={0 0}, upload=300711]: Begin upload segment... Jul 9 01:13:20.769876747 (376h2m20.653290386s) [seg_pos={0 0}, upload=300711]: Done begin upload segment. Jul 9 01:13:20.769954018 (376h2m20.653367648s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770052187 (376h2m20.653465935s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770659155 (376h2m20.6540728s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770677751 (376h2m20.654091394s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771126425 (376h2m20.654540056s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771277890 (376h2m20.654691526s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771476972 (376h2m20.654890603s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771804253 (376h2m20.655217892s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771829179 (376h2m20.655242816s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771909583 (376h2m20.655323221s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.772476215 (376h2m20.655889855s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.773678060 (376h2m20.657091699s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.774188246 (376h2m20.657601884s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.774573193 (376h2m20.657986824s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.775340796 (376h2m20.658754434s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.775786189 (376h2m20.659199809s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:21.378914824 (376h2m21.262328447s) [upload=300711]: Next returned nil segment Jul 9 01:13:21.378938224 (376h2m21.262351854s) [upload=300711]: Waiting for error group managing segments... Jul 9 01:13:21.517933925 (376h2m21.401347538s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.518663136 (376h2m21.402076749s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.522226581 (376h2m21.40564021s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.535639134 (376h2m21.419052751s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.539204781 (376h2m21.422618407s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.545311795 (376h2m21.428725407s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.548650688 (376h2m21.432064308s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.548766114 (376h2m21.432179732s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.552618914 (376h2m21.436032531s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.556234201 (376h2m21.439647819s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.556714631 (376h2m21.440128256s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.584297058 (376h2m21.467710677s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:50.371498503 (376h2m50.254912119s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:13:50.371544262 (376h2m50.254957891s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection Jul 9 01:13:50.371667019 (376h2m50.255080637s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection Jul 9 01:13:50.371684908 (376h2m50.255098535s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection Jul 9 01:13:50.371707098 (376h2m50.255120766s) [seg_pos={0 0}, upload=300711]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:13:50.484313152 (376h2m50.367726771s) [upload=300711]: Done uploading segments... Jul 9 01:13:50.484347418 (376h2m50.367761038s) [upload=300711]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.291653056s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.....XX.X.X......X.X.XXXXX.......XX....................X..X.] duration: 22.202680186s FAIL (2025-07-09 01:32:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.202983698s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 24 out of 60 is failed (threshold=3) [.XX.X...........XXXXX...X.........X..X...XXX..XXXX.XXXXX..X.] duration: 13.548437566s FAIL (2025-07-09 01:32:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.54845226s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 24 out of 60 is failed (threshold=3) [..XX.X...........XXXXX...X.........X..X...XXX..XXXX.XXXXX..X] duration: 1m0.000969375s FAIL (2025-07-09 01:32:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000990929s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.490816594s FAIL (2025-07-09 01:32:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== cessful: 13 optimal: 13 errs: <nil> Jul 9 01:12:51.706672230 (376h1m51.590085861s) [upload=300705]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:12:51.706677984 (376h1m51.590091617s) [upload=300705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:12:51.883113268 (376h1m51.76652692s) [upload=300705]: Done uploading segments... Jul 9 01:12:51.883135264 (376h1m51.766548909s) [upload=300705]: Upload finished. err: <nil> Jul 9 01:13:20.653341358 (376h2m20.536754991s) [upload=300711]: Starting upload Jul 9 01:13:20.653504915 (376h2m20.536918556s) [upload=300711]: Done starting upload Jul 9 01:13:20.653690252 (376h2m20.537103868s) [upload=300711]: Uploading segments... Jul 9 01:13:20.653990712 (376h2m20.537404332s) [upload=300711]: Got next segment. Inline: false Jul 9 01:13:20.654059122 (376h2m20.537472753s) [upload=300711]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:13:20.654065941 (376h2m20.537479574s) [upload=300711]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:13:20.769797764 (376h2m20.653211413s) [seg_pos={0 0}, upload=300711]: Begin upload segment... Jul 9 01:13:20.769876747 (376h2m20.653290386s) [seg_pos={0 0}, upload=300711]: Done begin upload segment. Jul 9 01:13:20.769954018 (376h2m20.653367648s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770052187 (376h2m20.653465935s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770659155 (376h2m20.6540728s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770677751 (376h2m20.654091394s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771126425 (376h2m20.654540056s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771277890 (376h2m20.654691526s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771476972 (376h2m20.654890603s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771804253 (376h2m20.655217892s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771829179 (376h2m20.655242816s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771909583 (376h2m20.655323221s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.772476215 (376h2m20.655889855s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.773678060 (376h2m20.657091699s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.774188246 (376h2m20.657601884s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.774573193 (376h2m20.657986824s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.775340796 (376h2m20.658754434s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.775786189 (376h2m20.659199809s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:21.378914824 (376h2m21.262328447s) [upload=300711]: Next returned nil segment Jul 9 01:13:21.378938224 (376h2m21.262351854s) [upload=300711]: Waiting for error group managing segments... Jul 9 01:13:21.517933925 (376h2m21.401347538s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.518663136 (376h2m21.402076749s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.522226581 (376h2m21.40564021s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.535639134 (376h2m21.419052751s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.539204781 (376h2m21.422618407s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.545311795 (376h2m21.428725407s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.548650688 (376h2m21.432064308s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.548766114 (376h2m21.432179732s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.552618914 (376h2m21.436032531s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.556234201 (376h2m21.439647819s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.556714631 (376h2m21.440128256s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.584297058 (376h2m21.467710677s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:50.371498503 (376h2m50.254912119s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:13:50.371544262 (376h2m50.254957891s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection Jul 9 01:13:50.371667019 (376h2m50.255080637s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection Jul 9 01:13:50.371684908 (376h2m50.255098535s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection Jul 9 01:13:50.371707098 (376h2m50.255120766s) [seg_pos={0 0}, upload=300711]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:13:50.484313152 (376h2m50.367726771s) [upload=300711]: Done uploading segments... Jul 9 01:13:50.484347418 (376h2m50.367761038s) [upload=300711]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.490849547s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [......XX.X.X......X.X.XXXXX.......XX....................X..X] duration: 30.001245564s FAIL (2025-07-09 01:31:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001375311s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.272149423s FAIL (2025-07-09 01:31:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== cessful: 13 optimal: 13 errs: <nil> Jul 9 01:12:51.706672230 (376h1m51.590085861s) [upload=300705]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:12:51.706677984 (376h1m51.590091617s) [upload=300705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:12:51.883113268 (376h1m51.76652692s) [upload=300705]: Done uploading segments... Jul 9 01:12:51.883135264 (376h1m51.766548909s) [upload=300705]: Upload finished. err: <nil> Jul 9 01:13:20.653341358 (376h2m20.536754991s) [upload=300711]: Starting upload Jul 9 01:13:20.653504915 (376h2m20.536918556s) [upload=300711]: Done starting upload Jul 9 01:13:20.653690252 (376h2m20.537103868s) [upload=300711]: Uploading segments... Jul 9 01:13:20.653990712 (376h2m20.537404332s) [upload=300711]: Got next segment. Inline: false Jul 9 01:13:20.654059122 (376h2m20.537472753s) [upload=300711]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:13:20.654065941 (376h2m20.537479574s) [upload=300711]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:13:20.769797764 (376h2m20.653211413s) [seg_pos={0 0}, upload=300711]: Begin upload segment... Jul 9 01:13:20.769876747 (376h2m20.653290386s) [seg_pos={0 0}, upload=300711]: Done begin upload segment. Jul 9 01:13:20.769954018 (376h2m20.653367648s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770052187 (376h2m20.653465935s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770659155 (376h2m20.6540728s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770677751 (376h2m20.654091394s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771126425 (376h2m20.654540056s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771277890 (376h2m20.654691526s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771476972 (376h2m20.654890603s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771804253 (376h2m20.655217892s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771829179 (376h2m20.655242816s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771909583 (376h2m20.655323221s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.772476215 (376h2m20.655889855s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.773678060 (376h2m20.657091699s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.774188246 (376h2m20.657601884s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.774573193 (376h2m20.657986824s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.775340796 (376h2m20.658754434s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.775786189 (376h2m20.659199809s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:21.378914824 (376h2m21.262328447s) [upload=300711]: Next returned nil segment Jul 9 01:13:21.378938224 (376h2m21.262351854s) [upload=300711]: Waiting for error group managing segments... Jul 9 01:13:21.517933925 (376h2m21.401347538s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.518663136 (376h2m21.402076749s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.522226581 (376h2m21.40564021s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.535639134 (376h2m21.419052751s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.539204781 (376h2m21.422618407s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.545311795 (376h2m21.428725407s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.548650688 (376h2m21.432064308s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.548766114 (376h2m21.432179732s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.552618914 (376h2m21.436032531s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.556234201 (376h2m21.439647819s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.556714631 (376h2m21.440128256s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.584297058 (376h2m21.467710677s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:50.371498503 (376h2m50.254912119s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:13:50.371544262 (376h2m50.254957891s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection Jul 9 01:13:50.371667019 (376h2m50.255080637s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection Jul 9 01:13:50.371684908 (376h2m50.255098535s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection Jul 9 01:13:50.371707098 (376h2m50.255120766s) [seg_pos={0 0}, upload=300711]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:13:50.484313152 (376h2m50.367726771s) [upload=300711]: Done uploading segments... Jul 9 01:13:50.484347418 (376h2m50.367761038s) [upload=300711]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.272177585s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.......XX.X.X......X.X.XXXXX.......XX....................X..] duration: 23.323356512s FAIL (2025-07-09 01:31:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.323378512s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 24 out of 60 is failed (threshold=3) [X..XX.X...........XXXXX...X.........X..X...XXX..XXXX.XXXXX..] duration: 49.768570271s FAIL (2025-07-09 01:31:13 UTC) orbiter warning: context deadline exceeded Duration: 49.768594006s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [........XX.X.X......X.X.XXXXX.......XX....................X.] duration: 24.969095449s FAIL (2025-07-09 01:30:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.96912314s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.41797237s FAIL (2025-07-09 01:30:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== cessful: 13 optimal: 13 errs: <nil> Jul 9 01:12:51.706672230 (376h1m51.590085861s) [upload=300705]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:12:51.706677984 (376h1m51.590091617s) [upload=300705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:12:51.883113268 (376h1m51.76652692s) [upload=300705]: Done uploading segments... Jul 9 01:12:51.883135264 (376h1m51.766548909s) [upload=300705]: Upload finished. err: <nil> Jul 9 01:13:20.653341358 (376h2m20.536754991s) [upload=300711]: Starting upload Jul 9 01:13:20.653504915 (376h2m20.536918556s) [upload=300711]: Done starting upload Jul 9 01:13:20.653690252 (376h2m20.537103868s) [upload=300711]: Uploading segments... Jul 9 01:13:20.653990712 (376h2m20.537404332s) [upload=300711]: Got next segment. Inline: false Jul 9 01:13:20.654059122 (376h2m20.537472753s) [upload=300711]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:13:20.654065941 (376h2m20.537479574s) [upload=300711]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:13:20.769797764 (376h2m20.653211413s) [seg_pos={0 0}, upload=300711]: Begin upload segment... Jul 9 01:13:20.769876747 (376h2m20.653290386s) [seg_pos={0 0}, upload=300711]: Done begin upload segment. Jul 9 01:13:20.769954018 (376h2m20.653367648s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770052187 (376h2m20.653465935s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770659155 (376h2m20.6540728s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770677751 (376h2m20.654091394s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771126425 (376h2m20.654540056s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771277890 (376h2m20.654691526s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771476972 (376h2m20.654890603s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771804253 (376h2m20.655217892s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771829179 (376h2m20.655242816s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771909583 (376h2m20.655323221s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.772476215 (376h2m20.655889855s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.773678060 (376h2m20.657091699s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.774188246 (376h2m20.657601884s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.774573193 (376h2m20.657986824s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.775340796 (376h2m20.658754434s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.775786189 (376h2m20.659199809s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:21.378914824 (376h2m21.262328447s) [upload=300711]: Next returned nil segment Jul 9 01:13:21.378938224 (376h2m21.262351854s) [upload=300711]: Waiting for error group managing segments... Jul 9 01:13:21.517933925 (376h2m21.401347538s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.518663136 (376h2m21.402076749s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.522226581 (376h2m21.40564021s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.535639134 (376h2m21.419052751s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.539204781 (376h2m21.422618407s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.545311795 (376h2m21.428725407s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.548650688 (376h2m21.432064308s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.548766114 (376h2m21.432179732s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.552618914 (376h2m21.436032531s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.556234201 (376h2m21.439647819s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.556714631 (376h2m21.440128256s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.584297058 (376h2m21.467710677s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:50.371498503 (376h2m50.254912119s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:13:50.371544262 (376h2m50.254957891s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection Jul 9 01:13:50.371667019 (376h2m50.255080637s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection Jul 9 01:13:50.371684908 (376h2m50.255098535s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection Jul 9 01:13:50.371707098 (376h2m50.255120766s) [seg_pos={0 0}, upload=300711]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:13:50.484313152 (376h2m50.367726771s) [upload=300711]: Done uploading segments... Jul 9 01:13:50.484347418 (376h2m50.367761038s) [upload=300711]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.418000834s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 24 out of 60 is failed (threshold=3) [.X..XX.X...........XXXXX...X.........X..X...XXX..XXXX.XXXXX.] duration: 44.36297404s FAIL (2025-07-09 01:30:23 UTC) orbiter warning: context deadline exceeded Duration: 44.362988877s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.........XX.X.X......X.X.XXXXX.......XX....................X] duration: 30.000657752s FAIL (2025-07-09 01:30:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000703444s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.464090689s FAIL (2025-07-09 01:30:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== cessful: 13 optimal: 13 errs: <nil> Jul 9 01:12:51.706672230 (376h1m51.590085861s) [upload=300705]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:12:51.706677984 (376h1m51.590091617s) [upload=300705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:12:51.883113268 (376h1m51.76652692s) [upload=300705]: Done uploading segments... Jul 9 01:12:51.883135264 (376h1m51.766548909s) [upload=300705]: Upload finished. err: <nil> Jul 9 01:13:20.653341358 (376h2m20.536754991s) [upload=300711]: Starting upload Jul 9 01:13:20.653504915 (376h2m20.536918556s) [upload=300711]: Done starting upload Jul 9 01:13:20.653690252 (376h2m20.537103868s) [upload=300711]: Uploading segments... Jul 9 01:13:20.653990712 (376h2m20.537404332s) [upload=300711]: Got next segment. Inline: false Jul 9 01:13:20.654059122 (376h2m20.537472753s) [upload=300711]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:13:20.654065941 (376h2m20.537479574s) [upload=300711]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:13:20.769797764 (376h2m20.653211413s) [seg_pos={0 0}, upload=300711]: Begin upload segment... Jul 9 01:13:20.769876747 (376h2m20.653290386s) [seg_pos={0 0}, upload=300711]: Done begin upload segment. Jul 9 01:13:20.769954018 (376h2m20.653367648s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770052187 (376h2m20.653465935s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770659155 (376h2m20.6540728s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770677751 (376h2m20.654091394s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771126425 (376h2m20.654540056s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771277890 (376h2m20.654691526s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771476972 (376h2m20.654890603s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771804253 (376h2m20.655217892s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771829179 (376h2m20.655242816s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771909583 (376h2m20.655323221s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.772476215 (376h2m20.655889855s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.773678060 (376h2m20.657091699s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.774188246 (376h2m20.657601884s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.774573193 (376h2m20.657986824s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.775340796 (376h2m20.658754434s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.775786189 (376h2m20.659199809s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:21.378914824 (376h2m21.262328447s) [upload=300711]: Next returned nil segment Jul 9 01:13:21.378938224 (376h2m21.262351854s) [upload=300711]: Waiting for error group managing segments... Jul 9 01:13:21.517933925 (376h2m21.401347538s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.518663136 (376h2m21.402076749s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.522226581 (376h2m21.40564021s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.535639134 (376h2m21.419052751s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.539204781 (376h2m21.422618407s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.545311795 (376h2m21.428725407s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.548650688 (376h2m21.432064308s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.548766114 (376h2m21.432179732s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.552618914 (376h2m21.436032531s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.556234201 (376h2m21.439647819s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.556714631 (376h2m21.440128256s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.584297058 (376h2m21.467710677s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:50.371498503 (376h2m50.254912119s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:13:50.371544262 (376h2m50.254957891s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection Jul 9 01:13:50.371667019 (376h2m50.255080637s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection Jul 9 01:13:50.371684908 (376h2m50.255098535s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection Jul 9 01:13:50.371707098 (376h2m50.255120766s) [seg_pos={0 0}, upload=300711]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:13:50.484313152 (376h2m50.367726771s) [upload=300711]: Done uploading segments... Jul 9 01:13:50.484347418 (376h2m50.367761038s) [upload=300711]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.464120553s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 24 out of 60 is failed (threshold=3) [..X..XX.X...........XXXXX...X.........X..X...XXX..XXXX.XXXXX] duration: 1m0.000751728s FAIL (2025-07-09 01:29:38 UTC) orbiter warning: context deadline exceeded Duration: 1m0.000826289s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 9.953107224s FAIL (2025-07-09 01:29:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== cessful: 13 optimal: 13 errs: <nil> Jul 9 01:12:51.706672230 (376h1m51.590085861s) [upload=300705]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:12:51.706677984 (376h1m51.590091617s) [upload=300705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:12:51.883113268 (376h1m51.76652692s) [upload=300705]: Done uploading segments... Jul 9 01:12:51.883135264 (376h1m51.766548909s) [upload=300705]: Upload finished. err: <nil> Jul 9 01:13:20.653341358 (376h2m20.536754991s) [upload=300711]: Starting upload Jul 9 01:13:20.653504915 (376h2m20.536918556s) [upload=300711]: Done starting upload Jul 9 01:13:20.653690252 (376h2m20.537103868s) [upload=300711]: Uploading segments... Jul 9 01:13:20.653990712 (376h2m20.537404332s) [upload=300711]: Got next segment. Inline: false Jul 9 01:13:20.654059122 (376h2m20.537472753s) [upload=300711]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:13:20.654065941 (376h2m20.537479574s) [upload=300711]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:13:20.769797764 (376h2m20.653211413s) [seg_pos={0 0}, upload=300711]: Begin upload segment... Jul 9 01:13:20.769876747 (376h2m20.653290386s) [seg_pos={0 0}, upload=300711]: Done begin upload segment. Jul 9 01:13:20.769954018 (376h2m20.653367648s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770052187 (376h2m20.653465935s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770659155 (376h2m20.6540728s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770677751 (376h2m20.654091394s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771126425 (376h2m20.654540056s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771277890 (376h2m20.654691526s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771476972 (376h2m20.654890603s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771804253 (376h2m20.655217892s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771829179 (376h2m20.655242816s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771909583 (376h2m20.655323221s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.772476215 (376h2m20.655889855s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.773678060 (376h2m20.657091699s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.774188246 (376h2m20.657601884s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.774573193 (376h2m20.657986824s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.775340796 (376h2m20.658754434s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.775786189 (376h2m20.659199809s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:21.378914824 (376h2m21.262328447s) [upload=300711]: Next returned nil segment Jul 9 01:13:21.378938224 (376h2m21.262351854s) [upload=300711]: Waiting for error group managing segments... Jul 9 01:13:21.517933925 (376h2m21.401347538s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.518663136 (376h2m21.402076749s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.522226581 (376h2m21.40564021s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.535639134 (376h2m21.419052751s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.539204781 (376h2m21.422618407s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.545311795 (376h2m21.428725407s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.548650688 (376h2m21.432064308s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.548766114 (376h2m21.432179732s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.552618914 (376h2m21.436032531s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.556234201 (376h2m21.439647819s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.556714631 (376h2m21.440128256s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.584297058 (376h2m21.467710677s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:50.371498503 (376h2m50.254912119s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:13:50.371544262 (376h2m50.254957891s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection Jul 9 01:13:50.371667019 (376h2m50.255080637s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection Jul 9 01:13:50.371684908 (376h2m50.255098535s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection Jul 9 01:13:50.371707098 (376h2m50.255120766s) [seg_pos={0 0}, upload=300711]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:13:50.484313152 (376h2m50.367726771s) [upload=300711]: Done uploading segments... Jul 9 01:13:50.484347418 (376h2m50.367761038s) [upload=300711]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.953139441s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..........XX.X.X......X.X.XXXXX.......XX....................] duration: 2.618774036s FAIL (2025-07-09 01:28:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.61879544s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...........XX.X.X......X.X.XXXXX.......XX...................] duration: 29.639296953s FAIL (2025-07-09 01:28:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.639319863s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 24 out of 60 is failed (threshold=3) [X..X..XX.X...........XXXXX...X.........X..X...XXX..XXXX.XXXX] duration: 1m0.001357893s FAIL (2025-07-09 01:28:38 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001374292s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.38037232s FAIL (2025-07-09 01:28:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== cessful: 13 optimal: 13 errs: <nil> Jul 9 01:12:51.706672230 (376h1m51.590085861s) [upload=300705]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:12:51.706677984 (376h1m51.590091617s) [upload=300705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:12:51.883113268 (376h1m51.76652692s) [upload=300705]: Done uploading segments... Jul 9 01:12:51.883135264 (376h1m51.766548909s) [upload=300705]: Upload finished. err: <nil> Jul 9 01:13:20.653341358 (376h2m20.536754991s) [upload=300711]: Starting upload Jul 9 01:13:20.653504915 (376h2m20.536918556s) [upload=300711]: Done starting upload Jul 9 01:13:20.653690252 (376h2m20.537103868s) [upload=300711]: Uploading segments... Jul 9 01:13:20.653990712 (376h2m20.537404332s) [upload=300711]: Got next segment. Inline: false Jul 9 01:13:20.654059122 (376h2m20.537472753s) [upload=300711]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:13:20.654065941 (376h2m20.537479574s) [upload=300711]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:13:20.769797764 (376h2m20.653211413s) [seg_pos={0 0}, upload=300711]: Begin upload segment... Jul 9 01:13:20.769876747 (376h2m20.653290386s) [seg_pos={0 0}, upload=300711]: Done begin upload segment. Jul 9 01:13:20.769954018 (376h2m20.653367648s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770052187 (376h2m20.653465935s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770659155 (376h2m20.6540728s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770677751 (376h2m20.654091394s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771126425 (376h2m20.654540056s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771277890 (376h2m20.654691526s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771476972 (376h2m20.654890603s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771804253 (376h2m20.655217892s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771829179 (376h2m20.655242816s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771909583 (376h2m20.655323221s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.772476215 (376h2m20.655889855s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.773678060 (376h2m20.657091699s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.774188246 (376h2m20.657601884s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.774573193 (376h2m20.657986824s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.775340796 (376h2m20.658754434s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.775786189 (376h2m20.659199809s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:21.378914824 (376h2m21.262328447s) [upload=300711]: Next returned nil segment Jul 9 01:13:21.378938224 (376h2m21.262351854s) [upload=300711]: Waiting for error group managing segments... Jul 9 01:13:21.517933925 (376h2m21.401347538s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.518663136 (376h2m21.402076749s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.522226581 (376h2m21.40564021s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.535639134 (376h2m21.419052751s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.539204781 (376h2m21.422618407s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.545311795 (376h2m21.428725407s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.548650688 (376h2m21.432064308s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.548766114 (376h2m21.432179732s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.552618914 (376h2m21.436032531s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.556234201 (376h2m21.439647819s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.556714631 (376h2m21.440128256s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.584297058 (376h2m21.467710677s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:50.371498503 (376h2m50.254912119s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:13:50.371544262 (376h2m50.254957891s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection Jul 9 01:13:50.371667019 (376h2m50.255080637s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection Jul 9 01:13:50.371684908 (376h2m50.255098535s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection Jul 9 01:13:50.371707098 (376h2m50.255120766s) [seg_pos={0 0}, upload=300711]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:13:50.484313152 (376h2m50.367726771s) [upload=300711]: Done uploading segments... Jul 9 01:13:50.484347418 (376h2m50.367761038s) [upload=300711]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.380410269s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 23 out of 60 is failed (threshold=3) [.X..X..XX.X...........XXXXX...X.........X..X...XXX..XXXX.XXX] duration: 1m0.003677144s FAIL (2025-07-09 01:27:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.003688965s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [............XX.X.X......X.X.XXXXX.......XX..................] duration: 20.694827247s FAIL (2025-07-09 01:27:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.694844662s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.751661633s FAIL (2025-07-09 01:27:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== cessful: 13 optimal: 13 errs: <nil> Jul 9 01:12:51.706672230 (376h1m51.590085861s) [upload=300705]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:12:51.706677984 (376h1m51.590091617s) [upload=300705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:12:51.883113268 (376h1m51.76652692s) [upload=300705]: Done uploading segments... Jul 9 01:12:51.883135264 (376h1m51.766548909s) [upload=300705]: Upload finished. err: <nil> Jul 9 01:13:20.653341358 (376h2m20.536754991s) [upload=300711]: Starting upload Jul 9 01:13:20.653504915 (376h2m20.536918556s) [upload=300711]: Done starting upload Jul 9 01:13:20.653690252 (376h2m20.537103868s) [upload=300711]: Uploading segments... Jul 9 01:13:20.653990712 (376h2m20.537404332s) [upload=300711]: Got next segment. Inline: false Jul 9 01:13:20.654059122 (376h2m20.537472753s) [upload=300711]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:13:20.654065941 (376h2m20.537479574s) [upload=300711]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:13:20.769797764 (376h2m20.653211413s) [seg_pos={0 0}, upload=300711]: Begin upload segment... Jul 9 01:13:20.769876747 (376h2m20.653290386s) [seg_pos={0 0}, upload=300711]: Done begin upload segment. Jul 9 01:13:20.769954018 (376h2m20.653367648s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770052187 (376h2m20.653465935s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770659155 (376h2m20.6540728s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770677751 (376h2m20.654091394s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771126425 (376h2m20.654540056s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771277890 (376h2m20.654691526s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771476972 (376h2m20.654890603s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771804253 (376h2m20.655217892s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771829179 (376h2m20.655242816s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771909583 (376h2m20.655323221s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.772476215 (376h2m20.655889855s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.773678060 (376h2m20.657091699s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.774188246 (376h2m20.657601884s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.774573193 (376h2m20.657986824s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.775340796 (376h2m20.658754434s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.775786189 (376h2m20.659199809s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:21.378914824 (376h2m21.262328447s) [upload=300711]: Next returned nil segment Jul 9 01:13:21.378938224 (376h2m21.262351854s) [upload=300711]: Waiting for error group managing segments... Jul 9 01:13:21.517933925 (376h2m21.401347538s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.518663136 (376h2m21.402076749s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.522226581 (376h2m21.40564021s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.535639134 (376h2m21.419052751s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.539204781 (376h2m21.422618407s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.545311795 (376h2m21.428725407s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.548650688 (376h2m21.432064308s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.548766114 (376h2m21.432179732s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.552618914 (376h2m21.436032531s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.556234201 (376h2m21.439647819s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.556714631 (376h2m21.440128256s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.584297058 (376h2m21.467710677s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:50.371498503 (376h2m50.254912119s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:13:50.371544262 (376h2m50.254957891s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection Jul 9 01:13:50.371667019 (376h2m50.255080637s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection Jul 9 01:13:50.371684908 (376h2m50.255098535s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection Jul 9 01:13:50.371707098 (376h2m50.255120766s) [seg_pos={0 0}, upload=300711]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:13:50.484313152 (376h2m50.367726771s) [upload=300711]: Done uploading segments... Jul 9 01:13:50.484347418 (376h2m50.367761038s) [upload=300711]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.751694017s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.............XX.X.X......X.X.XXXXX.......XX.................] duration: 25.149540716s FAIL (2025-07-09 01:27:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.14971145s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 22 out of 60 is failed (threshold=3) [..X..X..XX.X...........XXXXX...X.........X..X...XXX..XXXX.XX] duration: 1m0.000873375s FAIL (2025-07-09 01:26:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000894904s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.272486946s FAIL (2025-07-09 01:26:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== cessful: 13 optimal: 13 errs: <nil> Jul 9 01:12:51.706672230 (376h1m51.590085861s) [upload=300705]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:12:51.706677984 (376h1m51.590091617s) [upload=300705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:12:51.883113268 (376h1m51.76652692s) [upload=300705]: Done uploading segments... Jul 9 01:12:51.883135264 (376h1m51.766548909s) [upload=300705]: Upload finished. err: <nil> Jul 9 01:13:20.653341358 (376h2m20.536754991s) [upload=300711]: Starting upload Jul 9 01:13:20.653504915 (376h2m20.536918556s) [upload=300711]: Done starting upload Jul 9 01:13:20.653690252 (376h2m20.537103868s) [upload=300711]: Uploading segments... Jul 9 01:13:20.653990712 (376h2m20.537404332s) [upload=300711]: Got next segment. Inline: false Jul 9 01:13:20.654059122 (376h2m20.537472753s) [upload=300711]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:13:20.654065941 (376h2m20.537479574s) [upload=300711]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:13:20.769797764 (376h2m20.653211413s) [seg_pos={0 0}, upload=300711]: Begin upload segment... Jul 9 01:13:20.769876747 (376h2m20.653290386s) [seg_pos={0 0}, upload=300711]: Done begin upload segment. Jul 9 01:13:20.769954018 (376h2m20.653367648s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770052187 (376h2m20.653465935s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770659155 (376h2m20.6540728s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770677751 (376h2m20.654091394s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771126425 (376h2m20.654540056s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771277890 (376h2m20.654691526s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771476972 (376h2m20.654890603s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771804253 (376h2m20.655217892s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771829179 (376h2m20.655242816s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771909583 (376h2m20.655323221s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.772476215 (376h2m20.655889855s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.773678060 (376h2m20.657091699s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.774188246 (376h2m20.657601884s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.774573193 (376h2m20.657986824s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.775340796 (376h2m20.658754434s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.775786189 (376h2m20.659199809s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:21.378914824 (376h2m21.262328447s) [upload=300711]: Next returned nil segment Jul 9 01:13:21.378938224 (376h2m21.262351854s) [upload=300711]: Waiting for error group managing segments... Jul 9 01:13:21.517933925 (376h2m21.401347538s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.518663136 (376h2m21.402076749s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.522226581 (376h2m21.40564021s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.535639134 (376h2m21.419052751s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.539204781 (376h2m21.422618407s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.545311795 (376h2m21.428725407s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.548650688 (376h2m21.432064308s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.548766114 (376h2m21.432179732s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.552618914 (376h2m21.436032531s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.556234201 (376h2m21.439647819s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.556714631 (376h2m21.440128256s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.584297058 (376h2m21.467710677s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:50.371498503 (376h2m50.254912119s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:13:50.371544262 (376h2m50.254957891s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection Jul 9 01:13:50.371667019 (376h2m50.255080637s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection Jul 9 01:13:50.371684908 (376h2m50.255098535s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection Jul 9 01:13:50.371707098 (376h2m50.255120766s) [seg_pos={0 0}, upload=300711]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:13:50.484313152 (376h2m50.367726771s) [upload=300711]: Done uploading segments... Jul 9 01:13:50.484347418 (376h2m50.367761038s) [upload=300711]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.272515581s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.349112656s FAIL (2025-07-09 01:26:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== cessful: 13 optimal: 13 errs: <nil> Jul 9 01:12:51.706672230 (376h1m51.590085861s) [upload=300705]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:12:51.706677984 (376h1m51.590091617s) [upload=300705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:12:51.883113268 (376h1m51.76652692s) [upload=300705]: Done uploading segments... Jul 9 01:12:51.883135264 (376h1m51.766548909s) [upload=300705]: Upload finished. err: <nil> Jul 9 01:13:20.653341358 (376h2m20.536754991s) [upload=300711]: Starting upload Jul 9 01:13:20.653504915 (376h2m20.536918556s) [upload=300711]: Done starting upload Jul 9 01:13:20.653690252 (376h2m20.537103868s) [upload=300711]: Uploading segments... Jul 9 01:13:20.653990712 (376h2m20.537404332s) [upload=300711]: Got next segment. Inline: false Jul 9 01:13:20.654059122 (376h2m20.537472753s) [upload=300711]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:13:20.654065941 (376h2m20.537479574s) [upload=300711]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:13:20.769797764 (376h2m20.653211413s) [seg_pos={0 0}, upload=300711]: Begin upload segment... Jul 9 01:13:20.769876747 (376h2m20.653290386s) [seg_pos={0 0}, upload=300711]: Done begin upload segment. Jul 9 01:13:20.769954018 (376h2m20.653367648s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770052187 (376h2m20.653465935s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770659155 (376h2m20.6540728s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770677751 (376h2m20.654091394s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771126425 (376h2m20.654540056s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771277890 (376h2m20.654691526s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771476972 (376h2m20.654890603s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771804253 (376h2m20.655217892s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771829179 (376h2m20.655242816s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771909583 (376h2m20.655323221s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.772476215 (376h2m20.655889855s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.773678060 (376h2m20.657091699s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.774188246 (376h2m20.657601884s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.774573193 (376h2m20.657986824s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.775340796 (376h2m20.658754434s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.775786189 (376h2m20.659199809s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:21.378914824 (376h2m21.262328447s) [upload=300711]: Next returned nil segment Jul 9 01:13:21.378938224 (376h2m21.262351854s) [upload=300711]: Waiting for error group managing segments... Jul 9 01:13:21.517933925 (376h2m21.401347538s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.518663136 (376h2m21.402076749s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.522226581 (376h2m21.40564021s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.535639134 (376h2m21.419052751s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.539204781 (376h2m21.422618407s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.545311795 (376h2m21.428725407s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.548650688 (376h2m21.432064308s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.548766114 (376h2m21.432179732s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.552618914 (376h2m21.436032531s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.556234201 (376h2m21.439647819s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.556714631 (376h2m21.440128256s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.584297058 (376h2m21.467710677s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:50.371498503 (376h2m50.254912119s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:13:50.371544262 (376h2m50.254957891s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection Jul 9 01:13:50.371667019 (376h2m50.255080637s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection Jul 9 01:13:50.371684908 (376h2m50.255098535s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection Jul 9 01:13:50.371707098 (376h2m50.255120766s) [seg_pos={0 0}, upload=300711]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:13:50.484313152 (376h2m50.367726771s) [upload=300711]: Done uploading segments... Jul 9 01:13:50.484347418 (376h2m50.367761038s) [upload=300711]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.349198298s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X.............XX.X.X......X.X.XXXXX.......XX................] duration: 2.751783307s FAIL (2025-07-09 01:25:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.751807654s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 22 out of 60 is failed (threshold=3) [X..X..X..XX.X...........XXXXX...X.........X..X...XXX..XXXX.X] duration: 1m0.001160708s FAIL (2025-07-09 01:25:35 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001198775s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X.............XX.X.X......X.X.XXXXX.......XX...............] duration: 2.804845785s FAIL (2025-07-09 01:25:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.804866637s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..X.............XX.X.X......X.X.XXXXX.......XX..............] duration: 22.493246063s FAIL (2025-07-09 01:25:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.493278193s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.321456291s FAIL (2025-07-09 01:25:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== cessful: 13 optimal: 13 errs: <nil> Jul 9 01:12:51.706672230 (376h1m51.590085861s) [upload=300705]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:12:51.706677984 (376h1m51.590091617s) [upload=300705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:12:51.883113268 (376h1m51.76652692s) [upload=300705]: Done uploading segments... Jul 9 01:12:51.883135264 (376h1m51.766548909s) [upload=300705]: Upload finished. err: <nil> Jul 9 01:13:20.653341358 (376h2m20.536754991s) [upload=300711]: Starting upload Jul 9 01:13:20.653504915 (376h2m20.536918556s) [upload=300711]: Done starting upload Jul 9 01:13:20.653690252 (376h2m20.537103868s) [upload=300711]: Uploading segments... Jul 9 01:13:20.653990712 (376h2m20.537404332s) [upload=300711]: Got next segment. Inline: false Jul 9 01:13:20.654059122 (376h2m20.537472753s) [upload=300711]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:13:20.654065941 (376h2m20.537479574s) [upload=300711]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:13:20.769797764 (376h2m20.653211413s) [seg_pos={0 0}, upload=300711]: Begin upload segment... Jul 9 01:13:20.769876747 (376h2m20.653290386s) [seg_pos={0 0}, upload=300711]: Done begin upload segment. Jul 9 01:13:20.769954018 (376h2m20.653367648s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770052187 (376h2m20.653465935s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770659155 (376h2m20.6540728s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.770677751 (376h2m20.654091394s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771126425 (376h2m20.654540056s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771277890 (376h2m20.654691526s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771476972 (376h2m20.654890603s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771804253 (376h2m20.655217892s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771829179 (376h2m20.655242816s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.771909583 (376h2m20.655323221s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.772476215 (376h2m20.655889855s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.773678060 (376h2m20.657091699s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.774188246 (376h2m20.657601884s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.774573193 (376h2m20.657986824s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.775340796 (376h2m20.658754434s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:20.775786189 (376h2m20.659199809s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Uploading piece... Jul 9 01:13:21.378914824 (376h2m21.262328447s) [upload=300711]: Next returned nil segment Jul 9 01:13:21.378938224 (376h2m21.262351854s) [upload=300711]: Waiting for error group managing segments... Jul 9 01:13:21.517933925 (376h2m21.401347538s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.518663136 (376h2m21.402076749s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.522226581 (376h2m21.40564021s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.535639134 (376h2m21.419052751s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.539204781 (376h2m21.422618407s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.545311795 (376h2m21.428725407s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.548650688 (376h2m21.432064308s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.548766114 (376h2m21.432179732s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.552618914 (376h2m21.436032531s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.556234201 (376h2m21.439647819s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.556714631 (376h2m21.440128256s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:21.584297058 (376h2m21.467710677s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil> Jul 9 01:13:50.371498503 (376h2m50.254912119s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:13:50.371544262 (376h2m50.254957891s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection Jul 9 01:13:50.371667019 (376h2m50.255080637s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection Jul 9 01:13:50.371684908 (376h2m50.255098535s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection Jul 9 01:13:50.371707098 (376h2m50.255120766s) [seg_pos={0 0}, upload=300711]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:13:50.484313152 (376h2m50.367726771s) [upload=300711]: Done uploading segments... Jul 9 01:13:50.484347418 (376h2m50.367761038s) [upload=300711]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.32148551s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 21 out of 60 is failed (threshold=3) [.X..X..X..XX.X...........XXXXX...X.........X..X...XXX..XXXX.] duration: 1m0.001459032s FAIL (2025-07-09 01:24:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001482962s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)