Help - Can't pull from private GHCR

[yalihart]

time="2024-04-30T21:14:02Z" level=warning msg="Could not do a head request for \"ghcr.io/yalihart/redacted-project-name:e7c02f961274d359e326cee5e7d1d9f1c5811a7b\", falling back to regular pull." container=/redacted-project-name image="ghcr.io/yalihart/redacted-project-name:e7c02f961274d359e326cee5e7d1d9f1c5811a7b"
time="2024-04-30T21:14:02Z" level=warning msg="Reason: registry responded to head request with \"403 Forbidden\", auth: \"not present\"" container=/redacted-project-name image="ghcr.io/yalihart/redacted-project-name:e7c02f961274d359e326cee5e7d1d9f1c5811a7b"
time="2024-04-30T21:14:02Z" level=info msg="Unable to update container \"/redacted-project-name\": Error response from daemon: Head \"https://ghcr.io/v2/yalihart/redacted-project-name/manifests/e7c02f961274d359e326cee5e7d1d9f1c5811a7b\": unauthorized. Proceeding to next."
time="2024-04-30T21:14:02Z" level=info msg="Session done" Failed=0 Scanned=4 Updated=0 notify=no

I did docker login successfully and verified that the auth does appear in the config file.

Watchtower still fails to pull the image, why?

My Stack:

version: "3.4"

services: 
  watchtower:
    image: containrrr/watchtower
    environment:

        WATCHTOWER_POLL_INTERVAL: 10
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

Im also getting no creds found

[digininja]

Did you work this out?

I've got a private GitHub repo which I can pull from the command line, I can use the pass command to get the access token, but Watchtower is getting a 403 every time.

[lexuzieel]

Same issue here. I can pull it using docker compose pull but getting error with watchtower:

level=warning msg="Reason: registry responded to head request with \"403 Forbidden\", auth: \"not present\"" container=/app-app-1 image="<redacted>:edge"
time="2024-10-15T16:21:25Z" level=info msg="Unable to update container \"/app-app-1\": Error response from daemon: Head \"https://ghcr.io/v2/<redacted>/manifests/edge\": unauthorized. Proceeding to next."

Any ideas how to debug this?

[Vrashabh-Sontakke]

hey @lexuzieel , did you manage to debug this ? I am stuck on the same issue.

[lexuzieel]

hey @lexuzieel , did you manage to debug this ? I am stuck on the same issue.

Yes, in my case recreating docker compose stack worked. Maybe some token expired since I was pulling from private registry but I didn't debug it.

[Vrashabh-Sontakke]

Here is the message I am getting:

watchtower-1 | time="2024-10-25T18:25:03Z" level=info msg="Unable to update container "/deployment-pocketbase-1": Error response from daemon: Head "https://ghcr.io/v2/valueserveio/pocketbase/manifests/prod\": unauthorized. Proceeding to next."

I am logged in to the registry on the VPS.
Also the gh auth token has the permissions for package Read Write & Delete.

[digininja]

I was trying to use `pass` as the credential helper and it didn't work so I switched back to putting the creds directly into the config file and it's working again. It would be good to do it in the secure way. `pass` is able to retrieve the credentials so I know that bit was working correctly.

…

On Tue, 15 Oct 2024, 18:57 Aleksei Ivanov, ***@***.***> wrote: Same issue here. I can pull it using docker compose pull but getting error with watchtower: level=warning msg="Reason: registry responded to head request with \"403 Forbidden\", auth: \"not present\"" container=/app-app-1 image="<redacted>:edge" time="2024-10-15T16:21:25Z" level=info msg="Unable to update container \"/app-app-1\": Error response from daemon: Head \"https://ghcr.io/v2/<redacted>/manifests/edge\": unauthorized. Proceeding to next." Any ideas how to debug this? — Reply to this email directly, view it on GitHub <#1972 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA4SWLVRZZRSLU4DLP3UN3Z3VJRRAVCNFSM6AAAAABHBALSPSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAOJVGA4TQNQ> . You are receiving this because you commented.Message ID: ***@***.***>

[digininja]

How do you recreate the stack?

…

On Fri, 25 Oct 2024, 19:25 Aleksei Ivanov, ***@***.***> wrote: hey @lexuzieel <https://github.com/lexuzieel> , did you manage to debug this ? I am stuck on the same issue. Yes, in my case recreating docker compose stack worked. Maybe some token expired since I was pulling from private registry but I didn't debug it. — Reply to this email directly, view it on GitHub <#1972 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA4SWM2OE7RKN5G7LTNK7LZ5KEKXAVCNFSM6AAAAABHBALSPSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMBVGU4DGOA> . You are receiving this because you commented.Message ID: ***@***.***>

[lexuzieel]

How do you recreate the stack?

On Fri, 25 Oct 2024, 19:25 Aleksei Ivanov, @.***> wrote:

hey @lexuzieel https://github.com/lexuzieel , did you manage to debug

this ? I am stuck on the same issue.

Yes, in my case recreating docker compose stack worked. Maybe some token

expired since I was pulling from private registry but I didn't debug it.

—

Reply to this email directly, view it on GitHub

#1972 (reply in thread),

or unsubscribe

https://github.com/notifications/unsubscribe-auth/AAA4SWM2OE7RKN5G7LTNK7LZ5KEKXAVCNFSM6AAAAABHBALSPSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMBVGU4DGOA

.

You are receiving this because you commented.Message ID:

@.***>

docker compose down

docker compose up -d

[digininja]

I thought you meant something more complicated than that.

…

On Fri, 25 Oct 2024, 19:43 Aleksei Ivanov, ***@***.***> wrote: How do you recreate the stack? On Fri, 25 Oct 2024, 19:25 Aleksei Ivanov, *@*.***> wrote: hey @lexuzieel <https://github.com/lexuzieel> https://github.com/lexuzieel , did you manage to debug this ? I am stuck on the same issue. Yes, in my case recreating docker compose stack worked. Maybe some token expired since I was pulling from private registry but I didn't debug it. — Reply to this email directly, view it on GitHub #1972 (reply in thread) <#1972 (reply in thread)> , or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA4SWM2OE7RKN5G7LTNK7LZ5KEKXAVCNFSM6AAAAABHBALSPSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMBVGU4DGOA . You are receiving this because you commented.Message ID: *@*.***> docker compose down docker compose up -d — Reply to this email directly, view it on GitHub <#1972 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA4SWKOJTBOAYI566YNNZDZ5KGM7AVCNFSM6AAAAABHBALSPSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMBVGU4TMNA> . You are receiving this because you commented.Message ID: ***@***.***>

[lexuzieel]

That's what fixed it for me. Are you deploying watchtower using docker?

[digininja]

Yes I am. I'm sure I've tried restarting it like that but maybe not. I'll give it a try next week.

…

On Fri, 25 Oct 2024, 19:52 Aleksei Ivanov, ***@***.***> wrote: That's what fixed it for me. Are you deploying watchtower using docker? — Reply to this email directly, view it on GitHub <#1972 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA4SWN7CIGEQSQV6SKNIWLZ5KHQFAVCNFSM6AAAAABHBALSPSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMBVGYYDGNI> . You are receiving this because you commented.Message ID: ***@***.***>

[lexuzieel]

With docker you could try just docker rm -f command that should be equal to docker compose down

[digininja]

Did docker rm -f watchtower-watchtower-1 then docker compose up -d and I'm still getting the error.

This is my config.json file, is this all I need?

{
        "credsStore": "pass"
}

This returns the token:

pass docker-credential-helpers/Z2hjci5pbw\=\=/digininja

I've also tried it on the command line and this fails as well:

docker run --name watchtower --volume /var/run/docker.sock:/var/run/docker.sock --volume /home/robin/.docker/config.json:/config.json    containrrr/watchtower --interval 30

Error:

time="2024-10-25T19:42:17Z" level=warning msg="Could not do a head request for \"ghcr.io/digininja/a-api:main\", falling back to regular pull." container=/a-a-api-1 image="ghcr.io/digininja/a-api:main"
time="2024-10-25T19:42:17Z" level=warning msg="Reason: registry responded to head request with \"403 Forbidden\", auth: \"not present\"" container=/a-a-api-1 image="ghcr.io/digininja/a-api:main"

[digininja]

If I go back to this in my config file, it works fine:

{
        "auths": {
                "ghcr.io": {
                        "auth": "blahblahblah="
                }
        }
}

[dmuiX]

if docker would support secrets in a good way then this could have been done in a good way but so...its ugly unsecure but this is docker...