Multiple artifacts, directories, and S3 deployment

dmcgee's Avatar

dmcgee

01 May, 2014 07:43 PM

The quick version- I want to push some of the build artifacts from a ClickOnce publish to S3, and I'm running into some issues.

As a result of running MSBuild /t:Publish, I end up with a directory at ProjectName/bin/Debug/app.publish/ of which I need to push the contents, exactly as laid out, to S3.

My first step was to setup an artifact like so:

artifacts:
  - path: ProjectName\bin\Debug\app.publish\*
    name: published

This does result in the artifacts getting published individually; however, I also get published.zip files at multiple levels pushed with everything else, which I don't want. I see in at least one place in the documentation you use an undocumented type: zip option; is there a similar type: directory option to force it not to zip up the files needlessly?

The next step would be to push this artifact consisting of several files and directories to S3. However, I'm a bit unsure what I'm going to get there, as the artifact paths in the AppVeyor web interface include all the leading path components (e.g. ProjectName\bin\Debug\app.publish) which I don't want to show up in S3. Can or will these directory prefixes be stripped off?

Finally, this applies to any deployment, but can I push more than one artifact by name? I'm not sure how this would look in the YAML configuration file.

Thanks in advance!

  1. Support Staff 1 Posted by Feodor Fitsner on 01 May, 2014 08:24 PM

    Feodor Fitsner's Avatar

    OK, I see an issue here. Apparently, pattern ProjectName\bin\Debug\app.publish\* matches not only files, but folders too. Folders then are packaged to zips. This is a bug - this will be fixed and deployed today.

    Pushing zip artifact to S3 won't expand it. I think the first option (provided the bug is fixed) is better.

    When deploying with stripped directory prefixes do you need to preserve folders structure at S3 below app.publish?

  2. 2 Posted by dmcgee on 01 May, 2014 08:34 PM

    dmcgee's Avatar

    Yes, the structure below that point should be preserved. ClickOnce does something like this with the folder structure, and it embeds some logic of how to build the paths/URLs when it is looking for things:

    setup.exe
    ProjectManifest
    Application Files\MyProject_version\Something.dll.deploy
    Application Files\MyProject_version\AnotherFile.dll.deploy
    ...
    

    On the S3 topic, I tried deploying the published artifact to S3 now, but nothing seemed to make it there. The build log is sparse, and I can't really tell if it tried to do anything at all, this is the entire output of that stage:

    Deploying using S3 provider
    Build success
    

    Should a configuration like this have worked?

    deploy:
      provider: S3
      access_key_id: <key here>
      secret_access_key: <secret here>
      bucket: my-bucket-name
      artifact: published
    
  3. Support Staff 3 Posted by Feodor Fitsner on 01 May, 2014 08:37 PM

    Feodor Fitsner's Avatar

    Could you drop a screenshot of "Artifacts" tab?

  4. 4 Posted by dmcgee on 01 May, 2014 08:59 PM

    dmcgee's Avatar

    I screwed up my artifacts definition on my last deploy, sorry for the confusion. Once that was fixed it at least attempted to upload something.

  5. Support Staff 5 Posted by Feodor Fitsner on 01 May, 2014 09:01 PM

    Feodor Fitsner's Avatar

    You can look at the history?

  6. 6 Posted by dmcgee on 01 May, 2014 10:30 PM

    dmcgee's Avatar

    Here you go, wasn't exactly sure what you wanted to see in the screenshot.

  7. 7 Posted by dmcgee on 01 May, 2014 11:20 PM

    dmcgee's Avatar

    When I do get it uploading to S3, files end up named like this:

    ProjectName\bin\Debug\app.publish\Application Files\ProjectName_99_0_76_0\Microsoft.Office.Tools.v4.0.Framework.dll.deploy
    

    The \ characters are literals, those are NOT directory separators. So there is no nested-in-folders artifacts here, just a flat push to S3.

    Ideally, I would like the prefix part (from the artifacts configuration) to be dropped, have the Application Files/ProjectName_version/ folder get created, and the file placed inside, in this particular example.

  8. Support Staff 8 Posted by Feodor Fitsner on 02 May, 2014 12:42 AM

    Feodor Fitsner's Avatar

    I've found a better (with more control) way to solve your case :)

    I'm pushing artifacts in after_build section like that:

    after_build:
      - ps: $root = Resolve-Path .\SimpleApp; [IO.Directory]::GetFiles($root.Path, '*.*', 'AllDirectories') | % { Push-AppveyorArtifact $_ -FileName $_.Substring($root.Path.Length + 1) -DeploymentName test101 }
    

    Replace .\SimpleApp with your own .\ProjectName\bin\Debug\app.publish and test101 with published.

    Artifacts list I'm getting: https://ci.appveyor.com/project/feodorf1/appveyor-bot-0904-0540/bui...

    My S3 environment has settings as on attached screenshot.
    Deployment log is on the second screenshot.
    And finally s3 bucket contents is on the third screenshot.

    Hope that helps.

  9. 9 Posted by dmcgee on 02 May, 2014 02:55 PM

    dmcgee's Avatar

    OK, I think we're much closer now, but still having some hangups on the S3 stage. The prefix is now gone from the publish directory after switching to your PowerShell command, which is awesome. Here are the relevant pieces of my config.

    after_test:
      - ps: $root = Resolve-Path .\FeedStream\bin\Debug\app.publish; [IO.Directory]::GetFiles($root.Path, '*.*', 'AllDirectories') | % { Push-AppveyorArtifact $_ -FileName $_.Substring($root.Path.Length + 1) -DeploymentName to-publish }
    
    deploy:
      provider: S3
      access_key_id: <key>
      secret_access_key: <key>
      bucket: my-s3-bucket
      artifact: to-publish
    

    However, the upload still isn't working correctly. Here is a snippet from the console:

    Uploading artifact "setup.exe" (0 bytes) to S3 bucket "data-everywhere-beta" as "setup.exe"
    Uploading artifact "Application Files\MyProject_99_0_78_0\MyProject.dll.deploy" (0 bytes) to S3 bucket "my-s3-bucket" as "Application Files\MyProject_99_0_78_0\MyProject.dll.deploy"
    

    And the files end up in S3 with the full path name, all in the root of the bucket. Thoughts? My output differs from yours in the upload phase in two ways that I see: 1) \ instead of /, and 2) it always reports file sizes as zero bytes (although in S3, they all have seemingly correct sizes).

  10. Support Staff 10 Posted by Feodor Fitsner on 02 May, 2014 04:25 PM

    Feodor Fitsner's Avatar

    OK, let me try to reproduce the entire case with click-once deployment and appveyor.yml.

  11. Support Staff 11 Posted by Feodor Fitsner on 02 May, 2014 04:44 PM

    Feodor Fitsner's Avatar

    I've managed to reproduce the problem and going to work on the fix.

    https://ci.appveyor.com/project/feodorf1/click-once-test

  12. 12 Posted by dmcgee on 02 May, 2014 04:45 PM

    dmcgee's Avatar

    Awesome, thanks a bunch for looking into this. Happy to help in any way I can today.

  13. Support Staff 13 Posted by Feodor Fitsner on 02 May, 2014 06:49 PM

    Feodor Fitsner's Avatar

    OK, it's been fixed and update deployed. Give it another try and let me know how it goes.

    There was another bug too related to a deployment of artifact with a space in file name which is now fixed too.

  14. 14 Posted by dmcgee on 02 May, 2014 07:23 PM

    dmcgee's Avatar

    This is working great now, thanks! Can't say enough about your support and quick turnaround, much appreciated.

  15. Support Staff 15 Posted by Feodor Fitsner on 02 May, 2014 07:24 PM

    Feodor Fitsner's Avatar

    You're welcome!

  16. dmcgee closed this discussion on 12 Sep, 2014 01:46 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac

Recent Discussions

21 Aug, 2019 11:06 PM
21 Aug, 2019 02:08 PM
21 Aug, 2019 01:37 PM
21 Aug, 2019 10:21 AM
21 Aug, 2019 09:10 AM

 

21 Aug, 2019 05:56 AM
21 Aug, 2019 03:08 AM
20 Aug, 2019 11:33 PM
20 Aug, 2019 08:07 PM
20 Aug, 2019 05:50 PM
20 Aug, 2019 12:35 PM