WoWInterface

WoWInterface (https://www.wowinterface.com/forums/index.php)
-   Tutorials & Other Helpful Info. (https://www.wowinterface.com/forums/forumdisplay.php?f=12)
-   -   Guide: Automagically package and publish addons (https://www.wowinterface.com/forums/showthread.php?t=55801)

p3lim 10-14-17 02:34 PM

Guide: Automagically package and publish addons
 
Any seasoned developer will agree that automating repetitive tasks is a good idea,
and one of the most repetitive things we do as addon developers is publishing addon updates.

This guide will instruct you how you can automate packaging and publishing new releases,
so you can focus more on developing the addon itself.

With this tool, addons can be published at the two major addon sites; CurseForge/WoWAce (and thus Curse), and WoWInterface.
Addons can also be published (packaged) on GitHub in case you'd also want that.

To do this we're going to rely on a few tools.

Tools

You've probably already heard of Git, it's basically a tool to keep track of changes done to
your project's code, among other things.

If you've heard of (or used) Git, you've probably also heard of (and used) GitHub,
an online service that lets you host your Git repositories in the cloud.

But you've probably not heard of Travis CI, it's a tool exclusively made for GitHub that lets developers run
automated tests on their code. We'll be using this to package and release our addon(s).

Last but not least there is the BigWigs packager script which will do all of the actual work for us.

These tools combined will automatically do everything you would normally do when publishing
a new version for your addon(s), such as increasing the Version field in your TOC file,
fetching localization, embedding libraries and more, then finally zip everything and publish.

How it works

After the setup (see the next two posts), the following happens:
  1. You push your changes to GitHub, including a tag which signifies a new release/version.
  2. Travis CI gets notified by GitHub, which clones your repository and downloads the packager script.
  3. The packager runs all of the tasks it needs to create a zip.
  4. The packager uploads that zip to all the sites you've specified to release at.
  5. Addons are now uploaded and are awaiting approval by the site administrators.

So, all you have to do (once set up) is to push to GitHub with a tag, like so:
git tag -am "Tag v2.0" v2.0 && git push origin master --tags

I'm not going into how to use Git and GitHub, there are plenty of guides that will explain that
better than I can do here, so I'm going to focus on the specifics for the actual packaging tools;
Travis CI and the packager script.

Sidenote: You don't have to use GitHub and Travis CI, you could just as well just run the script locally,
or use it with other CI tools for GitLab or Bitbucket, but this guide focuses on GitHub and Travis CI.

p3lim 10-14-17 02:34 PM

2 Attachment(s)
Travis CI Setup

Visit the Travis CI website and log in by authenticating it with GitHub.
Once authenticated you can visit your profile page where you'll find a list of all your repositories on GitHub.

To enable Travis CI for a project, find it in this list, then "Settings" button next to it.
In here you'll want to make sure only "Build pushed branches" is enabled, like so:



Further down on this page you'll find a section called "Environment Variables", here you'll
enter your credentials for each site you want your addons to be released at.
Here's a list of what to put in the "Name" and "Value" boxes:
  • WoWInterface
  • CurseForge
  • GitHub
    • Name: GITHUB_OAUTH
    • Value: Generate a token (with full repo/org scopes) then copy/paste it.

Here's an example:



Once that's set up you'll need to create a ".travis.yml" file in the root of your repository,
right next to your TOC file. The naming of this file is crucial, as well as this content:

YAML Code:
  1. sudo: false
  2. language: c
  3.  
  4. addons:
  5.   apt:
  6.     packages:
  7.     - pandoc
  8.  
  9. script: curl -s https://raw.githubusercontent.com/BigWigsMods/packager/master/release.sh | bash
  10.  
  11. branches:
  12.   only:
  13.   - /^\d+\.\d+(\.\d+)?(-\S*)?$/

All this file does is letting Travis CI know what to do;
Install the required software, download the packager script and run it.

The only thing you need to keep in mind here is the "branches" part.
This will make sure that Travis CI only runs when there is a new tag.
It checks this by looking at the name of the branch (tags are techically branches),
then uses regular expressions to match it against your versioning schema.

You'll most likely need to tweak this to your own versioning schema, reply and I'll assist you.

And that's it for the Travis CI part!

p3lim 10-14-17 02:35 PM

1 Attachment(s)
Customizing the packaging

This part is specifically for instructing the packager on how to package your addon, and
where it should upload to.

TOC
You'll need to add additional fields to your addon's TOC file, this will let the packager know
where to upload your packaged addon to. You do this by finding the appropriate "IDs" for
the various supported sites and adding them to the TOC file.

The WoWInterface ID is easy to find, it's in the URL for your addon.
E.g. "22213" in wowinterface.com/downloads/info22213-BonusRollPreview.

For CurseForge you'll find it as "Project ID" under "About This Project" on the project page.
E.g. "54902" for wow.curseforge.com/projects/bonusrollpreview.



Add the ones you want to release on to the TOC file with their respective field names,
here's an example:

TOC Code:
  1. ## Interface: 70300
  2. ## Title: BonusRollPreview
  3.  
  4. ...
  5.  
  6. ## X-Curse-Project-ID: 54902
  7. ## X-WoWI-ID: 22213
  8.  
  9. ...

Packager Metadata
This file will deal with additional (all optional) things you'd like the packager to do to your
packaged addon, be it embedding external libraries, creating changelogs automatically, etc.

This is all done in a new file named ".pkgmeta" which needs to be located
at the root of your addon, next to the TOC file.

Here's a link to documentation on what you can include in this file:
https://authors.curseforge.com/knowl...ckagemeta-file

An example where I specify some libraries it should include:

YAML Code:
  1. enable-nolib-creation: no
  2.  
  3. externals:
  4.   libs/LibStub: svn://svn.wowace.com/wow/libstub/mainline/tags/1.0
  5.   libs/Wasabi:
  6.     url: git://github.com/p3lim-wow/Wasabi
  7.     tag: latest

In addition to this file, additional substitutions can be done to your addon's files (not just the TOC file),
such as filling in the Version field in the TOC file to your latest tag, inserting
localizations from CurseForge, and much more.
More info: https://authors.curseforge.com/knowl...-substitutions

I personally always use the automatic versioning, so my TOC file looks something like this (note the version field):

TOC Code:
  1. ## Interface: 70300
  2. ## Version: @[email protected]
  3. ## Title: Bonus Roll Preview
  4.  
  5. ...
The "@[email protected]" part will be replaced by the Git tag you used earlier for when it's packaged.

And that's it! Now you can tag and push to GitHub and the packager will do everything else!

p3lim 10-14-17 02:35 PM

Reserved. 10chars

p3lim 10-14-17 02:53 PM

If you're having trouble making the .pkgmeta and .travis.yml files, do the following:

1. Open up a terminal (e.g. Git Bash on Windows) in the root directory for your addon.
2. Type/paste in the following:
touch .pkgmeta .travis.yml
Windows is kind of backwards when it comes to naming files, this solves that.

Resike 10-15-17 08:28 AM

Nice, i was recently thinking about doing this, i just wasn't sure if this is even properly possible atm without hacks or not. Definitely gonna try it later.

CC_WOW 10-15-17 01:44 PM

I've a question regarding the Travis CI part:

I already have a simple travis.yml (based on https://github.com/WeakAuras/WeakAur...er/.travis.yml), all it does is run luacheck to catch any syntax errors if I forgot to run my offline build tools (which should be done automatically, but I haven't gotten around to that yet).

Can I just add the relevant parts to it without adding the "language: c" part, or must I add it for this script to work? Will it interfere with the execution if I do add it? Basically, I would want to combine the two to run the syntax check first and then deploy the file.

Also, for the regular expression:

My tags are usually named rXX-(alpha|beta)?(-YY), such as r21-beta-3 or r25-alpha or simply r27/r27-2/etc. (a release version). This was mostly so that Curse can detect the alpha/beta/release state properly, as I'm using their API via GitHub webhooks.

The packager script apparently would consider all of them beta, and worse still, package untagged commits as alpha versions when they aren't ready for deployment at all. Now, I think the branches option would filter out the untagged commits, but how do I have the script package as alpha/beta/release correctly without forking/modifying it or changing my versioning scheme? Maybe Travis could detect them and rename it before calling the script?

p3lim 10-15-17 03:03 PM

Quote:

Originally Posted by CC_WOW (Post 325523)
Can I just add the relevant parts to it without adding the "language: c" part, or must I add it for this script to work? Will it interfere with the execution if I do add it? Basically, I would want to combine the two to run the syntax check first and then deploy the file.

The only reason why "language: c" is chosen is because it's the fastest one that installs when Travis CI is setting up it's environment,
and the "default" language is (afaik) Ruby.
There is a "language: generic" that I'd prefer to use, but it's not documented so I'm staying away from using it.

You can add multiple scripts, but I'd recommend you use "after_script: " for the packager, to make sure that luacheck finishes first.

Quote:

Originally Posted by CC_WOW (Post 325523)
My tags are usually named rXX-(alpha|beta)?(-YY), such as r21-beta-3 or r25-alpha or simply r27/r27-2/etc. (a release version). This was mostly so that Curse can detect the alpha/beta/release state properly, as I'm using their API via GitHub webhooks.

The packager script apparently would consider all of them beta, and worse still, package untagged commits as alpha versions when they aren't ready for deployment at all. Now, I think the branches option would filter out the untagged commits, but how do I have the script package as alpha/beta/release correctly without forking/modifying it or changing my versioning scheme? Maybe Travis could detect them and rename it before calling the script?

Here's what the packager does to check what type of release your addon should be:
https://github.com/BigWigsMods/packa...sh#L1791-L1798

If the Git push was not a tag, it'll default to alpha.
If the Git push was a tag, it'll check if the version matches "1.2.3" or "v1.2.3" or has the word "release" in it, if not it'll be flagged as beta.

So, without modifying the script, your normal pushes will always be alpha, and your "releases" will be beta.
I'd recommend you fork the packager on GitHub, make the changes to match your versioning, then use that instead for the "after_script: ".
An alternative would be to apply a patch to the script before running it.

Something like this would do:
Code:

file_type=
if [ -n "$tag" ]; then
        if [[ "${tag,,}" == *"alpha"* ]]; then
                file_type=alpha
        elif [[ "${tag,,}" == *"beta"* ]]; then
                file_type=beta
        else
                file_type=release
        fi
else
        echo "Found no tag, exiting."
        exit 0
fi

Then use the following regex for branch checking in .travis.yml:
^r\d+(-(alpha|beta))?(-\d+)?$

CC_WOW 10-16-17 02:54 PM

Thanks for your reply. That is the exact same portion I was looking at earlier, and I don't really want to fork it unless I have to as that means I need to pull changes in to keep it updated... and I'm lazy :P

I have literally no clue how to use Travis or just bash scripts in it, and even after trying various things and googling a lot I can't seem to get it to accept the "patch" you posted as a valid parameter in my after_script block.

This is the best that I've got, though no matter what, none of the block indicators I found seems to work: GitHub gist for the invalid .travis.yml file

Right now the error message is
Code:

syntax error: (<unknown>): could not find expected ':' while scanning a simple key at line 19 column 1
but I've had a fair share of other ones, too, depending on what I tried doing.

p3lim 10-16-17 07:00 PM

Some issues with your yml file:

- You're ignoring the whitespace requirements of YAML (not really your fault, YAML is anal)
- You're running the "patch" before the release script is even downloaded
- You're attempting to start curl with "/curl", this is not WoW :P
- There's BBCode in the curl line (that's this forums fault for parsing links in code snippets)
- Incorrect syntax for the branch expression

I created a patch file:
https://gist.github.com/p3lim/d77edf...-release-patch

And fixed your yml file, using that patch file:
https://gist.github.com/p3lim/d77edf...ile-travis-yml

The major difference here (apart from the syntax errors here and there) is this:

- It downloads the release script and saves it to disk, instead of piping it directly into bash (like in the guide)
- It downloads and applies the patch (linked above)
- Lastly, it runs the script

Full changelog:
https://gist.github.com/p3lim/d77edf...8a45/revisions

CC_WOW 10-17-17 11:05 AM

Wow, thank you so much! I really have no clue what I'm doing. I just tried to somehow make it work with the info this guide gives and my very limited knowledge.
I forked your gist to have a backup of it in case you want to delete it, but otherwise I can just have Travis download yours if you don't mind?

The build completed successfully now, but I don't think the release script is doing what it should... yet. All it did was create a GitHub release (for my alpha file, no less...), which isn't ideal. It also skipped the localization and there was no mention of uploading anything to WOWI at all. This is the build log: Travis-CI.org

I'm sure I am still missing something, but from what I see the whole point of using it over the CurseForge + GitHub webhooks solution is to have WOWI included. So far my addons have only been on Curse, but I've updated a smaller addon yesterday to see if I could get it to work. CurseForge has not received any update (do I still need the old webhook?) and I'm not sure how the process works here.

p3lim 10-17-17 06:11 PM

Quote:

Originally Posted by CC_WOW (Post 325567)
Wow, thank you so much! I really have no clue what I'm doing. I just tried to somehow make it work with the info this guide gives and my very limited knowledge.
I forked your gist to have a backup of it in case you want to delete it, but otherwise I can just have Travis download yours if you don't mind?

Go ahead, I'll keep it up for archival purposes.

Quote:

Originally Posted by CC_WOW (Post 325567)
The build completed successfully now, but I don't think the release script is doing what it should... yet. All it did was create a GitHub release (for my alpha file, no less...), which isn't ideal. It also skipped the localization and there was no mention of uploading anything to WOWI at all. This is the build log: Travis-CI.org

I'm sure I am still missing something, but from what I see the whole point of using it over the CurseForge + GitHub webhooks solution is to have WOWI included. So far my addons have only been on Curse, but I've updated a smaller addon yesterday to see if I could get it to work. CurseForge has not received any update (do I still need the old webhook?) and I'm not sure how the process works here.

You forgot the first half of the 3rd post about the TOC metadata.
That's the only way (aside from arguments to the script directly, not very portable) the script will know which addon on their respective sites will be updated (aside from GitHub which already knows the repo).

Also notice the last part in the 3rd post, it will automagically update the version field in your TOC file also.

CC_WOW 10-19-17 01:23 AM

Everything seems to have worked now. Thank you! I should be able to use the same approach to upload my other addons here as well.

I am aware of the version substitution, but I had not added it to that specific addon yet.

alar 11-06-17 02:21 PM

I found an issue in the release.sh script.
From the old times of svn, we could directly refer to a subdirectory inside an external library. This survived ad is managed byt the curse packager but not from release.sh.
Here is the example

Code:

externals:
  libs/LibInit:
    url: https://github.com/alarofrunetotem/LibInit.git/LibInit
    tag: latest

the actual repository name ends at .git but the script attempts to use the full string and fails with:

fatal: repository 'https://github.com/alarofrunetotem/LibInit.git/LibInit/' not found

Actually, the right behavious would be:

Code:

git clone https://github.com/alarofrunetotem/LibInit.git tempdir
cp tempdir/LibInit releasedir/libs/LibInit

Best regards

p3lim 11-07-17 12:43 AM

Quote:

Originally Posted by alar (Post 325731)
I found an issue in the release.sh script.
From the old times of svn, we could directly refer to a subdirectory inside an external library. This survived ad is managed byt the curse packager but not from release.sh.
Here is the example

Code:

externals:
  libs/LibInit:
    url: https://github.com/alarofrunetotem/LibInit.git/LibInit
    tag: latest

the actual repository name ends at .git but the script attempts to use the full string and fails with:

fatal: repository 'https://github.com/alarofrunetotem/LibInit.git/LibInit/' not found

Actually, the right behavious would be:

Code:

git clone https://github.com/alarofrunetotem/LibInit.git tempdir
cp tempdir/LibInit releasedir/libs/LibInit

Best regards

The ability to target a specific subdirectory when "cloning" from SVN is actually a feature of SVN that Git doesn't have. The "fault" is with Git, not the script :)

nyyr 05-04-18 11:48 AM

Uploading to WowAce
 
Thank you for this detailled guide, I managed to set it up smoothly!

The packager script, however, does not upload to WowAce (and thus CurseForge). When looking at the script, it specifically checks whether the project_page is curseforge. In my case, however, it is a WowAce project page.

Is it possible at all to upload to WowAce?

p3lim 05-04-18 12:02 PM

Quote:

Originally Posted by nyyr (Post 327888)
Thank you for this detailled guide, I managed to set it up smoothly!

The packager script, however, does not upload to WowAce (and thus CurseForge). When looking at the script, it specifically checks whether the project_page is curseforge. In my case, however, it is a WowAce project page.

Is it possible at all to upload to WowAce?

They disabled support for wowace in the packager script at some point because wowace disabled the API on their end, but there's this PR on the repo if you'd like to follow the progress for fixing this:
https://github.com/BigWigsMods/packager/pull/15

nyyr 05-04-18 12:58 PM

Quote:

They disabled support for wowace in the packager script at some point because wowace disabled the API on their end, but there's this PR on the repo if you'd like to follow the progress for fixing this:
https://github.com/BigWigsMods/packager/pull/15
Thanks for your quick reply! I tested the PR and it is still functional. Let's see whether it will be integrated.

nyyr 05-06-18 05:40 AM

Quote:

Originally Posted by nyyr (Post 327890)
Thanks for your quick reply! I tested the PR and it is still functional. Let's see whether it will be integrated.

It's integrated again:
Quote:

Originally Posted by nebularg
I'll merge it but I still doubt it's intended. There's really no reason to start projects on WoWAce anymore 😐 I wish Curse would just merge the three addon sites into one.

Source: https://github.com/BigWigsMods/packager/pull/15

Ellypse 05-08-18 01:35 AM

Thanks!
 
Thank you p3lim for this guide. I have implemented this for my add-ons and (after a few trials and errors) it's working great.

Do you know if there is a way to provide CurseForge with the supported version? Right now, every build uploaded to CurseForge is flagged for 8.0 and I have to go and manually replace the flag for 7.3.5. It would be really great if there was a way to read the .toc files and use the interface version number to flag the build on CurseForge correctly.

Ammako 05-08-18 01:45 AM

.toc is a little bit of a weird one, because 70300 could mean 7.3 like it could mean 7.3.2 or 7.3.5, so it's probably not as simple as just reading from .toc.

Unless they finally started adding minor revision number to interface version number, at least, but I imagine they have reasons for why they haven't done that.

p3lim 05-08-18 01:59 AM

Quote:

Originally Posted by Ellypse (Post 327923)
Do you know if there is a way to provide CurseForge with the supported version? Right now, every build uploaded to CurseForge is flagged for 8.0 and I have to go and manually replace the flag for 7.3.5. It would be really great if there was a way to read the .toc files and use the interface version number to flag the build on CurseForge correctly.

Seems like CF hastily enabled 8.0 as the default. You can force a certain version by passing it through to the packager.
Modify your ".travis.yaml", changing the line that says "script" into this:

Code:

script: curl -sO https://raw.githubusercontent.com/BigWigsMods/packager/master/release.sh
after_script: bash release.sh -g 7.3.5

Quote:

Originally Posted by Ammako (Post 327924)
.toc is a little bit of a weird one, because 70300 could mean 7.3 like it could mean 7.3.2 or 7.3.5, so it's probably not as simple as just reading from .toc.

Unless they finally started adding minor revision number to interface version number, at least, but I imagine they have reasons for why they haven't done that.

At least the wowinterface API has a compatibility endpoint that returns a list of compatible game versions. If you match 70300 with it, it will "return" the latest game versions, in the current case 7.3.5, not 7.3 or 7.3.2. (although wowi doesn't seem to update game versions for the patch increments, they're still on 7.3.0)
https://api.wowinterface.com/addons/compatible.json

Ellypse 05-10-18 09:32 AM

Quote:

Originally Posted by p3lim (Post 327925)
Modify your ".travis.yaml", changing the line that says "script" into this:

Code:

script: curl -sO https://raw.githubusercontent.com/BigWigsMods/packager/master/release.sh
after_script: bash release.sh -g 7.3.5


Oh I'm so dumb, it was in the packager's documentation on GitHub. Well, that works perfectly. Since I have a dedicated branch for BfA alpha builds I have also set the Travis config to 8.0 on this specific branch.
Thank you again ^^

yess 05-11-18 03:18 PM

Thank you for this guide!

I already had a .pkgmeta for the CurseForge packer so setting this up for my addon was quite easy and I got my first automated release to WoWI :)

I am a bit confused about the release to GitHub. I used the "Draft a Release" feature on the side but that does not work with the packer since that already creates a release on the site?

Code:

Uploading ChocolateBar-v3.1.2.zip (7.3.0) to https://www.wowinterface.com/downloads/info12326
Success!

Creating GitHub release: https://github.com/zanony/ChocolateBar/releases/tag/v3.1.2
Error! (422)
{
  "message": "Validation Failed",
  "errors": [
    {
      "resource": "Release",
      "code": "already_exists",
      "field": "tag_name"
    }
  ],
  "documentation_url": "https://developer.github.com/v3/repos/releases/#create-a-release"
}

After trying with
Code:

git tag v3.1.3 && push --tags
the packer did release to GitHub but took the wrong tag (the one before the current).

Two side notes:
As you mentioned I had to adjust the regular expression in the .travis.yml to work with my version scheme. It has a leading "v". Since this version scheme is quite common you could include an info how to adjust regex for that scheme in the guide.

I noticed that the CurseForge link to the keywords substitutions has changed.

yess 05-11-18 04:36 PM

Quote:

Originally Posted by p3lim (Post 327945)
I'll take a look at it, I haven't used github releases myself.

Thanks. If It's to much of a hassle don't worry as git hub releases are not that important for me either :)

p3lim 05-11-18 05:03 PM

Quote:

Originally Posted by yess (Post 327943)
I am a bit confused about the release to GitHub. I used the "Draft a Release" feature on the side but that does not work with the packer since that already creates a release on the site?

Yes, but the script should delete the existing release on GitHub in case it already exists (force overwrite basically).
If that doesn't happen, report it on the packager repo.

Quote:

Originally Posted by yess (Post 327943)
After trying with
Code:

git tag v3.1.3 && push --tags
the packer did release to GitHub but took the wrong tag (the one before the current).

That should not happen, report it on the packager repo.

Quote:

Originally Posted by yess (Post 327943)
I noticed that the CurseForge link to the keywords substitutions has changed.

Thank you, I've updated the links.

nebula 05-13-18 04:00 PM

Quote:

Originally Posted by yess (Post 327943)
After trying with
Code:

git tag v3.1.3 && push --tags
the packer did release to GitHub but took the wrong tag (the one before the current).

You're using lightweight tags and tagged the same commit with both v3.1.2 and v3.1.3, git itself used the v3.1.2 tag to reference the commit. You should use annotated tags (add a message with -m) so there is an actual object for it.

Regarding the Curse version stuff, their new versions endpoint is lacking and they don't seem keen on improving it. The old platform included "in_development" and "internal_id" (toc) so it was pretty easy to pick the current version. Now the only context you get is the version name, so when the version isn't set manually all I can do is grab the latest one, which obviously doesn't work so well when there is a new version in testing. :mad: As a stopgap I committed using 7.3.5 as the default version for Curse uploads, but I really am hoping the endpoint gets better.

The WoWI versions json, on the other hand, is much nicer and has a "default" flag :p (Although it would be nice if they added new patch versions)

yess 05-15-18 05:58 AM

Quote:

Originally Posted by nebula (Post 327972)
You're using lightweight tags and tagged the same commit with both v3.1.2 and v3.1.3, git itself used the v3.1.2 tag to reference the commit. You should use annotated tags (add a message with -m) so there is an actual object for it.

Ah thanks. I usually use annotated tags but after the problem with the Github release I thought I try to tag the repo exactly like suggested in the guide.

p3lim 05-15-18 02:53 PM

Quote:

Originally Posted by yess (Post 327990)
Ah thanks. I usually use annotated tags but after the problem with the Github release I thought I try to tag the repo exactly like suggested in the guide.

Corrected that, thanks for pointing it out.

KyrosKrane 06-08-18 11:07 PM

Thank you for the excellent guide! You have tremendously simplified my workflow and made it much easier to push out new releases!

My question is related to the pkgmeta file. I'm relatively new to the addon writing world, and I'm trying to understand the difference between externals, dependencies, and tools. Could you point me to a good guide on when to use each? The Curse documentation is sorely lacking beyond the bare how-to; it assumes you already know what the difference is. Besides, the automated method bypasses the Curse packager anyway.

As an example, I've been using LibStub, CallbackHandler, and LibDataBroker in my addons, and until now, I've simply placed those files in my repository and included them in my .toc file. I tried to include them as externals, but I ended up with a massive amount of files I didn't need in my addon zip file, like test cases and documentation.

I'm now moving towards the Ace3 framework, so I thought to include Ace3 and the other libraries I'm using in my externals list - but I got with something like three copies of libstub and callbackhandler, including in directories under the Ace3 hierarchy! I played around with it a bit, but eventually gave up and went back to just including the files I want in my repo.

Any advice would be appreciated.

p3lim 06-08-18 11:24 PM

Quote:

Originally Posted by KyrosKrane (Post 328266)
Thank you for the excellent guide! You have tremendously simplified my workflow and made it much easier to push out new releases!

My question is related to the pkgmeta file. I'm relatively new to the addon writing world, and I'm trying to understand the difference between externals, dependencies, and tools. Could you point me to a good guide on when to use each? The Curse documentation is sorely lacking beyond the bare how-to; it assumes you already know what the difference is. Besides, the automated method bypasses the Curse packager anyway.

As an example, I've been using LibStub, CallbackHandler, and LibDataBroker in my addons, and until now, I've simply placed those files in my repository and included them in my .toc file. I tried to include them as externals, but I ended up with a massive amount of files I didn't need in my addon zip file, like test cases and documentation.

I'm now moving towards the Ace3 framework, so I thought to include Ace3 and the other libraries I'm using in my externals list - but I got with something like three copies of libstub and callbackhandler, including in directories under the Ace3 hierarchy! I played around with it a bit, but eventually gave up and went back to just including the files I want in my repo.

Any advice would be appreciated.

Externals are for telling the packager which libraries it should download for you.
Optional-/required-dependencies is metadata Curse uses for their client (afaik).
Tools-used I think is just something you'd put in so that tool would get some points for the rewards program (again, CF stuff).

In your example with the three libraries:
Code:

externals:
  Libs/LibStub: https://repos.wowace.com/wow/libstub/trunk
  Libs/CallbackHandler-1.0: https://repos.wowace.com/wow/callbackhandler/trunk/CallbackHandler-1.0
  Libs/LibDataBroker-1.1: https://repos.wowace.com/wow/libdatabroker-1-1

This would tell the packager to download those three libs and place them in the "Libs" subdirectory of your packaged addon. The first two are from SVN, the last one from Git (the packager is smart enough to figure out which is which).

Another field in .pkgmeta is the "ignore" list. Sadly, not enough libraries use this field, such as LibStub, so you'll end up with useless files like you mentioned. The packager will recursively check for the ignore list in the externals' .pkgmeta file, but it won't recursively get their externals (if any).
You can however add externals' files you want to ignore for the packaging, as it's the last thing it does.

As for Ace3, check out Phanx's addons (specifically Grid), she's using a bucketload of Ace3 modules in that addon, and uses this packager as well. While looking at it, notice how she's managed to get all of the libraries from externals alone, you only see them referenced in .pkgmeta and Grid.toc.

KyrosKrane 06-08-18 11:30 PM

Thanks again for the awesome advice, I'll check it out!

KyrosKrane 06-25-18 11:51 AM

Quote:

Originally Posted by p3lim (Post 328267)
You can however add externals' files you want to ignore for the packaging, as it's the last thing it does.

This unfortunately seems incorrect. I spent some time trying to exclude the tests subdir from LibStub, and it wouldn't work. I finally figured out why. In the release.sh script, the processing for the "ignore" directive in the .pkgmeta happens first (starting at line 549), but when externals are checked out, the ignore list is cleared before processing each checkout (line 1232). The upshot is, the release script will not apply ignore directives in your .pkgmeta to the contents of externals.

Proof of concept: Create a new project with the following .pkgmeta. Package it using release.sh. You'll see the tests subdir under Libs/LibStub is packaged.

Code:

ignore:
    - Libs/LibStub/tests

externals:
  Libs/LibStub: https://repos.wowace.com/wow/libstub/trunk


KyrosKrane 07-01-18 11:18 AM

I coded up the changes necessary to have the ignore directive apply to externals, and submitted it as a pull request to the main packager repository:

https://github.com/BigWigsMods/packager/pull/17

AcidWeb 07-17-18 02:05 AM

Code:

install: echo -e "[alias]\nclone = clone --insecure" > ~/.hgrc
Needed in .travis.yml if mercurial repositories are used. Dirty hack but it is not worth to invest time to fix it properly.

StormFX 08-01-18 04:22 PM

I'm a bit curious about this. I checked Travis-CI and it doesn't seem to have any native support for Lua. I'm assuming that without some additional scripts, it doesn't actually validate the code, but rather just packages it, yeah?

p3lim 08-01-18 05:00 PM

Quote:

Originally Posted by StormFX (Post 329238)
I'm a bit curious about this. I checked Travis-CI and it doesn't seem to have any native support for Lua. I'm assuming that without some additional scripts, it doesn't actually validate the code, but rather just packages it, yeah?

Travis CI is (basically) running a full Linux installation, you can install any software you need for testing or w/e on it, just as we're using pandoc for the packager.

StormFX 08-01-18 05:03 PM

Quote:

Originally Posted by p3lim (Post 329241)
Travis CI is (basically) running a full Linux installation, you can install any software you need for testing or w/e on it, just as we're using pandoc for the packager.

Right, but without third-party scripts, it doesn't actually validate anything.

Xruptor 08-11-18 09:30 AM

Quote:

Originally Posted by p3lim (Post 325496)
Travis CI Setup




To enable Travis CI for a project, find it in this list, then hit the big switch so it turns green,
then click the cogwheel next to it to continue with the setup.

p3lim I've recently registered for an account and have activated it. It syncs my repositories properly but I'm having issues with this guide. It looks like they've updated their website? The button above no longer exists on the default profile page. In fact I can't seem to find it anywhere.

Quote:

Originally Posted by p3lim (Post 325496)

In here you'll want to disable "Build pull request updates" and enable "Build only if .travis.yml is present", like so:




The only thing that seems to be still there is the Environment variables portion of the guide. The top parts have been moved or are no longer a feature. Please let me know what I can do to properly activate this.

p3lim 08-11-18 01:14 PM

Quote:

Originally Posted by Xruptor (Post 329458)
p3lim I've recently registered for an account and have activated it. It syncs my repositories properly but I'm having issues with this guide. It looks like they've updated their website? The button above no longer exists on the default profile page. In fact I can't seem to find it anywhere.

The only thing that seems to be still there is the Environment variables portion of the guide. The top parts have been moved or are no longer a feature. Please let me know what I can do to properly activate this.

The only option you should need to enable now is the "Build pushed branches" in each repo's settings.
Everything else is detected from the .travis.yml file.

Edit: I've updated the guide to reflect the changes from travis-ci.org to travis-ci.com.
(Although personally I've moved over to CircleCI instead, see packager#18 if you're interested)

p3lim 08-11-18 01:34 PM

Quote:

Originally Posted by StormFX (Post 329242)
Right, but without third-party scripts, it doesn't actually validate anything.

That is correct. If you want an example with tests/evaluations, see how Rainrider did it in his addon "oUF_Dispellable":
https://github.com/Rainrider/oUF_Dis...er/.travis.yml

(Sorry for the late response, I seem to have missed your reply.)

Xruptor 08-12-18 02:27 PM

Quote:

Originally Posted by p3lim (Post 329465)
The only option you should need to enable now is the "Build pushed branches" in each repo's settings.
Everything else is detected from the .travis.yml file.

Edit: I've updated the guide to reflect the changes from travis-ci.org to travis-ci.com.
(Although personally I've moved over to CircleCI instead, see packager#18 if you're interested)

Thanks for the update P3lim. Is there any major differences between TravisCI and CircleCI? Why the switch? I'll take a look at it since I see the packager update for it. Just curious :)

p3lim 08-12-18 02:30 PM

Quote:

Originally Posted by Xruptor (Post 329507)
Thanks for the update P3lim. Is there any major differences between TravisCI and CircleCI? Why the switch? I'll take a look at it since I see the packager update for it. Just curious :)

There are some differences, but for the packaging needs it's indifferent. It's slightly faster because it runs Docker images which we can pre-pack (like I did with the packager PR) instead of installing dependencies every time (like we do with Travis, although that can be cached to improve speeds).

I switched over because after TravisCI merged org to com my GitHub organization (where all my addons are hosted) got screwed up, and TravisCI support couldn't help me with it and told me to just wait for the merge to go through by itself (which could take months), so it was more of a necessity for me.
We still use TravisCI for oUF, and there's no real need to change.

Dajn 08-31-18 11:30 AM

Regex string
 
Hey, thanks for the great tutorial!
The only thing I would love to get some help with is the regex string, as I cant get it to work. x)

The format I usually use for my versions is either a tag with 8.0.1.18 for a full release or 8.0.1.18-alpha/beta for beta/alpha releases.

Any help with this would be wonderful!

With regards,
Dajn

p3lim 08-31-18 11:38 AM

Quote:

Originally Posted by Dajn (Post 329940)
Hey, thanks for the great tutorial!
The only thing I would love to get some help with is the regex string, as I cant get it to work. x)

The format I usually use for my versions is either a tag with 8.0.1.18 for a full release or 8.0.1.18-alpha/beta for beta/alpha releases.

Any help with this would be wonderful!

With regards,
Dajn

Code:

/^\d+\.\d+\.\d+\.\d+(-\w+)?$/
https://regexr.com/3upce look at the explaination at the bottom.

Dajn 08-31-18 12:39 PM

Quote:

Originally Posted by p3lim (Post 329942)
Code:

/^\d+\.\d+\.\d+\.\d+(-\w+)?$/
https://regexr.com/3upce look at the explaination at the bottom.

Ty!

Been trying to get the hang of regex lately as I will start use it at work a lot, so this tutorial was a step in the right direction to start to learn it. :)

With regards,
Dajn

Xruptor 10-01-18 03:53 PM

Hey p3lim I'm having an issue where the packager isn't uploading properly to wowinterface.com

My tags are in the following format. v5.4, v.5.5, v5.6 etc..

so I changed the regex to the following

Code:

/^v?\d+\.\d+(\.\d+)?(-\S*)?$/
That seems to do the trick and picks up the 'v' in front of my tags. I didn't initially incorporate a .pkgmeta file as I don't need to do any changes to my addon and just want it upload exactly the way it is. Travis CI did pickup the tag and did the whole sing and dance it does. It completed with a green all clear. However when I went to wowinterface the file wasn't uploaded and the version didn't change either.

So I then added a .pkgmeta file but left it empty. That didn't seem to work. So I'm thinking that something needs to go into this file. I don't need externals or anything. Just simply upload the addon as is to wowinterface.

BTW: Yes I did make the modifications to the .toc file and added the addon IDs for both curse and wowinterface.com

Code:

## X-Curse-Project-ID: 28603
## X-WoWI-ID: 18787

Hope you can help. I may be missing something. Thanks for your help advance p3lim! You rock :D

Xruptor 10-01-18 04:09 PM

Lol!!!!
 
Quote:

Originally Posted by p3lim (Post 325496)
YAML Code:
  1. sudo: false
  2. language: c
  3.  
  4. addons:
  5.   apt:
  6.     packages:
  7.     - pandoc
  8.  
  9. script: curl -s [url]https://raw.githubusercontent.com/BigWigsMods/packager/master/release.sh[/url] | bash
  10.  
  11. branches:
  12.   only:
  13.   - /^\d+\.\d+(\.\d+)?(-\S*)?$/

LOLOLOL I figured it out @p3lim!! Your instructions has the curl with URL tags. It should be like this.

Code:


script: curl -s https://raw.githubusercontent.com/BigWigsMods/packager/master/release.sh | bash

It worked perfectly when I did this. It's all working fine now :) Turns out no you don't need the .pkgmeta file if you want to just upload it the way it is without modifications.

p3lim 10-01-18 10:16 PM

Quote:

Originally Posted by Xruptor (Post 330323)
LOLOLOL I figured it out @p3lim!! Your instructions has the curl with URL tags. It should be like this.

It worked perfectly when I did this. It's all working fine now :) Turns out no you don't need the .pkgmeta file if you want to just upload it the way it is without modifications.

Thank you for spotting that, fixed it up.
Happy it all works for you :).

Rythal 10-24-18 01:02 PM

Trying to get this setup myself and not having much luck with the regex even tho it should work :/

Currently they get tagged as vA.B.C(.D) so can be v8.0.0 or v8.0.0.1

Trying - ^v\d+.\d+.\d+(.\d+)?$ which works in any self-tests I do, but on Travis i'm getting 'Branch "v8.0.1.4" not included per configuration.'

What have I done wrong? :)

p3lim 10-24-18 01:17 PM

1 Attachment(s)
Quote:

Originally Posted by Rythal (Post 330572)
Trying to get this setup myself and not having much luck with the regex even tho it should work :/

Currently they get tagged as vA.B.C(.D) so can be v8.0.0 or v8.0.0.1

Trying - ^v\d+.\d+.\d+(.\d+)?$ which works in any self-tests I do, but on Travis i'm getting 'Branch "v8.0.1.4" not included per configuration.'

What have I done wrong? :)

That should work fine, although I'd escape the punctuations (many regex parsers are picky).
Could you show me your .travis.yml file?

Rythal 10-24-18 01:50 PM

Quote:

Originally Posted by p3lim (Post 330573)
That should work fine, although I'd escape the punctuations (many regex parsers are picky).
Could you show me your .travis.yml file?

Code:

sudo: false
language: c
 
addons:
  apt:
    packages:
    - pandoc
 
script: curl -s https://raw.githubusercontent.com/BigWigsMods/packager/master/release.sh | bash
 
branches:
  only:
  - ^v\d+\.\d+\.\d+(\.\d+)?$

Just tried it with the escaped punctuation but still got not included by config error.

Bah I think I see the problem already... forgot the /'s

p3lim 10-24-18 02:04 PM

Quote:

Originally Posted by Rythal (Post 330576)
Bah I think I see the problem already... forgot the /'s

There you go :)

I mentioned an alternative to the Travis method here, which eliminates the need to verify tag structure (it just assumes any tag is a release-ready version).
(Plus, it's easier to set up and expand upon ;))

zork 10-25-18 06:36 AM

3 Attachment(s)
Thanks p3lim for providing this. I need some help.

I added the .travis.yml and the .pkgmeta to one of my addons in my github repo. Currently the .pkgmeta and the .travis.yml use CRLF as ending. Is that a problem? I told GIT bash for windows not to mess with line endings.

https://github.com/zorker/rothui/tre...w8.0/rVignette

I then used the git command you provided using git bash for windows in the git directory of rVignette.

I'm not sure but I think my whole repo got that tag?! Is that correct?

https://github.com/zorker/rothui/tree/v800.20181025

Once I started the git bash command I had to log into github and the tag completed.

But nothing else seem to have happened? What am I missing?

I had added the wowi token to the Travis page before. My travis page is all grey and says: There is no build on the default branch yet?

Ok I checked the travis request page. I can see some error messages now. Travis does seem to try to generate a new build when I commit with Github desktop to master. (The first two requests) Why is that?

*edit* Ohhh I see. It would run on any commit but when reading the config file would only react to branches that match the pattern.

That being said. To correct the error from my understanding I have to put the .travis.yml in the repo root right? What does that mean for the packager? Will he parse all directories and upload all of them at once? If I only make a change to one of the addons all will be updated? How do I do this for specific directories?

p3lim 10-25-18 07:11 AM

Quote:

Originally Posted by zork (Post 330594)
Thanks p3lim for providing this. I need some help.

I added the .travis.yml and the .pkgmeta to one of my addons in my github repo. Currently the .pkgmeta and the .travis.yml use CRLF as ending. Is that a problem? I told GIT bash for windows not to mess with line endings.

https://github.com/zorker/rothui/tre...w8.0/rVignette

I then used the git command you provided using git bash for windows in the git directory of rVignette.

I'm not sure but I think my whole repo got that tag?! Is that correct?

https://github.com/zorker/rothui/tree/v800.20181025

Once I started the git bash command I had to log into github and the tag completed.

But nothing else seem to have happened? What am I missing?

I had added the wowi token to the Travis page before. My travis page is all grey and says: There is no build on the default branch yet?

Ok I checked the travis request page. I can see some error messages now. Travis does seem to try to generate a new build when I commit with Github desktop to master. (The first two requests) Why is that?

*edit* Ohhh I see. It would run on any commit but when reading the config file would only react to branches that match the pattern.

That being said. To correct the error from my understanding I have to put the .travis.yml in the repo root right? What does that mean for the packager? Will he parse all directories and upload all of them at once? If I only make a change to one of the addons all will be updated? How do I do this for specific directories?

For reasons I do not understand, you are using a single repo for multiple projects, which is not how Git was intended to be used, thus no CI tool will be able to understand.
E.g, if you tag the repo (and tags only work on repo-level, as with branches), every project you have in that repo (as you've set it up) will get the same tag.

The packager and Travis expects Git to be used as it was intended; one repo per project, and each project has its own .travis.yml file at the root, and tagging the repo will trigger Travis/packager.

zork 10-25-18 07:12 AM

It relates back to Google Code and SVN days. When Google Code closed there was an option to move all of it to Github which I did.

That is why there is one huge mono repo for all of it.

Never saw a reason to change that.

p3lim 10-25-18 08:17 AM

Quote:

Originally Posted by zork (Post 330596)
It relates back to Google Code and SVN days. When Google Code closed there was an option to move all of it to Github which I did.

That is why there is one huge mono repo for all of it.

Never saw a reason to change that.

Git is not SVN, and monolithic repositories comes with a large amount of issues, especially when you want to expand upon just tracking history.
If you ever want to manage dependencies, external tooling, collaboration, or versioning (just to name a few), monorepos is going to become a problem, fast.

My suggestion to you is to spend some time splitting up your single repo into multiple, preferably one per addon, then you'll be able to fully utilize Git as it was intended, which includes being able to use the packager with TravisCI in an automated fashion.

Example of splitting while preserving history: https://www.atlassian.com/blog/git/t...sitory-git-way

Aiue 09-28-19 01:03 PM

Right, it would appear I'm doing something wrong with git, and I can't quite figure out what, so I'm turning here for help. Even tagging exactly in the manner suggested in the op, I keep tripping this:

Code:

        if [ -z "$TRAVIS_TAG" ]; then
                # don't need to run the packager if there is a tag pending
                TRAVIS_TAG=$( git -C "$TRAVIS_BUILD_DIR" tag --points-at )
                if [ -n "$TRAVIS_TAG" ]; then
                        echo "Found future tag \"${TRAVIS_TAG}\", not packaging."
                        exit 0
                fi

Which means the packager will only run if I have no tag (as I currently have it configured to do that, as, well, it's gotta run on something.)

I'm sure there's something obvious I'm missing, but I'm stumped, so any help would be appreciated.

The repo in question is at https://github.com/Aiue/AQT, in case an extra set of eyes looking at stuff helps.

Aiue 09-28-19 02:38 PM

Quote:

Originally Posted by Aiue (Post 334031)
Which means the packager will only run if I have no tag (as I currently have it configured to do that, as, well, it's gotta run on something.)

Turns out this was the issue. Setting it to run on anything will cause this. I guess that's what I get for not following instructions explicitly and improvising.

MooreaTv 09-30-19 12:14 PM

You can use Github Actions (just need to signup for it and afaik everyone asking gets immediate access to it) to run the packager with no other dependencies

here is an example:
https://github.com/mooreatv/AuctionD...packaging.yaml


All times are GMT -6. The time now is 10:30 PM.

vBulletin © 2020, Jelsoft Enterprises Ltd
© 2004 - 2020 MMOUI