Thread Tools Display Modes
10-14-17, 02:34 PM   #1
p3lim
A Pyroguard Emberseer
 
p3lim's Avatar
AddOn Author - Click to view addons
Join Date: Feb 2007
Posts: 1,710
Guide: Automagically package and publish addons

DISCLAIMER

The TravisCI section of this guide is no longer maintained, as the majority of the community has moved over to use GitHub Actions instead.
The rest of this guide is still valid, but if you want to use GitHub actions instead of TravisCI please see this: https://github.com/BigWigsMods/packa...tions-workflow

Original post:

Any seasoned developer will agree that automating repetitive tasks is a good idea,
and one of the most repetitive things we do as addon developers is publishing addon updates.

This guide will instruct you how you can automate packaging and publishing new releases,
so you can focus more on developing the addon itself.

With this tool, addons can be published at the two major addon sites; CurseForge/WoWAce (and thus Curse), and WoWInterface.
Addons can also be published (packaged) on GitHub in case you'd also want that.

To do this we're going to rely on a few tools.

Tools

You've probably already heard of Git, it's basically a tool to keep track of changes done to
your project's code, among other things.

If you've heard of (or used) Git, you've probably also heard of (and used) GitHub,
an online service that lets you host your Git repositories in the cloud.

But you've probably not heard of Travis CI, it's a tool exclusively made for GitHub that lets developers run
automated tests on their code. We'll be using this to package and release our addon(s).

Last but not least there is the BigWigs packager script which will do all of the actual work for us.

These tools combined will automatically do everything you would normally do when publishing
a new version for your addon(s), such as increasing the Version field in your TOC file,
fetching localization, embedding libraries and more, then finally zip everything and publish.

How it works

After the setup (see the next two posts), the following happens:
  1. You push your changes to GitHub, including a tag which signifies a new release/version.
  2. Travis CI gets notified by GitHub, which clones your repository and downloads the packager script.
  3. The packager runs all of the tasks it needs to create a zip.
  4. The packager uploads that zip to all the sites you've specified to release at.
  5. Addons are now uploaded and are awaiting approval by the site administrators.

So, all you have to do (once set up) is to push to GitHub with a tag, like so:
git tag -am "Tag v2.0" v2.0 && git push origin master --tags

I'm not going into how to use Git and GitHub, there are plenty of guides that will explain that
better than I can do here, so I'm going to focus on the specifics for the actual packaging tools;
Travis CI and the packager script.

Sidenote: You don't have to use GitHub and Travis CI, you could just as well just run the script locally,
or use it with other CI tools for GitLab or Bitbucket, but this guide focuses on GitHub and Travis CI.

Last edited by p3lim : 10-13-21 at 03:16 AM. Reason: Use annotated tags in example
  Reply With Quote
10-14-17, 02:34 PM   #2
p3lim
A Pyroguard Emberseer
 
p3lim's Avatar
AddOn Author - Click to view addons
Join Date: Feb 2007
Posts: 1,710
Travis CI Setup

Visit the Travis CI website and log in by authenticating it with GitHub.
Once authenticated you can visit your profile page where you'll find a list of all your repositories on GitHub.

To enable Travis CI for a project, find it in this list, then "Settings" button next to it.
In here you'll want to make sure only "Build pushed branches" is enabled, like so:



Further down on this page you'll find a section called "Environment Variables", here you'll
enter your credentials for each site you want your addons to be released at.
Here's a list of what to put in the "Name" and "Value" boxes:
  • WoWInterface
  • CurseForge
  • GitHub
    • Name: GITHUB_OAUTH
    • Value: Generate a token (with full repo/org scopes) then copy/paste it.

Here's an example:



Once that's set up you'll need to create a ".travis.yml" file in the root of your repository,
right next to your TOC file. The naming of this file is crucial, as well as this content:

YAML Code:
  1. sudo: false
  2. language: c
  3.  
  4. addons:
  5.   apt:
  6.     packages:
  7.     - pandoc
  8.  
  9. script: curl -s https://raw.githubusercontent.com/BigWigsMods/packager/master/release.sh | bash
  10.  
  11. branches:
  12.   only:
  13.   - /^\d+\.\d+(\.\d+)?(-\S*)?$/

All this file does is letting Travis CI know what to do;
Install the required software, download the packager script and run it.

The only thing you need to keep in mind here is the "branches" part.
This will make sure that Travis CI only runs when there is a new tag.
It checks this by looking at the name of the branch (tags are techically branches),
then uses regular expressions to match it against your versioning schema.

You'll most likely need to tweak this to your own versioning schema, reply and I'll assist you.

And that's it for the Travis CI part!
Attached Thumbnails
Click image for larger version

Name:	1.png
Views:	5765
Size:	15.8 KB
ID:	9082  Click image for larger version

Name:	3.png
Views:	6194
Size:	10.9 KB
ID:	9043  

Last edited by p3lim : 10-01-18 at 10:33 PM. Reason: Update instructions for travis
  Reply With Quote
08-11-18, 09:30 AM   #3
Xruptor
A Flamescale Wyrmkin
 
Xruptor's Avatar
AddOn Author - Click to view addons
Join Date: Mar 2005
Posts: 133
Originally Posted by p3lim View Post
Travis CI Setup




To enable Travis CI for a project, find it in this list, then hit the big switch so it turns green,
then click the cogwheel next to it to continue with the setup.
p3lim I've recently registered for an account and have activated it. It syncs my repositories properly but I'm having issues with this guide. It looks like they've updated their website? The button above no longer exists on the default profile page. In fact I can't seem to find it anywhere.

Originally Posted by p3lim View Post

In here you'll want to disable "Build pull request updates" and enable "Build only if .travis.yml is present", like so:



The only thing that seems to be still there is the Environment variables portion of the guide. The top parts have been moved or are no longer a feature. Please let me know what I can do to properly activate this.
__________________
Click HERE for the ultimate idiot test.

if (sizeof(sadness) > sizeof(happiness)) { initDepression(); }
  Reply With Quote
08-11-18, 01:14 PM   #4
p3lim
A Pyroguard Emberseer
 
p3lim's Avatar
AddOn Author - Click to view addons
Join Date: Feb 2007
Posts: 1,710
Originally Posted by Xruptor View Post
p3lim I've recently registered for an account and have activated it. It syncs my repositories properly but I'm having issues with this guide. It looks like they've updated their website? The button above no longer exists on the default profile page. In fact I can't seem to find it anywhere.

The only thing that seems to be still there is the Environment variables portion of the guide. The top parts have been moved or are no longer a feature. Please let me know what I can do to properly activate this.
The only option you should need to enable now is the "Build pushed branches" in each repo's settings.
Everything else is detected from the .travis.yml file.

Edit: I've updated the guide to reflect the changes from travis-ci.org to travis-ci.com.
(Although personally I've moved over to CircleCI instead, see packager#18 if you're interested)

Last edited by p3lim : 08-11-18 at 01:20 PM.
  Reply With Quote
08-12-18, 02:27 PM   #5
Xruptor
A Flamescale Wyrmkin
 
Xruptor's Avatar
AddOn Author - Click to view addons
Join Date: Mar 2005
Posts: 133
Originally Posted by p3lim View Post
The only option you should need to enable now is the "Build pushed branches" in each repo's settings.
Everything else is detected from the .travis.yml file.

Edit: I've updated the guide to reflect the changes from travis-ci.org to travis-ci.com.
(Although personally I've moved over to CircleCI instead, see packager#18 if you're interested)
Thanks for the update P3lim. Is there any major differences between TravisCI and CircleCI? Why the switch? I'll take a look at it since I see the packager update for it. Just curious
__________________
Click HERE for the ultimate idiot test.

if (sizeof(sadness) > sizeof(happiness)) { initDepression(); }
  Reply With Quote
08-12-18, 02:30 PM   #6
p3lim
A Pyroguard Emberseer
 
p3lim's Avatar
AddOn Author - Click to view addons
Join Date: Feb 2007
Posts: 1,710
Originally Posted by Xruptor View Post
Thanks for the update P3lim. Is there any major differences between TravisCI and CircleCI? Why the switch? I'll take a look at it since I see the packager update for it. Just curious
There are some differences, but for the packaging needs it's indifferent. It's slightly faster because it runs Docker images which we can pre-pack (like I did with the packager PR) instead of installing dependencies every time (like we do with Travis, although that can be cached to improve speeds).

I switched over because after TravisCI merged org to com my GitHub organization (where all my addons are hosted) got screwed up, and TravisCI support couldn't help me with it and told me to just wait for the merge to go through by itself (which could take months), so it was more of a necessity for me.
We still use TravisCI for oUF, and there's no real need to change.

Last edited by p3lim : 08-12-18 at 02:32 PM.
  Reply With Quote
10-01-18, 04:09 PM   #7
Xruptor
A Flamescale Wyrmkin
 
Xruptor's Avatar
AddOn Author - Click to view addons
Join Date: Mar 2005
Posts: 133
Lol!!!!

Originally Posted by p3lim View Post
YAML Code:
  1. sudo: false
  2. language: c
  3.  
  4. addons:
  5.   apt:
  6.     packages:
  7.     - pandoc
  8.  
  9. script: curl -s [url]https://raw.githubusercontent.com/BigWigsMods/packager/master/release.sh[/url] | bash
  10.  
  11. branches:
  12.   only:
  13.   - /^\d+\.\d+(\.\d+)?(-\S*)?$/
LOLOLOL I figured it out @p3lim!! Your instructions has the curl with URL tags. It should be like this.

Code:
script: curl -s https://raw.githubusercontent.com/BigWigsMods/packager/master/release.sh | bash
It worked perfectly when I did this. It's all working fine now Turns out no you don't need the .pkgmeta file if you want to just upload it the way it is without modifications.
__________________
Click HERE for the ultimate idiot test.

if (sizeof(sadness) > sizeof(happiness)) { initDepression(); }

Last edited by Xruptor : 10-01-18 at 04:45 PM.
  Reply With Quote
10-01-18, 10:16 PM   #8
p3lim
A Pyroguard Emberseer
 
p3lim's Avatar
AddOn Author - Click to view addons
Join Date: Feb 2007
Posts: 1,710
Originally Posted by Xruptor View Post
LOLOLOL I figured it out @p3lim!! Your instructions has the curl with URL tags. It should be like this.

It worked perfectly when I did this. It's all working fine now Turns out no you don't need the .pkgmeta file if you want to just upload it the way it is without modifications.
Thank you for spotting that, fixed it up.
Happy it all works for you .
  Reply With Quote
10-14-17, 02:35 PM   #9
p3lim
A Pyroguard Emberseer
 
p3lim's Avatar
AddOn Author - Click to view addons
Join Date: Feb 2007
Posts: 1,710
Customizing the packaging

This part is specifically for instructing the packager on how to package your addon, and
where it should upload to.

TOC
You'll need to add additional fields to your addon's TOC file, this will let the packager know
where to upload your packaged addon to. You do this by finding the appropriate "IDs" for
the various supported sites and adding them to the TOC file.

The WoWInterface ID is easy to find, it's in the URL for your addon.
E.g. "22213" in wowinterface.com/downloads/info22213-BonusRollPreview.

For CurseForge you'll find it as "Project ID" under "About This Project" on the project page.
E.g. "54902" for wow.curseforge.com/projects/bonusrollpreview.



Add the ones you want to release on to the TOC file with their respective field names,
here's an example:

TOC Code:
  1. ## Interface: 70300
  2. ## Title: BonusRollPreview
  3.  
  4. ...
  5.  
  6. ## X-Curse-Project-ID: 54902
  7. ## X-WoWI-ID: 22213
  8.  
  9. ...

Packager Metadata
This file will deal with additional (all optional) things you'd like the packager to do to your
packaged addon, be it embedding external libraries, creating changelogs automatically, etc.

This is all done in a new file named ".pkgmeta" which needs to be located
at the root of your addon, next to the TOC file.

Here's a link to documentation on what you can include in this file:
https://authors.curseforge.com/knowl...ckagemeta-file

An example where I specify some libraries it should include:

YAML Code:
  1. enable-nolib-creation: no
  2.  
  3. externals:
  4.   libs/LibStub: svn://svn.wowace.com/wow/libstub/mainline/tags/1.0
  5.   libs/Wasabi:
  6.     url: git://github.com/p3lim-wow/Wasabi
  7.     tag: latest

In addition to this file, additional substitutions can be done to your addon's files (not just the TOC file),
such as filling in the Version field in the TOC file to your latest tag, inserting
localizations from CurseForge, and much more.
More info: https://authors.curseforge.com/knowl...-substitutions

I personally always use the automatic versioning, so my TOC file looks something like this (note the version field):

TOC Code:
  1. ## Interface: 70300
  2. ## Version: @project-version@
  3. ## Title: Bonus Roll Preview
  4.  
  5. ...
The "@project-version@" part will be replaced by the Git tag you used earlier for when it's packaged.

And that's it! Now you can tag and push to GitHub and the packager will do everything else!
Attached Thumbnails
Click image for larger version

Name:	1.png
Views:	6025
Size:	4.7 KB
ID:	9044  

Last edited by p3lim : 05-14-20 at 02:06 AM. Reason: Updated links
  Reply With Quote
04-15-20, 01:23 PM   #10
LudiusMaximus
A Rage Talon Dragon Guard
 
LudiusMaximus's Avatar
AddOn Author - Click to view addons
Join Date: Mar 2018
Posts: 320
What a great guide! Thanks!
Updating these links in the original post would make it even more convenient:


https://authors.curseforge.com/knowledge-base/world-of-warcraft/527-preparing-the-packagemeta-file
->
https://authors.curseforge.com/knowledge-base/projects/527-preparing-the-packagemeta-file


https://authors.curseforge.com/knowledge-base/world-of-warcraft/532-repository-keyword-substitutions
->
https://authors.curseforge.com/knowledge-base/projects/532-repository-keyword-substitutions
__________________
~ Be the change you want to see in the world... of warcraft interface! ~
  Reply With Quote
10-14-17, 02:35 PM   #11
p3lim
A Pyroguard Emberseer
 
p3lim's Avatar
AddOn Author - Click to view addons
Join Date: Feb 2007
Posts: 1,710
Reserved. 10chars
  Reply With Quote
10-14-17, 02:53 PM   #12
p3lim
A Pyroguard Emberseer
 
p3lim's Avatar
AddOn Author - Click to view addons
Join Date: Feb 2007
Posts: 1,710
If you're having trouble making the .pkgmeta and .travis.yml files, do the following:

1. Open up a terminal (e.g. Git Bash on Windows) in the root directory for your addon.
2. Type/paste in the following:
touch .pkgmeta .travis.yml
Windows is kind of backwards when it comes to naming files, this solves that.
  Reply With Quote
10-15-17, 08:28 AM   #13
Resike
A Pyroguard Emberseer
AddOn Author - Click to view addons
Join Date: Mar 2010
Posts: 1,290
Nice, i was recently thinking about doing this, i just wasn't sure if this is even properly possible atm without hacks or not. Definitely gonna try it later.
  Reply With Quote
10-15-17, 01:44 PM   #14
CC_WOW
A Deviate Faerie Dragon
 
CC_WOW's Avatar
AddOn Author - Click to view addons
Join Date: Nov 2016
Posts: 19
I've a question regarding the Travis CI part:

I already have a simple travis.yml (based on https://github.com/WeakAuras/WeakAur...er/.travis.yml), all it does is run luacheck to catch any syntax errors if I forgot to run my offline build tools (which should be done automatically, but I haven't gotten around to that yet).

Can I just add the relevant parts to it without adding the "language: c" part, or must I add it for this script to work? Will it interfere with the execution if I do add it? Basically, I would want to combine the two to run the syntax check first and then deploy the file.

Also, for the regular expression:

My tags are usually named rXX-(alpha|beta)?(-YY), such as r21-beta-3 or r25-alpha or simply r27/r27-2/etc. (a release version). This was mostly so that Curse can detect the alpha/beta/release state properly, as I'm using their API via GitHub webhooks.

The packager script apparently would consider all of them beta, and worse still, package untagged commits as alpha versions when they aren't ready for deployment at all. Now, I think the branches option would filter out the untagged commits, but how do I have the script package as alpha/beta/release correctly without forking/modifying it or changing my versioning scheme? Maybe Travis could detect them and rename it before calling the script?
  Reply With Quote
10-15-17, 03:03 PM   #15
p3lim
A Pyroguard Emberseer
 
p3lim's Avatar
AddOn Author - Click to view addons
Join Date: Feb 2007
Posts: 1,710
Originally Posted by CC_WOW View Post
Can I just add the relevant parts to it without adding the "language: c" part, or must I add it for this script to work? Will it interfere with the execution if I do add it? Basically, I would want to combine the two to run the syntax check first and then deploy the file.
The only reason why "language: c" is chosen is because it's the fastest one that installs when Travis CI is setting up it's environment,
and the "default" language is (afaik) Ruby.
There is a "language: generic" that I'd prefer to use, but it's not documented so I'm staying away from using it.

You can add multiple scripts, but I'd recommend you use "after_script: " for the packager, to make sure that luacheck finishes first.

Originally Posted by CC_WOW View Post
My tags are usually named rXX-(alpha|beta)?(-YY), such as r21-beta-3 or r25-alpha or simply r27/r27-2/etc. (a release version). This was mostly so that Curse can detect the alpha/beta/release state properly, as I'm using their API via GitHub webhooks.

The packager script apparently would consider all of them beta, and worse still, package untagged commits as alpha versions when they aren't ready for deployment at all. Now, I think the branches option would filter out the untagged commits, but how do I have the script package as alpha/beta/release correctly without forking/modifying it or changing my versioning scheme? Maybe Travis could detect them and rename it before calling the script?
Here's what the packager does to check what type of release your addon should be:
https://github.com/BigWigsMods/packa...sh#L1791-L1798

If the Git push was not a tag, it'll default to alpha.
If the Git push was a tag, it'll check if the version matches "1.2.3" or "v1.2.3" or has the word "release" in it, if not it'll be flagged as beta.

So, without modifying the script, your normal pushes will always be alpha, and your "releases" will be beta.
I'd recommend you fork the packager on GitHub, make the changes to match your versioning, then use that instead for the "after_script: ".
An alternative would be to apply a patch to the script before running it.

Something like this would do:
Code:
file_type=
if [ -n "$tag" ]; then
	if [[ "${tag,,}" == *"alpha"* ]]; then
		file_type=alpha
	elif [[ "${tag,,}" == *"beta"* ]]; then
		file_type=beta
	else
		file_type=release
	fi
else
	echo "Found no tag, exiting."
	exit 0
fi
Then use the following regex for branch checking in .travis.yml:
^r\d+(-(alpha|beta))?(-\d+)?$

Last edited by p3lim : 10-15-17 at 03:08 PM.
  Reply With Quote
10-16-17, 02:54 PM   #16
CC_WOW
A Deviate Faerie Dragon
 
CC_WOW's Avatar
AddOn Author - Click to view addons
Join Date: Nov 2016
Posts: 19
Thanks for your reply. That is the exact same portion I was looking at earlier, and I don't really want to fork it unless I have to as that means I need to pull changes in to keep it updated... and I'm lazy :P

I have literally no clue how to use Travis or just bash scripts in it, and even after trying various things and googling a lot I can't seem to get it to accept the "patch" you posted as a valid parameter in my after_script block.

This is the best that I've got, though no matter what, none of the block indicators I found seems to work: GitHub gist for the invalid .travis.yml file

Right now the error message is
Code:
syntax error: (<unknown>): could not find expected ':' while scanning a simple key at line 19 column 1
but I've had a fair share of other ones, too, depending on what I tried doing.
  Reply With Quote
10-16-17, 07:00 PM   #17
p3lim
A Pyroguard Emberseer
 
p3lim's Avatar
AddOn Author - Click to view addons
Join Date: Feb 2007
Posts: 1,710
Some issues with your yml file:

- You're ignoring the whitespace requirements of YAML (not really your fault, YAML is anal)
- You're running the "patch" before the release script is even downloaded
- You're attempting to start curl with "/curl", this is not WoW :P
- There's BBCode in the curl line (that's this forums fault for parsing links in code snippets)
- Incorrect syntax for the branch expression

I created a patch file:
https://gist.github.com/p3lim/d77edf...-release-patch

And fixed your yml file, using that patch file:
https://gist.github.com/p3lim/d77edf...ile-travis-yml

The major difference here (apart from the syntax errors here and there) is this:

- It downloads the release script and saves it to disk, instead of piping it directly into bash (like in the guide)
- It downloads and applies the patch (linked above)
- Lastly, it runs the script

Full changelog:
https://gist.github.com/p3lim/d77edf...8a45/revisions

Last edited by p3lim : 10-17-17 at 06:14 PM.
  Reply With Quote
10-17-17, 11:05 AM   #18
CC_WOW
A Deviate Faerie Dragon
 
CC_WOW's Avatar
AddOn Author - Click to view addons
Join Date: Nov 2016
Posts: 19
Wow, thank you so much! I really have no clue what I'm doing. I just tried to somehow make it work with the info this guide gives and my very limited knowledge.
I forked your gist to have a backup of it in case you want to delete it, but otherwise I can just have Travis download yours if you don't mind?

The build completed successfully now, but I don't think the release script is doing what it should... yet. All it did was create a GitHub release (for my alpha file, no less...), which isn't ideal. It also skipped the localization and there was no mention of uploading anything to WOWI at all. This is the build log: Travis-CI.org

I'm sure I am still missing something, but from what I see the whole point of using it over the CurseForge + GitHub webhooks solution is to have WOWI included. So far my addons have only been on Curse, but I've updated a smaller addon yesterday to see if I could get it to work. CurseForge has not received any update (do I still need the old webhook?) and I'm not sure how the process works here.
  Reply With Quote
10-17-17, 06:11 PM   #19
p3lim
A Pyroguard Emberseer
 
p3lim's Avatar
AddOn Author - Click to view addons
Join Date: Feb 2007
Posts: 1,710
Originally Posted by CC_WOW View Post
Wow, thank you so much! I really have no clue what I'm doing. I just tried to somehow make it work with the info this guide gives and my very limited knowledge.
I forked your gist to have a backup of it in case you want to delete it, but otherwise I can just have Travis download yours if you don't mind?
Go ahead, I'll keep it up for archival purposes.

Originally Posted by CC_WOW View Post
The build completed successfully now, but I don't think the release script is doing what it should... yet. All it did was create a GitHub release (for my alpha file, no less...), which isn't ideal. It also skipped the localization and there was no mention of uploading anything to WOWI at all. This is the build log: Travis-CI.org

I'm sure I am still missing something, but from what I see the whole point of using it over the CurseForge + GitHub webhooks solution is to have WOWI included. So far my addons have only been on Curse, but I've updated a smaller addon yesterday to see if I could get it to work. CurseForge has not received any update (do I still need the old webhook?) and I'm not sure how the process works here.
You forgot the first half of the 3rd post about the TOC metadata.
That's the only way (aside from arguments to the script directly, not very portable) the script will know which addon on their respective sites will be updated (aside from GitHub which already knows the repo).

Also notice the last part in the 3rd post, it will automagically update the version field in your TOC file also.

Last edited by p3lim : 10-18-17 at 07:31 AM.
  Reply With Quote
10-19-17, 01:23 AM   #20
CC_WOW
A Deviate Faerie Dragon
 
CC_WOW's Avatar
AddOn Author - Click to view addons
Join Date: Nov 2016
Posts: 19
Everything seems to have worked now. Thank you! I should be able to use the same approach to upload my other addons here as well.

I am aware of the version substitution, but I had not added it to that specific addon yet.
  Reply With Quote

WoWInterface » Developer Discussions » Tutorials & Other Helpful Info. » Guide: Automagically package and publish addons

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off